Repository navigation
perf(openai-utils): discriminate response item unions - #2889
Merged
bxyu-nvidia merged 1 commit intoSep 9, 2026
Merged
bxyu-nvidia merged 1 commit into
bxyu-nvidia merged 1 commit into
Conversation
ananthsub
marked this pull request as draft
August 31, 2026 17:57
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Route response input and output items through callable Pydantic discriminators instead of trying every union member. Shape-based tags resolve shared message and function-call-output literals while preserving permissive typeless input handling. Require token metadata to remain atomic on both input and output items so an incomplete payload cannot select a base model and lose fields. Add regression coverage for concrete routing, training variants, and dump/revalidation. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
ananthsub
force-pushed
the
ananthsub/discriminated-response-items
branch
from
August 31, 2026 19:11
e8a487b to
7cba0c0
Compare
Contributor
Author
|
/ok to test 7cba0c0 |
ananthsub
marked this pull request as ready for review
August 31, 2026 19:15
5 tasks
Contributor
Author
5 tasks
bxyu-nvidia
approved these changes
Sep 9, 2026
ananthsub
added a commit
that referenced
this pull request
Sep 9, 2026
4 tasks done
bxyu-nvidia
pushed a commit
that referenced
this pull request
Sep 10, 2026
## Summary Restores the callable discriminated response item unions from #2889 on current `main` after #3183 reverted the original merge. `NeMoGymResponseInputItem` and `NeMoGymResponseOutputItem` are large untagged Pydantic unions. Validating each response item tries union members sequentially, so validation cost grows with both union size and trajectory length. The callable discriminators select one concrete model using item type and stable shape fields while preserving permissive typeless input behavior. Complete token metadata selects the corresponding training model; partial metadata is rejected instead of being silently discarded by a base model. This version is compatible with the refusal models added by #3165. Replay normalization uses `exclude_unset=True`, which can omit a refusal content part’s default `type="refusal"`. The content discriminator now recognizes that serialized shape by its `refusal` field, keeping refusal-only assistant messages routed to `NeMoGymResponseOutputMessage`. ## Pydantic validation microbenchmark The original #2889 measurements compared mixed message, reasoning, function, MCP, and training payloads against its then-current base: - 80 items: input union 22.19 -> 1.47 microseconds/item (15.1x); output union 22.70 -> 1.57 (14.4x); full request model 22.12 -> 1.60 (13.9x) - 320 items: input union 22.40 -> 1.48 microseconds/item (15.2x); output union 23.53 -> 1.58 (14.8x); full request model 22.24 -> 1.58 (14.1x) ## Test plan - [x] `uv run pytest tests/unit_tests/test_openai_utils.py tests/unit_tests/test_responses_api_model_streaming.py tests/unit_tests/test_responses_converter.py -q` (501 passed) - [x] `uv run pytest responses_api_models/inference_provider/tests/test_app.py -q` (23 passed) - [x] `uv run pytest tests/unit_tests/ -q` (4477 passed, 104 skipped, 20 subtests passed) - [x] `uv run pre-commit run --files nemo_gym/openai_utils.py tests/unit_tests/test_openai_utils.py` Related: #2889, #3165, #3183 Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
jkyi-nvidia
pushed a commit
that referenced
this pull request
Sep 11, 2026
## Summary `NeMoGymResponseInputItem` and `NeMoGymResponseOutputItem` are large untagged Pydantic unions. Validating each response item tries the union members in sequence to determine which model fits. This work repeats for every item whenever a request or response is validated, making validation cost grow with both the number of union members and the trajectory length. This change converts the unions to callable discriminated unions. The discriminator selects one tagged model from the item's `type` and, where multiple models share a type, from stable fields such as message role, content shape, ID, and status. Typeless input keeps its existing permissive fallback behavior. Complete token metadata selects the corresponding training model, while partial metadata is rejected instead of being discarded by a base model. Tests cover concrete class selection, all training variants, field preservation, and dump/revalidation. ## Pydantic validation microbenchmark Compared with `upstream/main` using mixed payloads containing messages, reasoning, function calls, function results, MCP items, training variants, and typeless tool turns. - 80 items: input union 22.19 -> 1.47 microseconds/item (15.1x); output union 22.70 -> 1.57 (14.4x); full request model 22.12 -> 1.60 (13.9x) - 320 items: input union 22.40 -> 1.48 microseconds/item (15.2x); output union 23.53 -> 1.58 (14.8x); full request model 22.24 -> 1.58 (14.1x) ## Test plan - [x] `uv run pytest tests/unit_tests/test_openai_utils.py tests/unit_tests/test_responses_api_model_streaming.py tests/unit_tests/test_responses_converter.py -q` - [x] scoped pre-commit checks for changed files - [x] `uv run pytest tests/unit_tests/ -q` Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
jkyi-nvidia
pushed a commit
that referenced
this pull request
Sep 11, 2026
Reverts #2889 Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
jkyi-nvidia
pushed a commit
that referenced
this pull request
Sep 11, 2026
## Summary Restores the callable discriminated response item unions from #2889 on current `main` after #3183 reverted the original merge. `NeMoGymResponseInputItem` and `NeMoGymResponseOutputItem` are large untagged Pydantic unions. Validating each response item tries union members sequentially, so validation cost grows with both union size and trajectory length. The callable discriminators select one concrete model using item type and stable shape fields while preserving permissive typeless input behavior. Complete token metadata selects the corresponding training model; partial metadata is rejected instead of being silently discarded by a base model. This version is compatible with the refusal models added by #3165. Replay normalization uses `exclude_unset=True`, which can omit a refusal content part’s default `type="refusal"`. The content discriminator now recognizes that serialized shape by its `refusal` field, keeping refusal-only assistant messages routed to `NeMoGymResponseOutputMessage`. ## Pydantic validation microbenchmark The original #2889 measurements compared mixed message, reasoning, function, MCP, and training payloads against its then-current base: - 80 items: input union 22.19 -> 1.47 microseconds/item (15.1x); output union 22.70 -> 1.57 (14.4x); full request model 22.12 -> 1.60 (13.9x) - 320 items: input union 22.40 -> 1.48 microseconds/item (15.2x); output union 23.53 -> 1.58 (14.8x); full request model 22.24 -> 1.58 (14.1x) ## Test plan - [x] `uv run pytest tests/unit_tests/test_openai_utils.py tests/unit_tests/test_responses_api_model_streaming.py tests/unit_tests/test_responses_converter.py -q` (501 passed) - [x] `uv run pytest responses_api_models/inference_provider/tests/test_app.py -q` (23 passed) - [x] `uv run pytest tests/unit_tests/ -q` (4477 passed, 104 skipped, 20 subtests passed) - [x] `uv run pre-commit run --files nemo_gym/openai_utils.py tests/unit_tests/test_openai_utils.py` Related: #2889, #3165, #3183 Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NeMoGymResponseInputItemandNeMoGymResponseOutputItemare large untagged Pydantic unions. Validating each response item tries the union members in sequence to determine which model fits. This work repeats for every item whenever a request or response is validated, making validation cost grow with both the number of union members and the trajectory length.This change converts the unions to callable discriminated unions. The discriminator selects one tagged model from the item's
typeand, where multiple models share a type, from stable fields such as message role, content shape, ID, and status. Typeless input keeps its existing permissive fallback behavior. Complete token metadata selects the corresponding training model, while partial metadata is rejected instead of being discarded by a base model.Tests cover concrete class selection, all training variants, field preservation, and dump/revalidation.
Pydantic validation microbenchmark
Compared with
upstream/mainusing mixed payloads containing messages, reasoning, function calls, function results, MCP items, training variants, and typeless tool turns.Test plan
uv run pytest tests/unit_tests/test_openai_utils.py tests/unit_tests/test_responses_api_model_streaming.py tests/unit_tests/test_responses_converter.py -quv run pytest tests/unit_tests/ -q