Skip to content

perf(openai-utils): discriminate response item unions - #2889

Merged
bxyu-nvidia merged 1 commit into
NVIDIA-NeMo:mainfrom
ananthsub:ananthsub/discriminated-response-items
Sep 9, 2026
Merged

bxyu-nvidia merged 1 commit into
NVIDIA-NeMo:mainfrom
ananthsub:ananthsub/discriminated-response-items

Conversation

@ananthsub

@ananthsub ananthsub commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Summary

NeMoGymResponseInputItem and NeMoGymResponseOutputItem are large untagged Pydantic unions. Validating each response item tries the union members in sequence to determine which model fits. This work repeats for every item whenever a request or response is validated, making validation cost grow with both the number of union members and the trajectory length.

This change converts the unions to callable discriminated unions. The discriminator selects one tagged model from the item's type and, where multiple models share a type, from stable fields such as message role, content shape, ID, and status. Typeless input keeps its existing permissive fallback behavior. Complete token metadata selects the corresponding training model, while partial metadata is rejected instead of being discarded by a base model.

Tests cover concrete class selection, all training variants, field preservation, and dump/revalidation.

Pydantic validation microbenchmark

Compared with upstream/main using mixed payloads containing messages, reasoning, function calls, function results, MCP items, training variants, and typeless tool turns.

  • 80 items: input union 22.19 -> 1.47 microseconds/item (15.1x); output union 22.70 -> 1.57 (14.4x); full request model 22.12 -> 1.60 (13.9x)
  • 320 items: input union 22.40 -> 1.48 microseconds/item (15.2x); output union 23.53 -> 1.58 (14.8x); full request model 22.24 -> 1.58 (14.1x)

Test plan

  • uv run pytest tests/unit_tests/test_openai_utils.py tests/unit_tests/test_responses_api_model_streaming.py tests/unit_tests/test_responses_converter.py -q
  • scoped pre-commit checks for changed files
  • uv run pytest tests/unit_tests/ -q

@ananthsub
ananthsub marked this pull request as draft August 31, 2026 17:57
@copy-pr-bot

copy-pr-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Route response input and output items through callable Pydantic
discriminators instead of trying every union member. Shape-based tags resolve
shared message and function-call-output literals while preserving permissive
typeless input handling.

Require token metadata to remain atomic on both input and output items so an
incomplete payload cannot select a base model and lose fields. Add regression
coverage for concrete routing, training variants, and dump/revalidation.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@ananthsub
ananthsub force-pushed the ananthsub/discriminated-response-items branch from e8a487b to 7cba0c0 Compare August 31, 2026 19:11
@copy-pr-bot

copy-pr-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ananthsub ananthsub added the r0.6.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge. label Aug 31, 2026
@ananthsub

Copy link
Copy Markdown
Contributor Author

/ok to test 7cba0c0

@ananthsub
ananthsub requested a review from bxyu-nvidia August 31, 2026 19:15
@ananthsub
ananthsub marked this pull request as ready for review August 31, 2026 19:15
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Sep 1, 2026
@ananthsub

Copy link
Copy Markdown
Contributor Author

Tracked by the performance roadmap in #2998. This PR directly implements #3005.

@ananthsub ananthsub linked an issue Sep 3, 2026 that may be closed by this pull request
5 tasks
@bxyu-nvidia
bxyu-nvidia merged commit c61f85b into NVIDIA-NeMo:main Sep 9, 2026
41 checks passed
bxyu-nvidia pushed a commit that referenced this pull request Sep 10, 2026
## Summary

Restores the callable discriminated response item unions from #2889 on
current `main` after #3183 reverted the original merge.

`NeMoGymResponseInputItem` and `NeMoGymResponseOutputItem` are large
untagged Pydantic unions. Validating each response item tries union
members sequentially, so validation cost grows with both union size and
trajectory length. The callable discriminators select one concrete model
using item type and stable shape fields while preserving permissive
typeless input behavior. Complete token metadata selects the
corresponding training model; partial metadata is rejected instead of
being silently discarded by a base model.

This version is compatible with the refusal models added by #3165.
Replay normalization uses `exclude_unset=True`, which can omit a refusal
content part’s default `type="refusal"`. The content discriminator now
recognizes that serialized shape by its `refusal` field, keeping
refusal-only assistant messages routed to
`NeMoGymResponseOutputMessage`.

## Pydantic validation microbenchmark

The original #2889 measurements compared mixed message, reasoning,
function, MCP, and training payloads against its then-current base:

- 80 items: input union 22.19 -> 1.47 microseconds/item (15.1x); output
union 22.70 -> 1.57 (14.4x); full request model 22.12 -> 1.60 (13.9x)
- 320 items: input union 22.40 -> 1.48 microseconds/item (15.2x); output
union 23.53 -> 1.58 (14.8x); full request model 22.24 -> 1.58 (14.1x)

## Test plan

- [x] `uv run pytest tests/unit_tests/test_openai_utils.py
tests/unit_tests/test_responses_api_model_streaming.py
tests/unit_tests/test_responses_converter.py -q` (501 passed)
- [x] `uv run pytest
responses_api_models/inference_provider/tests/test_app.py -q` (23
passed)
- [x] `uv run pytest tests/unit_tests/ -q` (4477 passed, 104 skipped, 20
subtests passed)
- [x] `uv run pre-commit run --files nemo_gym/openai_utils.py
tests/unit_tests/test_openai_utils.py`

Related: #2889, #3165, #3183

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
jkyi-nvidia pushed a commit that referenced this pull request Sep 11, 2026
## Summary

`NeMoGymResponseInputItem` and `NeMoGymResponseOutputItem` are large
untagged Pydantic unions. Validating each response item tries the union
members in sequence to determine which model fits. This work repeats for
every item whenever a request or response is validated, making
validation cost grow with both the number of union members and the
trajectory length.

This change converts the unions to callable discriminated unions. The
discriminator selects one tagged model from the item's `type` and, where
multiple models share a type, from stable fields such as message role,
content shape, ID, and status. Typeless input keeps its existing
permissive fallback behavior. Complete token metadata selects the
corresponding training model, while partial metadata is rejected instead
of being discarded by a base model.

Tests cover concrete class selection, all training variants, field
preservation, and dump/revalidation.

## Pydantic validation microbenchmark

Compared with `upstream/main` using mixed payloads containing messages,
reasoning, function calls, function results, MCP items, training
variants, and typeless tool turns.

- 80 items: input union 22.19 -> 1.47 microseconds/item (15.1x); output
union 22.70 -> 1.57 (14.4x); full request model 22.12 -> 1.60 (13.9x)
- 320 items: input union 22.40 -> 1.48 microseconds/item (15.2x); output
union 23.53 -> 1.58 (14.8x); full request model 22.24 -> 1.58 (14.1x)

## Test plan
- [x] `uv run pytest tests/unit_tests/test_openai_utils.py
tests/unit_tests/test_responses_api_model_streaming.py
tests/unit_tests/test_responses_converter.py -q`
- [x] scoped pre-commit checks for changed files
- [x] `uv run pytest tests/unit_tests/ -q`

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
jkyi-nvidia pushed a commit that referenced this pull request Sep 11, 2026
Reverts #2889

Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
jkyi-nvidia pushed a commit that referenced this pull request Sep 11, 2026
## Summary

Restores the callable discriminated response item unions from #2889 on
current `main` after #3183 reverted the original merge.

`NeMoGymResponseInputItem` and `NeMoGymResponseOutputItem` are large
untagged Pydantic unions. Validating each response item tries union
members sequentially, so validation cost grows with both union size and
trajectory length. The callable discriminators select one concrete model
using item type and stable shape fields while preserving permissive
typeless input behavior. Complete token metadata selects the
corresponding training model; partial metadata is rejected instead of
being silently discarded by a base model.

This version is compatible with the refusal models added by #3165.
Replay normalization uses `exclude_unset=True`, which can omit a refusal
content part’s default `type="refusal"`. The content discriminator now
recognizes that serialized shape by its `refusal` field, keeping
refusal-only assistant messages routed to
`NeMoGymResponseOutputMessage`.

## Pydantic validation microbenchmark

The original #2889 measurements compared mixed message, reasoning,
function, MCP, and training payloads against its then-current base:

- 80 items: input union 22.19 -> 1.47 microseconds/item (15.1x); output
union 22.70 -> 1.57 (14.4x); full request model 22.12 -> 1.60 (13.9x)
- 320 items: input union 22.40 -> 1.48 microseconds/item (15.2x); output
union 23.53 -> 1.58 (14.8x); full request model 22.24 -> 1.58 (14.1x)

## Test plan

- [x] `uv run pytest tests/unit_tests/test_openai_utils.py
tests/unit_tests/test_responses_api_model_streaming.py
tests/unit_tests/test_responses_converter.py -q` (501 passed)
- [x] `uv run pytest
responses_api_models/inference_provider/tests/test_app.py -q` (23
passed)
- [x] `uv run pytest tests/unit_tests/ -q` (4477 passed, 104 skipped, 20
subtests passed)
- [x] `uv run pre-commit run --files nemo_gym/openai_utils.py
tests/unit_tests/test_openai_utils.py`

Related: #2889, #3165, #3183

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: Junkeun Yi <jkyi@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — 7cba0c01 Deployed Aug 31, 2026 by copy-pr-bot[bot] via release / finalize / notify #2329
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

r0.6.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge. sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use discriminated unions for Responses API items

2 participants