Skip to content

fix(arena): recognize marked context-window rejections in metrics - #5208

Merged
AdamRajfer merged 3 commits into
mainfrom
arajfer/fix-arena-context-diagnostic
Oct 30, 2026
Merged

AdamRajfer merged 3 commits into
mainfrom
arajfer/fix-arena-context-diagnostic

Conversation

@AdamRajfer

@AdamRajfer AdamRajfer commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Arena now recognizes vllm_model's explicit context-window rejection marker when the response reports zero token usage. Previously this response entered the output-token-limit metric bucket. Historical saved responses without the marker retain their missing-usage fallback.

The regression covers the current marked response, the historical representation, an ordinary token-limit response and a reasoning-only response. On unchanged main, the output-limit rate is 0.4 instead of the expected 0.2; the patch passes all 183 Arena tests. All pre-commit hooks pass. Tested on public main 0d0f6c0.

This focused metadata/metric mismatch is covered directly by the regression; no separate issue is needed. Live model-backed diagnostic validation remains pending.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer AdamRajfer added bug Something isn't working area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters labels Oct 28, 2026
@AdamRajfer AdamRajfer self-assigned this Oct 28, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

laszkiewiczp
laszkiewiczp previously approved these changes Oct 29, 2026
Cover invalid_output_token_parameter responses with zero usage and no context-window marker. Keep the existing marked context-exceeded case and verify the separate output-limit rate.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer
AdamRajfer marked this pull request as ready for review October 29, 2026 11:04
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 29, 2026 11:04
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Oct 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

LGTM — no concerns found. The metric classification matches vllm_model’s explicit marker while preserving historical missing-usage fallback, unmarked zero-usage output-limit classification, and existing scoring behavior.

The updated regression includes the previously requested unmarked zero-usage case and retains the marked-context, ordinary truncation, and reasoning-only cases.

Static review of both changed files and affected producer/consumer paths completed. Tests and model-backed validation were not executed in this review.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@AdamRajfer
AdamRajfer enabled auto-merge October 29, 2026 11:56
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40874333991

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 29, 2026
@AdamRajfer
AdamRajfer added this pull request to the merge queue Oct 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40958426199

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 29, 2026
@AdamRajfer
AdamRajfer added this pull request to the merge queue Oct 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41049389730

Merged via the queue into main with commit e0867da Oct 30, 2026
48 of 51 checks passed
@AdamRajfer
AdamRajfer deleted the arajfer/fix-arena-context-diagnostic branch October 30, 2026 12:16

This branch was previously deployed

1 inactive deployment
public — 6697e68b Deployed Oct 29, 2026 by copy-pr-bot[bot] via release / finalize / notify #8105
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants