Repository navigation
fix(arena): recognize marked context-window rejections in metrics - #5208
Conversation
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Cover invalid_output_token_parameter responses with zero usage and no context-window marker. Keep the existing marked context-exceeded case and verify the separate output-limit rate. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
🌿 Preview your docs: https://nvidia-preview-pr-5208-f76e658317f0.docs.buildwithfern.com/nemo/gym |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
LGTM — no concerns found. The metric classification matches vllm_model’s explicit marker while preserving historical missing-usage fallback, unmarked zero-usage output-limit classification, and existing scoring behavior.
The updated regression includes the previously requested unmarked zero-usage case and retains the marked-context, ordinary truncation, and reasoning-only cases.
Static review of both changed files and affected producer/consumer paths completed. Tests and model-backed validation were not executed in this review.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40874333991 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40958426199 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41049389730 |
Arena now recognizes vllm_model's explicit context-window rejection marker when the response reports zero token usage. Previously this response entered the output-token-limit metric bucket. Historical saved responses without the marker retain their missing-usage fallback.
The regression covers the current marked response, the historical representation, an ordinary token-limit response and a reasoning-only response. On unchanged main, the output-limit rate is 0.4 instead of the expected 0.2; the patch passes all 183 Arena tests. All pre-commit hooks pass. Tested on public main 0d0f6c0.
This focused metadata/metric mismatch is covered directly by the regression; no separate issue is needed. Live model-backed diagnostic validation remains pending.