Repository navigation
fix(frontierscience): accept numeric-only rubric grades - #5415
Conversation
Signed-off-by: Igor Gitman <igitman@nvidia.com>
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE
RISK — Supply the required before/after benchmark reward numbers
This changes grading outcomes for the fixed FrontierScience Research benchmark: a completed response of 0 now contributes a graded zero instead of entering the judge-failure sidecar. The verification and metric paths preserve that zero, so restored coverage can intentionally change aggregate results.
The PR description reports real judge-only recovery and unchanged previously successful grades, but provides no before/after reward numbers or linked comparison. The pinned review rubric requires this evidence for fixed-benchmark verifier changes. The historical reproduction results do not compare this change. This is a validation-evidence gap, not a demonstrated scoring defect.
Correction: Could you add a compact before/after table from the reported saved-answer recovery? Identify the policy/judge configuration and task/repeat counts; report graded and judge-failure counts, solved rate, and mean raw rubric score with explicit denominators. Separate previously successful judgments from newly accepted numeric-only judgments, and cite the corresponding recovery/profile output. No code change or committed rollout artifacts are requested.
Preservation and acceptance: On the identical saved policy-answer cohort, the comparison should show unchanged grades for previously successful judgments and reconcile every newly graded row and remaining failure. Keep the paper prompt, judge settings, 0–10 scale, and threshold 7 unchanged. Aggregate shifts caused solely by restored coverage are expected; the evidence should distinguish those from unintended grade changes. These are proposed evidence checks, not checks executed during this review.
Static inspection covered all three changed files and directly affected parser, verification, completion/error, configuration, and metric paths. No implementation defects were found. Tests were not executed.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
|
Thanks; the denominator change is intentional and needs to be explicit. I cannot publish the saved-answer recovery cohort or its model-level results, so the earlier real-run statement is not independently reproducible evidence for that comparison. I ran this separate, fully disclosed synthetic profile against the parent parser and this PR's parser. It is a parser/aggregation regression example, not a model benchmark result. Five fixed judge texts, one occurrence each; no policy or judge API calls:
The prose without a final verdict remains a failure; all three previously accepted grades remain identical. The still-incomplete cohort should not be advertised as a completed benchmark score. This demonstrates the expected coverage/denominator effect but does not substitute for the requested publishable model-cohort comparison. The paper prompt, judge settings, scale, and threshold are unchanged by the patch. |
Signed-off-by: Igor Gitman <igitman@nvidia.com>
|
Merged upstream main 95 FrontierScience tests pass. The earlier request for a publishable before/after model-cohort comparison remains open: the disclosed synthetic comparison demonstrates the denominator effect, but is not model-benchmark evidence. No new cohort or benchmark score is claimed.
|
|
The before/after evidence is now available from a saved real-evaluation cohort. This supersedes my earlier statement that the model-level results could not be published. I applied the old and patched
Exactly one saved judge reply consists solely of Configuration: Qwen3.8-2.4T-A95B NVFP4, revision Method and provenance:
Retained evidence SHA-256: The full raw artifacts and comparison script are retained; the hashes identify them but do not make the raw transcripts publicly downloadable. This is evidence of this parser change's effect on one saved real-evaluation cohort, not a fresh model evaluation of current head or an estimate of numeric-only reply frequency across models. |
|
/ok to test 15da2dd |
|
/review |
yaoyu-33
left a comment
There was a problem hiding this comment.
Thanks. The fallback is tight: it only accepts a response that is a single number on the rubric scale, and explanations still need the final VERDICT: line. The 960-reply replay is the right evidence for the denominator change: 959 grades are unchanged, and the one numeric-only zero is now graded.
One nit inline. CI is green apart from the Autobot check.
|
🤖 On it — handling this as babysit-ready-to-merge. |
1 similar comment
|
🤖 On it — handling this as babysit-ready-to-merge. |
|
I'll watch this ready-to-merge PR and report substantive progress. Running on |
|
💬 CI diagnosis is blocked: the platform can't resolve this session's workstation authority, and the identical retry also failed. I couldn't read the failing jobs or logs, so I haven't diagnosed the failure or requested a CI rerun. Triage needs platform access restored; I'll retain the PR watch. |
Signed-off-by: Igor Gitman <igitman@nvidia.com>
|
Addressed in |
|
💬 The workstation recovered, but the GitHub read and its one retry both failed with HTTP 503: “native Git authority is unavailable.” I still can't inspect the PR or CI logs, so I haven't requested a CI rerun. Diagnosis needs the Git broker restored; the PR watch remains active. |
|
💬 Head |
|
💬 The latest refresh confirmed the PR is still open at |
|
💬 The latest PR refresh failed twice because the Git access service returned HTTP 503. I couldn’t verify the current PR, main, CI, or review state. The PR watch remains active while access is unavailable. |
|
💬 The GitHub refresh succeeded. The PR remains open at |
|
/ok to test 16f8a91 |
|
💬 Trusted CI has started for |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42090132662 |
|
💬 All three required checks pass for |
|
The PR is no longer an open ready-to-merge PR targeting main. |
A completed FrontierScience Research judge response containing only
0is rejected because it lacks theVERDICT:prefix. Accept a response consisting entirely of one numeric score within the configured rubric bounds. Explanations still require the finalVERDICT:line; intermediate scores, fractions, non-finite values and out-of-range scores remain invalid.The judge prompt, model settings, scoring scale and solved threshold are unchanged. This extends formatting compatibility from #4891 and allows previously rejected unambiguous grades to contribute their actual score. No separate issue is needed for this focused parser correction.
Validation:
pre-commit run --all-filespassed; DCO sign-off included.Repository CI remains for the standard review workflow.
Saved-cohort before/after validation: replayed both parsers on identical judge replies for 60 tasks × 16 repeats from Qwen3.8 NVFP4 (low effort), judged by GPT-6.1 Sol (high effort). Coverage increases from 959/960 to 960/960 by accepting one numeric-only zero; all 959 previously accepted grades are unchanged, and all 960 patched scores/rewards match the stored evaluation. Pooled solved/graded changes from 140/959 to 140/960, and mean raw rubric score from 4.1516649635 to 4.1473403125. No new policy/judge calls. The cohort includes 61 incomplete policy responses. Full comparison, configuration, method, caveats and artifact hashes.