Skip to content

fix(frontierscience): accept numeric-only rubric grades - #5415

Merged
adil-a merged 3 commits into
mainfrom
codex/frontierscience-numeric-verdict
Nov 5, 2026
Merged

adil-a merged 3 commits into
mainfrom
codex/frontierscience-numeric-verdict

Conversation

@Kipok

@Kipok Kipok commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

A completed FrontierScience Research judge response containing only 0 is rejected because it lacks the VERDICT: prefix. Accept a response consisting entirely of one numeric score within the configured rubric bounds. Explanations still require the final VERDICT: line; intermediate scores, fractions, non-finite values and out-of-range scores remain invalid.

The judge prompt, model settings, scoring scale and solved threshold are unchanged. This extends formatting compatibility from #4891 and allows previously rejected unambiguous grades to contribute their actual score. No separate issue is needed for this focused parser correction.

Validation:

  • 116 focused FrontierScience server and shared judge tests passed, including numeric-only zero scoring and rejection boundaries.
  • pre-commit run --all-files passed; DCO sign-off included.
  • Tested in a real judge-only recovery with saved policy answers. Completion and integrity checks confirmed full coverage, unchanged policy responses and unchanged previously successful grades.

Repository CI remains for the standard review workflow.

Saved-cohort before/after validation: replayed both parsers on identical judge replies for 60 tasks × 16 repeats from Qwen3.8 NVFP4 (low effort), judged by GPT-6.1 Sol (high effort). Coverage increases from 959/960 to 960/960 by accepting one numeric-only zero; all 959 previously accepted grades are unchanged, and all 960 patched scores/rewards match the stored evaluation. Pooled solved/graded changes from 140/959 to 140/960, and mean raw rubric score from 4.1516649635 to 4.1473403125. No new policy/judge calls. The cohort includes 61 incomplete policy responses. Full comparison, configuration, method, caveats and artifact hashes.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok Kipok added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Nov 1, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE

RISK — Supply the required before/after benchmark reward numbers

This changes grading outcomes for the fixed FrontierScience Research benchmark: a completed response of 0 now contributes a graded zero instead of entering the judge-failure sidecar. The verification and metric paths preserve that zero, so restored coverage can intentionally change aggregate results.

The PR description reports real judge-only recovery and unchanged previously successful grades, but provides no before/after reward numbers or linked comparison. The pinned review rubric requires this evidence for fixed-benchmark verifier changes. The historical reproduction results do not compare this change. This is a validation-evidence gap, not a demonstrated scoring defect.

Correction: Could you add a compact before/after table from the reported saved-answer recovery? Identify the policy/judge configuration and task/repeat counts; report graded and judge-failure counts, solved rate, and mean raw rubric score with explicit denominators. Separate previously successful judgments from newly accepted numeric-only judgments, and cite the corresponding recovery/profile output. No code change or committed rollout artifacts are requested.

Preservation and acceptance: On the identical saved policy-answer cohort, the comparison should show unchanged grades for previously successful judgments and reconcile every newly graded row and remaining failure. Keep the paper prompt, judge settings, 0–10 scale, and threshold 7 unchanged. Aggregate shifts caused solely by restored coverage are expected; the evidence should distinguish those from unintended grade changes. These are proposed evidence checks, not checks executed during this review.

Static inspection covered all three changed files and directly affected parser, verification, completion/error, configuration, and metric paths. No implementation defects were found. Tests were not executed.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@Kipok

Kipok commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks; the denominator change is intentional and needs to be explicit. I cannot publish the saved-answer recovery cohort or its model-level results, so the earlier real-run statement is not independently reproducible evidence for that comparison.

I ran this separate, fully disclosed synthetic profile against the parent parser and this PR's parser. It is a parser/aggregation regression example, not a model benchmark result. Five fixed judge texts, one occurrence each; no policy or judge API calls: VERDICT: 10, VERDICT: 7, VERDICT: 3, 0, and The answer mentions 8 but no final verdict. The scale is 0–10 and the solved threshold is 7.

Metric Parent PR
Graded / total 3/5 4/5
Judge failures / total 2/5 1/5
Solved / graded 2/3 (66.67%) 2/4 (50%)
Raw score sum / graded 20/3 (6.6667) 20/4 (5.0)
Previously graded scores 10, 7, 3 10, 7, 3
Numeric-only 0 judge failure score 0, unsolved

The prose without a final verdict remains a failure; all three previously accepted grades remain identical. The still-incomplete cohort should not be advertised as a completed benchmark score. This demonstrates the expected coverage/denominator effect but does not substitute for the requested publishable model-cohort comparison. The paper prompt, judge settings, scale, and threshold are unchanged by the patch.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok

Kipok commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Merged upstream main 483674910 in 15da2dd73403eeb252ef341dd72e1c485aa0eafb and reconciled the package-layout changes.

95 FrontierScience tests pass. The earlier request for a publishable before/after model-cohort comparison remains open: the disclosed synthetic comparison demonstrates the denominator effect, but is not model-benchmark evidence. No new cohort or benchmark score is claimed.

pre-commit run --all-files, diff whitespace checks and 724 public-API/import-boundary tests pass. This update merges main and adapts imports; no new model rollout or full local coverage run is claimed. Trusted CI needs vetter authorization for the new head.

@Kipok

Kipok commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

The before/after evidence is now available from a saved real-evaluation cohort. This supersedes my earlier statement that the model-level results could not be published.

I applied the old and patched parse_rubric_score functions to exactly the same 960 saved judge replies: 60 FrontierScience Research tasks × 16 repeats. This is an offline counterfactual parser replay, not two independently generated evaluations. No new policy or judge calls were made.

Metric Before After
Graded / total 959/960 960/960
Parser failures / total 1/960 0/960
Solved / graded 140/959 (14.598540%) 140/960 (14.583333%)
Raw rubric score sum / graded 3981.4466999999/959 3981.4466999999/960
Mean raw rubric score (0–10) 4.1516649635 4.1473403125
Previously accepted grades changed — 0/959
Patched scores/rewards disagreeing with stored eval — 0/960

Exactly one saved judge reply consists solely of 0: it becomes a valid zero/unsolved grade. All previously accepted grades remain identical. The aggregate decreases come entirely from including that zero in the denominator; solved count and score sum are unchanged. These solved rates use pooled solved/graded denominators, not a recomputation of Gym's unequal-coverage repeat-weighted baseline.

Configuration: Qwen3.8-2.4T-A95B NVFP4, revision a6abb8bfb82b6965166d4129275f012d8809bc40, low reasoning effort, thinking enabled, temperature 1.0, top-p 0.95, top-k 20, min-p 0, presence penalty 0, repetition penalty 1.0, remaining-context output budget. Judge: openai/openai/gpt-6.1-sol, high reasoning effort, Chat Completions judge path, no explicit output-token cap, the pinned Research prompt, scale 0–10 and solved threshold ≥7. The cohort includes 61 incomplete policy responses, including the newly accepted zero; all remain in the fixed cohort. “960 graded” describes verifier coverage, not 960 completed policy generations.

Method and provenance:

  • Before parser: 0c7ac5119c99e0aa0dfeda7be59157c589680ffd; after: 6f5bbc0589238e99b864ed20158f8bfb416d311e. Extracted the exact parser function and regex constant via Python AST and applied them to every saved judge_output.
  • Current head 15da2dd73403eeb252ef341dd72e1c485aa0eafb has the identical parser AST and produces the same 960 results. Research prompt bytes are identical at all three revisions.
  • Reconciled all 960 unique task/repeat pairs against materialized inputs: no missing or duplicate pairs. Patched scores and thresholded rewards match all stored results.
  • The historical recovery separately carried 957 successful grades unchanged and retried three judge failures, preserving policy responses. Those retries are not three parser-fix effects: the comparison above freezes the final replies and isolates one acceptance difference.

Retained evidence SHA-256:

rollouts.jsonl
7acdd1f7a72b0994f29f60848f5ac2fd76f247c646ffa22f003dd630985228d7
rollouts_materialized_inputs.jsonl
0ff744ca7ba64849d54fe9c2cc96282475fd014fb312f24e1e05e7274f688888
rollouts_aggregate_metrics.json
f694bea24328f4b64b2b5418a372c1c499edb220ff34c3e04204bc60c79da456
research_judge.yaml
82b210ee17b3c6d1c99e6c3b59af7cc8d4a1f39334035550a6e801d6cb8f0daf

The full raw artifacts and comparison script are retained; the hashes identify them but do not make the raw transcripts publicly downloadable. This is evidence of this parser change's effect on one saved real-evaluation cohort, not a fresh model evaluation of current head or an estimate of numeric-only reply frequency across models.

@cmunley1

cmunley1 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

/ok to test 15da2dd

@cmunley1

cmunley1 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

/review

yaoyu-33
yaoyu-33 previously approved these changes Nov 4, 2026

@yaoyu-33 yaoyu-33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The fallback is tight: it only accepts a response that is a single number on the rubric scale, and explanations still need the final VERDICT: line. The 960-reply replay is the right evidence for the denominator change: 959 grades are unchanged, and the one numeric-only zero is now graded.

One nit inline. CI is green apart from the Autobot check.

@yaoyu-33 yaoyu-33 added ready-to-merge PR is approved, current, and waiting for final CI or merge and removed needs-review PR is ready for code review and waiting on a reviewer labels Nov 4, 2026

Copy link
Copy Markdown
Contributor

🤖 On it — handling this as babysit-ready-to-merge.

1 similar comment

Copy link
Copy Markdown
Contributor

🤖 On it — handling this as babysit-ready-to-merge.

Copy link
Copy Markdown
Contributor

I'll watch this ready-to-merge PR and report substantive progress. Running on nim-gpt-6-1-sol.

Copy link
Copy Markdown
Contributor

💬 CI diagnosis is blocked: the platform can't resolve this session's workstation authority, and the identical retry also failed. I couldn't read the failing jobs or logs, so I haven't diagnosed the failure or requested a CI rerun. Triage needs platform access restored; I'll retain the PR watch.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok

Kipok commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in 16f8a9137: compiled _NUMERIC_SCORE_RE at module scope and reused its number grammar in _RUBRIC_VERDICT_RE. All 95 FrontierScience tests pass, as do pre-commit and whitespace checks. This is a parser refactor with unchanged accepted grammar; the published 960-reply replay evidence remains applicable. New head needs trusted-CI authorization.

Copy link
Copy Markdown
Contributor

💬 The workstation recovered, but the GitHub read and its one retry both failed with HTTP 503: “native Git authority is unavailable.” I still can't inspect the PR or CI logs, so I haven't requested a CI rerun. Diagnosis needs the Git broker restored; the PR watch remains active.

Copy link
Copy Markdown
Contributor

💬 Head 16f8a913 passes lint, DCO, copyright and secrets checks, but has no unit/server CI results; its CI request job was skipped. Gym requires fresh trusted-CI authorization after a head change, so that needs a qualified maintainer’s attention—not a failed-job rerun. The PR remains open and mergeable. I’ll retain the watch; older failure logs remain unread because the broker denied access.

Copy link
Copy Markdown
Contributor

💬 The latest refresh confirmed the PR is still open at 16f8a913, but the main-ref and workflow-run reads failed with credential_rejected. Those reads remain unverified until broker access recovers. I haven’t requested a CI retry; the PR watch remains active.

Copy link
Copy Markdown
Contributor

💬 The latest PR refresh failed twice because the Git access service returned HTTP 503. I couldn’t verify the current PR, main, CI, or review state. The PR watch remains active while access is unavailable.

Copy link
Copy Markdown
Contributor

💬 The GitHub refresh succeeded. The PR remains open at 16f8a913: four checks passed, but the CI request was skipped and no unit/server test run is reported for this head. There’s no failed run to retry, so I’m continuing the watch.

@adil-a
adil-a enabled auto-merge November 5, 2026 18:35
@adil-a

adil-a commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

/ok to test 16f8a91

Copy link
Copy Markdown
Contributor

💬 Trusted CI has started for 16f8a913. The reported lint run passed. The observed failure is in the non-blocking KiloCode conformance probe: a controlled model error became an HTTP 500 instead of the expected structured failure response. I’m comparing it with current main before attributing it to this PR. Core/server tests were still running in the latest result.

@adil-a
adil-a added this pull request to the merge queue Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42090132662

Copy link
Copy Markdown
Contributor

💬 All three required checks pass for 16f8a913, and GitHub records an approval. The remaining red checks are non-blocking KiloCode conformance checks; the same scenario passes on main, which includes the matching error-handling fix from #5377. I’ll keep watching until GitHub confirms the merge.

Merged via the queue into main with commit 42faef8 Nov 5, 2026
62 of 65 checks passed
@adil-a
adil-a deleted the codex/frontierscience-numeric-verdict branch November 5, 2026 19:23

Copy link
Copy Markdown
Contributor

The PR is no longer an open ready-to-merge PR targeting main.

This branch was previously deployed

1 inactive deployment
public — 16f8a913 Deployed Nov 5, 2026 by copy-pr-bot[bot] via release / finalize / notify #10421
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working ready-to-merge PR is approved, current, and waiting for final CI or merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants