Repository navigation
feat(eval): extend generic slice metrics and preserve HLE labels - #4658
Conversation
Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>
…report_slice_fields
Add an opt-in `report_slice_fields` knob to the LLM judge server. For each
configured row field (HLE-Diamond uses `partition`) compute_metrics emits
slices/<field>/<label>/{question_count,rollout_count,mean/reward} plus a
missing-label count, and get_key_metrics promotes the slice means next to
mean/reward and judgement_parsing_issue_rate. The default (empty list)
leaves every existing metric unchanged. Additive to the domain slices in
PR #4658, which read the same row fields under domains/.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
… metrics
Initialising `metrics` from `_compute_slice_metrics` replaced the `if not
judged: return {}` block, the same lines that PR #4658 (report_hle_domains)
rewrites, so whichever landed second needed a manual one-line resolution.
Update `metrics` with the slices just before the final return instead: the
two hunks are now separated by unchanged lines and `git merge-file` against
origin/codex/hle-domain-metrics merges cleanly (exit 0).
The only behavioural change is the degenerate run in which no rollout was
judged: it now emits no metrics at all, as before the hook, and the test pins
that case for a server with `report_slice_fields: [partition]`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>
yaoyu-33
left a comment
There was a problem hiding this comment.
Thanks for adding domain and modality slices. Reusing compute_aggregate_metrics per slice, instead of a separate CI implementation, is a nice choice, and the flag-off path is byte-identical to before.
One blocker, found independently by two reviewers: in a real run the labels never reach the hook, so every HLE run would report missing_label_rollouts equal to all rollouts and no slices. The tests build aggregate rows that already carry the labels, so they never exercise the verify() → aggregate boundary. A short real smoke (even a few tasks) would confirm the fix.
What we verified: masked rows are excluded before the hook, quote(label, safe="") is injective, single-repeat and unequal-coverage slices suppress the CI, HLE-Verified prepare keeps all three fields for text and vision, and the 13 new tests plus the 57 existing equivalence_llm_judge tests pass.
…gation Signed-off-by: Igor Gitman <igitman@nvidia.com>
|
/ok to test 4be539b |
|
💬 Main contains the matching Kilo model-error repair in #5377 ( |
|
💬 At |
Signed-off-by: Igor Gitman <igitman@nvidia.com>
|
Updated to main Merged current main. All 158 HLE domain/metrics and equivalence-judge tests passed; full pre-commit passed. The test harness uses the same local hostname-resolution workaround as the previous validation. The old nonblocking Kilo infrastructure check is fixed upstream by #5377, now included. New-head CI is pending and may require qualified-maintainer trust authorization; local checks are not a claim that CI has passed. |
|
💬 The new head |
|
The model provider is unavailable. Work is paused and automatic retries have stopped. Send a new message or retry when the provider is available to resume. |
|
The label application does not own this session. |
|
I'll watch this ready-to-merge PR and report substantive progress. Running on |
|
/ok to test 596b7d9 |
adil-a
left a comment
There was a problem hiding this comment.
Thanks for adding domain and modality reporting; per-category results for HLE are useful. One structural concern before this lands: equivalence_llm_judge is the shared answer-equivalence grader, not an HLE server. FinanceBench, SecQue and the Nemotron 3.5 Super HLE config use it too. This PR builds HLE's schema and an HLE-named flag into it.
The server already has a generic mechanism for this, report_slice_fields, added for HLE-Diamond's partition. HLE could set report_slice_fields: [category, raw_subject, modality] and get per-label accuracy today. What this PR adds on top would benefit every benchmark on this grader if it extended report_slice_fields instead of sitting behind report_hle_domains: per-slice CIs across repeats, output-token means, and the modality-by-category cross-slice.
Suggested shape:
- Extend
report_slice_fieldswith the repeat CI and output-token stats, plus an optional cross-slice, for examplereport_slice_cross_fieldsor afield/fieldentry. - Have
benchmarks/hle/prepare.pyandhle_verified/prepare.pywritemodalitydirectly, instead of the server mappinghas_image→modality. - Drop
report_hle_domainsand the HLE fields from the shared request, task-data and verify-response models.extra="allow"already carries them through.
Inline comments below, including one regression for non-HLE users when the flag is off.
Signed-off-by: Igor Gitman <igitman@nvidia.com>
|
Pushed Addressed the structural review and the non-HLE regression. The shared grader now extends report_slice_fields and optional report_slice_cross_fields; HLE prepares modality directly, and extra row labels are never stripped. Removed the HLE-specific flag, shared-schema additions, and domains/ namespace. Existing partition slicing and negative configured rewards remain supported. 159 tests and a fresh two-answer real-judge smoke passed; full pre-commit passed. Updated the PR description to describe the final implementation and actual validation. New-head trusted CI still needs maintainer authorization. |
Signed-off-by: Igor Gitman <igitman@nvidia.com>
|
Merged current main ( Validation on |
Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>
|
/ok to test 79f8664 |
yfw
left a comment
There was a problem hiding this comment.
Approving the 11-09 rework at 79f8664. It follows the structural review: the HLE flag and HLE fields are gone from the shared grader, the statistics sit behind the generic report_slice_fields and report_slice_cross_fields, the HLE prepare scripts write modality, and row labels survive verify with slicing on or off. The new tests check each of these. Two small follow-ups, not blocking: tests/unit_tests/test_hle_domains.py tests server code, so AGENTS.md puts it in tests/unit_tests/benchmarks/; and a slice with a missing reward now leaves out mean/reward, where it used to count that reward as 0.
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43520300658 |
Extend the shared equivalence judge's existing
report_slice_fieldswith output-token means and repeat confidence intervals, and add optionalreport_slice_cross_fieldspairs. Arbitrary row labels survive verification through the existing open schemas. No HLE-specific flag or dataset fields are added to the shared grader.HLE and HLE-Verified preparation preserve category/raw-subject labels and emit
modalitydirectly. Their configs request category, subject, modality, and modality-by-category/subject slices. Metrics useslices/<field>/<label>/...andslices/<field1>/<field2>/<label1>/<label2>/..., with encoded field names and labels. Existing slice accuracy/count keys remain supported. The previous draft'sreport_hle_domainsanddomains/keys are removed; cached HLE data must be prepared again to include modality.Each slice reports question/rollout counts, reward/token means, missing observations, and Student-t intervals from Gym's public repeat aggregator. CIs require at least two repeats with matching question coverage. Invalid repeat identities suppress intervals without discarding row means; missing tokens suppress token statistics. Configured negative rewards remain valid. Slice work runs off the event loop, and masked outcomes remain excluded. Empty slice config preserves existing metrics; grading is unchanged.
This is a focused reporting feature requested for HLE; no separate issue is needed.
Validation:
openai/openai/gpt-5.6-luna: synthetic answers 4 and 5 to “What is 2 + 2?” scored 1 and 0. Text/vision labels survived verification and generic cross-slice aggregation; overall reward was 0.5 with no numeric has_image metric. These modality labels exercise metadata plumbing, not vision reasoning. Inspected both judge verdicts and aggregate output. No benchmark inference or score-parity claim.pre-commit run --all-filesand diff whitespace checks passed; DCO sign-off included.Validated head:
06df59914a515a376059c13c95169404f827d0b9. New-head trusted CI and reviewer approval remain pending.Current-main refresh (2026-11-13): includes
d50f0763ff03f3b7efd1e1e476d507001588ccf9. Validated head79f8664eaa41e814062e7809164d3992b8bbcd53: 1191 passed, 10 warnings. All-file pre-commit, whitespace and DCO checks passed. Three independent local review passes: SHIP using the protected-main rubric; this is not hosted approval. Hosted CI and maintainer review for this head remain independent.