Skip to content

feat(eval): extend generic slice metrics and preserve HLE labels - #4658

Merged
yfw merged 18 commits into
mainfrom
codex/hle-domain-metrics
Nov 13, 2026
Merged

yfw merged 18 commits into
mainfrom
codex/hle-domain-metrics

Conversation

@Kipok

@Kipok Kipok commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Extend the shared equivalence judge's existing report_slice_fields with output-token means and repeat confidence intervals, and add optional report_slice_cross_fields pairs. Arbitrary row labels survive verification through the existing open schemas. No HLE-specific flag or dataset fields are added to the shared grader.

HLE and HLE-Verified preparation preserve category/raw-subject labels and emit modality directly. Their configs request category, subject, modality, and modality-by-category/subject slices. Metrics use slices/<field>/<label>/... and slices/<field1>/<field2>/<label1>/<label2>/..., with encoded field names and labels. Existing slice accuracy/count keys remain supported. The previous draft's report_hle_domains and domains/ keys are removed; cached HLE data must be prepared again to include modality.

Each slice reports question/rollout counts, reward/token means, missing observations, and Student-t intervals from Gym's public repeat aggregator. CIs require at least two repeats with matching question coverage. Invalid repeat identities suppress intervals without discarding row means; missing tokens suppress token statistics. Configured negative rewards remain valid. Slice work runs off the event loop, and masked outcomes remain excluded. Empty slice config preserves existing metrics; grading is unchanged.

This is a focused reporting feature requested for HLE; no separate issue is needed.

Validation:

  • 159 tests passed across the equivalence judge, HLE-Diamond, generic slices, and HLE/HLE-Verified preparation. Includes arbitrary-label verification-to-aggregation, disabled slicing, cross-slices, repeat coverage, missing observations, negative rewards, and unchanged HLE-Diamond behavior.
  • Fresh real-judge smoke with openai/openai/gpt-5.6-luna: synthetic answers 4 and 5 to “What is 2 + 2?” scored 1 and 0. Text/vision labels survived verification and generic cross-slice aggregation; overall reward was 0.5 with no numeric has_image metric. These modality labels exercise metadata plumbing, not vision reasoning. Inspected both judge verdicts and aggregate output. No benchmark inference or score-parity claim.
  • Full pre-commit run --all-files and diff whitespace checks passed; DCO sign-off included.

Validated head: 06df59914a515a376059c13c95169404f827d0b9. New-head trusted CI and reviewer approval remain pending.

Current-main refresh (2026-11-13): includes d50f0763ff03f3b7efd1e1e476d507001588ccf9. Validated head 79f8664eaa41e814062e7809164d3992b8bbcd53: 1191 passed, 10 warnings. All-file pre-commit, whitespace and DCO checks passed. Three independent local review passes: SHIP using the protected-main rubric; this is not hosted approval. Hosted CI and maintainer review for this head remain independent.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Kipok Kipok added area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters feature New capabilities, enhancements, or enablement work labels Oct 15, 2026
Kipok added 3 commits October 15, 2026 16:53
Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok Kipok changed the title feat(eval): preserve HLE domains and report accuracy and token intervals feat(eval): preserve HLE labels and report domain and modality metrics Oct 16, 2026
@Kipok
Kipok marked this pull request as ready for review October 16, 2026 00:13
@Kipok Kipok added the needs-review PR is ready for code review and waiting on a reviewer label Oct 16, 2026
@yaoyu-33 yaoyu-33 added the complexity:medium Single-domain change with interacting parts or a moderate review surface label Oct 16, 2026
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 19, 2026
rimashahbazyan added a commit that referenced this pull request Oct 21, 2026
…report_slice_fields

Add an opt-in `report_slice_fields` knob to the LLM judge server. For each
configured row field (HLE-Diamond uses `partition`) compute_metrics emits
slices/<field>/<label>/{question_count,rollout_count,mean/reward} plus a
missing-label count, and get_key_metrics promotes the slice means next to
mean/reward and judgement_parsing_issue_rate. The default (empty list)
leaves every existing metric unchanged. Additive to the domain slices in
PR #4658, which read the same row fields under domains/.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
rimashahbazyan added a commit that referenced this pull request Oct 21, 2026
… metrics

Initialising `metrics` from `_compute_slice_metrics` replaced the `if not
judged: return {}` block, the same lines that PR #4658 (report_hle_domains)
rewrites, so whichever landed second needed a manual one-line resolution.
Update `metrics` with the slices just before the final return instead: the
two hunks are now separated by unchanged lines and `git merge-file` against
origin/codex/hle-domain-metrics merges cleanly (exit 0).

The only behavioural change is the degenerate run in which no rollout was
judged: it now emits no metrics at all, as before the hook, and the test pins
that case for a server with `report_slice_fields: [partition]`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Kipok added 2 commits October 21, 2026 21:24
Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>

@yaoyu-33 yaoyu-33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding domain and modality slices. Reusing compute_aggregate_metrics per slice, instead of a separate CI implementation, is a nice choice, and the flag-off path is byte-identical to before.

One blocker, found independently by two reviewers: in a real run the labels never reach the hook, so every HLE run would report missing_label_rollouts equal to all rollouts and no slices. The tests build aggregate rows that already carry the labels, so they never exercise the verify() → aggregate boundary. A short real smoke (even a few tasks) would confirm the fix.

What we verified: masked rows are excluded before the hook, quote(label, safe="") is injective, single-repeat and unequal-coverage slices suppress the CI, HLE-Verified prepare keeps all three fields for text and vision, and the 13 new tests plus the 57 existing equivalence_llm_judge tests pass.

@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 22, 2026
…gation

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@marcromeyn

Copy link
Copy Markdown
Contributor

/ok to test 4be539b

Copy link
Copy Markdown
Contributor

💬 Main contains the matching Kilo model-error repair in #5377 (a38b363d), but this PR doesn't include it. The next step is to incorporate that fix and rerun conformance. I haven't verified the live probe on current main: CI skipped Kilo, and the managed test adapter has no conformance selector. I'll keep watching the remaining CI and review.

Copy link
Copy Markdown
Contributor

💬 At 731796e8, all four core-test shards, all eight server-test shards, sandbox tests, combined coverage and session-routing E2E passed. The completed workflow's only failed jobs were the previously diagnosed Kilo conformance probe and its result publisher. The PR remains open and labeled.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok

Kipok commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Updated to main 2ac7908c41438e025f763e8cbf00134ce2f80906; new head 596b7d971dd6b510d273c0dee5e77d6e9f29a416.

Merged current main. All 158 HLE domain/metrics and equivalence-judge tests passed; full pre-commit passed. The test harness uses the same local hostname-resolution workaround as the previous validation. The old nonblocking Kilo infrastructure check is fixed upstream by #5377, now included.

New-head CI is pending and may require qualified-maintainer trust authorization; local checks are not a claim that CI has passed.

Copy link
Copy Markdown
Contributor

💬 The new head 596b7d97 now includes the upstream Kilo model-error repair. Lint, copyright, secrets and DCO checks passed. Full test and conformance CI hasn't appeared on this head yet, so the previous test results don't establish readiness for it.

Copy link
Copy Markdown
Contributor

The model provider is unavailable. Work is paused and automatic retries have stopped. Send a new message or retry when the provider is available to resume.

Copy link
Copy Markdown
Contributor

The label application does not own this session.

Copy link
Copy Markdown
Contributor

I'll watch this ready-to-merge PR and report substantive progress. Running on nim-gpt-6-1-sol.

@adil-a

adil-a commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

/ok to test 596b7d9

@adil-a adil-a left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding domain and modality reporting; per-category results for HLE are useful. One structural concern before this lands: equivalence_llm_judge is the shared answer-equivalence grader, not an HLE server. FinanceBench, SecQue and the Nemotron 3.5 Super HLE config use it too. This PR builds HLE's schema and an HLE-named flag into it.

The server already has a generic mechanism for this, report_slice_fields, added for HLE-Diamond's partition. HLE could set report_slice_fields: [category, raw_subject, modality] and get per-label accuracy today. What this PR adds on top would benefit every benchmark on this grader if it extended report_slice_fields instead of sitting behind report_hle_domains: per-slice CIs across repeats, output-token means, and the modality-by-category cross-slice.

Suggested shape:

  • Extend report_slice_fields with the repeat CI and output-token stats, plus an optional cross-slice, for example report_slice_cross_fields or a field/field entry.
  • Have benchmarks/hle/prepare.py and hle_verified/prepare.py write modality directly, instead of the server mapping has_image → modality.
  • Drop report_hle_domains and the HLE fields from the shared request, task-data and verify-response models. extra="allow" already carries them through.

Inline comments below, including one regression for non-HLE users when the flag is off.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok Kipok changed the title feat(eval): preserve HLE labels and report domain and modality metrics feat(eval): extend generic slice metrics and preserve HLE labels Nov 9, 2026
@Kipok

Kipok commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 06df59914a515a376059c13c95169404f827d0b9.

Addressed the structural review and the non-HLE regression. The shared grader now extends report_slice_fields and optional report_slice_cross_fields; HLE prepares modality directly, and extra row labels are never stripped. Removed the HLE-specific flag, shared-schema additions, and domains/ namespace. Existing partition slicing and negative configured rewards remain supported. 159 tests and a fresh two-answer real-judge smoke passed; full pre-commit passed.

Updated the PR description to describe the final implementation and actual validation. New-head trusted CI still needs maintainer authorization.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok

Kipok commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Merged current main (22f6c7971fd36cf225870ffefc35ef7a645a9e0b) and resolved the HLE-Diamond test conflict. HLE and HLE-Verified now retain main's case-insensitive verdict-label setting alongside this PR's generic slice fields.

Validation on 6f6e72db43: 178 targeted tests passed; all-file pre-commit passed. This is an integration/test-expectation correction; the previously recorded model smoke remains the runtime evidence.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
Signed-off-by: Igor Gitman <igitman@nvidia.com>
@yfw

yfw commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

/ok to test 79f8664

@nemo-automation-bot
nemo-automation-bot Bot requested review from a team November 13, 2026 22:09

@yfw yfw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the 11-09 rework at 79f8664. It follows the structural review: the HLE flag and HLE fields are gone from the shared grader, the statistics sit behind the generic report_slice_fields and report_slice_cross_fields, the HLE prepare scripts write modality, and row labels survive verify with slicing on or off. The new tests check each of these. Two small follow-ups, not blocking: tests/unit_tests/test_hle_domains.py tests server code, so AGENTS.md puts it in tests/unit_tests/benchmarks/; and a slice with a missing reward now leaves out mean/reward, where it used to count that reward as 0.

@yfw
yfw added this pull request to the merge queue Nov 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43520300658

Merged via the queue into main with commit 7f2a663 Nov 13, 2026
68 checks passed
@yfw
yfw deleted the codex/hle-domain-metrics branch November 13, 2026 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work ready-to-merge PR is approved, current, and waiting for final CI or merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants