Skip to content

fix(benchmarks): restore the Core and SWE batch repeat counts after #5042 - #6095

Open
fsiino-nvidia wants to merge 5 commits into
mainfrom
fsiino/batch-recipes-restore-repeats
Open

fsiino-nvidia wants to merge 5 commits into
mainfrom
fsiino/batch-recipes-restore-repeats

Conversation

@fsiino-nvidia

@fsiino-nvidia fsiino-nvidia commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Restores the Nemotron 3.5 Super Core and SWE batch recipes' repeat counts, which #5042 cut to one attempt per task for most members.

Since #5042, a run-level num_repeats replaces each dataset's own. The Core recipe listed only APEX (16) and LMArena (3), with _default: 1, so every other member dropped from 8, 10 or 16 attempts to 1: 9,427 rollouts instead of 23,801. SWE's run-level num_repeats: 1 did the same to its 3 attempts: 800 rollouts instead of 2,400.

The fix removes the run-level num_repeats from both recipes, so each member's count comes from one place, its own dataset:

  • benchmarks/nemotron_3.5_super/batch_configs/core.yaml: no run-level num_repeats. Nine members already set their counts in their own configs. APEX and LMArena repeat only in this batch, so the recipe sets their datasets' counts (16 and 3), as it already does for Omniscience and IFBench. num_repeats_add_seed is unchanged.
  • benchmarks/nemotron_3.5_super/batch_configs/swe.yaml: no run-level num_repeats; the SWE datasets already set 3.
  • tests/unit_tests/benchmarks/test_super35_batch_launcher.py: checks that neither recipe sets a run-level num_repeats, that APEX and LMArena get 16 and 3, and that every SWE dataset gets 3.
  • benchmarks/nemotron_3.5_super/README.md: the repeat example and explanation.

Validation

.venv/bin/pytest -o addopts= tests/unit_tests/benchmarks/test_super35_batch_launcher.py

39 passed, 6 skipped. Pre-commit clean.

Rollouts

Materialized rollouts per member, counted from rollouts_materialized_inputs.jsonl of cluster runs on d86234de7 (Super 3.5 step-120):

Batch Run Rollouts
Core Slurm 8999722 23,801: tau2 2,224, tau3 776, SciCode 520, HLE 2,158, GPQA 1,584, Omniscience 6,000, AA-LCR 1,600, APEX 752, LMArena 4,425, LiveCodeBench 1,362, IFBench 2,400
SWE Slurm 8999114 2,400: Verified 1,500, Multilingual 900

Both totals match the counts before #5042. Each member's copies of a task form one task with rollout indexes 0 to N-1 (for example GPQA 198 tasks x 8, APEX 47 x 16, SWE-bench Verified 500 x 3), and every APEX and LMArena row carries its repeat's seed. The Core run set CODE_GEN_SANDBOX_IMAGE for LiveCodeBench (see below).

Compatibility and benchmark impact

Core and SWE batch runs go back to the attempts per task they had before #5042, so their rollout counts and runtime grow about 2.5x (Core) and 3x (SWE), and scores are again averaged over the intended repeats. No code or config key changes. A run-level num_repeats (for example +num_repeats=1 for a quick check) still replaces every member's count.

Not in this PR: since #4958, Core on Gym main also needs a LiveCodeBench sandbox image (CODE_GEN_SANDBOX_IMAGE), which the recipe does not set.

🤖 Generated with Claude Code

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Documentation added or updated, or N/A with justification.
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
  • New source files include the required Apache-2.0 SPDX header.
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

fsiino-nvidia and others added 2 commits November 9, 2026 23:56
…5042

Since #5042, data preparation writes each task once whenever a run-level num_repeats is set, so the run-level value replaces each dataset's own. The Core recipe listed only APEX (16) and LMArena (3) with a default of 1, so every other member dropped from 8, 10 or 16 attempts per task to 1 (23,801 rollouts before, 9,427 now). The SWE recipe's run-level 1 did the same to its 3 attempts (2,400 rollouts before, 800 now).

List every Core member's intended count in the run-level num_repeats, set num_repeats: 3 in swe.yaml, update the launcher test, and update the README repeat paragraphs. num_repeats_add_seed is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…ark's repeat count

Drop _default: 1 from the Core recipe's run-level num_repeats, so a member without an entry stops the run instead of silently getting one attempt. Check in the launcher test that each member's batch count equals its own benchmark's count (APEX and LMArena take theirs from the batch alone), and that SWE's run-level count matches its datasets, so the two sources cannot drift. Update the README example and its explanation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

The Core mapping covers all 11 members, and SWE uses three collection attempts without adding seeds. I traced preparation and collection and found no runtime correctness defect in these changes.

One optional NOTE: the updated test no longer pins the two batch-only repeat counts.

Static review only; I didn't run tests or independently verify the reported cluster artifacts.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

for server in members:
own = {dataset.num_repeats for dataset in server.datasets}
if server.name in run_level_only:
assert own <= {None, 1} and rollout.num_repeats[server.name] > 1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE — Suggestion: retain exact expectations for the batch-only counts.

The base assertion pinned APEX to 16 and LMArena to 3. This replacement checks only > 1: changing either batch count to 2 would now pass. These two exceptions have no dataset-side expected count to catch that drift. The current recipe is correct; this is lost regression protection, not a current runtime failure.

Could you replace the run_level_only set with the expected-count mapping below, then use its value in this branch?

run_level_only = {
    "apex_shortlist_math_with_judge_simple_agent": 16,
    "lmarena_v2_benchmark_agent": 3,
}

With that dependent change, replace the anchored assertion with:

Suggested change
assert own <= {None, 1} and rollout.num_repeats[server.name] > 1
assert own <= {None, 1}
assert rollout.num_repeats[server.name] == run_level_only[server.name]

Dictionary membership preserves the branch selection. Keep the dataset-side guard, exact Core member-key check, ordinary members' dataset comparisons, SWE comparison, and seed assertions unchanged.

Proposed acceptance checks, not executed here: the current recipe passes; changing only APEX to 2 or only LMArena to 2 fails; setting either exception's dataset repeats above 1 still fails; removing a Core mapping entry or adding _default still fails. Verify the suggested code before adopting it.

fsiino-nvidia and others added 2 commits November 13, 2026 22:05
…ataset

A run-level num_repeats replaces every dataset's own count (#5042), so the recipes had to repeat every member's count. Drop it from both recipes: each member's dataset sets its count, and the Core recipe sets APEX's and LMArena's datasets to 16 and 3, since they repeat only in this batch. The launcher test checks that neither recipe sets a run-level num_repeats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…restore-repeats

Signed-off-by: Frankie Siino <fsiino@nvidia.com>
@fsiino-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test b086a70

@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team November 14, 2026 00:04
@yaoyu-33
yaoyu-33 requested a review from init-nikhil November 14, 2026 00:35
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Nov 16, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added bug Something isn't working area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters needs-review PR is ready for code review and waiting on a reviewer labels Nov 16, 2026
…ataset

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
@fsiino-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test 889ca13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants