Repository navigation
fix(benchmarks): restore the Core and SWE batch repeat counts after #5042 - #6095
fsiino-nvidia wants to merge 5 commits into
Conversation
…5042 Since #5042, data preparation writes each task once whenever a run-level num_repeats is set, so the run-level value replaces each dataset's own. The Core recipe listed only APEX (16) and LMArena (3) with a default of 1, so every other member dropped from 8, 10 or 16 attempts per task to 1 (23,801 rollouts before, 9,427 now). The SWE recipe's run-level 1 did the same to its 3 attempts (2,400 rollouts before, 800 now). List every Core member's intended count in the run-level num_repeats, set num_repeats: 3 in swe.yaml, update the launcher test, and update the README repeat paragraphs. num_repeats_add_seed is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…ark's repeat count Drop _default: 1 from the Core recipe's run-level num_repeats, so a member without an entry stops the run instead of silently getting one attempt. Check in the launcher test that each member's batch count equals its own benchmark's count (APEX and LMArena take theirs from the batch alone), and that SWE's run-level count matches its datasets, so the two sources cannot drift. Update the README example and its explanation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
The Core mapping covers all 11 members, and SWE uses three collection attempts without adding seeds. I traced preparation and collection and found no runtime correctness defect in these changes.
One optional NOTE: the updated test no longer pins the two batch-only repeat counts.
Static review only; I didn't run tests or independently verify the reported cluster artifacts.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| for server in members: | ||
| own = {dataset.num_repeats for dataset in server.datasets} | ||
| if server.name in run_level_only: | ||
| assert own <= {None, 1} and rollout.num_repeats[server.name] > 1 |
There was a problem hiding this comment.
NOTE — Suggestion: retain exact expectations for the batch-only counts.
The base assertion pinned APEX to 16 and LMArena to 3. This replacement checks only > 1: changing either batch count to 2 would now pass. These two exceptions have no dataset-side expected count to catch that drift. The current recipe is correct; this is lost regression protection, not a current runtime failure.
Could you replace the run_level_only set with the expected-count mapping below, then use its value in this branch?
run_level_only = {
"apex_shortlist_math_with_judge_simple_agent": 16,
"lmarena_v2_benchmark_agent": 3,
}With that dependent change, replace the anchored assertion with:
| assert own <= {None, 1} and rollout.num_repeats[server.name] > 1 | |
| assert own <= {None, 1} | |
| assert rollout.num_repeats[server.name] == run_level_only[server.name] |
Dictionary membership preserves the branch selection. Keep the dataset-side guard, exact Core member-key check, ordinary members' dataset comparisons, SWE comparison, and seed assertions unchanged.
Proposed acceptance checks, not executed here: the current recipe passes; changing only APEX to 2 or only LMArena to 2 fails; setting either exception's dataset repeats above 1 still fails; removing a Core mapping entry or adding _default still fails. Verify the suggested code before adopting it.
…ataset A run-level num_repeats replaces every dataset's own count (#5042), so the recipes had to repeat every member's count. Drop it from both recipes: each member's dataset sets its count, and the Core recipe sets APEX's and LMArena's datasets to 16 and 3, since they repeat only in this batch. The launcher test checks that neither recipe sets a run-level num_repeats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…restore-repeats Signed-off-by: Frankie Siino <fsiino@nvidia.com>
|
/ok to test b086a70 |
…ataset Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
|
/ok to test 889ca13 |
What does this PR do?
Restores the Nemotron 3.5 Super Core and SWE batch recipes' repeat counts, which #5042 cut to one attempt per task for most members.
Since #5042, a run-level
num_repeatsreplaces each dataset's own. The Core recipe listed only APEX (16) and LMArena (3), with_default: 1, so every other member dropped from 8, 10 or 16 attempts to 1: 9,427 rollouts instead of 23,801. SWE's run-levelnum_repeats: 1did the same to its 3 attempts: 800 rollouts instead of 2,400.The fix removes the run-level
num_repeatsfrom both recipes, so each member's count comes from one place, its own dataset:benchmarks/nemotron_3.5_super/batch_configs/core.yaml: no run-levelnum_repeats. Nine members already set their counts in their own configs. APEX and LMArena repeat only in this batch, so the recipe sets their datasets' counts (16 and 3), as it already does for Omniscience and IFBench.num_repeats_add_seedis unchanged.benchmarks/nemotron_3.5_super/batch_configs/swe.yaml: no run-levelnum_repeats; the SWE datasets already set 3.tests/unit_tests/benchmarks/test_super35_batch_launcher.py: checks that neither recipe sets a run-levelnum_repeats, that APEX and LMArena get 16 and 3, and that every SWE dataset gets 3.benchmarks/nemotron_3.5_super/README.md: the repeat example and explanation.Validation
39 passed, 6 skipped. Pre-commit clean.
Rollouts
Materialized rollouts per member, counted from
rollouts_materialized_inputs.jsonlof cluster runs ond86234de7(Super 3.5 step-120):Both totals match the counts before #5042. Each member's copies of a task form one task with rollout indexes 0 to N-1 (for example GPQA 198 tasks x 8, APEX 47 x 16, SWE-bench Verified 500 x 3), and every APEX and LMArena row carries its repeat's seed. The Core run set
CODE_GEN_SANDBOX_IMAGEfor LiveCodeBench (see below).Compatibility and benchmark impact
Core and SWE batch runs go back to the attempts per task they had before #5042, so their rollout counts and runtime grow about 2.5x (Core) and 3x (SWE), and scores are again averaged over the intended repeats. No code or config key changes. A run-level
num_repeats(for example+num_repeats=1for a quick check) still replaces every member's count.Not in this PR: since #4958, Core on Gym main also needs a LiveCodeBench sandbox image (
CODE_GEN_SANDBOX_IMAGE), which the recipe does not set.🤖 Generated with Claude Code
Checklist
pre-commit run --all-files) (so CI lint/format checks pass).git commit -s) (so the DCO check passes).