Repository navigation
fix(cli): stop --num-repeats from multiplying a benchmark's num_repeats - #5042
Merged
marcromeyn merged 3 commits intoOct 27, 2026
Merged
Conversation
`gym eval run --help` describes --num-repeats as "Number of rollouts per task", but with --benchmark it multiplied the benchmark's own num_repeats: data preparation wrote each task num_repeats times, then rollout collection ran every row --num-repeats times. terminal_bench_2_1/terminus_2 (num_repeats: 8) with --num-repeats 3 ran 24 per task. When num_repeats is set for the run, data preparation now writes each task once, so it alone sets the count. Validation and the metrics sidecars still use the dataset's num_repeats, so they don't change. Without it, a benchmark runs its own num_repeats as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kelly <kbuchanan@nvidia.com>
AdamRajfer
approved these changes
Oct 26, 2026
Contributor
|
/ok to test 59fbcf8 |
Contributor
|
Thanks, the fix looks right: |
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40529681001 |
7 tasks done
7 tasks done
fsiino-nvidia
added a commit
that referenced
this pull request
Nov 2, 2026
…5042 Since #5042, data preparation writes each task once whenever a run-level num_repeats is set, so the run-level value replaces each dataset's own. The Core recipe listed only APEX (16) and LMArena (3) with a default of 1, so every other member dropped from 8, 10 or 16 attempts per task to 1 (23,801 rollouts before, 9,427 now). The SWE recipe's run-level 1 did the same to its 3 attempts (2,400 rollouts before, 800 now). List every Core member's intended count in the run-level num_repeats, set num_repeats: 3 in swe.yaml, and update the matching expectations in test_super35_batch_launcher.py. num_repeats_add_seed is unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
7 tasks done
fsiino-nvidia
added a commit
that referenced
this pull request
Nov 13, 2026
…ataset A run-level num_repeats replaces every dataset's own count (#5042), so the recipes had to repeat every member's count. Drop it from both recipes: each member's dataset sets its count, and the Core recipe sets APEX's and LMArena's datasets to 16 and 3, since they repeat only in this batch. The launcher test checks that neither recipe sets a run-level num_repeats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #5041.
Problem
gym eval run --helpdescribes--num-repeatsas "Number of rollouts per task", but with--benchmarkit multiplies the benchmark's ownnum_repeatsinstead of replacing it.gym eval run --benchmark terminal_bench_2_1/terminus_2 --num-repeats 3runs 24 rollouts per task, not 3, because TB2.1 setsnum_repeats: 8. 48 of the 149 benchmarks on main set their ownnum_repeatsabove 1; the issue has the full table.Changes
nemo_gym/train_data_utils.pynum_repeats(--num-repeats), data preparation writes each task once instead of the dataset'snum_repeatstimes, so rollout collection's repeats alone set the count. Validation and the metrics sidecars still use the dataset'snum_repeats, so committed*_metrics.jsonfiles don't conflict.tests/unit_tests/test_train_data_utils.pytest_collate_writes_each_task_once_when_the_run_sets_num_repeats: anum_repeats: 3dataset is written 3× per task without a run value, and once with one.Compatibility
num_repeats: unchanged; each benchmark runs its ownnum_repeats.--num-repeats, ornum_repeatsin a run config): each task now runs exactly that many times. READMEs that assumed this become correct, e.g.proofnetandaime25-x(--num-repeats 32ran 1,024 per task).num_repeats_add_seedand--interleave-repeatsworking for every attempt.Testing
pytest tests/unit_tests/test_train_data_utils.py test_task_materialization.py test_cli_eval.py test_indic_aime_2026.py: 135 passed. The new test fails on main.TrainDataProcessor.runon a benchmark-type dataset withnum_repeats: 8(2 tasks):num_repeats: 3The second run passed the metrics-sidecar check.
ruff check/ruff format: clean.Model rollout: N/A. The change is in data preparation, before any agent or model runs.
Docs: no change needed. The help text and
cli-commands.mdxalready say "rollouts per task".Proposed labels:
bug,area:evaluation.🤖 Generated with Claude Code