Skip to content

fix(cli): stop --num-repeats from multiplying a benchmark's num_repeats - #5042

Merged
marcromeyn merged 3 commits into
NVIDIA-NeMo:mainfrom
ekellbuch:kbuchanan/num-repeats-sets-attempts
Oct 27, 2026
Merged

marcromeyn merged 3 commits into
NVIDIA-NeMo:mainfrom
ekellbuch:kbuchanan/num-repeats-sets-attempts

Conversation

@ekellbuch

@ekellbuch ekellbuch commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Closes #5041.

Problem

gym eval run --help describes --num-repeats as "Number of rollouts per task", but with --benchmark it multiplies the benchmark's own num_repeats instead of replacing it. gym eval run --benchmark terminal_bench_2_1/terminus_2 --num-repeats 3 runs 24 rollouts per task, not 3, because TB2.1 sets num_repeats: 8. 48 of the 149 benchmarks on main set their own num_repeats above 1; the issue has the full table.

Changes

File Change
nemo_gym/train_data_utils.py When the run sets num_repeats (--num-repeats), data preparation writes each task once instead of the dataset's num_repeats times, so rollout collection's repeats alone set the count. Validation and the metrics sidecars still use the dataset's num_repeats, so committed *_metrics.json files don't conflict.
tests/unit_tests/test_train_data_utils.py test_collate_writes_each_task_once_when_the_run_sets_num_repeats: a num_repeats: 3 dataset is written 3× per task without a run value, and once with one.

Compatibility

  • Runs without a run-level num_repeats: unchanged; each benchmark runs its own num_repeats.
  • Runs that set one (--num-repeats, or num_repeats in a run config): each task now runs exactly that many times. READMEs that assumed this become correct, e.g. proofnet and aime25-x (--num-repeats 32 ran 1,024 per task).
  • The Nemotron 3.5 Super batch configs relied on the multiplication. fix(benchmarks): keep the Super 3.5 batch repeat counts when --num-repeats replaces num_repeats #5043 updates them so their counts stay the same; merge it together with this PR.
  • Running repeats in rollout collection keeps per-agent counts, num_repeats_add_seed and --interleave-repeats working for every attempt.

Testing

  • pytest tests/unit_tests/test_train_data_utils.py test_task_materialization.py test_cli_eval.py test_indic_aime_2026.py: 135 passed. The new test fails on main.

  • TrainDataProcessor.run on a benchmark-type dataset with num_repeats: 8 (2 tasks):

    No run value Run num_repeats: 3
    main 16 rows 16 rows (→ 48 rollouts)
    this PR 16 rows 2 rows (→ 6 rollouts)

    The second run passed the metrics-sidecar check.

  • ruff check / ruff format: clean.

  • Model rollout: N/A. The change is in data preparation, before any agent or model runs.

  • Docs: no change needed. The help text and cli-commands.mdx already say "rollouts per task".

Proposed labels: bug, area:evaluation.

🤖 Generated with Claude Code

`gym eval run --help` describes --num-repeats as "Number of rollouts per task", but with --benchmark it multiplied
the benchmark's own num_repeats: data preparation wrote each task num_repeats times, then rollout collection ran
every row --num-repeats times. terminal_bench_2_1/terminus_2 (num_repeats: 8) with --num-repeats 3 ran 24 per task.

When num_repeats is set for the run, data preparation now writes each task once, so it alone sets the count.
Validation and the metrics sidecars still use the dataset's num_repeats, so they don't change. Without it, a
benchmark runs its own num_repeats as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kelly <kbuchanan@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Oct 24, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Oct 24, 2026
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 26, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Oct 26, 2026
@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 26, 2026
@marcromeyn

Copy link
Copy Markdown
Contributor

/ok to test 59fbcf8

@marcromeyn

Copy link
Copy Markdown
Contributor

Thanks, the fix looks right: _iter_dataset_lines repeats each task back to back, so keeping every num_repeats-th row keeps exactly one copy of each. One non-blocking note: the skip is gated on num_repeats is not None, so a per-agent --num-repeats dict that doesn't name an agent (and has no _default) also drops that agent's dataset repeats and falls back to 1. Probably fine and arguably correct, but worth a line in the help text or a follow-up if you'd rather keep the dataset value in that case.

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40529681001

Merged via the queue into NVIDIA-NeMo:main with commit 778a37b Oct 27, 2026
78 of 86 checks passed
@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-maintainers Waiting on maintainers to respond label Oct 27, 2026
@athitten athitten mentioned this pull request Oct 28, 2026
7 tasks done
fsiino-nvidia added a commit that referenced this pull request Nov 2, 2026
…5042

Since #5042, data preparation writes each task once whenever a run-level num_repeats is set, so the run-level value replaces each dataset's own. The Core recipe listed only APEX (16) and LMArena (3) with a default of 1, so every other member dropped from 8, 10 or 16 attempts per task to 1 (23,801 rollouts before, 9,427 now). The SWE recipe's run-level 1 did the same to its 3 attempts (2,400 rollouts before, 800 now).

List every Core member's intended count in the run-level num_repeats, set num_repeats: 3 in swe.yaml, and update the matching expectations in test_super35_batch_launcher.py. num_repeats_add_seed is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
fsiino-nvidia added a commit that referenced this pull request Nov 13, 2026
…ataset

A run-level num_repeats replaces every dataset's own count (#5042), so the recipes had to repeat every member's count. Drop it from both recipes: each member's dataset sets its count, and the Core recipe sets APEX's and LMArena's datasets to 16 and 3, since they repeat only in this batch. The launcher test checks that neither recipe sets a run-level num_repeats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — 59fbcf8c Deployed Oct 27, 2026 by copy-pr-bot[bot] via release / finalize / notify #7603
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters bug Something isn't working community-request Issue reported or requested by someone from the community needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gym eval run --num-repeats multiplies a benchmark's own num_repeats instead of setting rollouts per task

4 participants