Skip to content

gym eval run --num-repeats multiplies a benchmark's own num_repeats instead of setting rollouts per task #5041

Closed

Description

@ekellbuch

gym eval run --help describes --num-repeats as "Number of rollouts per task", but with --benchmark it multiplies the benchmark's own num_repeats instead of replacing it.

Example

gym eval run --benchmark terminal_bench_2_1/terminus_2 --num-repeats 3 runs 24 rollouts per task, not 3, because TB2.1 sets num_repeats: 8 in its config. --num-repeats 1 still runs 8.

Why

The benchmark config's num_repeats and the --num-repeats flag are applied at different stages:

  1. The benchmark config's num_repeats: data preparation writes each task that many times into the task list (TrainDataProcessor._iter_dataset_lines, train_data_utils.py).
  2. The --num-repeats flag: rollout collection then runs every row of that list that many times (rollout_collection.py).

Scope

48 of the 149 benchmarks on main set num_repeats above 1, so the flag's result depends on the benchmark, and sometimes on the agent. For example, SWE-bench Pro runs 3 per task with swebench/pro/opencode and 1 with swebench/pro/hermes.

Default num_repeats Benchmarks (config) --num-repeats 3 gives per task
50 ioi, ioi26 150
32 aime24, aime24-x, aime25, aime25-x, minif2f, mobench, proofnet, putnam_bench 96
16 aalcr, apex_shortlist (opencode, pi), birdbench, spider2_lite 48
10 deepswe/opencode, pinchbench 30
8 terminal_bench_2_1 (terminus_2, opencode, inkling_small), tau2 (4 configs), gpqa, gpqa-x, indic/gpqa_diamond, bunsenbench_chemistry_mcq, fukuyamabench, omniscience, scicode 24
5 assaybench, critpt, gdp_pdf, ifbench, litmus-bench (5 configs), longmt_pg19, longmt_wmt24pp 15
4 biomnibench_da, biomysterybench, bixbench_hypothesis, genebench_pro, hmmt_feb25, indic/aime_2026, mrcr, nemotron_imo_dev30, xstest 12
3 swebench (verified, pro, multilingual)/opencode, livecodebench (v5, v6, + cascade), openthoughts_tblite (+ terminus_2), finance_sec_search (2 configs), labbench2_vlm, partner_if 9
2 swe_together/opencode 6

Some benchmark READMEs already assume the flag replaces the count, so their commands run far more rollouts than intended:

README Command Benchmark's own num_repeats Rollouts per task today
benchmarks/proofnet --num-repeats 32 32 1,024
benchmarks/aime25-x --num-repeats 32 32 1,024
benchmarks/swe_together --num-repeats 2 2 4

The Nemotron 3.5 Super batch configs (benchmarks/nemotron_3.5_super/batch_configs/, #3201) rely on the multiplication and document it ("Collection repeats multiply any repeats already in the prepared inputs").

Options

  1. Set every benchmark's num_repeats to 1, so --num-repeats alone sets the count. This changes the default of all 48 benchmarks above: a plain gym eval run --benchmark aime25 would run 1 rollout instead of 32.
  2. Make a run's num_repeats replace the benchmark's own when it's set. Without it, each benchmark keeps its default.

The linked PRs implement (2), and update the Nemotron 3.5 Super batch configs so their counts don't change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    community-requestIssue reported or requested by someone from the community

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions