Repository navigation
Description
gym eval run --help describes --num-repeats as "Number of rollouts per task", but with --benchmark it multiplies the benchmark's own num_repeats instead of replacing it.
Example
gym eval run --benchmark terminal_bench_2_1/terminus_2 --num-repeats 3 runs 24 rollouts per task, not 3, because TB2.1 sets num_repeats: 8 in its config. --num-repeats 1 still runs 8.
Why
The benchmark config's num_repeats and the --num-repeats flag are applied at different stages:
- The benchmark config's
num_repeats: data preparation writes each task that many times into the task list (TrainDataProcessor._iter_dataset_lines,train_data_utils.py). - The
--num-repeatsflag: rollout collection then runs every row of that list that many times (rollout_collection.py).
Scope
48 of the 149 benchmarks on main set num_repeats above 1, so the flag's result depends on the benchmark, and sometimes on the agent. For example, SWE-bench Pro runs 3 per task with swebench/pro/opencode and 1 with swebench/pro/hermes.
Default num_repeats |
Benchmarks (config) | --num-repeats 3 gives per task |
|---|---|---|
| 50 | ioi, ioi26 | 150 |
| 32 | aime24, aime24-x, aime25, aime25-x, minif2f, mobench, proofnet, putnam_bench | 96 |
| 16 | aalcr, apex_shortlist (opencode, pi), birdbench, spider2_lite | 48 |
| 10 | deepswe/opencode, pinchbench | 30 |
| 8 | terminal_bench_2_1 (terminus_2, opencode, inkling_small), tau2 (4 configs), gpqa, gpqa-x, indic/gpqa_diamond, bunsenbench_chemistry_mcq, fukuyamabench, omniscience, scicode | 24 |
| 5 | assaybench, critpt, gdp_pdf, ifbench, litmus-bench (5 configs), longmt_pg19, longmt_wmt24pp | 15 |
| 4 | biomnibench_da, biomysterybench, bixbench_hypothesis, genebench_pro, hmmt_feb25, indic/aime_2026, mrcr, nemotron_imo_dev30, xstest | 12 |
| 3 | swebench (verified, pro, multilingual)/opencode, livecodebench (v5, v6, + cascade), openthoughts_tblite (+ terminus_2), finance_sec_search (2 configs), labbench2_vlm, partner_if | 9 |
| 2 | swe_together/opencode | 6 |
Some benchmark READMEs already assume the flag replaces the count, so their commands run far more rollouts than intended:
| README | Command | Benchmark's own num_repeats |
Rollouts per task today |
|---|---|---|---|
benchmarks/proofnet |
--num-repeats 32 |
32 | 1,024 |
benchmarks/aime25-x |
--num-repeats 32 |
32 | 1,024 |
benchmarks/swe_together |
--num-repeats 2 |
2 | 4 |
The Nemotron 3.5 Super batch configs (benchmarks/nemotron_3.5_super/batch_configs/, #3201) rely on the multiplication and document it ("Collection repeats multiply any repeats already in the prepared inputs").
Options
- Set every benchmark's
num_repeatsto 1, so--num-repeatsalone sets the count. This changes the default of all 48 benchmarks above: a plaingym eval run --benchmark aime25would run 1 rollout instead of 32. - Make a run's
num_repeatsreplace the benchmark's own when it's set. Without it, each benchmark keeps its default.
The linked PRs implement (2), and update the Nemotron 3.5 Super batch configs so their counts don't change.
Activity