Skip to content

feat(eval): write a completion file per agent as soon as its rollouts are settled - #5386

Open
fsiino-nvidia wants to merge 26 commits into
mainfrom
fsiino/batched-evals-agent-complete
Open

fsiino-nvidia wants to merge 26 commits into
mainfrom
fsiino/batched-evals-agent-complete

Conversation

@fsiino-nvidia

@fsiino-nvidia fsiino-nvidia commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Writes a completion file for each agent in a multi-benchmark (batch) run as soon as every rollout of that agent is settled, so a finished benchmark's numbers are available while other benchmarks in the same run are still running, stuck, or killed.

Today a batch run (several benchmarks in one gym eval job, such as the Nemotron 3.5 Super Core and SWE batches) aggregates once, after the whole dispatch loop ends, into <stem>_aggregate_metrics.json. One benchmark that never finishes therefore withholds every other benchmark's score, even when those finished hours earlier. This blocks the picking up of finished benchmarks' scores.

How it works

The collection changes are in nemo_gym/rollouts/rollout_collection.py; nemo_gym/rollouts/rollout_records.py and nemo_gym/rollouts/rollout_store.py gain the settled-rollout rule.

  1. Setup. At startup, if aggregation is enabled and the materialized rows aggregate into more than one environment server, Gym records each agent's expected (task, rollout) keys. An agent here is the environment server that scores its rows, as the run-wide aggregate groups them (_metrics_group), so a task routed to a server and a row naming the agent behind it are one agent. Its pending keys are the expected keys minus the rollouts already settled, so a resumed job counts what an earlier job finished. Other runs stop here and behave as before.
  2. Track. Each time a rollout's outcome is recorded (a success, a failure or an intentional omission), its key leaves the pending set if the outcome is settled: RolloutRecords.settled, the complement of the rule that decides what a resume runs again. A success masked by infrastructure, or a failure a resume will retry, stays pending. When a set empties, a background task starts for that agent. Agents already finished by an earlier job start before dispatch.
  3. Aggregate and write. The task reads the agent's selected rows through one handle per file, adds the failures the run's count_failure_classes_as_zero counts as zero (the same helpers as the end-of-run pass), sorts by (task, rollout), calls the environment server's /aggregate_metrics while servers are still up, and writes <stem>_agent_complete/<agent>.json atomically (for results/rollouts.jsonl, results/rollouts_agent_complete/). _call_aggregate_metrics is split into _aggregate_entries(results, rows, *, labels=None) plus a thin wrapper, and both paths label entries through one helper (_server_agents), with labels computed over the full run so a per-agent call labels its entry as the full run does.
  4. End of run. The tasks are awaited, then the run-wide <stem>_aggregate_metrics.json is written exactly as before. A fresh run deletes the completion directory.

The file holds schema_version, agent_ref, _ng_environment_server, expected_rollouts, completed_rollouts (scored), failed_rollouts, completed_at, aggregation_status (complete or error) and aggregate (the agent's entry from the run-wide aggregate). The filename percent-encodes characters other than letters, digits, ., _ and -, so distinct agent names never share a file.

Behavior to know:

  • Complete means every rollout of the agent is settled. A failure no resume retries (a terminal class or an exhausted attempt budget) settles its rollout and enters the score as the end of the run would count it. A failure a resume will retry, or a result masked by infrastructure, keeps the agent pending until the rerun, so a file never goes stale and its entry equals the run-wide one, apart from _ng_run_health_metrics, the run-level health summary Gym adds to the first run-wide entry after the run.
  • A persisted result without a reward is not a scored rollout. An agent with an execute-only or reward-less result gets no file; the result stays in the output and the run-wide aggregate.
  • Failures never fail the run. An /aggregate_metrics error, or a call that gives no answer within 30 minutes (AGENT_COMPLETE_AGGREGATION_TIMEOUT_S), is recorded in the file (aggregation_status: "error"), so a hung environment server cannot hold up the run-wide aggregate. Any other failure writing the file is logged and no file is written. A run that stops early cancels and waits for its completion tasks and their in-flight /aggregate_metrics requests.
  • The run-wide aggregate is unchanged and authoritative. It is first written at the end of collection. A partial one is never written, because readers (nemo_gym/rollouts/batch_status.py, nemo_gym/eval/comparison/loading.py, benchmarks/nemotron_3.5_super/export_to_csv.py) treat it as final. A later gym eval aggregate repair rewrites it but not the completion files, so after a repair the run-wide entry wins over a file that recorded an aggregation error.
  • Re-judging keeps the files current. A judge failure keeps its saved answer and settles its rollout, so its agent can complete during the job. The saved-answer re-judge pass at the end of gym eval run and the judge-only recovery job can then grade that answer again and recompute the run-wide aggregate in place. Both now refresh each completion file to its agent's recomputed entry and counts (refresh_agent_complete_files), and remove the files when that fails, or when a run that requires every rollout stays incomplete after the pass and so gets no recomputed metrics.
  • A resume rewrites the file of an agent a previous job already finished, so completed_at becomes the resuming job's time. It deletes the file of an agent it still has rollouts to run for, until they settle.
  • An agent fronted by several environment servers is labeled by the environment server in its file, while the run-wide aggregate falls back to the agent name when only one of those servers ends with scored rollouts.

Files: nemo_gym/rollouts/rollout_collection.py, nemo_gym/cli/eval.py (re-judge refresh), nemo_gym/rollouts/rollout_records.py, nemo_gym/rollouts/rollout_store.py, nemo_gym/utils/path_utils.py (agent_complete_dir_for), tests/unit_tests/rollouts/test_rollout_collection.py, tests/unit_tests/eval/test_rejudge_complete_only.py, tests/unit_tests/eval/test_judge_only_settle_receipt.py, benchmarks/nemotron_3.5_super/README.md, fern/versions/latest/pages/evaluation/aggregate-metrics.mdx.

Validation

On the current head:

.venv/bin/pre-commit run --files nemo_gym/rollouts/rollout_collection.py nemo_gym/rollouts/rollout_records.py \
  nemo_gym/rollouts/rollout_store.py tests/unit_tests/rollouts/test_rollout_collection.py \
  benchmarks/nemotron_3.5_super/README.md fern/versions/latest/pages/evaluation/aggregate-metrics.mdx
.venv/bin/pytest -o addopts= -o log_cli=false -p no:cacheprovider -n 8 \
  tests/unit_tests/rollouts tests/unit_tests/benchmarks/test_super35_batch_launcher.py \
  <the cli, eval, orchestration and telemetry test files that use the rollout store, coverage or labels>

Result: pre-commit clean, 3,669 tests passed. One timing-sensitive case in tests/unit_tests/rollouts/test_legacy_deadline_collection.py failed once under parallel load and passed on rerun, with and without this change. tests/unit_tests/artifacts/test_exporters.py and test_live_export_receipt.py need mlflow, which the local environment lacks.

The completion-file tests in tests/unit_tests/rollouts/test_rollout_collection.py cover:

  • A finished agent's file exists while another agent is still running, and the run-wide aggregate does not, with results kept in memory or read back from the file.
  • Repeats are aggregated in (task, rollout) order even when the store hands rows back reversed, so metrics over a task's first repeats match the final aggregate.
  • Resume: an agent finished earlier gets its file and is not dispatched again.
  • A masked rollout keeps its agent pending in the job, and after a resume reruns it the file equals the run-wide entry (the reviewer's reproduction).
  • A final failure settles its rollout, with the failure left out or counted as zero; a retryable failure keeps the agent pending until a resume settles it.
  • A routed task and a row naming its agent are one agent; a run that aggregates into one environment server writes no files; two environment servers fronting one agent each get the run-wide label.
  • Execute-only and reward-less results suppress the file; a reward of zero does not.
  • Agent names that differ only in punctuation get distinct files.
  • A judge failure graded again by the re-judge pass refreshes its agent's file to the recomputed run-wide entry and counts, an unchanged file keeps its completed_at, and a file with no recomputed entry is removed; the end-of-run pass and the judge-only job remove the files when the recompute or the refresh fails, and a strict run left incomplete removes them.
  • No files for a single-agent run or with disable_aggregation; an /aggregate_metrics 500 is recorded in the file; a failed file write is logged and the run-wide aggregate is still written.
  • A resume deletes the file of an agent it reruns; an early /aggregate_metrics call that never answers is recorded as a TimeoutError while the run-wide aggregate is still written; a cancelled run stops its completion tasks and their requests before it returns.

Rollouts

Cluster runs with the Super 3.5 step-120 checkpoint, through the NeMo Evaluator Factory benchmark runner, whose batch-status reads the files. Run 1 is on a850e8dd9, which has this review round's changes (grouping by environment server, settled rollouts, filename encoding). Runs 2 and 3 are on b1b8f0d28; runs 4 to 6 are on earlier commits (c11ceb6c6 and 08d13588f).

  1. Core smoke on a850e8dd9, all 11 agents, 190 rollouts (Slurm 8798318, 46 min, exit 0). The same small Core recipe as run 2. All 11 completion files were written during the job, between 02:56:30 (tau2) and 03:21:26 (IFBench), while the run-wide aggregate was written at 03:21:35. Every file carries failed_rollouts (0 here). Ten files are identical to their entries in the final run-wide aggregate. The eleventh, AA-LCR, has identical scores; its final entry adds only _ng_run_health_metrics, the run-level health summary Gym attaches to the first run-wide entry after the run. batch-status shows all 11 accepted. No tracebacks.
  2. Core smoke, all 11 agents, 190 rollouts (Slurm 8555339, b1b8f0d28, 43 min, exit 0). A small version of the real Core recipe: a few tasks per agent, repeats above 1 on several agents, and 40 LMArena tasks x 3 so LMArena's own failure-rate check passes. All 11 completion files were written during the job, between 19:55:55 (tau2) and 20:21:38 (IFBench), while the job ended at 20:23:36. All 11 are identical to their entries in the final run-wide aggregate, including the multi-repeat benchmarks, and batch-status shows all 11 accepted. HLE completed with its current judge. No tracebacks.
  3. SWE smoke, 3 Verified + 30 Multilingual rollouts (Slurm 8553607, b1b8f0d28, 32 min). Verified's file was written at 18:15:29 and Multilingual's at 18:33:05, and the job ended at 18:34. Both files are identical to the final aggregate, and batch-status accepted both. No tracebacks.
  4. Earlier two-agent smoke, 3 SWE-bench Verified + 30 Multilingual rollouts (Slurm 8415667, 51 min). Verified's file appeared at 19:31:52 while Multilingual was at 10 of 30, and batch-status accepted Verified then. Both files later matched the final aggregate field for field.
  5. Full SWE batch, 500 Verified + 300 Multilingual rollouts (Slurm 8433527, 1h46m). Verified's file was written 48 minutes before the job ended. Both files were identical to the final aggregate (mean reward 0.694 and 0.66, all rollouts scored). No tracebacks or retries.
  6. Full Core batch, 11 agents (Slurm 8455414), stopped before finishing. Nine agents wrote files while the job ran. The job hit its 4-hour limit with LMArena unfinished (3,872 of 4,425), so no run-wide aggregate was ever written, yet the nine files gave batch-status nine finished benchmarks' scores that would otherwise all have been pending. HLE never finished because its judge at the time (azure/openai/gpt-4o) returned 403, which is unrelated to this change and since replaced in the recipe. It made HLE a natural stuck benchmark.

Not exercised: a full-size Core run on the current head, and a benchmark whose rollouts hang forever (the unit tests cover a blocked /run).

Compatibility and benchmark impact

Additive. The new files go in a new <stem>_agent_complete/ directory next to the existing outputs. No config key or CLI flag was added, and the run-wide aggregate, its format and its readers are unchanged. Benchmark results are unaffected. nemo_gym/rollouts/batch_status.py does not read the new files. Outside consumers such as the EFB runner's batch-status do, and fall back to the run-wide aggregate when they are absent.

The early scores are not written into batch_status.json: that file exists only when a run is given batch_manifest_fpath, which the EFB batches do not use, and it is a progress and status file rather than a place for scored results. The completion files need no manifest, so they work for any multi-agent run.

The Core and SWE batch repeat-count restore (needed since #5042) first landed here and has moved to its own PR, #6095, so this PR no longer changes any recipe.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Documentation added or updated, or N/A with justification.
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
  • New source files include the required Apache-2.0 SPDX header.
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

🤖 Generated with Claude Code

… are scored

Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 30, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…agent-complete

Signed-off-by: Frankie Siino <fsiino@nvidia.com>
@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters feature New capabilities, enhancements, or enablement work labels Oct 30, 2026
fsiino-nvidia and others added 3 commits October 31, 2026 00:30
…agent-complete

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

# Conflicts:
#	nemo_gym/rollout_collection.py
…5042

Since #5042, data preparation writes each task once whenever a run-level num_repeats is set, so the run-level value replaces each dataset's own. The Core recipe listed only APEX (16) and LMArena (3) with a default of 1, so every other member dropped from 8, 10 or 16 attempts per task to 1 (23,801 rollouts before, 9,427 now). The SWE recipe's run-level 1 did the same to its 3 attempts (2,400 rollouts before, 800 now).

List every Core member's intended count in the run-level num_repeats, set num_repeats: 3 in swe.yaml, and update the matching expectations in test_super35_batch_launcher.py. num_repeats_add_seed is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…agent-complete

Signed-off-by: Frankie Siino <fsiino@nvidia.com>
@fsiino-nvidia fsiino-nvidia changed the title Fsiino/batched evals agent complete feat(eval): write a completion file per agent as soon its rollouts are scored Nov 3, 2026
@fsiino-nvidia
fsiino-nvidia marked this pull request as ready for review November 3, 2026 00:29

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD

One BLOCKER can publish a partial benchmark score as complete. Four RISK findings cover unscored results, colliding artifact filenames, inconsistent aggregate labels, and stale repeat-count instructions.

I inspected all eight changed files and traced the affected routing, persistence/resume, aggregation, and data-preparation contracts. The ordinary alpha/beta completion, repeat ordering, aggregation-error recording, and write-failure isolation have targeted regression coverage. I couldn't execute tests in this read-only review; the acceptance checks below are proposed.

Note: formal approval is not available for this request, so this review is posted as a comment only.


Inline comments (escaped v1):

  • nemo_gym/rollout_collection.py line 4775: BLOCKER — Track completion by aggregation identity, not dispatch spelling.

    A supported mixed batch can contain a materialized task routed to env_a and a flat row naming agent alpha, where env_a fronts alpha. The adapter accepts both forms, but _dispatch_name groups them separately as env_a and alpha. Both groups subsequently aggregate into the same environment and write alpha.json.

    If the native task scores 1 while the flat task remains blocked, the first file declares 1/1 complete prematurely. If the flat task later scores 0, its writer replaces that file with another one-row aggregate. Neither matches the final mean of 0.5. Adding a third benchmark makes this a multi-benchmark batch; without it, the split also incorrectly enables completion files for a single aggregation group. The base aggregator already combined these rows.

    Correction: give _agent_completion_plan the resolved server map and group with _metrics_group, then use that same identity in the success trigger. For example, build completion_servers_by_agent = _environment_servers_by_agent(identity_client.global_config_dict) once, pass it into the plan, replace this grouping assignment with dispatch_name = _metrics_group(row, lambda: completion_servers_by_agent), and replace the success lookup with:

    completion_name = _metrics_group(row, lambda: completion_servers_by_agent)
    pending_keys = agent_pending_keys.get(completion_name)
    if pending_keys is not None:
        pending_keys.discard((row[TASK_INDEX_KEY_NAME], row[ROLLOUT_INDEX_KEY_NAME]))
        if not pending_keys:
            del agent_pending_keys[completion_name]
            start_agent_complete(completion_name)

    Keep progress/logging dispatch names unchanged. The invariant is one expected inventory per final aggregation environment, regardless of input representation. Preserve separate inventories for different environments fronting the same agent, including the existing distinct-server labels, and retain rollout-order sorting.

    Proposed acceptance, not executed: mix native and flat alpha tasks through the same single_agent_turn_legacy server, plus blocked beta. Assert no alpha file after only the native task finishes; after both finish, assert one file with counts 2/2, mean 0.5, and equality with alpha's final entry. Cover both retention modes and resume with the native success already persisted. With only those two alpha tasks, assert no completion directory. Retain ordinary alpha/beta early completion and distinct files for two different environments binding one agent.

  • nemo_gym/rollout_collection.py line 4842: RISK — A persisted success isn't necessarily a scored rollout.

    The new pending-set calculation and this check count every selected success as scored. However, normalization explicitly permits native results without reward and legacy {"execute_only": true, "response": {}} results. Store coverage records these as unscored.

    For example, alpha can return one reward-1 result and one execute-only result while beta remains blocked. Alpha's file then claims completed_rollouts: 2, expected_rollouts: 2, and aggregation_status: "complete", although one rollout has no score. Resume publishes the same false completion from saved unscored successes.

    Correction: retain those valid results in persistence, recovery, and final aggregation, but suppress the scored-completion artifact when any selected result is unscored. After the key-equality check and before aggregation, this small guard matches the store's reward distinction:

    if any(type(result.get("reward")) not in (int, float) for result in results):
        return None

    Preserve numeric zero rewards and numeric masked results; don't reject execute-only or reward-less native results from collection or turn them into zeros.

    Proposed acceptance, not executed: collect alpha with one scored and one execute-only result, plus beta; assert both alpha results persist, coverage reports one unscored rollout, and no alpha completion file appears. Repeat with a reward-less native result and with that result already persisted before resume. A neighboring case containing reward: 0.0 and a numeric masked result should still produce a file equal to its final aggregate. Retain the execute-only progress test's two 100% reward assertions.

  • nemo_gym/rollout_collection.py line 2770: RISK — Distinct agent names can overwrite the same completion file.

    alpha beta and alpha_beta both map to alpha_beta.json. When both benchmarks finish, the atomic writer replaces the earlier file, silently removing one benchmark's early artifact. Names are strings without this character restriction in the instance contract; the collector's fixtures also include names with spaces. Run-wide metrics remain intact, but the per-agent artifact guarantee fails.

    Correction: use an injective filename encoding. A small suggested replacement here, with from urllib.parse import quote added to the imports, is:

    return agent_complete_dir_for(output_fpath) / (quote(label, safe="._-") + ".json")

    This preserves ordinary alphanumeric/underscore/dot/hyphen filenames while keeping spaces, literal percent signs, and path separators distinct. Document the encoding in the completion-file docs for consumers constructing filenames. Verify the suggestion before adopting it.

    Proposed acceptance, not executed: run one-rollout agents named alpha beta and alpha_beta; assert two files remain, with the correct identities and aggregates matching their respective final entries. Check that a/b, a_b, and a%2Fb produce distinct basenames within the completion directory. Retain the existing alpha.json/beta.json names and the assertion that alpha's complete JSON is visible while beta is still blocked.

  • nemo_gym/rollout_collection.py line 4801: RISK — Failed sibling groups can make early and final labels disagree.

    Let stamped environments env_a and env_b front the same agent alpha. This resolver includes both materialized inventories, so a successful env_a completion is labelled env_a. If every env_b rollout fails and failures aren't counted as zero, the final aggregation contains only env_a's results. It then recomputes labels from those results and labels the same entry alpha. The early filename/payload therefore doesn't identify the entry as the final aggregate does, contrary to the documented equality contract. This differs from the mixed-routing defect: the environments here really are distinct.

    Correction: resolve the competing contracts explicitly. If exact early/final entry equality is required, compute one inventory-based label map for collection and pass it through _call_aggregate_metrics into _aggregate_entries for the final pass too; update affected overrides/tests to accept the optional mapping. Preserve result-only default labeling for direct/offline callers without that inventory, and document the changed final labels for partial multi-environment runs. If unchanged final labeling is mandatory instead, the completion schema/docs must stop promising an identical final-labelled entry and provide a stable environment-based join contract. I haven't supplied a patch because preserving both current promises is impossible when final labels depend on which sibling results eventually arrive.

    Preserve ordinary one-server-per-agent names, distinct names when both servers succeed, and the existing direct-call error persistence/rethrow behavior.

    Proposed acceptance, not executed: two stamped environments bind alpha; one scores a task and the other returns a typed failure, without failure-to-zero scoring. Assert that the completed file follows the chosen documented identity contract and that the failed environment gets no file. If equality is retained, compare the complete entry with the corresponding final entry, including agent_ref. Repeat with both environments successful and retain the existing distinct-label assertions.

  • benchmarks/nemotron_3.5_super/README.md line 83: RISK — The SWE instructions undo the repeat-count correction.

    This paragraph still tells users to override collection with num_repeats=1, whereas the updated SWE recipe now correctly sets 3. When a run-level value is present, preparation emits each source task only once. Following this instruction therefore gives one attempt per task, not three, recreating the under-count this PR fixes.

    Correction: replace this paragraph with:

    The SWE recipe sets `num_repeats=3` and `num_repeats_add_seed=false`. With a run-level repeat setting, preparation writes each task once and collection creates the three attempts. Use `num_repeats=1` only when collecting separately prepared inputs that already contain all three copies; don't apply that override to the end-to-end batch recipe.
    

    Also qualify the nearby multiplication statement: collection multiplies copies actually present in supplied input, but end-to-end preparation suppresses dataset-level duplication when a run-level value is set. Preserve that distinction rather than recommending 3 for every pre-expanded --no-serve input.

    Proposed acceptance, not executed: prepare two synthetic one-task SWE members whose dataset repeat count is 3 using the recipe's run-level value. Assert preparation yields one row per member and collection materializes three attempts per member, six total, without added seeds. Separately retain the valid pre-expanded-input case: three supplied copies with collection repeats 1 must remain three, not nine.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…cored agents

Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Replacing every character outside letters, digits and . _ - with an underscore made names such as 'alpha beta' and 'alpha_beta' share one completion file, so the later write silently replaced the earlier agent's file. Percent-encode the name instead, which keeps ordinary names unchanged, keeps distinct names distinct, and cannot leave the completion directory.

Document the encoding for consumers that build the filename, and note how the completion label differs from the run-wide label for an agent behind several environment servers.

Signed-off-by: Frankie Siino <fsiino@nvidia.com>
@fsiino-nvidia

Copy link
Copy Markdown
Contributor Author

Addressed the 5 issues raised above:

  1. BLOCKER: partial score published as complete
    Completion is now tracked per environment server (_metrics_group), so the file is written only when all of its rows are scored. Added tests.

  2. RISK: unscored results counted as scored
    _write_agent_complete now skips the file if any of the agent's results has no numeric reward (e.g. execute-only). Those results stay in the output and the final aggregate, and a reward of 0 still counts as scored.

  3. RISK: colliding completion filenames
    _agent_complete_fpath now percent-encodes the name (quote(label, safe="._-")), so distinct names get distinct files.

  4. RISK: early and final labels disagree for sibling environments
    No code change, since fixing it would alter existing final labels. Documented the difference. It doesn't apply to one environment server per agent (Core and SWE).

  5. RISK: README undoes the repeat-count fix
    Updated the SWE paragraph, the multiplication statement and the YAML example to match the run-level num_repeats behavior (num_repeats=3).

…agent-complete

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

# Conflicts:
#	nemo_gym/rollouts/rollout_collection.py
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Nov 7, 2026
@adil-a

adil-a commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

/review reset=True

@adil-a

adil-a commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

/ok to test 15210cc

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD

Two BLOCKER findings affect completion accuracy and async termination. One RISK covers cancellation leaving aggregation requests alive.

I inspected all eight changed files and traced routing, selected-attempt persistence/resume, aggregation, cancellation, and repeat preparation. The earlier mixed-routing, missing-reward, filename-collision, and repeat-instruction cases are fixed. The sibling-environment label difference is now documented in the batch guide.

This was a static review. I couldn't execute tests in the read-only environment; the acceptance checks below are proposed, not executed.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread nemo_gym/rollouts/rollout_collection.py Outdated
)
if len(agent_expected_keys) > 1:
agent_pending_keys = {
name: keys - persisted_success_keys for name, keys in agent_expected_keys.items()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Keep retryable masked attempts out of completion.

Failure: collect two alpha tasks, one scoring 1 and one returning {"reward": 0.0, "mask_sample": true, "failure_kind": "session_lost", "response": {}}, while beta remains blocked. The Store deliberately retries that masked success and subtracts it from completion while retries remain. However, the new pending set counts every persisted success, the success trigger empties alpha's set, and the writer accepts both numeric rewards. Alpha gets a 2/2, aggregation_status: "complete" artifact even though recovery still owes it an attempt. Since masked rows leave quality metrics, its reported mean is 1 from only one measurement.

Resume can leave that score permanently obsolete. Startup again marks alpha finished and removes its pending entry. With the retry blocked, the completion worker can read the old selected records and publish mean 1. When the retry later scores 0, no pending entry remains to trigger a rewrite; the final alpha mean is 0.5 while its completion artifact still says 1. Existing files also survive resume because cleanup is fresh-run-only.

Correction: keep a completion-specific set of settled successful keys; don't change persisted_success_keys, which other accounting uses. Share the Store's retryability and attempt-cap decision through a small accessor. At startup and after each saved outcome, leave retryable masked keys pending. Recheck the writer's selected attempts and eligibility after asynchronous reads, and invalidate an existing completion artifact when recovery reopens one of its keys. The invariant is that no announced group contains an attempt collection will retry, and a replacement settled attempt can trigger a new artifact. The inspected Store has no public per-rollout settled-success accessor, so this isn't a safe drop-in predicate change.

Preserve: ordinary rewards including zero; numeric masks the Store intentionally retains, including no-kind/model-side masks, protected saved answers, and masks settled at their attempt cap; suppression of execute-only/reward-less artifacts; mixed-routing grouping and rollout-order sorting. Don't suppress every masked result or convert masks to zeros.

Proposed acceptance: run the alpha/beta case above and assert no alpha artifact while masked_retryable == 1. Resume with the retry blocked and assert no stale artifact remains; release it with reward 0 and assert a new 2/2 artifact with mean 0.5, equal to alpha's final entry, in both retention modes. Retain the existing attempt-supersession assertions, and add completion-artifact assertions for the neighboring kept-mask and saved-answer cases.


# Completion files are written before the run-wide aggregate, which they must not race.
if agent_complete_tasks:
await asyncio.gather(*agent_complete_tasks)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Bound optional early aggregation before awaiting it.

Failure: in a two-agent run, let both /run requests succeed, but hold alpha's first, early /aggregate_metrics request indefinitely while subsequent aggregation requests would succeed. This gather never returns, so the authoritative final aggregation is never attempted. Catching exceptions in _write_agent_complete doesn't handle an await that never raises. The aggregation call and body read supply no deadline or retry bound; the internal transport can retry without a bound. This new optional request therefore adds a way to withhold every benchmark's final metrics after collection has finished.

Correction: give _write_agent_complete a finite, completion-only deadline covering aggregation through response-body parsing. Log a timeout and return without publishing an artifact. Cancel and join its nested fetch tasks when the deadline expires; otherwise timing out the outer worker leaves the request running. Keep the final _call_aggregate_metrics policy unchanged. The invariant is that unfinished optional aggregation cannot prevent the final pass from starting indefinitely. The inspected contracts don't establish an appropriate deadline value, so I haven't invented one in a patch.

Preserve: successful early publication while another agent is blocked, HTTP aggregation errors recorded as error artifacts, file-write failure isolation, and the final aggregator's existing error persistence/rethrow behavior.

Proposed acceptance: make alpha's first aggregation await an event indefinitely while all rollout requests and subsequent aggregation calls succeed. Inject a short completion deadline and assert the helper reaches final aggregation, writes both final entries, and leaves no early fetch pending. Retain the early-visibility and HTTP-500 assertions in the completion tests.

Comment thread nemo_gym/rollouts/rollout_collection.py Outdated
if owned_token_source is not None:
output_files.push_async_callback(owned_token_source.close)
output_files.enter_context(store)
output_files.callback(lambda: [task.cancel() for task in agent_complete_tasks])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Join completion workers and their requests during cancellation.

Failure: cancel run_from_config after alpha's early aggregation starts but before it replies. This callback only calls cancel(); it doesn't join the worker. More importantly, _aggregate_entries schedules fetches through asyncio.as_completed without retaining and cancelling those tasks in finally. Cancelling the outer completion worker leaves the independently scheduled HTTP fetch alive. A library caller that catches cancellation and keeps its event loop running can retain requests and rollout payloads after collection unwinds. The dispatch iterator already cancels and joins its resident tasks; the new completion path needs equivalent ownership.

Correction: replace this callback with an async cleanup owner:

async def stop_agent_complete() -> None:
    for task in agent_complete_tasks:
        task.cancel()
    await asyncio.gather(*agent_complete_tasks, return_exceptions=True)

output_files.push_async_callback(stop_agent_complete)

Also create explicit fetch tasks in _aggregate_entries, retain them, and put the existing as_completed processing inside try/finally. In finally, cancel unfinished fetches and await asyncio.gather(*tasks, return_exceptions=True). Both changes are necessary: joining the outer worker alone doesn't stop its child requests. Verify the suggested code before adopting it.

Preserve: completion-order diagnostics, final label sorting, aggregation-error entries, direct callers' original exception rethrow, and successful early publication. Cleanup must propagate caller cancellation rather than convert it into success or an aggregation-error artifact.

Proposed acceptance: block alpha's early aggregation, wait until its request starts, cancel collection, and catch CancelledError in a still-running event loop. Assert the fake request observes cancellation before the helper returns and no completion worker or fetch remains pending. Retain the ordinary early-publication and write-failure assertions, including both final alpha/beta entries.

@adil-a adil-a left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Getting a finished benchmark's score out while another benchmark in the batch is still running fills a real gap. Computing it in-process while the servers are up is the right call: I checked the alternatives, and none meets the goal as well.

  • gym eval aggregate +merge_shards=false -o <path> needs every environment server restarted after a kill, scores unfinished agents too, and is a manual step. It is still worth documenting as the fallback for a killed job.
  • One job per benchmark gives up the shared serving deployment the batch exists for.
  • dispatch_budget_s only stops new rollouts. It would be a good complement in the batch recipes, neither of which sets it, but it does not bound a rollout that hangs.

Before merging I'd like to see the items inline. In short:

  1. A masked-but-retryable rollout counts as scored, so a completion file can go stale after resume. I reproduced this.
  2. A single rollout that fails finally keeps its agent's file from ever appearing in that job.
  3. Please split the num_repeats recipe fix into its own PR.
  4. Consider feeding batch_status.json instead of adding a second schema-versioned artifact that Gym itself doesn't read. If the manifest requirement rules that out for the EFB runner, please say so in the description.
  5. A few smaller items and test gaps, also inline.

Docs: the field descriptions match the code. Please also say that a fresh run deletes <stem>_agent_complete/ and that a resume rewrites a finished agent's completed_at, since external pollers will see both. In the mdx, list all the fields (schema_version, agent_ref, _ng_environment_server) and the complete/error values. The description also has stale paths (nemo_gym/rollout_collection.py, _read_jsonl_for_keys), and its cluster runs predate the regroup-by-environment and filename-encoding commits, so a short Core smoke on the current head would be good.

Comment thread nemo_gym/rollouts/rollout_collection.py Outdated
completion_name = completion_group(row)
pending_keys = agent_pending_keys.get(completion_name)
if pending_keys is not None:
pending_keys.discard((row[TASK_INDEX_KEY_NAME], row[ROLLOUT_INDEX_KEY_NAME]))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A masked rollout that will be retried counts as scored, so the completion file can go stale.

A success with mask_sample and an evaluation-side failure_kind (for example provider_unavailable) is selected as the success but stays retryable: coverage() reports it as masked_retryable ("a resume runs it again, so it is not complete"). Here, and when persisted_success_keys is subtracted at 4110, its key leaves the pending set anyway, so the agent gets a complete file while coverage says it is unfinished.

On resume the file is rewritten from the masked row before dispatch. The rollout is then re-run, and the file is never updated, so it no longer equals the run-wide entry.

Repro: a two-agent resume where alpha has (0,0) scored 1.0 and (0,1) masked with provider_unavailable. Alpha's file carries coverage/masked_rollouts: 1. The run-wide alpha entry after the re-run differs, so afile["aggregate"] == final["alpha"] fails.

Could a key count only once its selected success is settled? That means skipping the discard for a masked-infrastructure row that is still retryable, and building the resume pending set from settled successes rather than every selected success. gym eval recover --rerun-masked-without-artifact rows hit the same path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 93aa5cd. If infrastructure masked a result and a resume will run it again, that rollout now stays pending, both during the run and when a resume starts. The check lives in RolloutRecords.masked_retryable, and coverage() now uses it too.

Comment thread nemo_gym/rollouts/rollout_collection.py Outdated
)
if len(agent_expected_keys) > 1:
agent_pending_keys = {
name: keys - persisted_success_keys for name, keys in agent_expected_keys.items()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One final failure keeps the agent's file from ever appearing in this job.

Only the success branch removes keys from the pending set; the failure branch at ~4409-4434 never touches it. So a single rollout that fails finally after its in-run retries blocks that agent's completion file for the rest of the job, even though the end-of-run pass still aggregates the agent, leaving failures out or counting them as zero under count_failure_classes_as_zero / count_missing_rollouts_as_zero.

Core routes failures to the sidecar and LMArena tolerates a 1% failure rate, so I'd expect this to be common, and it undercuts the goal. Could completion mean "every expected rollout has settled" (scored, or failed finally), with the early aggregate computed under the same failure policy as the final pass? Then the early entry still equals the run-wide one.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d9f509d. An agent is complete once every rollout is settled (RolloutRecords.settled, the opposite of pending): a success that won't be rerun, or a failure/omission that won't be retried.

- benchmarks/nemotron_3.5_super/shared_model_endpoints.yaml

num_samples_in_parallel: 512
# A run-level num_repeats replaces each dataset's own, so every member lists its attempts per task.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move the repeat restore into its own PR.

The analysis is right: since #5042 a run-level num_repeats replaces each dataset's own, and these counts restore the earlier ones. But it is unrelated to completion files, and it changes benchmark cost and results: Core goes from 9,427 to 23,801 rollouts and SWE from 800 to 2,400. It hasn't run on the cluster, and it needs review from the benchmark owners. That covers this file, swe.yaml, the launcher test and the README repeat paragraphs.

It also leaves the dataset-level num_repeats (omniscience 10, ifbench 8) as dead config beside the new dict, and _default: 1 silently gives any new member one attempt. That PR should settle which source is authoritative. Splitting also keeps this feature easy to revert on its own.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted in 81e71d3. Will follow up with new PR.

entry = entries[0]
label = entry[AGENT_REF_KEY_NAME]["name"]
fpath = _agent_complete_fpath(output_fpath, label)
_atomic_write_json(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gym eval aggregate doesn't refresh or remove completion files.

The documented repair for an aggregation failure is gym eval aggregate, which rewrites <stem>_aggregate_metrics.json in place but leaves <stem>_agent_complete/<agent>.json at aggregation_status: "error". A consumer that prefers the completion files keeps showing the failure after the repair. Consider having a metrics-only aggregate into the same stem rewrite or remove the directory, or document that the run-wide file wins once it exists.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented this in 3c14869. Once the run-wide aggregate exists, it is the source of truth and takes priority over a completion file after a repair.

Comment thread nemo_gym/rollouts/rollout_collection.py Outdated
for record, row in zip(selected.values(), inputs)
if (row[TASK_INDEX_KEY_NAME], row[ROLLOUT_INDEX_KEY_NAME]) in keys
]
results = await asyncio.to_thread(lambda: [record.read() for record in records])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (shared storage): reuse one file handle. record.read() opens the rollouts file once per record, so a full Core batch makes about 23k opens on the output file, in bursts at agent completion and at resume startup. store.selected() already reads through one handle per path; open the file once and use record._read(handle).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 92ddb23. The writer now reads all its records through one open file per path (read_record_rows), and RolloutRecords.selected uses the same function.

}
return keys_by_group, samples

def _agent_complete_labels(self, samples: Mapping[str, Mapping[str, Any]]) -> Dict[str, str]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shared label logic, and a test for it.

This re-derives the grouping that the end-of-run labeling does, and the docs already note the two can differ for an agent behind several environment servers. A helper shared with _aggregate_entries would remove that divergence.

The path is also untested: replacing the computed labels with labels=None still passes all 762 tests in test_rollout_collection.py. Could you add a case where two environment servers front one agent, asserting each file's name and agent_ref.name, and equality with the run-wide entry?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 92ddb23. Both paths now get their labels from 1 helper, _server_agents. A new test with 2 env servers in front of 1 agent checks each file's name, its agent_ref.name, and that it matches the run-wide entry. Test fails with labels=None.

assert orjson.loads((complete_dir / "beta.json").read_bytes())["aggregation_status"] == "complete"

@pytest.mark.parametrize("retain_results_in_memory", [False, True])
async def test_agent_completion_file_reads_repeats_in_rollout_order(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test passes with the results.sort(key=_rollout_order_key) line removed, because RolloutState.selected_records already yields rollouts in materialized order. Either drop the sort and note that the store's order is the contract, or feed records out of order here, for example by stubbing selected_records, so the sort is what keeps the test green.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 92ddb23. The test now makes the store return records in reverse order, so it fails if the sort is removed.

fsiino-nvidia and others added 7 commits November 9, 2026 23:52
… own PR

Reverts the repeats parts of 6abbf26 and 174c5e6

Signed-off-by: Frankie Siino <fsiino@nvidia.com>
A success masked by infrastructure stays retryable: coverage reports it as masked_retryable and a resume runs it again. The completion code counted it as done, so the agent's file was written from the masked row and went stale once the resume reran it. A masked retryable success now stays pending, during the run and when a resume rebuilds the pending set, so the file is written after the rerun and equals the run-wide entry. The rule moves into RolloutRecords.masked_retryable, which coverage now uses too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
… final failures

Only successes left the pending set, so one failure no resume would retry blocked its agent's completion file for the rest of the job. A rollout now leaves it once its outcome is settled (RolloutRecords.settled, the complement of pending): a success no resume reruns, or a failure or omission none retries. The early aggregate adds the failures the run counts as zero, with the same helpers as the end of the run, so it still equals the run-wide entry, and the file gains failed_rollouts. A failure a resume will retry still keeps the agent pending, so the file never goes stale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…ion files

Read an agent's records through one handle per file (read_record_rows, also used by RolloutRecords.selected) instead of one open per record. Derive the agent that labels each environment server's entry in one helper (_server_agents) for both the run-wide aggregate and the completion files, and test two environment servers fronting one agent. Make the rollout-order test hand records back out of order, so only the completion file's sort keeps it green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
gym eval aggregate repairs the run-wide aggregate in place but leaves the completion files, so one that recorded an aggregation error can be stale after a repair. Document that the run-wide entry is authoritative once it exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…ery field

List every field of a completion file and the complete/error values, state when an agent is complete, that a fresh run deletes the directory and a resume rewrites completed_at, and give gym eval aggregate +merge_shards=false as the fallback for a job killed before its run-wide aggregate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…ntry

After a run, Gym attaches _ng_run_health_metrics, a run-level health summary, to the first entry of the run-wide aggregate. A completion file's aggregate equals its run-wide entry apart from that key, so say so where the docs promise equality.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
@fsiino-nvidia fsiino-nvidia changed the title feat(eval): write a completion file per agent as soon its rollouts are scored feat(eval): write a completion file per agent as soon as its rollouts are scored Nov 10, 2026
fsiino-nvidia and others added 2 commits November 11, 2026 19:18
A resume deletes the completion file of an agent it still runs. The early /aggregate_metrics call gives up after 30 minutes and records an aggregation error, so a hung server cannot hold the run-wide aggregate. A run that stops early cancels and waits for its completion tasks and their in-flight requests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…agent-complete

Signed-off-by: Frankie Siino <fsiino@nvidia.com>
@fsiino-nvidia

Copy link
Copy Markdown
Contributor Author

@adil-a All inline points are addressed and answered in their threads.

Also addressed bot callouts from @svcnvidia-nemo-ci in 9d62087:

  • Stale file on resume: a resume deletes the completion file of any agent with pending rollouts
  • Unbound early aggregation: /aggregate_metrics gives up after 30 min (AGENT_COMPLETE_AGGREGATION_TIMEOUT_S) and records an aggregation error. Nested fetches are cancelled and joined.
  • Workers on cancellation: the run cancels and joins its completion tasks before the store closes.
  • Masked rollouts: fixed earlier in 93aa5cd and d9f509d.

fsiino-nvidia and others added 2 commits November 12, 2026 01:07
The re-judge pass at the end of gym eval run grades judge failures again and recomputes the run-wide aggregate, which left completion files written during collection with the old scores. Each file now takes its agent's recomputed entry and rollout counts; if the metrics or the files cannot be refreshed, the files are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
The judge-only recovery job recomputes the run-wide metrics in place and now refreshes the completion files, or removes them when it cannot. A strict run that the re-judge pass leaves incomplete gets no recomputed metrics, so its completion files are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
@fsiino-nvidia fsiino-nvidia changed the title feat(eval): write a completion file per agent as soon as its rollouts are scored feat(eval): write a completion file per agent as soon as its rollouts are settled Nov 12, 2026
@fsiino-nvidia
fsiino-nvidia requested a review from adil-a November 13, 2026 23:33
…agent-complete

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

# Conflicts:
#	nemo_gym/rollouts/rollout_collection.py
@fsiino-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test 4a9477b

@nemo-automation-bot
nemo-automation-bot Bot requested review from a team November 14, 2026 00:02
fsiino-nvidia and others added 2 commits November 16, 2026 17:53
…agent-complete

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

# Conflicts:
#	nemo_gym/rollouts/rollout_records.py
Match main's masked_exhausted coverage (#6070), so no completion file is written while such a rollout is unmeasured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
@fsiino-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test 613794c

@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team November 16, 2026 18:20
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Nov 17, 2026

This branch was previously deployed

1 inactive (outdated) deployment
public — 15210cc2 Deployed Nov 8, 2026 by copy-pr-bot[bot] via release / finalize / notify #10888
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants