Repository navigation
feat(eval): write a completion file per agent as soon as its rollouts are settled - #5386
fsiino-nvidia wants to merge 26 commits into
Conversation
… are scored Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…agent-complete Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…agent-complete Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # nemo_gym/rollout_collection.py
…5042 Since #5042, data preparation writes each task once whenever a run-level num_repeats is set, so the run-level value replaces each dataset's own. The Core recipe listed only APEX (16) and LMArena (3) with a default of 1, so every other member dropped from 8, 10 or 16 attempts per task to 1 (23,801 rollouts before, 9,427 now). The SWE recipe's run-level 1 did the same to its 3 attempts (2,400 rollouts before, 800 now). List every Core member's intended count in the run-level num_repeats, set num_repeats: 3 in swe.yaml, and update the matching expectations in test_super35_batch_launcher.py. num_repeats_add_seed is unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…agent-complete Signed-off-by: Frankie Siino <fsiino@nvidia.com>
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD
One BLOCKER can publish a partial benchmark score as complete. Four RISK findings cover unscored results, colliding artifact filenames, inconsistent aggregate labels, and stale repeat-count instructions.
I inspected all eight changed files and traced the affected routing, persistence/resume, aggregation, and data-preparation contracts. The ordinary alpha/beta completion, repeat ordering, aggregation-error recording, and write-failure isolation have targeted regression coverage. I couldn't execute tests in this read-only review; the acceptance checks below are proposed.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Inline comments (escaped v1):
-
nemo_gym/rollout_collection.pyline 4775: BLOCKER — Track completion by aggregation identity, not dispatch spelling.A supported mixed batch can contain a materialized task routed to
env_aand a flat row naming agentalpha, whereenv_afrontsalpha. The adapter accepts both forms, but_dispatch_namegroups them separately asenv_aandalpha. Both groups subsequently aggregate into the same environment and writealpha.json.If the native task scores 1 while the flat task remains blocked, the first file declares
1/1complete prematurely. If the flat task later scores 0, its writer replaces that file with another one-row aggregate. Neither matches the final mean of 0.5. Adding a third benchmark makes this a multi-benchmark batch; without it, the split also incorrectly enables completion files for a single aggregation group. The base aggregator already combined these rows.Correction: give
_agent_completion_planthe resolved server map and group with_metrics_group, then use that same identity in the success trigger. For example, buildcompletion_servers_by_agent = _environment_servers_by_agent(identity_client.global_config_dict)once, pass it into the plan, replace this grouping assignment withdispatch_name = _metrics_group(row, lambda: completion_servers_by_agent), and replace the success lookup with:completion_name = _metrics_group(row, lambda: completion_servers_by_agent) pending_keys = agent_pending_keys.get(completion_name) if pending_keys is not None: pending_keys.discard((row[TASK_INDEX_KEY_NAME], row[ROLLOUT_INDEX_KEY_NAME])) if not pending_keys: del agent_pending_keys[completion_name] start_agent_complete(completion_name)
Keep progress/logging dispatch names unchanged. The invariant is one expected inventory per final aggregation environment, regardless of input representation. Preserve separate inventories for different environments fronting the same agent, including the existing distinct-server labels, and retain rollout-order sorting.
Proposed acceptance, not executed: mix native and flat alpha tasks through the same
single_agent_turn_legacyserver, plus blocked beta. Assert no alpha file after only the native task finishes; after both finish, assert one file with counts2/2, mean0.5, and equality with alpha's final entry. Cover both retention modes and resume with the native success already persisted. With only those two alpha tasks, assert no completion directory. Retain ordinary alpha/beta early completion and distinct files for two different environments binding one agent. -
nemo_gym/rollout_collection.pyline 4842: RISK — A persisted success isn't necessarily a scored rollout.The new pending-set calculation and this check count every selected success as scored. However, normalization explicitly permits native results without
rewardand legacy{"execute_only": true, "response": {}}results. Store coverage records these as unscored.For example, alpha can return one reward-1 result and one execute-only result while beta remains blocked. Alpha's file then claims
completed_rollouts: 2,expected_rollouts: 2, andaggregation_status: "complete", although one rollout has no score. Resume publishes the same false completion from saved unscored successes.Correction: retain those valid results in persistence, recovery, and final aggregation, but suppress the scored-completion artifact when any selected result is unscored. After the key-equality check and before aggregation, this small guard matches the store's reward distinction:
if any(type(result.get("reward")) not in (int, float) for result in results): return None
Preserve numeric zero rewards and numeric masked results; don't reject execute-only or reward-less native results from collection or turn them into zeros.
Proposed acceptance, not executed: collect alpha with one scored and one execute-only result, plus beta; assert both alpha results persist, coverage reports one unscored rollout, and no alpha completion file appears. Repeat with a reward-less native result and with that result already persisted before resume. A neighboring case containing
reward: 0.0and a numeric masked result should still produce a file equal to its final aggregate. Retain the execute-only progress test's two 100% reward assertions. -
nemo_gym/rollout_collection.pyline 2770: RISK — Distinct agent names can overwrite the same completion file.alpha betaandalpha_betaboth map toalpha_beta.json. When both benchmarks finish, the atomic writer replaces the earlier file, silently removing one benchmark's early artifact. Names are strings without this character restriction in the instance contract; the collector's fixtures also include names with spaces. Run-wide metrics remain intact, but the per-agent artifact guarantee fails.Correction: use an injective filename encoding. A small suggested replacement here, with
from urllib.parse import quoteadded to the imports, is:return agent_complete_dir_for(output_fpath) / (quote(label, safe="._-") + ".json")
This preserves ordinary alphanumeric/underscore/dot/hyphen filenames while keeping spaces, literal percent signs, and path separators distinct. Document the encoding in the completion-file docs for consumers constructing filenames. Verify the suggestion before adopting it.
Proposed acceptance, not executed: run one-rollout agents named
alpha betaandalpha_beta; assert two files remain, with the correct identities and aggregates matching their respective final entries. Check thata/b,a_b, anda%2Fbproduce distinct basenames within the completion directory. Retain the existingalpha.json/beta.jsonnames and the assertion that alpha's complete JSON is visible while beta is still blocked. -
nemo_gym/rollout_collection.pyline 4801: RISK — Failed sibling groups can make early and final labels disagree.Let stamped environments
env_aandenv_bfront the same agentalpha. This resolver includes both materialized inventories, so a successfulenv_acompletion is labelledenv_a. If everyenv_brollout fails and failures aren't counted as zero, the final aggregation contains onlyenv_a's results. It then recomputes labels from those results and labels the same entryalpha. The early filename/payload therefore doesn't identify the entry as the final aggregate does, contrary to the documented equality contract. This differs from the mixed-routing defect: the environments here really are distinct.Correction: resolve the competing contracts explicitly. If exact early/final entry equality is required, compute one inventory-based label map for collection and pass it through
_call_aggregate_metricsinto_aggregate_entriesfor the final pass too; update affected overrides/tests to accept the optional mapping. Preserve result-only default labeling for direct/offline callers without that inventory, and document the changed final labels for partial multi-environment runs. If unchanged final labeling is mandatory instead, the completion schema/docs must stop promising an identical final-labelled entry and provide a stable environment-based join contract. I haven't supplied a patch because preserving both current promises is impossible when final labels depend on which sibling results eventually arrive.Preserve ordinary one-server-per-agent names, distinct names when both servers succeed, and the existing direct-call error persistence/rethrow behavior.
Proposed acceptance, not executed: two stamped environments bind alpha; one scores a task and the other returns a typed failure, without failure-to-zero scoring. Assert that the completed file follows the chosen documented identity contract and that the failed environment gets no file. If equality is retained, compare the complete entry with the corresponding final entry, including
agent_ref. Repeat with both environments successful and retain the existing distinct-label assertions. -
benchmarks/nemotron_3.5_super/README.mdline 83: RISK — The SWE instructions undo the repeat-count correction.This paragraph still tells users to override collection with
num_repeats=1, whereas the updated SWE recipe now correctly sets 3. When a run-level value is present, preparation emits each source task only once. Following this instruction therefore gives one attempt per task, not three, recreating the under-count this PR fixes.Correction: replace this paragraph with:
The SWE recipe sets `num_repeats=3` and `num_repeats_add_seed=false`. With a run-level repeat setting, preparation writes each task once and collection creates the three attempts. Use `num_repeats=1` only when collecting separately prepared inputs that already contain all three copies; don't apply that override to the end-to-end batch recipe.Also qualify the nearby multiplication statement: collection multiplies copies actually present in supplied input, but end-to-end preparation suppresses dataset-level duplication when a run-level value is set. Preserve that distinction rather than recommending 3 for every pre-expanded
--no-serveinput.Proposed acceptance, not executed: prepare two synthetic one-task SWE members whose dataset repeat count is 3 using the recipe's run-level value. Assert preparation yields one row per member and collection materializes three attempts per member, six total, without added seeds. Separately retain the valid pre-expanded-input case: three supplied copies with collection repeats 1 must remain three, not nine.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…cored agents Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Replacing every character outside letters, digits and . _ - with an underscore made names such as 'alpha beta' and 'alpha_beta' share one completion file, so the later write silently replaced the earlier agent's file. Percent-encode the name instead, which keeps ordinary names unchanged, keeps distinct names distinct, and cannot leave the completion directory. Document the encoding for consumers that build the filename, and note how the completion label differs from the run-wide label for an agent behind several environment servers. Signed-off-by: Frankie Siino <fsiino@nvidia.com>
|
Addressed the 5 issues raised above:
|
…agent-complete Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # nemo_gym/rollouts/rollout_collection.py
|
/review reset=True |
|
/ok to test 15210cc |
|
🌿 Preview your docs: https://nvidia-preview-pr-5386-613794cea466.docs.buildwithfern.com/nemo/gym |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD
Two BLOCKER findings affect completion accuracy and async termination. One RISK covers cancellation leaving aggregation requests alive.
I inspected all eight changed files and traced routing, selected-attempt persistence/resume, aggregation, cancellation, and repeat preparation. The earlier mixed-routing, missing-reward, filename-collision, and repeat-instruction cases are fixed. The sibling-environment label difference is now documented in the batch guide.
This was a static review. I couldn't execute tests in the read-only environment; the acceptance checks below are proposed, not executed.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| ) | ||
| if len(agent_expected_keys) > 1: | ||
| agent_pending_keys = { | ||
| name: keys - persisted_success_keys for name, keys in agent_expected_keys.items() |
There was a problem hiding this comment.
BLOCKER — Keep retryable masked attempts out of completion.
Failure: collect two alpha tasks, one scoring 1 and one returning {"reward": 0.0, "mask_sample": true, "failure_kind": "session_lost", "response": {}}, while beta remains blocked. The Store deliberately retries that masked success and subtracts it from completion while retries remain. However, the new pending set counts every persisted success, the success trigger empties alpha's set, and the writer accepts both numeric rewards. Alpha gets a 2/2, aggregation_status: "complete" artifact even though recovery still owes it an attempt. Since masked rows leave quality metrics, its reported mean is 1 from only one measurement.
Resume can leave that score permanently obsolete. Startup again marks alpha finished and removes its pending entry. With the retry blocked, the completion worker can read the old selected records and publish mean 1. When the retry later scores 0, no pending entry remains to trigger a rewrite; the final alpha mean is 0.5 while its completion artifact still says 1. Existing files also survive resume because cleanup is fresh-run-only.
Correction: keep a completion-specific set of settled successful keys; don't change persisted_success_keys, which other accounting uses. Share the Store's retryability and attempt-cap decision through a small accessor. At startup and after each saved outcome, leave retryable masked keys pending. Recheck the writer's selected attempts and eligibility after asynchronous reads, and invalidate an existing completion artifact when recovery reopens one of its keys. The invariant is that no announced group contains an attempt collection will retry, and a replacement settled attempt can trigger a new artifact. The inspected Store has no public per-rollout settled-success accessor, so this isn't a safe drop-in predicate change.
Preserve: ordinary rewards including zero; numeric masks the Store intentionally retains, including no-kind/model-side masks, protected saved answers, and masks settled at their attempt cap; suppression of execute-only/reward-less artifacts; mixed-routing grouping and rollout-order sorting. Don't suppress every masked result or convert masks to zeros.
Proposed acceptance: run the alpha/beta case above and assert no alpha artifact while masked_retryable == 1. Resume with the retry blocked and assert no stale artifact remains; release it with reward 0 and assert a new 2/2 artifact with mean 0.5, equal to alpha's final entry, in both retention modes. Retain the existing attempt-supersession assertions, and add completion-artifact assertions for the neighboring kept-mask and saved-answer cases.
|
|
||
| # Completion files are written before the run-wide aggregate, which they must not race. | ||
| if agent_complete_tasks: | ||
| await asyncio.gather(*agent_complete_tasks) |
There was a problem hiding this comment.
BLOCKER — Bound optional early aggregation before awaiting it.
Failure: in a two-agent run, let both /run requests succeed, but hold alpha's first, early /aggregate_metrics request indefinitely while subsequent aggregation requests would succeed. This gather never returns, so the authoritative final aggregation is never attempted. Catching exceptions in _write_agent_complete doesn't handle an await that never raises. The aggregation call and body read supply no deadline or retry bound; the internal transport can retry without a bound. This new optional request therefore adds a way to withhold every benchmark's final metrics after collection has finished.
Correction: give _write_agent_complete a finite, completion-only deadline covering aggregation through response-body parsing. Log a timeout and return without publishing an artifact. Cancel and join its nested fetch tasks when the deadline expires; otherwise timing out the outer worker leaves the request running. Keep the final _call_aggregate_metrics policy unchanged. The invariant is that unfinished optional aggregation cannot prevent the final pass from starting indefinitely. The inspected contracts don't establish an appropriate deadline value, so I haven't invented one in a patch.
Preserve: successful early publication while another agent is blocked, HTTP aggregation errors recorded as error artifacts, file-write failure isolation, and the final aggregator's existing error persistence/rethrow behavior.
Proposed acceptance: make alpha's first aggregation await an event indefinitely while all rollout requests and subsequent aggregation calls succeed. Inject a short completion deadline and assert the helper reaches final aggregation, writes both final entries, and leaves no early fetch pending. Retain the early-visibility and HTTP-500 assertions in the completion tests.
| if owned_token_source is not None: | ||
| output_files.push_async_callback(owned_token_source.close) | ||
| output_files.enter_context(store) | ||
| output_files.callback(lambda: [task.cancel() for task in agent_complete_tasks]) |
There was a problem hiding this comment.
RISK — Join completion workers and their requests during cancellation.
Failure: cancel run_from_config after alpha's early aggregation starts but before it replies. This callback only calls cancel(); it doesn't join the worker. More importantly, _aggregate_entries schedules fetches through asyncio.as_completed without retaining and cancelling those tasks in finally. Cancelling the outer completion worker leaves the independently scheduled HTTP fetch alive. A library caller that catches cancellation and keeps its event loop running can retain requests and rollout payloads after collection unwinds. The dispatch iterator already cancels and joins its resident tasks; the new completion path needs equivalent ownership.
Correction: replace this callback with an async cleanup owner:
async def stop_agent_complete() -> None:
for task in agent_complete_tasks:
task.cancel()
await asyncio.gather(*agent_complete_tasks, return_exceptions=True)
output_files.push_async_callback(stop_agent_complete)Also create explicit fetch tasks in _aggregate_entries, retain them, and put the existing as_completed processing inside try/finally. In finally, cancel unfinished fetches and await asyncio.gather(*tasks, return_exceptions=True). Both changes are necessary: joining the outer worker alone doesn't stop its child requests. Verify the suggested code before adopting it.
Preserve: completion-order diagnostics, final label sorting, aggregation-error entries, direct callers' original exception rethrow, and successful early publication. Cleanup must propagate caller cancellation rather than convert it into success or an aggregation-error artifact.
Proposed acceptance: block alpha's early aggregation, wait until its request starts, cancel collection, and catch CancelledError in a still-running event loop. Assert the fake request observes cancellation before the helper returns and no completion worker or fetch remains pending. Retain the ordinary early-publication and write-failure assertions, including both final alpha/beta entries.
adil-a
left a comment
There was a problem hiding this comment.
Thanks for this. Getting a finished benchmark's score out while another benchmark in the batch is still running fills a real gap. Computing it in-process while the servers are up is the right call: I checked the alternatives, and none meets the goal as well.
gym eval aggregate +merge_shards=false -o <path>needs every environment server restarted after a kill, scores unfinished agents too, and is a manual step. It is still worth documenting as the fallback for a killed job.- One job per benchmark gives up the shared serving deployment the batch exists for.
dispatch_budget_sonly stops new rollouts. It would be a good complement in the batch recipes, neither of which sets it, but it does not bound a rollout that hangs.
Before merging I'd like to see the items inline. In short:
- A masked-but-retryable rollout counts as scored, so a completion file can go stale after resume. I reproduced this.
- A single rollout that fails finally keeps its agent's file from ever appearing in that job.
- Please split the
num_repeatsrecipe fix into its own PR. - Consider feeding
batch_status.jsoninstead of adding a second schema-versioned artifact that Gym itself doesn't read. If the manifest requirement rules that out for the EFB runner, please say so in the description. - A few smaller items and test gaps, also inline.
Docs: the field descriptions match the code. Please also say that a fresh run deletes <stem>_agent_complete/ and that a resume rewrites a finished agent's completed_at, since external pollers will see both. In the mdx, list all the fields (schema_version, agent_ref, _ng_environment_server) and the complete/error values. The description also has stale paths (nemo_gym/rollout_collection.py, _read_jsonl_for_keys), and its cluster runs predate the regroup-by-environment and filename-encoding commits, so a short Core smoke on the current head would be good.
| completion_name = completion_group(row) | ||
| pending_keys = agent_pending_keys.get(completion_name) | ||
| if pending_keys is not None: | ||
| pending_keys.discard((row[TASK_INDEX_KEY_NAME], row[ROLLOUT_INDEX_KEY_NAME])) |
There was a problem hiding this comment.
A masked rollout that will be retried counts as scored, so the completion file can go stale.
A success with mask_sample and an evaluation-side failure_kind (for example provider_unavailable) is selected as the success but stays retryable: coverage() reports it as masked_retryable ("a resume runs it again, so it is not complete"). Here, and when persisted_success_keys is subtracted at 4110, its key leaves the pending set anyway, so the agent gets a complete file while coverage says it is unfinished.
On resume the file is rewritten from the masked row before dispatch. The rollout is then re-run, and the file is never updated, so it no longer equals the run-wide entry.
Repro: a two-agent resume where alpha has (0,0) scored 1.0 and (0,1) masked with provider_unavailable. Alpha's file carries coverage/masked_rollouts: 1. The run-wide alpha entry after the re-run differs, so afile["aggregate"] == final["alpha"] fails.
Could a key count only once its selected success is settled? That means skipping the discard for a masked-infrastructure row that is still retryable, and building the resume pending set from settled successes rather than every selected success. gym eval recover --rerun-masked-without-artifact rows hit the same path.
There was a problem hiding this comment.
Fixed in 93aa5cd. If infrastructure masked a result and a resume will run it again, that rollout now stays pending, both during the run and when a resume starts. The check lives in RolloutRecords.masked_retryable, and coverage() now uses it too.
| ) | ||
| if len(agent_expected_keys) > 1: | ||
| agent_pending_keys = { | ||
| name: keys - persisted_success_keys for name, keys in agent_expected_keys.items() |
There was a problem hiding this comment.
One final failure keeps the agent's file from ever appearing in this job.
Only the success branch removes keys from the pending set; the failure branch at ~4409-4434 never touches it. So a single rollout that fails finally after its in-run retries blocks that agent's completion file for the rest of the job, even though the end-of-run pass still aggregates the agent, leaving failures out or counting them as zero under count_failure_classes_as_zero / count_missing_rollouts_as_zero.
Core routes failures to the sidecar and LMArena tolerates a 1% failure rate, so I'd expect this to be common, and it undercuts the goal. Could completion mean "every expected rollout has settled" (scored, or failed finally), with the early aggregate computed under the same failure policy as the final pass? Then the early entry still equals the run-wide one.
There was a problem hiding this comment.
Fixed in d9f509d. An agent is complete once every rollout is settled (RolloutRecords.settled, the opposite of pending): a success that won't be rerun, or a failure/omission that won't be retried.
| - benchmarks/nemotron_3.5_super/shared_model_endpoints.yaml | ||
|
|
||
| num_samples_in_parallel: 512 | ||
| # A run-level num_repeats replaces each dataset's own, so every member lists its attempts per task. |
There was a problem hiding this comment.
Please move the repeat restore into its own PR.
The analysis is right: since #5042 a run-level num_repeats replaces each dataset's own, and these counts restore the earlier ones. But it is unrelated to completion files, and it changes benchmark cost and results: Core goes from 9,427 to 23,801 rollouts and SWE from 800 to 2,400. It hasn't run on the cluster, and it needs review from the benchmark owners. That covers this file, swe.yaml, the launcher test and the README repeat paragraphs.
It also leaves the dataset-level num_repeats (omniscience 10, ifbench 8) as dead config beside the new dict, and _default: 1 silently gives any new member one attempt. That PR should settle which source is authoritative. Splitting also keeps this feature easy to revert on its own.
There was a problem hiding this comment.
Reverted in 81e71d3. Will follow up with new PR.
| entry = entries[0] | ||
| label = entry[AGENT_REF_KEY_NAME]["name"] | ||
| fpath = _agent_complete_fpath(output_fpath, label) | ||
| _atomic_write_json( |
There was a problem hiding this comment.
gym eval aggregate doesn't refresh or remove completion files.
The documented repair for an aggregation failure is gym eval aggregate, which rewrites <stem>_aggregate_metrics.json in place but leaves <stem>_agent_complete/<agent>.json at aggregation_status: "error". A consumer that prefers the completion files keeps showing the failure after the repair. Consider having a metrics-only aggregate into the same stem rewrite or remove the directory, or document that the run-wide file wins once it exists.
There was a problem hiding this comment.
Documented this in 3c14869. Once the run-wide aggregate exists, it is the source of truth and takes priority over a completion file after a repair.
| for record, row in zip(selected.values(), inputs) | ||
| if (row[TASK_INDEX_KEY_NAME], row[ROLLOUT_INDEX_KEY_NAME]) in keys | ||
| ] | ||
| results = await asyncio.to_thread(lambda: [record.read() for record in records]) |
There was a problem hiding this comment.
Nit (shared storage): reuse one file handle. record.read() opens the rollouts file once per record, so a full Core batch makes about 23k opens on the output file, in bursts at agent completion and at resume startup. store.selected() already reads through one handle per path; open the file once and use record._read(handle).
There was a problem hiding this comment.
Fixed in 92ddb23. The writer now reads all its records through one open file per path (read_record_rows), and RolloutRecords.selected uses the same function.
| } | ||
| return keys_by_group, samples | ||
|
|
||
| def _agent_complete_labels(self, samples: Mapping[str, Mapping[str, Any]]) -> Dict[str, str]: |
There was a problem hiding this comment.
Shared label logic, and a test for it.
This re-derives the grouping that the end-of-run labeling does, and the docs already note the two can differ for an agent behind several environment servers. A helper shared with _aggregate_entries would remove that divergence.
The path is also untested: replacing the computed labels with labels=None still passes all 762 tests in test_rollout_collection.py. Could you add a case where two environment servers front one agent, asserting each file's name and agent_ref.name, and equality with the run-wide entry?
There was a problem hiding this comment.
Fixed in 92ddb23. Both paths now get their labels from 1 helper, _server_agents. A new test with 2 env servers in front of 1 agent checks each file's name, its agent_ref.name, and that it matches the run-wide entry. Test fails with labels=None.
| assert orjson.loads((complete_dir / "beta.json").read_bytes())["aggregation_status"] == "complete" | ||
|
|
||
| @pytest.mark.parametrize("retain_results_in_memory", [False, True]) | ||
| async def test_agent_completion_file_reads_repeats_in_rollout_order( |
There was a problem hiding this comment.
This test passes with the results.sort(key=_rollout_order_key) line removed, because RolloutState.selected_records already yields rollouts in materialized order. Either drop the sort and note that the store's order is the contract, or feed records out of order here, for example by stubbing selected_records, so the sort is what keeps the test green.
There was a problem hiding this comment.
Fixed in 92ddb23. The test now makes the store return records in reverse order, so it fails if the sort is removed.
A success masked by infrastructure stays retryable: coverage reports it as masked_retryable and a resume runs it again. The completion code counted it as done, so the agent's file was written from the masked row and went stale once the resume reran it. A masked retryable success now stays pending, during the run and when a resume rebuilds the pending set, so the file is written after the rerun and equals the run-wide entry. The rule moves into RolloutRecords.masked_retryable, which coverage now uses too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
… final failures Only successes left the pending set, so one failure no resume would retry blocked its agent's completion file for the rest of the job. A rollout now leaves it once its outcome is settled (RolloutRecords.settled, the complement of pending): a success no resume reruns, or a failure or omission none retries. The early aggregate adds the failures the run counts as zero, with the same helpers as the end of the run, so it still equals the run-wide entry, and the file gains failed_rollouts. A failure a resume will retry still keeps the agent pending, so the file never goes stale. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…ion files Read an agent's records through one handle per file (read_record_rows, also used by RolloutRecords.selected) instead of one open per record. Derive the agent that labels each environment server's entry in one helper (_server_agents) for both the run-wide aggregate and the completion files, and test two environment servers fronting one agent. Make the rollout-order test hand records back out of order, so only the completion file's sort keeps it green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
gym eval aggregate repairs the run-wide aggregate in place but leaves the completion files, so one that recorded an aggregation error can be stale after a repair. Document that the run-wide entry is authoritative once it exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…ery field List every field of a completion file and the complete/error values, state when an agent is complete, that a fresh run deletes the directory and a resume rewrites completed_at, and give gym eval aggregate +merge_shards=false as the fallback for a job killed before its run-wide aggregate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…ntry After a run, Gym attaches _ng_run_health_metrics, a run-level health summary, to the first entry of the run-wide aggregate. A completion file's aggregate equals its run-wide entry apart from that key, so say so where the docs promise equality. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
A resume deletes the completion file of an agent it still runs. The early /aggregate_metrics call gives up after 30 minutes and records an aggregation error, so a hung server cannot hold the run-wide aggregate. A run that stops early cancels and waits for its completion tasks and their in-flight requests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…agent-complete Signed-off-by: Frankie Siino <fsiino@nvidia.com>
|
@adil-a All inline points are addressed and answered in their threads. Also addressed bot callouts from @svcnvidia-nemo-ci in 9d62087:
|
The re-judge pass at the end of gym eval run grades judge failures again and recomputes the run-wide aggregate, which left completion files written during collection with the old scores. Each file now takes its agent's recomputed entry and rollout counts; if the metrics or the files cannot be refreshed, the files are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
The judge-only recovery job recomputes the run-wide metrics in place and now refreshes the completion files, or removes them when it cannot. A strict run that the re-judge pass leaves incomplete gets no recomputed metrics, so its completion files are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…agent-complete Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # nemo_gym/rollouts/rollout_collection.py
|
/ok to test 4a9477b |
…agent-complete Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # nemo_gym/rollouts/rollout_records.py
Match main's masked_exhausted coverage (#6070), so no completion file is written while such a rollout is unmeasured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Frankie Siino <fsiino@nvidia.com>
|
/ok to test 613794c |
What does this PR do?
Writes a completion file for each agent in a multi-benchmark (batch) run as soon as every rollout of that agent is settled, so a finished benchmark's numbers are available while other benchmarks in the same run are still running, stuck, or killed.
Today a batch run (several benchmarks in one
gym evaljob, such as the Nemotron 3.5 Super Core and SWE batches) aggregates once, after the whole dispatch loop ends, into<stem>_aggregate_metrics.json. One benchmark that never finishes therefore withholds every other benchmark's score, even when those finished hours earlier. This blocks the picking up of finished benchmarks' scores.How it works
The collection changes are in
nemo_gym/rollouts/rollout_collection.py;nemo_gym/rollouts/rollout_records.pyandnemo_gym/rollouts/rollout_store.pygain the settled-rollout rule.(task, rollout)keys. An agent here is the environment server that scores its rows, as the run-wide aggregate groups them (_metrics_group), so a task routed to a server and a row naming the agent behind it are one agent. Its pending keys are the expected keys minus the rollouts already settled, so a resumed job counts what an earlier job finished. Other runs stop here and behave as before.RolloutRecords.settled, the complement of the rule that decides what a resume runs again. A success masked by infrastructure, or a failure a resume will retry, stays pending. When a set empties, a background task starts for that agent. Agents already finished by an earlier job start before dispatch.count_failure_classes_as_zerocounts as zero (the same helpers as the end-of-run pass), sorts by(task, rollout), calls the environment server's/aggregate_metricswhile servers are still up, and writes<stem>_agent_complete/<agent>.jsonatomically (forresults/rollouts.jsonl,results/rollouts_agent_complete/)._call_aggregate_metricsis split into_aggregate_entries(results, rows, *, labels=None)plus a thin wrapper, and both paths label entries through one helper (_server_agents), with labels computed over the full run so a per-agent call labels its entry as the full run does.<stem>_aggregate_metrics.jsonis written exactly as before. A fresh run deletes the completion directory.The file holds
schema_version,agent_ref,_ng_environment_server,expected_rollouts,completed_rollouts(scored),failed_rollouts,completed_at,aggregation_status(completeorerror) andaggregate(the agent's entry from the run-wide aggregate). The filename percent-encodes characters other than letters, digits,.,_and-, so distinct agent names never share a file.Behavior to know:
_ng_run_health_metrics, the run-level health summary Gym adds to the first run-wide entry after the run./aggregate_metricserror, or a call that gives no answer within 30 minutes (AGENT_COMPLETE_AGGREGATION_TIMEOUT_S), is recorded in the file (aggregation_status: "error"), so a hung environment server cannot hold up the run-wide aggregate. Any other failure writing the file is logged and no file is written. A run that stops early cancels and waits for its completion tasks and their in-flight/aggregate_metricsrequests.nemo_gym/rollouts/batch_status.py,nemo_gym/eval/comparison/loading.py,benchmarks/nemotron_3.5_super/export_to_csv.py) treat it as final. A latergym eval aggregaterepair rewrites it but not the completion files, so after a repair the run-wide entry wins over a file that recorded an aggregation error.gym eval runand the judge-only recovery job can then grade that answer again and recompute the run-wide aggregate in place. Both now refresh each completion file to its agent's recomputed entry and counts (refresh_agent_complete_files), and remove the files when that fails, or when a run that requires every rollout stays incomplete after the pass and so gets no recomputed metrics.completed_atbecomes the resuming job's time. It deletes the file of an agent it still has rollouts to run for, until they settle.Files:
nemo_gym/rollouts/rollout_collection.py,nemo_gym/cli/eval.py(re-judge refresh),nemo_gym/rollouts/rollout_records.py,nemo_gym/rollouts/rollout_store.py,nemo_gym/utils/path_utils.py(agent_complete_dir_for),tests/unit_tests/rollouts/test_rollout_collection.py,tests/unit_tests/eval/test_rejudge_complete_only.py,tests/unit_tests/eval/test_judge_only_settle_receipt.py,benchmarks/nemotron_3.5_super/README.md,fern/versions/latest/pages/evaluation/aggregate-metrics.mdx.Validation
On the current head:
Result: pre-commit clean, 3,669 tests passed. One timing-sensitive case in
tests/unit_tests/rollouts/test_legacy_deadline_collection.pyfailed once under parallel load and passed on rerun, with and without this change.tests/unit_tests/artifacts/test_exporters.pyandtest_live_export_receipt.pyneedmlflow, which the local environment lacks.The completion-file tests in
tests/unit_tests/rollouts/test_rollout_collection.pycover:(task, rollout)order even when the store hands rows back reversed, so metrics over a task's first repeats match the final aggregate.completed_at, and a file with no recomputed entry is removed; the end-of-run pass and the judge-only job remove the files when the recompute or the refresh fails, and a strict run left incomplete removes them.disable_aggregation; an/aggregate_metrics500 is recorded in the file; a failed file write is logged and the run-wide aggregate is still written./aggregate_metricscall that never answers is recorded as aTimeoutErrorwhile the run-wide aggregate is still written; a cancelled run stops its completion tasks and their requests before it returns.Rollouts
Cluster runs with the Super 3.5 step-120 checkpoint, through the NeMo Evaluator Factory benchmark runner, whose
batch-statusreads the files. Run 1 is ona850e8dd9, which has this review round's changes (grouping by environment server, settled rollouts, filename encoding). Runs 2 and 3 are onb1b8f0d28; runs 4 to 6 are on earlier commits (c11ceb6c6and08d13588f).a850e8dd9, all 11 agents, 190 rollouts (Slurm 8798318, 46 min, exit 0). The same small Core recipe as run 2. All 11 completion files were written during the job, between 02:56:30 (tau2) and 03:21:26 (IFBench), while the run-wide aggregate was written at 03:21:35. Every file carriesfailed_rollouts(0 here). Ten files are identical to their entries in the final run-wide aggregate. The eleventh, AA-LCR, has identical scores; its final entry adds only_ng_run_health_metrics, the run-level health summary Gym attaches to the first run-wide entry after the run.batch-statusshows all 11 accepted. No tracebacks.b1b8f0d28, 43 min, exit 0). A small version of the real Core recipe: a few tasks per agent, repeats above 1 on several agents, and 40 LMArena tasks x 3 so LMArena's own failure-rate check passes. All 11 completion files were written during the job, between 19:55:55 (tau2) and 20:21:38 (IFBench), while the job ended at 20:23:36. All 11 are identical to their entries in the final run-wide aggregate, including the multi-repeat benchmarks, andbatch-statusshows all 11 accepted. HLE completed with its current judge. No tracebacks.b1b8f0d28, 32 min). Verified's file was written at 18:15:29 and Multilingual's at 18:33:05, and the job ended at 18:34. Both files are identical to the final aggregate, andbatch-statusaccepted both. No tracebacks.batch-statusaccepted Verified then. Both files later matched the final aggregate field for field.batch-statusnine finished benchmarks' scores that would otherwise all have beenpending. HLE never finished because its judge at the time (azure/openai/gpt-4o) returned 403, which is unrelated to this change and since replaced in the recipe. It made HLE a natural stuck benchmark.Not exercised: a full-size Core run on the current head, and a benchmark whose rollouts hang forever (the unit tests cover a blocked
/run).Compatibility and benchmark impact
Additive. The new files go in a new
<stem>_agent_complete/directory next to the existing outputs. No config key or CLI flag was added, and the run-wide aggregate, its format and its readers are unchanged. Benchmark results are unaffected.nemo_gym/rollouts/batch_status.pydoes not read the new files. Outside consumers such as the EFB runner'sbatch-statusdo, and fall back to the run-wide aggregate when they are absent.The early scores are not written into
batch_status.json: that file exists only when a run is givenbatch_manifest_fpath, which the EFB batches do not use, and it is a progress and status file rather than a place for scored results. The completion files need no manifest, so they work for any multi-agent run.The Core and SWE batch repeat-count restore (needed since #5042) first landed here and has moved to its own PR, #6095, so this PR no longer changes any recipe.
Checklist
pre-commit run --all-files) (so CI lint/format checks pass).git commit -s) (so the DCO check passes).🤖 Generated with Claude Code