Repository navigation
test: mirror nemo_gym packages in the unit test tree - #5589
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
75efb95 to
79d18be
Compare
79d18be to
9cc41e6
Compare
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE
One RISK: a Harbor server test still reads a fixture from its removed location, so it fails before exercising the server.
I reviewed the relocation changes and their affected imports, fixture readers, repository-root lookups, collection configuration, and CI consumers. This was a static review; I didn't execute tests or collection checks.
Inline comments (escaped v1):
-
environment_servers/harbor/tests/test_multi_turn.pyline 2667: RISK — Update the server test's reference to the moved fixtureRunning
test_a_slopcodebench_task_runs_its_checkpoints_with_their_app_artifactsnow raisesFileNotFoundErrorat itsshutil.copytreecall, before exercising the server. This constant names the old fixture directory, which exists in the base tree but is absent from the head tree. The PR moved the synthetic task under the Harbor unit-test package. This breaks the Harbor server suite, though it doesn't change runtime behavior.Could you replace this line with:
Path(__file__).parents[3] / "tests/unit_tests/data/tasks/harbor/fixtures/harbor_slopcodebench/code_search"Keep the fixture at its new location: the relocated loader test correctly reads it there. Both tests must resolve the same committed task without duplicating or moving it back.
Acceptance checks, not executed here: rerun the server test and retain its assertions that all five steps run, the reward is
1.5, and every turn stagesapp/code_search/cli.pycontainingimport sys\n. Also retain the loader regression: it must reject the unbuiltSHELLimage, then load the recorded build with five steps and the existing artifact and reward gates. The suggestion changes only the fixture source path; please verify it before adopting it.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
9cc41e6 to
12512c7
Compare
12512c7 to
e50e404
Compare
|
🌿 Preview your docs: https://nvidia-preview-pr-5589-05f0f17f6f41.docs.buildwithfern.com/nemo/gym |
e50e404 to
2c1adbf
Compare
Move the flat files in tests/unit_tests/ into package folders that match nemo_gym/, so each test sits beside its package's other tests and can share that package's owners. Tests of benchmark and resources server code that the core suite covers go to tests/unit_tests/benchmarks/, and tests of scripts/ and CI go to tests/unit_tests/tooling/ (pytest skips any folder named scripts). Tests of the nemo_gym root modules and repo-wide checks stay flat. Only paths change: test-to-test imports, repo-root lookups through Path(__file__).parents, fixture locations, the core shard durations, the harness conformance path prefix, the migrate_imports keep list, the secrets baseline entries of moved fixtures, and docs that name a moved test. Saved-run fixtures that trajectory and orchestration tests share stay in the shared fixtures/ folder. Add an __init__.py to rollouts/harness_capabilities/ so its test modules import under their package names, like every other test folder. Refs #5166 Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
nemo_gym/progress_utils.py imports only the standard library and tqdm, so it belongs in nemo_gym/utils/ rather than at the package root, where it needed its own entry beside utils in the import-linter layers. It holds one class, the async tqdm subclass, so the module takes that name. The module has not been released, so the old path gets no deprecation alias. Its two importers and its test move with it. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
AGENTS.md describes the package layers but not where a new module or test belongs, which is how a helper landed at the nemo_gym/ root and tests piled up flat in tests/unit_tests/. Add the layout the previous commits apply: library code in its package (helpers without nemo_gym imports in utils/), unit tests mirroring nemo_gym/ with tooling/ and benchmarks/ for code outside it, an __init__.py in every test folder, fixtures beside their readers, and what to update when a test moves. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
2f0869c to
05f0f17
Compare
…ilience Clean merge; #5589 moved test_e2b_provider.py to tests/unit_tests/sandbox/. Signed-off-by: Hemil Desai <hemildesai@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No conflicts; git merged every file on its own, following the test files #5589 moved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Main now mirrors nemo_gym's packages in tests/unit_tests (#5589). The journal health test covers nemo_gym/rollouts/rollout_health.py, so it moves to tests/unit_tests/rollouts/, and its import of the re-judge test helpers follows that module to tests/unit_tests/eval/. Without the import change the settled-judge-answer test fails with ModuleNotFoundError on the merged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…st tree Main now mirrors nemo_gym's packages in tests/unit_tests (#5589). The journal re-verification test covers nemo_gym/eval/rollout_reverification.py, so it moves to tests/unit_tests/eval/, and its import of the re-judge test helpers follows that module to the same folder. Without the import change the settled-judge test fails with ModuleNotFoundError on the merged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
…-NeMo#3881) * feat(failures): typed failure reporting from servers, agents and collection Report every failure a rollout meets as a typed, causal record instead of an untyped error or a silent zero, so recovery can decide from saved evidence which failures to retry, which are final and which only need grading again. What this layer contains: - A failure-kind registry (`nemo_gym/core/failure_kinds.py`) by boundary: model requests, agents and harnesses, environments and verifiers, sandboxes, judges and collection, each with its documented default action. - Failure reports and events carried end to end: response headers and stream events between Gym servers (bounded, with an explicit "omitted" marker), a per-request failure log, and the collector's failure records (`_ng_failure_record`, `_ng_failure_events`) for each failed attempt. - Adapters that type their failures: model servers (transport, HTTP status, context limit, policy response timeout), agents and harnesses (route failures, timeouts, termination receipts), Harbor and legacy environments, sandbox providers, verifiers and judges (judge failure, unparseable verdict, verifier error), with a judge-call retry before a row is masked. - Collection that keeps a failed attempt's evidence, resends a refused /run within a connect grace period, retries a taken server port once, and marks placeholders that hold no generation. - Trial observations and grading records that bind a saved answer to its grading, used by the acceptance gate of the recovery layer. This commit squashes the branch's history (116 commits and its merges of main) into one, so the PR can be reviewed and its sign-offs checked as a unit. The tree is identical to the previous head, 04375cf71f90da06ade767af2653f1655e34c626. Four of the squashed commits were authored by the co-authors below and carried into this branch with the author's sign-off; the rest were authored by the author below. Co-authored-by: Wojciech Prazuch <wprazuch@nvidia.com> Co-authored-by: Jonathan Li <jonli@nvidia.com> Co-authored-by: Jeremy Bottleson <jbottleson@nvidia.com> Co-authored-by: Carlyn Aarish <caarish@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * feat(recovery): decide recovery from saved evidence, with budgets and acceptance gates Recover a saved evaluation run from its persisted evidence instead of re-running it. `gym eval resubmit --recover` (and `--inspect` to only read) collects the run's identities, saved results, failure records, quality summary and the job chain's stored budgets, and decides one action: RESUBMIT the identities a rollout store would run again, REVERIFY saved answers that only need grading again (never calling the policy), NOOP when the run is complete, or STOP with a typed reason. What this layer contains, on top of the failure observations of #5072: - Recovery decisions per identity: scored, model-final, retryable, exhausted, judge-recoverable, masked, omitted, terminal and unknown, driven by the typed failure kinds and a per-run outcome policy. - Budgets that are spent, never reset: infra retries, walltime and the per-identity attempt cap, raised only by explicit, journaled options (`--extend-retries`, `--extend-attempts`, `--max-walltime`), and an operator continuation for runs whose chain accounting is unknown. - The job chain's guard: continue, retry on an infrastructure cause, or stop and write the reason (`.gym_chain_incomplete`), including resume refusal, spin-up timeouts, install failures and credential refusals. - Re-judging saved answers: `gym eval reverify` and a judge-only job for GPU-served runs, on one CPU node (`--judge-partition`, `--judge-qos`) or, for a run that serves its judge on a node pool of its own, on that pool, settling into the source run under a receipt (`--settle-receipt`). Every `gym` command a rendered job script runs is tested against the real CLI. - Acceptance: a gate summary with response and trial health over the expected trials, the 2% evaluation-side limit, metric coverage, and MLflow delivery reported as a fact. - Store readers on the run's pinned revision, repin of a run to a newer commit (previewed with `--inspect`), and resume-fingerprint rules for settings added later. - Trial observation from a saved run's own grading evidence: saved verifier output, a self-grading agent's reward or rubric score, an agent rubric's guardrail-gated counts, a native scorer's saved metrics, a truncated answer saved in an agent's own envelope, a graded row that ended on the model's own failed request, and judge transport retries and cut judge replies in the trial join. A masked result is measured again: a saved patch is re-verified on a verifier that replays it, and a sandboxed session's lost workspace is retried. SciCode records how each sub-step's grading process ended. A SWE-bench Verified or Multilingual evaluation that ends without a verdict is masked as a verifier error with its reason and the patch digest (an intact patch that does not apply stays a graded zero); such rows are counted, not re-measured, until SWE-bench verifiers replay a saved patch. A verifiers rollout records the error it ended on, as a recorded fact that recovery and the gate do not read yet. - Operator tooling for chains the scheduler has already purged: a chain whose job ids squeue no longer knows is judged by sacct (a requeued job by its latest incarnation), so the post-run export and saved-answer re-judges are not blocked as "scheduler unknown"; and `gym eval resubmit ... --run-dir <run dir>` restores a lost job-index entry from the run's own record, journaled. - Skills and docs for operating the above. This commit squashes the branch's history (342 commits and the merges of #5072 and main) into one, so the PR can be reviewed and its sign-offs checked as a unit. Every squashed commit was authored and signed off by the author below. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * feat(rollout): record every rollout run on the Store and report its saved progress Give every rollout run, not only `gym eval run`, a recorded shard on the rollout Store with the same failure records and manifest, and let an operator read a run's saved progress without opening its files. What this layer contains, on top of the recovery layer of #5343: - `run_examples` callers (training frameworks) write their rollouts to a run-folder shard through the collector's outcome builder, so their failures are typed and recorded as in evaluation, with per-rollout enrichment exposed to the caller and failure-evidence captures kept. - A recorded shard gets a verified run manifest with its attempts reserved, a writer identified by a token (not an inode), and refuses rows without indices before replacing anything; every dispatched copy is recorded under the shard's run. - Saved progress per shard: a status file written under the writer lock, throttled and ended current, kept current when a step after collection fails. - `gym eval status <run dir | job id>` reads the Store for every run and reports completed and remaining rollouts, failure kinds, each exporter's receipt and the live tracking run id. - Docs for recording rollout evidence from a training framework, and the gym-eval skill's use of `gym eval status`. This commit squashes the branch's own history (22 commits and its merges of #5343) into one, so the PR can be reviewed and its sign-offs checked as a unit. Three of the squashed commits were authored by Marc Romeyn (one co-authored with Pranav Thombre) and carried his sign-off; the rest were authored by the author below. Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: Pranav Thombre <pthombre@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(eval): read selected attempts for health on the recovery stack Keep health checks on the Store snapshots used for scoring and adapt automatic, standalone, and shard reporting entry points to the recovery stack. Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> * fix(recovery): a verifier sandbox whose teardown failed after grading keeps the trial observed A failed provider close or client release after the grading is recorded by the sandbox lifecycle producer as outcome sandbox_error (or unknown) with the cleanup error as its error_type. The reader counted that as a failed grading sandbox and left a trial with a bound grade unobserved, against its own rule that an unconfirmed teardown does not. An observation whose only failure is its teardown no longer counts; a failed execution, or an error other than the teardown's, still does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(skills): gym-eval carries the recovery facts an operator needs on a pinned run A recovery reads the run inside its pinned Gym, so a run pinned before a rule existed is read by the older code until it is repinned; --inspect --repin previews the repinned plan without dispatching. Say that a queued action may exit 0 while the gate still reads RESUMING and that ok and job_id come from the JSON line, what an inspect that could not read reports and when a larger NEMO_GYM_REMOTE_READER_MAX is the remedy, and that the judge-only job needs a Gym that defines --settle-receipt. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(skills): gym-eval review nit (reader step limit) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * test(recovery): import SandboxSpec from its defining module Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * feat(failures): add typed boundary details Select sandbox/model details using the shared event schema. Move upload evidence and its policy readers together; keep event IDs and causes through transport and persistence. Unmerged draft fields have no compatibility alias. Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> * fix(recovery): a masked row that keeps a saved answer is never sampled again A masked result whose saved transcript held no gradeable artifact (a patch never collected or not intact, or a verifier that graded the agent's own sandbox) was run again as an infrastructure retry, although it kept the model's saved answer. On resume that regenerated the answer: a saved answer is never sampled again, and a limits-exceeded cleanup failure is a model-side stop. The Store keeps such a row as the result again; a masked row without a saved answer is still retried by its evaluation-side kind, and an intact saved patch is still re-verified. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * Record where and on which sandbox a sandboxed episode failed When a sandbox provider ended a container under an episode, the saved failure said little about where: the coding agent's provider exception reached the route wrapper with only its type, the Harbor agent's message named only Harbor's exception type, and the harness agent's masked row kept only the exception text. A record could not answer in which phase, on which sandbox, or with which provider status the episode failed. Each of the three producers now starts its failure_reason with a fixed "[sandbox phase=... provider=... sandbox=... reason=... http=... exit=... signal=... command=...]" prefix of the facts it knows, built by one helper (exception_inspection.sandbox_failure_prefix): - the coding agent's own command (phase agent): provider, sandbox id, the provider's error code (e.g. SANDBOX::ENDED) and HTTP status, or the unclassified result's error type and exit code. A provider exception is reported as the same agent_route_failed failure the route wrapper produced before, raised from the provider error, so the reply and the provider's own message (foreign_error) are unchanged; - the Harbor agent: the phase Harbor's trial was in (the last timed phase that started and did not finish: setup, agent, verify), the environment type and Harbor's exception type; - the harness agent: agent or verify phase, sandbox id, the provider's error code and HTTP status. Record only: no failure kind, mask, retry or score changes, and no reader classifies a failure from the prefix. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 45edada54fbfd913068ad08a649cd0decd6a13ca) * docs(skills): gym-eval lists the outcomes the stack does not decide for the operator A sandbox killed for memory during the agent's own command, a sandbox that ended for its own limit, SkillsBench agent timeouts, AutomationBench rows graded on a partial world and SciCode grading subprocesses that were killed are retried or scored in a way the operator must correct or report until the matching policy lands. Name the field to read for each: failure_reason, sandbox_resource_limit, rollout_error and step_exits. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(skills): gym-eval review nit (exit 137 is Terminal-Bench 4.0) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * feat(recovery): a reader at the reader-step cap queues for a slot instead of reading nothing At the per-user cap of reader steps, a recovery reader with the login-host fallback forbidden ran nowhere and the inspect read UNKNOWN. It now queues: it counts the reader steps again every 2 s for up to NEMO_GYM_REMOTE_READER_QUEUE_WAIT_S (default 1800; 0 keeps the old behaviour), prints a progress line saying how many it is queued behind, and runs in its step once one ends. A cap still reached after the wait gives the old reason with how long it queued. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> (cherry picked from commit a0c76c85ac0c6fce6dc8adf8dac4258b3c8fc43d) * feat(recovery): plan an old-pin run on the installed Gym and repin it only when the plan needs it A recovery read a run's Store inside the run's pinned Gym, so a run pinned before a recovery rule existed was planned by the older code, and a judge-only re-judge on it was refused unless the operator passed --repin. The official command could not recover such a run on its own. A run pinned at another full commit than the launcher's is now planned with the installed Gym's rules. When the plan needs the installed revision to run (a re-judge, whose job runs the installed recovery source, or a resubmit the pinned revision plans differently), the recovery moves the run to the installed commit itself, journaled as an automatic repin with the reason; otherwise the pin stays. The outcome reports the decision (pin_plan) and --inspect shows the same one. --no-auto-repin keeps the pin and refuses that case, naming --repin; --repin stays the explicit way to move a run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> (cherry picked from commit 697bf3725734c367e90f46e978e3688cd8c1685c) * fix(recovery): the reader queue waits 10 minutes by default and polls every 10 s Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> (cherry picked from commit fb1e87ff0dc016ab9543bec85896a156e971a6e9) * refactor: trim sandbox failure details Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> * fix: export typed failure details publicly Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> * fix(recovery): a judge-only job captures its model calls in its own folder A judge-only job's gym eval reverify takes the run's settings, which turn observability on, but only the submitted evaluation's script derives the model-call capture directory. The job therefore started with observability on and no capture directory, and its policy server refused to start ("model_call_capture_dir is required when observability_enabled=true"). Both judge-only paths now apply the same rule as a submitted evaluation (_with_default_capture_dir): observability on by default, capturing into the job's own reverification folder unless the run names a directory. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(recovery): the CPU-only re-judge captures its model calls in its own folder Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(resume): an omitted failure report never blocks resume; name what resume leaves to recovery A failure report over the 4096-byte transport bound is saved as omitted. The resume gate treated that flag alone as an unclassified execution, so the rollout was never run again whatever its own failure record said: a transport size limit decided retries, contrary to the documented "an omitted report changes no outcome". The flag now carries no veto; the row's own record and kind decide, as for a row without a report. The answer gates (a possibly delivered request that kept a response, a saved generation, a recorded grade, a partial response) still keep a saved answer from ever being regenerated. Judge failures, kept answers and unclassified executions are left to recovery and not run again, but neither resume path said so. Store resume now prints how many such rollouts it leaves, and coverage reports them as stopped_not_retried (a part of failed_final). The legacy resume summary names them on their own line. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a31c4741a9b0f840a8a35b4419e292b3dce0ff7d) * docs(failure-reporting): align three doc statements with the code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> (cherry picked from commit d00e6520e93538a705dae8735a0c956518766ae9) * fix(failures): keep an unknown kind on the wire, judge every judge failure again, log dropped events A forwarded failure event whose kind the receiving Gym does not register (a server pinned to another version) made the whole report invalid, so it decoded as omitted and every event in it was lost. Header and stream decoding now keep such an event under "unregistered:<kind>"; any other invalid report is still omitted. The judge failures whose saved answer is judged again were spelled out as ("judge_failed", "judge_invalid") at seven sites; the registered judge_unparseable kind was in none of them, so its rows could be run again instead of re-judged. One shared JUDGE_FAILURE_KINDS now holds all three (judge_invalid stays: it is collection's own legacy class). A request's failure log that fills up counted its dropped events but never said so; it now logs once when it starts dropping. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2f04f0617fd7deddfd63b2d9ee191be990b68b98) * test(resume): the summary names rollouts left to recovery; an omitted report keeps a terminal failure final Covers the legacy resume summary line for a judge failure left to verification recovery, and a terminal failure whose report the size bound omitted: the omitted flag no longer vetoes a retry, and the terminal mark still keeps the rollout final. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 9265edd70bb67aad03a03d4dc31ce4194f8f30b4) * test(resume): an omitted report on a model 500 is retried like any other 500 The body-store test pinned the old gate, in which a report the size bound omitted kept the rollout from ever running again. With that veto gone the model 500 (no answer was produced) is retried on resume; a failed body behind a 2xx header (truncated, timeout) is still never run again. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e601af569a8bfcc976a54933cdbbbf0da4677de8) * fix(merge): import names from their defining modules; layer the moved session-conformance helpers The main merge-up rewrote some test imports to the nemo_gym.rollouts facade, and the explicit server_runtime layer list did not name the nemo_gym.server_runtime.testing package main added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(merge): TB4 reports rejected sandbox credentials at the sandbox boundary sandbox_credentials_rejected no longer takes a boundary: a provider that refuses the credentials is a sandbox failure. The terminal-bench resources server still passed boundary=ENVIRONMENT, so every session setup that raised failed with a TypeError instead of the typed rejection. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(merge): tests import names from their defining modules after the rollouts move Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * test(export): a run_dir above the rollouts leaves out the resume identity beside them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(exports): leave out a resume identity anywhere under the uploaded run directory The EvalRDL upload is now the benchmark directory above the rollouts, so a resume identity can sit in any directory under it (beside the rollouts, or in a re-verification directory). Only the one beside the rollouts was left out of the upload. An identity can hold credentials, so the upload mirror now leaves out every file with the identity suffix under the run directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(resume): name the operator continuation when chain accounting is unknown An accounting-unknown stop said only to reconcile the queue before recovery. The log line now also names the continuation the gym-eval skill and the resumable-runs guide give: gym eval resubmit --recover with --max-walltime and --walltime-reason, plus --extend-retries when the retries are spent. The incomplete-chain marker keeps the reason alone, since its readers match that text. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(recovery): correct the omission-summary example and a comment typo The omission summary prints the stage as "(stage_index 1, ...)", the form accepted_stage_omission returns. Comment and docstring only. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(orchestration): align resume, resubmit and driver-install comments with the code ResumeConfig.max_retries counts infrastructure retries only; a generic FAILED, an out-of-memory kill or a deadline stops the chain. A --recover resubmit keeps its budget, and plain resubmit still starts a fresh one. The private-uv paragraph of render_driver_entrypoint is re-worded and the cleanup option is documented. Comments and docstrings only. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(eval): drop a repeated source-snapshot guard and a shadowing import The CPU-only re-judge branch checked evidence.source_digests for None twice with nothing in between that could change it; the first check stays. _prepare_run_health_metrics re-imported subprocess, which the module already imports. No behaviour change. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(eval): fix comments and docstrings that no longer match the code The final-zero comment in collection moves back above check_completion, which it explains; the capture field comment gets its apostrophes; the verifiers agent lists its kind candidates in the order they are matched; and the exporter docstring describes the version-1 receipt check. Comments and docstrings only. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(eval): describe this layer's recovery as the code does Skills and the resume guides now match the code at this head: - the accounting-unknown stops: an empty sacct reply is asked up to six times, 10 s apart, before the chain stops, and the stop names the continuation; repeated allocation ends charge one attempt from the second end on; - the gate fields (allocation cause, output-budget refits, agent adaptations, retry budget, pin plan, observed grading recovery), the full list of grading artifacts and the rule for unobserved servers; - the coverage fields (masked_retryable, final_zeros, stopped_not_retried, attempt_extension, completion_fraction); - the chain end, the reader queue wait and login fallback, the reader time bound, the UNVERIFIED status of an unobserved trial, two chain files, and smaller corrections (config path, GDP-PDF and Stirrup defaults, smoke limit); - two exit-check tables that a pointer paragraph had split. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(recovery): a reader step srun refuses for a submit limit or busy nodes is submitted again A recovery reader whose Slurm step srun refused outright (a per-user submit or QOS limit, or no node free for an immediate start) was treated as a step that did not start: it was cancelled and the read gave up, so an inspect or recover read nothing on a busy CPU partition. Such a submission was never queued, so the reader now submits it again after 15 s, doubling up to 60 s, with a progress line each time, for as long as the reader queue wait allows. Any other srun failure, and a refusal that outlasts the wait, end as before; the reason then says how long it resubmitted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): a segment with nothing eligible for retry ends its own chain A driver that exits 76 (no task is eligible for another attempt) failed the job, so Slurm started its afternotok follow-up on a full allocation, often GPU nodes, only for that follow-up's prologue to write the stop and exit. The segment's epilogue now writes the no-eligible-retry stop itself, cancels the sentinel and the follow-up it queued, and journals the end in chain-end-journal.jsonl, the same way a segment that completes the run does. A deliberate stop already in the marker stays; the job keeps exit 76. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): a remote reader's install waits as long for uv's lock as a Gym run does A remote Store reader runs from a cleared environment on the cluster's login host, so a cold install of the run's pinned Gym there took uv's own default lock timeout instead of the caller's UV_LOCK_TIMEOUT or the 1800 s a Gym run uses, and an inspect on a busy host failed in the install. The reader command now exports UV_LOCK_TIMEOUT (the caller's value, else 1800) on the login host and before srun, so a reader step gets it as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(skills): gym-eval reads as one path from a cell state to the official act The gym-eval skill is rewritten around the operator path, with the per-benchmark notes in references/benchmark-notes.md, and states what this PR adds: a refused reader step is submitted again, a remote reader takes UV_LOCK_TIMEOUT, and a segment with nothing eligible for retry stops its own chain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): a typed driver failure that exits 76 keeps its own stop A driver that stops on a typed failure (rejected provider credentials, a refused collection resume) leaves its kind in the driver failure file and exits 76, and the follow-up prologue turns that kind into its stop and remedy. The chain-end epilogue wrote the generic no-eligible-retry stop for every exit 76 and cancelled the follow-up, so that typed stop was lost. The epilogue now ends the chain only when the driver left no failure kind; otherwise the job keeps its follow-up as before. The reader's UV_LOCK_TIMEOUT check accepts only ASCII digits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): exit 76 keeps its follow-up only for a kind the next guard stops on The previous change kept the follow-up for every exit 76 that left a driver failure kind. The follow-up's prologue writes a stop only for four kinds (spin-up timeout, install failed, resume refused, credentials rejected); for any other kind, such as a server that failed to start, it would start the run again. The segment now ends its own chain on exit 76 unless the kind it left is one of those four. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * feat(recovery): an explicit, journaled rerun of a masked row with no gradeable artifact A masked result that kept a saved answer is retained and never sampled again. When it also has no recorded grade and no re-verification cause (for example a patch artifact that was never collected), nothing can ever measure it: resume keeps it and re-verification cannot grade it. `gym eval resubmit --recover --rerun-masked-without-artifact --rerun-reason TEXT` is the explicit opt-in. The default is unchanged: without the flag these rows stay MASKED, and `--inspect` now names them in their own bucket (`masked_without_artifact`, with a per-row reason) instead of lumping them in with the other masked rows. With the flag, the identities the plan finds qualifying at that moment are appended with the reason, the time and each attempt count as one line to `rerun-masked-journal.jsonl` beside the rollouts file, just before the job is queued (and taken back if it is not). The Store reads that journal: an approved row counts one failed attempt, is retried within its attempt cap and is superseded by its next attempt. An approval names the attempt it covered, so a later attempt is never reopened by it, and the row must still qualify. The reason is required and must be non-empty; the flag needs `--recover`; `--inspect` never writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * feat(recovery): an explicit, journaled acknowledgement of failed judge-only jobs When the last two judge-only jobs of a run ended without settling a saved answer, recovery queues no third and stops. Nothing could clear that state, because it is derived from the tail of the judge-job journal. `gym eval resubmit --recover --acknowledge-failed-judge-jobs --acknowledge-reason TEXT` allows exactly one more judge-only job. The reason is required and non-empty, the flag needs `--recover`, and `--inspect` never writes. The acknowledgement is one line (time, user, reason, the ids of the failed jobs it covers) in a sibling journal, `judge-job-acknowledgement-journal.jsonl`, because the judge-job journal's readers treat every line as a job. It is written only once the scheduler has accepted the new job. It covers only the pair of jobs it names, so if the extra job fails too, the next pair needs an acknowledgement of its own. The stop text and `--inspect` name the flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): name the rerun-masked journal after its rollouts file The approvals were journaled under one fixed name per directory, so two rollouts files in the same directory would have shared them. The journal is now named after the rollouts file it sits beside, the same way the attempt extension is (rollouts.jsonl -> rollouts_rerun_masked_journal.jsonl). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * refactor(recovery): one list of the failure kinds the next segment's guard stops on The exit-76 epilogue named the four kinds the follow-up's prologue turns into its own stop as a second, separate list. Both now come from GUARDED_DRIVER_FAILURE_KINDS, and a test checks that the prologue has a stop for every kind in it, so a kind added to one side cannot be missed on the other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * test(recovery): a blocking judge-only job without an id cannot be acknowledged Covers the stop text for a failed pair that includes a job without an id: it names the unknown job, says it cannot be acknowledged, and offers no acknowledgement flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): a judge-only job does not wait for the policy endpoint A judge-only re-verification job judges saved answers and never calls the policy model, whose served endpoint ended with the run's allocation. It still waited for every model endpoint's readiness, so it timed out after the readiness limit and exited 1 before judging anything. The rendered job command now sets model_endpoint_readiness_timeout_seconds to 0 (a run argument that set it is dropped), which the job's pinned Gym reads as "skip this check". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(submit): retry the gym_install ref check and say when it is the network that failed `gym eval submit` verifies driver.gym_install.ref with one git ls-remote against the repo. A short network hiccup on the submitting host made that call time out, and the submit failed with "Could not verify driver.gym_install.ref '<ref>' in <repo>: ... timed out" and created no job, which read like a bad ref. The check now retries a failure to reach the repo after 10, 30 and 60 s. A full commit hash that a remote-tracking ref of the running Gym's own clone of that repo already contains is verified locally, with no network call; a local commit that was never pushed is still checked over the network. When every attempt fails, the error says the network check failed, how many times, and that this is not evidence the ref is missing. A missing git binary is not retried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(submit): the local fallback of the ref check runs only after the retries and a pruning fetch The local check accepted a full commit hash that a remote-tracking ref of the running Gym's own clone contained, without asking the repo. Such a ref can be stale: after a branch is deleted or a stack is squash-merged, the clone keeps the old remote branch until its next pruning fetch while the repo no longer serves the commit, and the job's install would fail later. The network check now always decides when the repo answers, and a ref it does not have fails the submit. Only when every attempt failed to reach the repo is the clone consulted, and only after a successful `git fetch --prune` of that remote, so a stale ref never vouches for a commit; the submit then logs that the local fallback was used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * test(submit): record only the ref check's own retry waits The fixture replaced the global time.sleep and asserted the exact list of waits, so a sleep elsewhere in the same test process (a subprocess poll, a thread left by another test) joined the list and the assertion failed under parallel runs. The fixture now stubs only the time module the ref check uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(skills): gym-eval and gym-add-model-config, v23 gym-eval: install the gym CLI at the pinned commit (the runner's own form), say the page links need a full Gym checkout, name the reader-step srun variable with an example, describe the chain end without assuming the pin, the chain bookkeeping files and their permissions, the judge-only acknowledgement, and a caveat that a verifier-timeout row can read as measured in coverage while recovery marks it masked. gym-add-model-config: point at the gym-eval steps that exist, and check that the served chat template renders tools (and how to see in the rollouts that it did). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): the remote reader installs the MLflow client; name the score flags The remote reader installs the run's pinned Gym without extras, so its query of the exported run's status on the tracking server failed with ModuleNotFoundError: No module named 'mlflow', and MLflow delivery stayed unverified for a run that had exported. The reader now installs the same mlflow extra the login-host export uses. No gate rule changes. The gate's "score unavailable: no requested metric was read" now names the flags that read a score: --score-key, and --score-agent or --score-environment for a Gym aggregate, which is a list of entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * refactor(recovery): move the evidence collector script into collect_evidence.py COLLECT_SCRIPT, the self-contained script recovery runs on the run's host, and the six constants it embeds move from recovery.py into nemo_gym/orchestration/collect_evidence.py, as a mechanical move. The script is byte-identical: its sha256 and those of the constants are the same before and after. recovery.py imports them back, so every name stays importable from it, and the Slurm executor runs the same object. A new test pins that, and that the script compiles. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(resume): a refused follow-up submission is a typed stop, not unknown accounting When sbatch answers the follow-up with the scheduler's own refusal (for example a QOS limit on submitted jobs), nothing was queued and the accounting is known. The chain recorded it as unknown accounting, which needs an explicit walltime budget to continue. It now writes follow_up_submission_refused with sbatch's exit code and error lines to .gym_chain_incomplete, still queues nothing, and its log line names the continuation, gym eval resubmit --recover. Recovery reports the stop. A reply that is missing, or says the request or its reply was lost (a timeout or a connection failure), stays unknown accounting as before; its log line now includes sbatch's reply. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(resume): a segment that completes the run supersedes the chain's earlier stop A stop recorded earlier in the chain (a refused or unanswered follow-up submission, a walltime cap) described a run with rollouts missing. When a segment then completes the run, .gym_chain_incomplete stayed, so a later reader saw an incomplete run. The segment now removes it, logs the stop it supersedes, and journals the chain end as run_complete_stop_superseded. A run that still lacks its required export receipt records that instead, as before. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(metrics): keep a per-task row for a task with no saved rollout profile_from_data dropped a task none of whose rollouts was saved, so a partial run's per-task report silently listed fewer tasks than its inventory. Such a task now gets a row with counts only: num_rollouts 0, expected and missing rollouts, and a completion of 0%. The agent and repeat statistics still come from the saved rollouts, so scores are unchanged. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(resume): count only a definite scheduler refusal as a refused follow-up A reply such as a broken pipe or an unspecified error can come back for a job the scheduler already accepted. The follow-up is now taken as refused only when sbatch names a definite refusal (an invalid request, a policy or limit it violates, a resource or dependency that is not available) and nothing says the request was lost; any other error stays unknown accounting. Tests cover both kinds of reply, and that recovery resubmits a run whose follow-up was refused instead of stopping on unknown accounting. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): carry the gym-eval corrections onto the rewritten skill The rewritten gym-eval skill dropped corrections made to its previous text. They are carried over: the empty-sacct stop and its continuation, a refused follow-up submission and a completed run superseding an earlier stop, attempt charging from the second allocation end, the gate fields (pin_plan, the allocation cause, output-budget refits, agent adaptations, retry_budget, observed grading recovery), compute.cluster.forward_env, the smoke-run limit, and in the benchmark notes the GDP-PDF and Stirrup defaults, the full list of grading artifacts and the rule for unobserved servers. The skill no longer instructs gym eval status, which this version does not have: progress is read with the read-only recovery inspection or from the run's coverage report. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(recovery): a server that failed to start before any rollout restarts the run fresh, once A segment whose driver died at server start (for example a port already in use), before the run saved any rollout, left the kind server_start_failed. The follow-up treated the FAILED job as having no infrastructure cause, asked the saved evidence, found no rollout store manifest and stopped with "no eligible retries", which hid the cause and named a recovery that would refuse as well. Now the follow-up starts the run again fresh, as an infra retry within the chain's retry budget, and at most once: the restarting segment leaves a marker, and a second server start failure, or a spent budget, stops with the kind, the job, "fresh restart exhausted" and the fresh resubmit command. A run with saved rollouts keeps the saved-evidence path unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(skills): install the launcher with the mlflow extra The inspection's MLflow delivery check imports the mlflow client. A launcher installed without it reports the delivery as unverified with a missing-module error, although the gate still passes. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(recovery): restart fresh only after a job that ended FAILED A server_start_failed file beside a job that ended OUT_OF_MEMORY or DEADLINE keeps its existing stop; the one fresh restart applies to a job that failed before any rollout and left no manifest. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): drop pin-dependent and outdated wording from gym-eval The skill now describes this version only: no notes about older pins (the sentinel that ends a chain, attempt extensions, the old re-judge NOOP, a reverify option an older Gym lacks), no "until the fix lands" caveats, and the GDP-PDF and grading notes no longer compare against an unnamed branch or stack version. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): gym-eval comparison guidance and the agent-timeout opt-out Keep a policy-model timeout on the model's side when two checkpoints, or a checkpoint and its quantized export, are compared; keep compared runs on one Gym commit. An agent timeout that kept no answer is retried by default, and the skill now names the recovery_outcome_policy entry that keeps it final. The forward reference to a later status command is removed. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * gym-add-model-config: eval-vs-deployment nuances and a complete smoke step Pin a full commit, never a branch. Name --gpu-memory-utilization as safe to change and keep quantization and dtypes identical across a comparison. Explain what the client sends and leaves to the server (sampling, the output-budget refit, system prompts), the reasoning-parser default, the policy_model_type clash, and give the smoke step its submit command. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(resume): the walltime-cap stop names a continuation the CLI accepts The stop said to continue with --max-walltime alone, which gym eval resubmit refuses without --walltime-reason. It now names both. A test runs the hinted options through the CLI and checks they reach the recovery. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(eval): document recovery as the code does at this version The CLI reference lists every gym eval resubmit option. The resumable-runs guide names the typed chain stops, the chain-end journal values and the split between a refused and an unanswered follow-up submission, and states the walltime budget rule: max_walltime is the chain total, a cap equal to the job walltime is spent by the first timeout, and Gym never adds time on its own. The vLLM page documents refit_output_budget_to_window, and the max_retries description is corrected. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(recovery): an unread score of an incomplete run names the run, not the options The gate said to pass --score-key and --score-agent whenever no score was read, also when both were passed and the run simply had no aggregate yet. An incomplete run now says so, with its settled and expected rollouts; the options hint stays for a complete run. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): gym-eval progress, guard exit, dispatch credentials and rollout upload Progress while a chain runs is the line count of rollouts.jsonl; the coverage report's completion is final only when the chain stops or the run ends. A guard job that writes a stop ends FAILED by design. A run submitted from the login node keeps its credentials as runtime entries the dispatch does not check, so the dispatching shell must set them. A run that does not upload its rollouts keeps an unverified MLflow delivery by configuration. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(resume): name the continuation when less than a minute of walltime is left The guard stopped with only "Less than one minute remains in the walltime budget." It now gives the stored and capped seconds and the same continuation as the cap stop, --recover --max-walltime with --walltime-reason. When the infra-retry budget is spent and the walltime is too, the recovery stop names both and says to raise both in one call, and the gym-eval skill says so in its two budget rows. Tests run the hinted options through the CLI and cover the stop with and without a spent walltime. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): rename gym-eval to run-and-babysit-eval and gym-add-model-config to add-model-eval-config The two public skills get names that say what they do. The skill directories, their .claude and .codex links, the name fields, the cross-references between the skills (and from add-benchmark and add-harness), the benchmark-notes title, the agent prompts and the agent-skills docs page follow. The wording inside the skills is otherwise unchanged. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): add per-benchmark findings to benchmark-notes Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): a segment must outlast the longest task A rollout cut by a segment end restarts from its first step, so a segment walltime shorter than the longest task repeats the same work in every segment. Say so in run-and-babysit-eval next to the three limits, and in the Terminal-Bench notes, where single tasks have needed more than 90 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(eval): reverify: a collection --limit no longer caps the rows a re-judge pass may re-verify A collection limit counts tasks, while reverify reads limit as the largest number of rows to re-verify. The automatic re-judge after collection and the judge-only job kept the run's limit, so a run collected with a limit re-judged at most that many rows per pass and never reached the rest. Both now drop it with the other collection-only settings. Tests cover the automatic pass (a run collected with limit 4 re-judges all 10 due judge failures) and both judge-only job paths. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): spin-up limit and judge-key environment for babysitting a run Before submitting: the server spin-up limit (default 600 s) covers dependency installation and model loading, so raise it for a large model, a fresh install or a loaded cluster. A spin-up timeout is an infra stop that leaves rows untouched and is never scored as zero; the chain waits for a recover with a raised limit. Judge-only re-judges take the judge key and the other forwarded credentials from the environment of the dispatching shell, so export the valid key there before dispatching. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(slurm): pin the policy node's Ray bootstrap to a tested version with the serve extra A container without the ray CLI installed `ray[default]` unpinned before `ray start` and the Serve gateway. The gateway imports `ray.serve`, whose dependencies the `default` extra does not declare, and an unpinned install takes the newest Ray. With a Ray release whose Serve import needs a package outside `default`, every start-up of such a policy service fails the same way at the gateway's import. Install `ray[default,serve]==2.59.0` instead. A container that ships its own Ray is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(rollouts): scope a recorded shard's rollout ids to its run run_examples records a shard on copies of the rows that name the shard's run, but the copies kept the row's own rollout id, and with it the capture key the model calls are staged under. Capture files are kept by default, so a later shard that dispatched the same task and rollout (a row an earlier shard returned, fed back, or indices numbered again each step) merged the earlier shard's calls into its own records. Each dispatched copy of a recorded shard now carries the row's own id (its _ng_rollout_id, or <task>-<rollout>) prefixed with the shard's run id. A row an earlier shard returned has that prefix replaced, not nested. The capture key is that id with the usual attempt suffix, so capture_key and the /ng-rollout/<capture key> path keep their form; only the key is now the shard's own. The Store, the manifest, the records and the returned rows use the same id, so a caller reads a capture's key from the returned row. enrich_results without recording keeps the caller's keys. Tests: two shards that dispatch the same task and rollout, with the row fed back or the indices numbered again, each record only their own calls (they fail without the change). The shard tests stage captures under the dispatched key, as the model server does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(rollouts): give a recorded shard the manifest its resume computes A recorded shard's manifest hashed the caller's rows and the recording options, while the manifest gym eval run computes on resume hashes the source file's bytes and the resolved configuration. Resuming a shard with resume_from_cache was therefore refused for its source dataset and its configuration. The earlier test passed the saved manifest in as the current one, so it never compared the two. The writer now builds the shard's manifest with RunManifest.create, the identity function collection uses. Its inputs are the shard's materialized inputs as the source file, the rows as dispatched, a collection config that names the shard's input and output paths and leaves every other setting at its default, and the call's server configuration. The identity file is saved beside the manifest, as for any run, and replaced with the shard. So gym eval run +input_jsonl_fpath=<dir>/rollouts-<shard>_materialized_inputs.jsonl +output_jsonl_fpath=<dir>/rollouts-<shard>.jsonl +resume_from_cache=true verifies and runs only what the shard still lacks. An operational difference resumes with the usual drift note, and a semantic one is refused and named. If the running configuration gives no identity (for example it does not name the rows' agent), the shard is still recorded, with a warning, and a resume refuses it and says why. Tests: a shard recorded with one failure resumes through run_from_config, dispatches only the failed row and ends with both rows scored; a shard without a configuration identity is recorded and warns. Both fail without the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(observability): shard capture keys and resuming a recorded shard The training-framework section now says that a recorded shard dispatches each row under its rollout id prefixed with the shard's run id, so a capture's key is read from the returned row, and that a shard resumes with gym eval run on its materialized inputs and its rollouts file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * test(rollouts): a run that is not a shard keeps its rollout ids and capture keys Only a recorded shard prefixes its rollout ids with its run id. A plain gym eval run and run_examples with enrich_results alone dispatch every row under the id Gym derived before: <task>-<rollout>, or the row's own _ng_rollout_id. The saved results carry the same ids, so recovery and re-verification of such runs read the identities they always did. The test passes before and after the shard change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(rollouts): end a shard's status current when its last row is an unkept omitted row With record_omitted_rollouts left off, the shard writer returned for an omitted row before its status write. When that row was the last one expected and status writes were throttled, the final counts never reached the status file: with two scored rows and an omitted third, gym eval status reported one completed result instead of two. Closing the iterator corrected it, but iterating to the end does not close it. The early return now flushes the status when it handles the last expected row. As with any flush, a writer that another writer superseded leaves the newer status alone. The saved results do not change. Test: the throttled status test also writes two scored rows and an unkept omitted row, and expects two completed and one remaining (it reports one completed without the change). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(rollouts): refuse recording rows that would share a rollout in the shard A recorded shard prefixes each row's rollout id with its run id, after removing the prefix an earlier shard put on a row it returned. A row fed back as `<earlier run>.0` and a new row whose own id is `0` therefore both became `<this run>.0` in the same call. They would then share capture files and one manifest identity. A training framework hits this when a step resubmits a returned row next to fresh rows whose numbering restarted. run_examples now refuses such a call before dispatch, and before the existing shard is replaced. The error names both rows and both of their own ids, so the caller can give each row a distinct _ng_rollout_id or distinct task and rollout indices. Test: `<earlier run>.0` beside `0` is refused with both ids named, nothing is dispatched and the existing shard is kept. `<earlier run>.0` beside `1` still dispatches, as `<run>.0` and `<run>.1`. It fails without the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(rollouts): refuse a shard rollout id too long for its capture files The capture key, a rollout id with its attempt suffix, also names the model server's capture files. The longest of them is `<key>.capture.incomplete`, and a file name holds at most 255 bytes. A recorded shard adds a 33-character run-id prefix to each row's id, so a long explicit id could pass that limit and fail only when the first model call was captured, mid-rollout. A shared check, check_rollout_id_length in nemo_gym.core.episode_context, allows a rollout id of up to 230 bytes. That leaves room for attempt suffixes up to -a9999 and the longest capture file suffix, and also keeps a key under a 256-character id limit. run_examples applies it to each recorded row's id after the prefix is added, before dispatch and before the existing shard is replaced. The error names the id, its length and the limit. Test: an id one byte over the limit, after the prefix, is refused with nothing dispatched and no shard written. The longest id that fits dispatches, and its capture file name at attempt 9999 is exactly 255 bytes. It fails without the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(reference): placeholders the MDX generator can parse The library reference page generated from the judge_profile module failed to parse: its module docstring wrote the pool node variable as ``${GYM_POOL_<POOL>_NODES}``, and the generated MDX read `<POOL>` as a tag. The docs preview of any change that rebuilds the full reference failed on it. The docstring now names the variable as ``GYM_POOL_*_NODES``. No other docstring combines a ${...} reference with an angle-bracket placeholder, and the other pages built from docstrings parsed in the same build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(reference): name the judge profile's default QOS source as a constant The library reference page for the judge_profile module failed to parse. The page generator writes each field's default into an MDX attribute as its Python repr, and JudgeExecutionProfile.qos_source defaulted to the string "the partition's default". Because of the apostrophe, the repr is double-quoted, so the generated line was <ParamField path="qos_source" type="str = "the partition's default""> and MDX stopped at the `'` ("Unexpected character `'` in attribute name"). The default is now the module constant PARTITION_DEFAULT_QOS, with the same text, so the page shows `str = PARTITION_DEFAULT_QOS`, as it does for `kind: str = CPU_PROFILE`. The two other places that set the same source use the constant too. The text in the operator's log does not change. No other plain string default in nemo_gym contains an apostrophe. Verified by generating the page locally with the docs CLI and compiling it with MDX 3: the old page fails with the error above, the new one compiles. The judge profile tests pass unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(eval): normalize health snapshot paths and validate record identity Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> * test(rollouts): place the journal health test in the mirrored test tree Main now mirrors nemo_gym's packages in tests/unit_tests (#5589). The journal health test covers nemo_gym/rollouts/rollout_health.py, so it moves to tests/unit_tests/rollouts/, and its import of the re-judge test helpers follows that module to tests/unit_tests/eval/. Without the import change the settled-judge-answer test fails with ModuleNotFoundError on the merged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> --------- Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Signed-off-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> Co-authored-by: Adam Rajfer <arajfer@nvidia.com> Co-authored-by: Wojciech Prazuch <wprazuch@nvidia.com> Co-authored-by: Jonathan Li <jonli@nvidia.com> Co-authored-by: Jeremy Bottleson <jbottleson@nvidia.com> Co-authored-by: Carlyn Aarish <caarish@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: Pranav Thombre <pthombre@nvidia.com> Co-authored-by: Giulio Lovisotto <glovisotto@nvidia.com>
Brings in the unit test tree that mirrors the nemo_gym packages (#5589), with the rest of main. Resolutions, following the layout #5589 adds to AGENTS.md (a test sits with the package of the module it is named for): - The 24 recovery test modules that #5589 placed in tests/unit_tests/orchestration/ go to tests/unit_tests/eval/recovery/, beside the package they now test. The export receipt, grading result binding and saved-judge rejudge tests stay where #5589 put them (artifacts/, trajectory/, eval/): each is named for a module outside the recovery package. They keep this branch's imports. - Test-to-test imports follow both moves, and the moved tests keep #5589's path changes. - The five recorded-run fixtures this branch keeps as patches are removed from tests/unit_tests/orchestration/fixtures/, where #5589 moved them, and their patches move there, beside their healthy base and the loader that reads them. The secrets baseline follows: main's entries, without the five full fixtures, with the patch files at their new paths. - No moved test has an entry in the core shard durations. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) * feat(failures): typed failure reporting from servers, agents and collection Report every failure a rollout meets as a typed, causal record instead of an untyped error or a silent zero, so recovery can decide from saved evidence which failures to retry, which are final and which only need grading again. What this layer contains: - A failure-kind registry (`nemo_gym/core/failure_kinds.py`) by boundary: model requests, agents and harnesses, environments and verifiers, sandboxes, judges and collection, each with its documented default action. - Failure reports and events carried end to end: response headers and stream events between Gym servers (bounded, with an explicit "omitted" marker), a per-request failure log, and the collector's failure records (`_ng_failure_record`, `_ng_failure_events`) for each failed attempt. - Adapters that type their failures: model servers (transport, HTTP status, context limit, policy response timeout), agents and harnesses (route failures, timeouts, termination receipts), Harbor and legacy environments, sandbox providers, verifiers and judges (judge failure, unparseable verdict, verifier error), with a judge-call retry before a row is masked. - Collection that keeps a failed attempt's evidence, resends a refused /run within a connect grace period, retries a taken server port once, and marks placeholders that hold no generation. - Trial observations and grading records that bind a saved answer to its grading, used by the acceptance gate of the recovery layer. This commit squashes the branch's history (116 commits and its merges of main) into one, so the PR can be reviewed and its sign-offs checked as a unit. The tree is identical to the previous head, 04375cf71f90da06ade767af2653f1655e34c626. Four of the squashed commits were authored by the co-authors below and carried into this branch with the author's sign-off; the rest were authored by the author below. Co-authored-by: Wojciech Prazuch <wprazuch@nvidia.com> Co-authored-by: Jonathan Li <jonli@nvidia.com> Co-authored-by: Jeremy Bottleson <jbottleson@nvidia.com> Co-authored-by: Carlyn Aarish <caarish@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * feat(eval): rescore selected answers from resumed runs Read completed answers selected by the saved run, retain source provenance in separate outputs, and preserve the parent stacks judge-only settlement and retry policy. Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> * feat(recovery): decide recovery from saved evidence, with budgets and acceptance gates Recover a saved evaluation run from its persisted evidence instead of re-running it. `gym eval resubmit --recover` (and `--inspect` to only read) collects the run's identities, saved results, failure records, quality summary and the job chain's stored budgets, and decides one action: RESUBMIT the identities a rollout store would run again, REVERIFY saved answers that only need grading again (never calling the policy), NOOP when the run is complete, or STOP with a typed reason. What this layer contains, on top of the failure observations of #5072: - Recovery decisions per identity: scored, model-final, retryable, exhausted, judge-recoverable, masked, omitted, terminal and unknown, driven by the typed failure kinds and a per-run outcome policy. - Budgets that are spent, never reset: infra retries, walltime and the per-identity attempt cap, raised only by explicit, journaled options (`--extend-retries`, `--extend-attempts`, `--max-walltime`), and an operator continuation for runs whose chain accounting is unknown. - The job chain's guard: continue, retry on an infrastructure cause, or stop and write the reason (`.gym_chain_incomplete`), including resume refusal, spin-up timeouts, install failures and credential refusals. - Re-judging saved answers: `gym eval reverify` and a judge-only job for GPU-served runs, on one CPU node (`--judge-partition`, `--judge-qos`) or, for a run that serves its judge on a node pool of its own, on that pool, settling into the source run under a receipt (`--settle-receipt`). Every `gym` command a rendered job script runs is tested against the real CLI. - Acceptance: a gate summary with response and trial health over the expected trials, the 2% evaluation-side limit, metric coverage, and MLflow delivery reported as a fact. - Store readers on the run's pinned revision, repin of a run to a newer commit (previewed with `--inspect`), and resume-fingerprint rules for settings added later. - Trial observation from a saved run's own grading evidence: saved verifier output, a self-grading agent's reward or rubric score, an agent rubric's guardrail-gated counts, a native scorer's saved metrics, a truncated answer saved in an agent's own envelope, a graded row that ended on the model's own failed request, and judge transport retries and cut judge replies in the trial join. A masked result is measured again: a saved patch is re-verified on a verifier that replays it, and a sandboxed session's lost workspace is retried. SciCode records how each sub-step's grading process ended. A SWE-bench Verified or Multilingual evaluation that ends without a verdict is masked as a verifier error with its reason and the patch digest (an intact patch that does not apply stays a graded zero); such rows are counted, not re-measured, until SWE-bench verifiers replay a saved patch. A verifiers rollout records the error it ended on, as a recorded fact that recovery and the gate do not read yet. - Operator tooling for chains the scheduler has already purged: a chain whose job ids squeue no longer knows is judged by sacct (a requeued job by its latest incarnation), so the post-run export and saved-answer re-judges are not blocked as "scheduler unknown"; and `gym eval resubmit ... --run-dir <run dir>` restores a lost job-index entry from the run's own record, journaled. - Skills and docs for operating the above. This commit squashes the branch's history (342 commits and the merges of #5072 and main) into one, so the PR can be reviewed and its sign-offs checked as a unit. Every squashed commit was authored and signed off by the author below. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * feat(rollout): record every rollout run on the Store and report its saved progress Give every rollout run, not only `gym eval run`, a recorded shard on the rollout Store with the same failure records and manifest, and let an operator read a run's saved progress without opening its files. What this layer contains, on top of the recovery layer of #5343: - `run_examples` callers (training frameworks) write their rollouts to a run-folder shard through the collector's outcome builder, so their failures are typed and recorded as in evaluation, with per-rollout enrichment exposed to the caller and failure-evidence captures kept. - A recorded shard gets a verified run manifest with its attempts reserved, a writer identified by a token (not an inode), and refuses rows without indices before replacing anything; every dispatched copy is recorded under the shard's run. - Saved progress per shard: a status file written under the writer lock, throttled and ended current, kept current when a step after collection fails. - `gym eval status <run dir | job id>` reads the Store for every run and reports completed and remaining rollouts, failure kinds, each exporter's receipt and the live tracking run id. - Docs for recording rollout evidence from a training framework, and the gym-eval skill's use of `gym eval status`. This commit squashes the branch's own history (22 commits and its merges of #5343) into one, so the PR can be reviewed and its sign-offs checked as a unit. Three of the squashed commits were authored by Marc Romeyn (one co-authored with Pranav Thombre) and carried his sign-off; the rest were authored by the author below. Co-authored-by: Marc Romeyn <mromeijn@nvidia.com> Co-authored-by: Pranav Thombre <pthombre@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(eval): read selected attempts for health on the recovery stack Keep health checks on the Store snapshots used for scoring and adapt automatic, standalone, and shard reporting entry points to the recovery stack. Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> * test(eval): settle a result scored in a separate rescoring output into its run The resolution of the saved-result re-verification against the rescoring of selected answers came with one test that the merge of the two sides did not carry, because the test file merged without a conflict. It checks that a result scored in a separate rescoring output is still settled into its run. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 75ccf48720068f2cff6d4b99f84abb3c7bdafe55) * fix(recovery): a verifier sandbox whose teardown failed after grading keeps the trial observed A failed provider close or client release after the grading is recorded by the sandbox lifecycle producer as outcome sandbox_error (or unknown) with the cleanup error as its error_type. The reader counted that as a failed grading sandbox and left a trial with a bound grade unobserved, against its own rule that an unconfirmed teardown does not. An observation whose only failure is its teardown no longer counts; a failed execution, or an error other than the teardown's, still does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(skills): gym-eval carries the recovery facts an operator needs on a pinned run A recovery reads the run inside its pinned Gym, so a run pinned before a rule existed is read by the older code until it is repinned; --inspect --repin previews the repinned plan without dispatching. Say that a queued action may exit 0 while the gate still reads RESUMING and that ok and job_id come from the JSON line, what an inspect that could not read reports and when a larger NEMO_GYM_REMOTE_READER_MAX is the remedy, and that the judge-only job needs a Gym that defines --settle-receipt. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(skills): gym-eval review nit (reader step limit) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * test(recovery): import SandboxSpec from its defining module Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * feat(failures): add typed boundary details Select sandbox/model details using the shared event schema. Move upload evidence and its policy readers together; keep event IDs and causes through transport and persistence. Unmerged draft fields have no compatibility alias. Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> * fix(recovery): a masked row that keeps a saved answer is never sampled again A masked result whose saved transcript held no gradeable artifact (a patch never collected or not intact, or a verifier that graded the agent's own sandbox) was run again as an infrastructure retry, although it kept the model's saved answer. On resume that regenerated the answer: a saved answer is never sampled again, and a limits-exceeded cleanup failure is a model-side stop. The Store keeps such a row as the result again; a masked row without a saved answer is still retried by its evaluation-side kind, and an intact saved patch is still re-verified. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * Record where and on which sandbox a sandboxed episode failed When a sandbox provider ended a container under an episode, the saved failure said little about where: the coding agent's provider exception reached the route wrapper with only its type, the Harbor agent's message named only Harbor's exception type, and the harness agent's masked row kept only the exception text. A record could not answer in which phase, on which sandbox, or with which provider status the episode failed. Each of the three producers now starts its failure_reason with a fixed "[sandbox phase=... provider=... sandbox=... reason=... http=... exit=... signal=... command=...]" prefix of the facts it knows, built by one helper (exception_inspection.sandbox_failure_prefix): - the coding agent's own command (phase agent): provider, sandbox id, the provider's error code (e.g. SANDBOX::ENDED) and HTTP status, or the unclassified result's error type and exit code. A provider exception is reported as the same agent_route_failed failure the route wrapper produced before, raised from the provider error, so the reply and the provider's own message (foreign_error) are unchanged; - the Harbor agent: the phase Harbor's trial was in (the last timed phase that started and did not finish: setup, agent, verify), the environment type and Harbor's exception type; - the harness agent: agent or verify phase, sandbox id, the provider's error code and HTTP status. Record only: no failure kind, mask, retry or score changes, and no reader classifies a failure from the prefix. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 45edada54fbfd913068ad08a649cd0decd6a13ca) * docs(skills): gym-eval lists the outcomes the stack does not decide for the operator A sandbox killed for memory during the agent's own command, a sandbox that ended for its own limit, SkillsBench agent timeouts, AutomationBench rows graded on a partial world and SciCode grading subprocesses that were killed are retried or scored in a way the operator must correct or report until the matching policy lands. Name the field to read for each: failure_reason, sandbox_resource_limit, rollout_error and step_exits. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(skills): gym-eval review nit (exit 137 is Terminal-Bench 4.0) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * feat(recovery): a reader at the reader-step cap queues for a slot instead of reading nothing At the per-user cap of reader steps, a recovery reader with the login-host fallback forbidden ran nowhere and the inspect read UNKNOWN. It now queues: it counts the reader steps again every 2 s for up to NEMO_GYM_REMOTE_READER_QUEUE_WAIT_S (default 1800; 0 keeps the old behaviour), prints a progress line saying how many it is queued behind, and runs in its step once one ends. A cap still reached after the wait gives the old reason with how long it queued. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> (cherry picked from commit a0c76c85ac0c6fce6dc8adf8dac4258b3c8fc43d) * feat(recovery): plan an old-pin run on the installed Gym and repin it only when the plan needs it A recovery read a run's Store inside the run's pinned Gym, so a run pinned before a recovery rule existed was planned by the older code, and a judge-only re-judge on it was refused unless the operator passed --repin. The official command could not recover such a run on its own. A run pinned at another full commit than the launcher's is now planned with the installed Gym's rules. When the plan needs the installed revision to run (a re-judge, whose job runs the installed recovery source, or a resubmit the pinned revision plans differently), the recovery moves the run to the installed commit itself, journaled as an automatic repin with the reason; otherwise the pin stays. The outcome reports the decision (pin_plan) and --inspect shows the same one. --no-auto-repin keeps the pin and refuses that case, naming --repin; --repin stays the explicit way to move a run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> (cherry picked from commit 697bf3725734c367e90f46e978e3688cd8c1685c) * fix(recovery): the reader queue waits 10 minutes by default and polls every 10 s Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> (cherry picked from commit fb1e87ff0dc016ab9543bec85896a156e971a6e9) * refactor: trim sandbox failure details Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> * fix: export typed failure details publicly Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> * fix(recovery): a judge-only job captures its model calls in its own folder A judge-only job's gym eval reverify takes the run's settings, which turn observability on, but only the submitted evaluation's script derives the model-call capture directory. The job therefore started with observability on and no capture directory, and its policy server refused to start ("model_call_capture_dir is required when observability_enabled=true"). Both judge-only paths now apply the same rule as a submitted evaluation (_with_default_capture_dir): observability on by default, capturing into the job's own reverification folder unless the run names a directory. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(recovery): the CPU-only re-judge captures its model calls in its own folder Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(resume): an omitted failure report never blocks resume; name what resume leaves to recovery A failure report over the 4096-byte transport bound is saved as omitted. The resume gate treated that flag alone as an unclassified execution, so the rollout was never run again whatever its own failure record said: a transport size limit decided retries, contrary to the documented "an omitted report changes no outcome". The flag now carries no veto; the row's own record and kind decide, as for a row without a report. The answer gates (a possibly delivered request that kept a response, a saved generation, a recorded grade, a partial response) still keep a saved answer from ever being regenerated. Judge failures, kept answers and unclassified executions are left to recovery and not run again, but neither resume path said so. Store resume now prints how many such rollouts it leaves, and coverage reports them as stopped_not_retried (a part of failed_final). The legacy resume summary names them on their own line. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a31c4741a9b0f840a8a35b4419e292b3dce0ff7d) * docs(failure-reporting): align three doc statements with the code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> (cherry picked from commit d00e6520e93538a705dae8735a0c956518766ae9) * fix(failures): keep an unknown kind on the wire, judge every judge failure again, log dropped events A forwarded failure event whose kind the receiving Gym does not register (a server pinned to another version) made the whole report invalid, so it decoded as omitted and every event in it was lost. Header and stream decoding now keep such an event under "unregistered:<kind>"; any other invalid report is still omitted. The judge failures whose saved answer is judged again were spelled out as ("judge_failed", "judge_invalid") at seven sites; the registered judge_unparseable kind was in none of them, so its rows could be run again instead of re-judged. One shared JUDGE_FAILURE_KINDS now holds all three (judge_invalid stays: it is collection's own legacy class). A request's failure log that fills up counted its dropped events but never said so; it now logs once when it starts dropping. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2f04f0617fd7deddfd63b2d9ee191be990b68b98) * test(resume): the summary names rollouts left to recovery; an omitted report keeps a terminal failure final Covers the legacy resume summary line for a judge failure left to verification recovery, and a terminal failure whose report the size bound omitted: the omitted flag no longer vetoes a retry, and the terminal mark still keeps the rollout final. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 9265edd70bb67aad03a03d4dc31ce4194f8f30b4) * test(resume): an omitted report on a model 500 is retried like any other 500 The body-store test pinned the old gate, in which a report the size bound omitted kept the rollout from ever running again. With that veto gone the model 500 (no answer was produced) is retried on resume; a failed body behind a 2xx header (truncated, timeout) is still never run again. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e601af569a8bfcc976a54933cdbbbf0da4677de8) * fix(merge): import names from their defining modules; layer the moved session-conformance helpers The main merge-up rewrote some test imports to the nemo_gym.rollouts facade, and the explicit server_runtime layer list did not name the nemo_gym.server_runtime.testing package main added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(merge): TB4 reports rejected sandbox credentials at the sandbox boundary sandbox_credentials_rejected no longer takes a boundary: a provider that refuses the credentials is a sandbox failure. The terminal-bench resources server still passed boundary=ENVIRONMENT, so every session setup that raised failed with a TypeError instead of the typed rejection. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(merge): tests import names from their defining modules after the rollouts move Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * test(export): a run_dir above the rollouts leaves out the resume identity beside them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(exports): leave out a resume identity anywhere under the uploaded run directory The EvalRDL upload is now the benchmark directory above the rollouts, so a resume identity can sit in any directory under it (beside the rollouts, or in a re-verification directory). Only the one beside the rollouts was left out of the upload. An identity can hold credentials, so the upload mirror now leaves out every file with the identity suffix under the run directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(resume): name the operator continuation when chain accounting is unknown An accounting-unknown stop said only to reconcile the queue before recovery. The log line now also names the continuation the gym-eval skill and the resumable-runs guide give: gym eval resubmit --recover with --max-walltime and --walltime-reason, plus --extend-retries when the retries are spent. The incomplete-chain marker keeps the reason alone, since its readers match that text. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(recovery): correct the omission-summary example and a comment typo The omission summary prints the stage as "(stage_index 1, ...)", the form accepted_stage_omission returns. Comment and docstring only. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(orchestration): align resume, resubmit and driver-install comments with the code ResumeConfig.max_retries counts infrastructure retries only; a generic FAILED, an out-of-memory kill or a deadline stops the chain. A --recover resubmit keeps its budget, and plain resubmit still starts a fresh one. The private-uv paragraph of render_driver_entrypoint is re-worded and the cleanup option is documented. Comments and docstrings only. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(eval): drop a repeated source-snapshot guard and a shadowing import The CPU-only re-judge branch checked evidence.source_digests for None twice with nothing in between that could change it; the first check stays. _prepare_run_health_metrics re-imported subprocess, which the module already imports. No behaviour change. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(eval): fix comments and docstrings that no longer match the code The final-zero comment in collection moves back above check_completion, which it explains; the capture field comment gets its apostrophes; the verifiers agent lists its kind candidates in the order they are matched; and the exporter docstring describes the version-1 receipt check. Comments and docstrings only. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(eval): describe this layer's recovery as the code does Skills and the resume guides now match the code at this head: - the accounting-unknown stops: an empty sacct reply is asked up to six times, 10 s apart, before the chain stops, and the stop names the continuation; repeated allocation ends charge one attempt from the second end on; - the gate fields (allocation cause, output-budget refits, agent adaptations, retry budget, pin plan, observed grading recovery), the full list of grading artifacts and the rule for unobserved servers; - the coverage fields (masked_retryable, final_zeros, stopped_not_retried, attempt_extension, completion_fraction); - the chain end, the reader queue wait and login fallback, the reader time bound, the UNVERIFIED status of an unobserved trial, two chain files, and smaller corrections (config path, GDP-PDF and Stirrup defaults, smoke limit); - two exit-check tables that a pointer paragraph had split. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(recovery): a reader step srun refuses for a submit limit or busy nodes is submitted again A recovery reader whose Slurm step srun refused outright (a per-user submit or QOS limit, or no node free for an immediate start) was treated as a step that did not start: it was cancelled and the read gave up, so an inspect or recover read nothing on a busy CPU partition. Such a submission was never queued, so the reader now submits it again after 15 s, doubling up to 60 s, with a progress line each time, for as long as the reader queue wait allows. Any other srun failure, and a refusal that outlasts the wait, end as before; the reason then says how long it resubmitted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): a segment with nothing eligible for retry ends its own chain A driver that exits 76 (no task is eligible for another attempt) failed the job, so Slurm started its afternotok follow-up on a full allocation, often GPU nodes, only for that follow-up's prologue to write the stop and exit. The segment's epilogue now writes the no-eligible-retry stop itself, cancels the sentinel and the follow-up it queued, and journals the end in chain-end-journal.jsonl, the same way a segment that completes the run does. A deliberate stop already in the marker stays; the job keeps exit 76. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): a remote reader's install waits as long for uv's lock as a Gym run does A remote Store reader runs from a cleared environment on the cluster's login host, so a cold install of the run's pinned Gym there took uv's own default lock timeout instead of the caller's UV_LOCK_TIMEOUT or the 1800 s a Gym run uses, and an inspect on a busy host failed in the install. The reader command now exports UV_LOCK_TIMEOUT (the caller's value, else 1800) on the login host and before srun, so a reader step gets it as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(skills): gym-eval reads as one path from a cell state to the official act The gym-eval skill is rewritten around the operator path, with the per-benchmark notes in references/benchmark-notes.md, and states what this PR adds: a refused reader step is submitted again, a remote reader takes UV_LOCK_TIMEOUT, and a segment with nothing eligible for retry stops its own chain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): a typed driver failure that exits 76 keeps its own stop A driver that stops on a typed failure (rejected provider credentials, a refused collection resume) leaves its kind in the driver failure file and exits 76, and the follow-up prologue turns that kind into its stop and remedy. The chain-end epilogue wrote the generic no-eligible-retry stop for every exit 76 and cancelled the follow-up, so that typed stop was lost. The epilogue now ends the chain only when the driver left no failure kind; otherwise the job keeps its follow-up as before. The reader's UV_LOCK_TIMEOUT check accepts only ASCII digits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): exit 76 keeps its follow-up only for a kind the next guard stops on The previous change kept the follow-up for every exit 76 that left a driver failure kind. The follow-up's prologue writes a stop only for four kinds (spin-up timeout, install failed, resume refused, credentials rejected); for any other kind, such as a server that failed to start, it would start the run again. The segment now ends its own chain on exit 76 unless the kind it left is one of those four. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * feat(recovery): an explicit, journaled rerun of a masked row with no gradeable artifact A masked result that kept a saved answer is retained and never sampled again. When it also has no recorded grade and no re-verification cause (for example a patch artifact that was never collected), nothing can ever measure it: resume keeps it and re-verification cannot grade it. `gym eval resubmit --recover --rerun-masked-without-artifact --rerun-reason TEXT` is the explicit opt-in. The default is unchanged: without the flag these rows stay MASKED, and `--inspect` now names them in their own bucket (`masked_without_artifact`, with a per-row reason) instead of lumping them in with the other masked rows. With the flag, the identities the plan finds qualifying at that moment are appended with the reason, the time and each attempt count as one line to `rerun-masked-journal.jsonl` beside the rollouts file, just before the job is queued (and taken back if it is not). The Store reads that journal: an approved row counts one failed attempt, is retried within its attempt cap and is superseded by its next attempt. An approval names the attempt it covered, so a later attempt is never reopened by it, and the row must still qualify. The reason is required and must be non-empty; the flag needs `--recover`; `--inspect` never writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * feat(recovery): an explicit, journaled acknowledgement of failed judge-only jobs When the last two judge-only jobs of a run ended without settling a saved answer, recovery queues no third and stops. Nothing could clear that state, because it is derived from the tail of the judge-job journal. `gym eval resubmit --recover --acknowledge-failed-judge-jobs --acknowledge-reason TEXT` allows exactly one more judge-only job. The reason is required and non-empty, the flag needs `--recover`, and `--inspect` never writes. The acknowledgement is one line (time, user, reason, the ids of the failed jobs it covers) in a sibling journal, `judge-job-acknowledgement-journal.jsonl`, because the judge-job journal's readers treat every line as a job. It is written only once the scheduler has accepted the new job. It covers only the pair of jobs it names, so if the extra job fails too, the next pair needs an acknowledgement of its own. The stop text and `--inspect` name the flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): name the rerun-masked journal after its rollouts file The approvals were journaled under one fixed name per directory, so two rollouts files in the same directory would have shared them. The journal is now named after the rollouts file it sits beside, the same way the attempt extension is (rollouts.jsonl -> rollouts_rerun_masked_journal.jsonl). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * refactor(recovery): one list of the failure kinds the next segment's guard stops on The exit-76 epilogue named the four kinds the follow-up's prologue turns into its own stop as a second, separate list. Both now come from GUARDED_DRIVER_FAILURE_KINDS, and a test checks that the prologue has a stop for every kind in it, so a kind added to one side cannot be missed on the other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * test(recovery): a blocking judge-only job without an id cannot be acknowledged Covers the stop text for a failed pair that includes a job without an id: it names the unknown job, says it cannot be acknowledged, and offers no acknowledgement flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): a judge-only job does not wait for the policy endpoint A judge-only re-verification job judges saved answers and never calls the policy model, whose served endpoint ended with the run's allocation. It still waited for every model endpoint's readiness, so it timed out after the readiness limit and exited 1 before judging anything. The rendered job command now sets model_endpoint_readiness_timeout_seconds to 0 (a run argument that set it is dropped), which the job's pinned Gym reads as "skip this check". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(submit): retry the gym_install ref check and say when it is the network that failed `gym eval submit` verifies driver.gym_install.ref with one git ls-remote against the repo. A short network hiccup on the submitting host made that call time out, and the submit failed with "Could not verify driver.gym_install.ref '<ref>' in <repo>: ... timed out" and created no job, which read like a bad ref. The check now retries a failure to reach the repo after 10, 30 and 60 s. A full commit hash that a remote-tracking ref of the running Gym's own clone of that repo already contains is verified locally, with no network call; a local commit that was never pushed is still checked over the network. When every attempt fails, the error says the network check failed, how many times, and that this is not evidence the ref is missing. A missing git binary is not retried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(submit): the local fallback of the ref check runs only after the retries and a pruning fetch The local check accepted a full commit hash that a remote-tracking ref of the running Gym's own clone contained, without asking the repo. Such a ref can be stale: after a branch is deleted or a stack is squash-merged, the clone keeps the old remote branch until its next pruning fetch while the repo no longer serves the commit, and the job's install would fail later. The network check now always decides when the repo answers, and a ref it does not have fails the submit. Only when every attempt failed to reach the repo is the clone consulted, and only after a successful `git fetch --prune` of that remote, so a stale ref never vouches for a commit; the submit then logs that the local fallback was used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * test(submit): record only the ref check's own retry waits The fixture replaced the global time.sleep and asserted the exact list of waits, so a sleep elsewhere in the same test process (a subprocess poll, a thread left by another test) joined the list and the assertion failed under parallel runs. The fixture now stubs only the time module the ref check uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(skills): gym-eval and gym-add-model-config, v23 gym-eval: install the gym CLI at the pinned commit (the runner's own form), say the page links need a full Gym checkout, name the reader-step srun variable with an example, describe the chain end without assuming the pin, the chain bookkeeping files and their permissions, the judge-only acknowledgement, and a caveat that a verifier-timeout row can read as measured in coverage while recovery marks it masked. gym-add-model-config: point at the gym-eval steps that exist, and check that the served chat template renders tools (and how to see in the rollouts that it did). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(recovery): the remote reader installs the MLflow client; name the score flags The remote reader installs the run's pinned Gym without extras, so its query of the exported run's status on the tracking server failed with ModuleNotFoundError: No module named 'mlflow', and MLflow delivery stayed unverified for a run that had exported. The reader now installs the same mlflow extra the login-host export uses. No gate rule changes. The gate's "score unavailable: no requested metric was read" now names the flags that read a score: --score-key, and --score-agent or --score-environment for a Gym aggregate, which is a list of entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * refactor(recovery): move the evidence collector script into collect_evidence.py COLLECT_SCRIPT, the self-contained script recovery runs on the run's host, and the six constants it embeds move from recovery.py into nemo_gym/orchestration/collect_evidence.py, as a mechanical move. The script is byte-identical: its sha256 and those of the constants are the same before and after. recovery.py imports them back, so every name stays importable from it, and the Slurm executor runs the same object. A new test pins that, and that the script compiles. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(resume): a refused follow-up submission is a typed stop, not unknown accounting When sbatch answers the follow-up with the scheduler's own refusal (for example a QOS limit on submitted jobs), nothing was queued and the accounting is known. The chain recorded it as unknown accounting, which needs an explicit walltime budget to continue. It now writes follow_up_submission_refused with sbatch's exit code and error lines to .gym_chain_incomplete, still queues nothing, and its log line names the continuation, gym eval resubmit --recover. Recovery reports the stop. A reply that is missing, or says the request or its reply was lost (a timeout or a connection failure), stays unknown accounting as before; its log line now includes sbatch's reply. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(resume): a segment that completes the run supersedes the chain's earlier stop A stop recorded earlier in the chain (a refused or unanswered follow-up submission, a walltime cap) described a run with rollouts missing. When a segment then completes the run, .gym_chain_incomplete stayed, so a later reader saw an incomplete run. The segment now removes it, logs the stop it supersedes, and journals the chain end as run_complete_stop_superseded. A run that still lacks its required export receipt records that instead, as before. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(metrics): keep a per-task row for a task with no saved rollout profile_from_data dropped a task none of whose rollouts was saved, so a partial run's per-task report silently listed fewer tasks than its inventory. Such a task now gets a row with counts only: num_rollouts 0, expected and missing rollouts, and a completion of 0%. The agent and repeat statistics still come from the saved rollouts, so scores are unchanged. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(resume): count only a definite scheduler refusal as a refused follow-up A reply such as a broken pipe or an unspecified error can come back for a job the scheduler already accepted. The follow-up is now taken as refused only when sbatch names a definite refusal (an invalid request, a policy or limit it violates, a resource or dependency that is not available) and nothing says the request was lost; any other error stays unknown accounting. Tests cover both kinds of reply, and that recovery resubmits a run whose follow-up was refused instead of stopping on unknown accounting. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): carry the gym-eval corrections onto the rewritten skill The rewritten gym-eval skill dropped corrections made to its previous text. They are carried over: the empty-sacct stop and its continuation, a refused follow-up submission and a completed run superseding an earlier stop, attempt charging from the second allocation end, the gate fields (pin_plan, the allocation cause, output-budget refits, agent adaptations, retry_budget, observed grading recovery), compute.cluster.forward_env, the smoke-run limit, and in the benchmark notes the GDP-PDF and Stirrup defaults, the full list of grading artifacts and the rule for unobserved servers. The skill no longer instructs gym eval status, which this version does not have: progress is read with the read-only recovery inspection or from the run's coverage report. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(recovery): a server that failed to start before any rollout restarts the run fresh, once A segment whose driver died at server start (for example a port already in use), before the run saved any rollout, left the kind server_start_failed. The follow-up treated the FAILED job as having no infrastructure cause, asked the saved evidence, found no rollout store manifest and stopped with "no eligible retries", which hid the cause and named a recovery that would refuse as well. Now the follow-up starts the run again fresh, as an infra retry within the chain's retry budget, and at most once: the restarting segment leaves a marker, and a second server start failure, or a spent budget, stops with the kind, the job, "fresh restart exhausted" and the fresh resubmit command. A run with saved rollouts keeps the saved-evidence path unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(skills): install the launcher with the mlflow extra The inspection's MLflow delivery check imports the mlflow client. A launcher installed without it reports the delivery as unverified with a missing-module error, although the gate still passes. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(recovery): restart fresh only after a job that ended FAILED A server_start_failed file beside a job that ended OUT_OF_MEMORY or DEADLINE keeps its existing stop; the one fresh restart applies to a job that failed before any rollout and left no manifest. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): drop pin-dependent and outdated wording from gym-eval The skill now describes this version only: no notes about older pins (the sentinel that ends a chain, attempt extensions, the old re-judge NOOP, a reverify option an older Gym lacks), no "until the fix lands" caveats, and the GDP-PDF and grading notes no longer compare against an unnamed branch or stack version. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): gym-eval comparison guidance and the agent-timeout opt-out Keep a policy-model timeout on the model's side when two checkpoints, or a checkpoint and its quantized export, are compared; keep compared runs on one Gym commit. An agent timeout that kept no answer is retried by default, and the skill now names the recovery_outcome_policy entry that keeps it final. The forward reference to a later status command is removed. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * gym-add-model-config: eval-vs-deployment nuances and a complete smoke step Pin a full commit, never a branch. Name --gpu-memory-utilization as safe to change and keep quantization and dtypes identical across a comparison. Explain what the client sends and leaves to the server (sampling, the output-budget refit, system prompts), the reasoning-parser default, the policy_model_type clash, and give the smoke step its submit command. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(resume): the walltime-cap stop names a continuation the CLI accepts The stop said to continue with --max-walltime alone, which gym eval resubmit refuses without --walltime-reason. It now names both. A test runs the hinted options through the CLI and checks they reach the recovery. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(eval): document recovery as the code does at this version The CLI reference lists every gym eval resubmit option. The resumable-runs guide names the typed chain stops, the chain-end journal values and the split between a refused and an unanswered follow-up submission, and states the walltime budget rule: max_walltime is the chain total, a cap equal to the job walltime is spent by the first timeout, and Gym never adds time on its own. The vLLM page documents refit_output_budget_to_window, and the max_retries description is corrected. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(recovery): an unread score of an incomplete run names the run, not the options The gate said to pass --score-key and --score-agent whenever no score was read, also when both were passed and the run simply had no aggregate yet. An incomplete run now says so, with its settled and expected rollouts; the options hint stays for a complete run. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): gym-eval progress, guard exit, dispatch credentials and rollout upload Progress while a chain runs is the line count of rollouts.jsonl; the coverage report's completion is final only when the chain stops or the run ends. A guard job that writes a stop ends FAILED by design. A run submitted from the login node keeps its credentials as runtime entries the dispatch does not check, so the dispatching shell must set them. A run that does not upload its rollouts keeps an unverified MLflow delivery by configuration. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(resume): name the continuation when less than a minute of walltime is left The guard stopped with only "Less than one minute remains in the walltime budget." It now gives the stored and capped seconds and the same continuation as the cap stop, --recover --max-walltime with --walltime-reason. When the infra-retry budget is spent and the walltime is too, the recovery stop names both and says to raise both in one call, and the gym-eval skill says so in its two budget rows. Tests run the hinted options through the CLI and cover the stop with and without a spent walltime. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): rename gym-eval to run-and-babysit-eval and gym-add-model-config to add-model-eval-config The two public skills get names that say what they do. The skill directories, their .claude and .codex links, the name fields, the cross-references between the skills (and from add-benchmark and add-harness), the benchmark-notes title, the agent prompts and the agent-skills docs page follow. The wording inside the skills is otherwise unchanged. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): add per-benchmark findings to benchmark-notes Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): a segment must outlast the longest task A rollout cut by a segment end restarts from its first step, so a segment walltime shorter than the longest task repeats the same work in every segment. Say so in run-and-babysit-eval next to the three limits, and in the Terminal-Bench notes, where single tasks have needed more than 90 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(eval): reverify: a collection --limit no longer caps the rows a re-judge pass may re-verify A collection limit counts tasks, while reverify reads limit as the largest number of rows to re-verify. The automatic re-judge after collection and the judge-only job kept the run's limit, so a run collected with a limit re-judged at most that many rows per pass and never reached the rest. Both now drop it with the other collection-only settings. Tests cover the automatic pass (a run collected with limit 4 re-judges all 10 due judge failures) and both judge-only job paths. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(skills): spin-up limit and judge-key environment for babysitting a run Before submitting: the server spin-up limit (default 600 s) covers dependency installation and model loading, so raise it for a large model, a fresh install or a loaded cluster. A spin-up timeout is an infra stop that leaves rows untouched and is never scored as zero; the chain waits for a recover with a raised limit. Judge-only re-judges take the judge key and the other forwarded credentials from the environment of the dispatching shell, so export the valid key there before dispatching. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(slurm): pin the policy node's Ray bootstrap to a tested version with the serve extra A container without the ray CLI installed `ray[default]` unpinned before `ray start` and the Serve gateway. The gateway imports `ray.serve`, whose dependencies the `default` extra does not declare, and an unpinned install takes the newest Ray. With a Ray release whose Serve import needs a package outside `default`, every start-up of such a policy service fails the same way at the gateway's import. Install `ray[default,serve]==2.59.0` instead. A container that ships its own Ray is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(rollouts): scope a recorded shard's rollout ids to its run run_examples records a shard on copies of the rows that name the shard's run, but the copies kept the row's own rollout id, and with it the capture key the model calls are staged under. Capture files are kept by default, so a later shard that dispatched the same task and rollout (a row an earlier shard returned, fed back, or indices numbered again each step) merged the earlier shard's calls into its own records. Each dispatched copy of a recorded shard now carries the row's own id (its _ng_rollout_id, or <task>-<rollout>) prefixed with the shard's run id. A row an earlier shard returned has that prefix replaced, not nested. The capture key is that id with the usual attempt suffix, so capture_key and the /ng-rollout/<capture key> path keep their form; only the key is now the shard's own. The Store, the manifest, the records and the returned rows use the same id, so a caller reads a capture's key from the returned row. enrich_results without recording keeps the caller's keys. Tests: two shards that dispatch the same task and rollout, with the row fed back or the indices numbered again, each record only their own calls (they fail without the change). The shard tests stage captures under the dispatched key, as the model server does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(rollouts): give a recorded shard the manifest its resume computes A recorded shard's manifest hashed the caller's rows and the recording options, while the manifest gym eval run computes on resume hashes the source file's bytes and the resolved configuration. Resuming a shard with resume_from_cache was therefore refused for its source dataset and its configuration. The earlier test passed the saved manifest in as the current one, so it never compared the two. The writer now builds the shard's manifest with RunManifest.create, the identity function collection uses. Its inputs are the shard's materialized inputs as the source file, the rows as dispatched, a collection config that names the shard's input and output paths and leaves every other setting at its default, and the call's server configuration. The identity file is saved beside the manifest, as for any run, and replaced with the shard. So gym eval run +input_jsonl_fpath=<dir>/rollouts-<shard>_materialized_inputs.jsonl +output_jsonl_fpath=<dir>/rollouts-<shard>.jsonl +resume_from_cache=true verifies and runs only what the shard still lacks. An operational difference resumes with the usual drift note, and a semantic one is refused and named. If the running configuration gives no identity (for example it does not name the rows' agent), the shard is still recorded, with a warning, and a resume refuses it and says why. Tests: a shard recorded with one failure resumes through run_from_config, dispatches only the failed row and ends with both rows scored; a shard without a configuration identity is recorded and warns. Both fail without the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(observability): shard capture keys and resuming a recorded shard The training-framework section now says that a recorded shard dispatches each row under its rollout id prefixed with the shard's run id, so a capture's key is read from the returned row, and that a shard resumes with gym eval run on its materialized inputs and its rollouts file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * test(rollouts): a run that is not a shard keeps its rollout ids and capture keys Only a recorded shard prefixes its rollout ids with its run id. A plain gym eval run and run_examples with enrich_results alone dispatch every row under the id Gym derived before: <task>-<rollout>, or the row's own _ng_rollout_id. The saved results carry the same ids, so recovery and re-verification of such runs read the identities they always did. The test passes before and after the shard change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(rollouts): end a shard's status current when its last row is an unkept omitted row With record_omitted_rollouts left off, the shard writer returned for an omitted row before its status write. When that row was the last one expected and status writes were throttled, the final counts never reached the status file: with two scored rows and an omitted third, gym eval status reported one completed result instead of two. Closing the iterator corrected it, but iterating to the end does not close it. The early return now flushes the status when it handles the last expected row. As with any flush, a writer that another writer superseded leaves the newer status alone. The saved results do not change. Test: the throttled status test also writes two scored rows and an unkept omitted row, and expects two completed and one remaining (it reports one completed without the change). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(eval): bind resumed rescoring to selected saved answers Record source run, rollout, attempt and answer fingerprints on separate rescoring results and failures. Validate every cached binding before cache filtering or verifier dispatch, refusing replaced or unbound answers with a fresh-output remedy. Grade-only settlement of an unchanged answer stays compatible, and source bookkeeping stays out of verifier requests. Read selected records lazily so --limit does not decode every trajectory. Document the managed-cache compatibility requirement and retain legacy JSONL, judge settlement and ATIF behavior. Validation: 864 focused recovery/reverification tests passed. The 18 stale cache regressions and limited-loading regression fail against 9f58da3b8. All-files pre-commit and Fern checks passed (one Fern warning); private guardwords catalog unavailable. Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> * fix(rollouts): refuse recording rows that would share a rollout in the shard A recorded shard prefixes each row's rollout id with its run id, after removing the prefix an earlier shard put on a row it returned. A row fed back as `<earlier run>.0` and a new row whose own id is `0` therefore both became `<this run>.0` in the same call. They would then share capture files and one manifest identity. A training framework hits this when a step resubmits a returned row next to fresh rows whose numbering restarted. run_examples now refuses such a call before dispatch, and before the existing shard is replaced. The error names both rows and both of their own ids, so the caller can give each row a distinct _ng_rollout_id or distinct task and rollout indices. Test: `<earlier run>.0` beside `0` is refused with both ids named, nothing is dispatched and the existing shard is kept. `<earlier run>.0` beside `1` still dispatches, as `<run>.0` and `<run>.1`. It fails without the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(rollouts): refuse a shard rollout id too long for its capture files The capture key, a rollout id with its attempt suffix, also names the model server's capture files. The longest of them is `<key>.capture.incomplete`, and a file name holds at most 255 bytes. A recorded shard adds a 33-character run-id prefix to each row's id, so a long explicit id could pass that limit and fail only when the first model call was captured, mid-rollout. A shared check, check_rollout_id_length in nemo_gym.core.episode_context, allows a rollout id of up to 230 bytes. That leaves room for attempt suffixes up to -a9999 and the longest capture file suffix, and also keeps a key under a 256-character id limit. run_examples applies it to each recorded row's id after the prefix is added, before dispatch and before the existing shard is replaced. The error names the id, its length and the limit. Test: an id one byte over the limit, after the prefix, is refused with nothing dispatched and no shard written. The longest id that fits dispatches, and its capture file name at attempt 9999 is exactly 255 bytes. It fails without the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(reference): placeholders the MDX generator can parse The library reference page generated from the judge_profile module failed to parse: its module docstring wrote the pool node variable as ``${GYM_POOL_<POOL>_NODES}``, and the generated MDX read `<POOL>` as a tag. The docs preview of any change that rebuilds the full reference failed on it. The docstring now names the variable as ``GYM_POOL_*_NODES``. No other docstring combines a ${...} reference with an angle-bracket placeholder, and the other pages built from docstrings parsed in the same build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(reference): name the judge profile's default QOS source as a constant The library reference page for the judge_profile module failed to parse. The page generator writes each field's default into an MDX attribute as its Python repr, and JudgeExecutionProfile.qos_source defaulted to the string "the partition's default". Because of the apostrophe, the repr is double-quoted, so the generated line was <ParamField path="qos_source" type="str = "the partition's default""> and MDX stopped at the `'` ("Unexpected character `'` in attribute name"). The default is now the module constant PARTITION_DEFAULT_QOS, with the same text, so the page shows `str = PARTITION_DEFAULT_QOS`, as it does for `kind: str = CPU_PROFILE`. The two other places that set the same source use the constant too. The text in the operator's log does not change. No other plain string default in nemo_gym contains an apostrophe. Verified by generating the page locally with the docs CLI and compiling it with MDX 3: the old page fails with the error above, the new one compiles. The judge profile tests pass unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(eval): check rescoring limit before advancing selected rows A selected-row iterator fingerprints its answer while advancing. Check the post-predicate limit before requesting another row so an unrequested completion without a response cannot reject a bounded rescoring pass. Keep unlimited iteration and missing-response errors for requested rows. Test the boundary through direct and aliased managed sources, and require exactly one decoded answer with limit=1. Validation: 868 recovery/reverification tests and all-files pre-commit pass. The three relevant boundary regressions fail against 382455798; unlimited controls pass. The shared generated judge_profile docs issue is unchanged. Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> * fix(eval): normalize health snapshot paths and validate record identity Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> * docs(eval): explain rescoring stop-on-error behavior Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> * test(rollouts): place the journal health test in the mirrored test tree Main now mirrors nemo_gym's packages in tests/unit_tests (#5589). The journal health test covers nemo_gym/rollouts/rollout_health.py, so it moves to tests/unit_tests/rollouts/, and its import of the re-judge test helpers follows that module to tests/unit_tests/eval/. Without the import change the settled-judge-answer test fails with ModuleNotFoundError on the merged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * test(eval): place the journal re-verification test in the mirrored test tree Main now mirrors nemo_gym's packages in tests/unit_tests (#5589). The journal re-verification test covers nemo_gym/eval/rollout_reverification.py, so it moves to tests/unit_tests/eval/, and its import of the re-judge test helpers follows that module to the same folder. Without the import change the settled-judge test fails with ModuleNotFoundError on the merged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> --------- Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> Signed-off-by: Marc Romeyn <mromeijn@nvidia.com> Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> Co-authored-by: Adam Rajfer <arajfer@nvidia.com> Co-authored-by: Wojciech Prazuch <wprazuch@nvidia.com> Co-authored-by: Jonathan Li <jonli@nvidia.com> Co-authored-by: Jeremy Bottleson <jbottleson@nvidia.com> Co-authored-by: Carlyn Aarish <caarish@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: …
…test tree #5589 mirrors nemo_gym packages under tests/unit_tests; the sibling README config tests live in tests/unit_tests/benchmarks and import nemo_gym.config.global_config (the nemo_gym.global_config path only resolves through the moved-module shim, with a warning). Follow them. Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
What changed and why
The package reorganization in #5166 moved a test only when its module moved. That left 254 test and helper files flat in
tests/unit_tests/, beside the package folders the stack created. This PR moves them into folders that mirrornemo_gym/, so each test sits with its package's other tests. The proposed per-package CODEOWNERS can then cover them throughtests/unit_tests/<package>/.The moves are renames. The only content changes are the path fixes the move needs:
test_harbor_tasksand the telemetry and verification tests importingtest_sandbox.Path(__file__).parents[N], which get one level deeper (three underdata/tasks/harbor/).sandbox/fixtures/because three sandbox compose tests and one Harbor test share them. The Harbor test reads them from there.orchestration/fixtures/, exceptcpu-mcqa-trial-operations.jsonandcpu-mcqa-programmatic-operations.json. The trajectory tests read those two as well, so they stay in the sharedfixtures/folder and the orchestration test reads them throughparents[1], as the trajectory test already does..github/workflows/config/.secrets.baseline, and the imports of moved test helpers inresponses_api_agents/miniswe_sandboxed_agent/tests/, and the Harbor environment server test that copies the Harbor slopcodebench fixture from its new folder.scripts/ci/core_test_durations.jsonkeeps each file's weight under its new path, so the core shards stay balanced.scripts/harness_conformance/ci.pymatches harness conformance tests by path prefix to decide which harnesses to run. The prefix now namestests/unit_tests/tooling/.KEEP_OLD_PATHSlist inscripts/migrate_imports.pyand the exporter guide skip intest_public_api_facades.py.Ruff re-sorted the imports whose module paths changed.
tests/unit_tests/rollouts/harness_capabilities/gains an__init__.py. Its tests already importtests.unit_tests.rollouts.harness_capabilities.synthetic, but without the file pytest named its test modules by basename (test_health,test_results), which can collide with any other basename-named module. Every folder of tests undertests/unit_tests/is now a package.artifacts/exporter_guide/stays without one: it holds docs example sources that its test loads by file path, not test modules.Moving
progress_utilsintoutilsThe second commit moves
nemo_gym/progress_utils.pytonemo_gym/utils/tqdm.py. The module imports only the standard library andtqdm, so it belongs inutils/rather than at the package root, where it needed its own entry besideutilsin the import-linter layers. It holds one class, the asynctqdmsubclass, so the module takes that name. Its test moves totests/unit_tests/utils/test_tqdm.py, and its two importers (eval/rollout_reverification.pyandrollouts/rollout_collection.py) import from the new path.The module was added in #4837 and is in no release, so the old path gets no deprecation alias.
Layout guidance in AGENTS.md
The third commit adds a "Where code and tests go" section to
AGENTS.md, so the layout this PR applies holds for new code: library code goes into its package (a helper withoutnemo_gymimports intoutils/), unit tests mirrornemo_gym/withtooling/andbenchmarks/for code outside it, every test folder has an__init__.py, fixtures sit beside the tests that read them, and a moved test updates its root lookups, imports, fixture paths and shard durations key.Where files went
sandbox/test_sandbox_*files, and thecompose_testing.pyhelperdata/tasks/harbor/test_harbor_*file, the static Harbor sandbox policy check, theharbor_build_helpers.pyandharbor_registry_fakes.pyhelpers, and the Harbor task fixturesorchestration/test_cli_server_*,test_cli_endpoint_readinessandtest_cli_eval_submitfiles, whose subject isserver_supervisororsubmit. Also the recovery gate tests from #5343 (trial coverage, grading, Store readers, resubmit and continuation), whose subject isorchestration/recovery.py,submit.pyor the executors, with their saved-run fixturescli/test_cli*files whose subject isnemo_gym.cli, thegym env validate --server-typestest, and the locked environment tests (nemo_gym/cli/_locked_env.py)eval/nemo_gym/eval/rollout_reverification.py)agents/nemo_gym/agents/)models/rollouts/,artifacts/,config/,core/,environment/,metrics/,server_runtime/,trajectory/,utils/artifacts/exporter_guide/.Two groups have no
nemo_gympackage:benchmarks/holds tests of benchmark prepare scripts, launchers and configs, and of resources server code that the core suite covers on purpose: README launch replays andtask_dataschemas. They can't move into those servers' owntests/folders. Those folders run in per-server environments outside the core coverage run, which is why these files are in the core suite.tooling/holds tests ofscripts/, of thedocker/installers, of wheel packaging (setup.py), and of CI and review configuration. The folder can't be namedscripts/, becausescriptsis in pytest'snorecursedirs, and those tests would silently stop being collected.What stays flat, and why
test_moved_paths.py,test_public_api.py,test_public_api_facades.py,test_import_linter_coverage.py,test_packaging_paths.py: they test thenemo_gym/root modules (_moved_paths.py,_public_api.py) or the package as a whole, so the flat folder is their mirror.fixtures/import_surface.jsonstays withtest_moved_paths.py.test_session_contract_repo.py: checks every agent and resources server in the repository.test_fern_docs_links.py: a docs check overfern/, run by path from the docs workflow andCONTRIBUTING.md.test_placeholder.py: a trivial placeholder test with no subject.conftest.pyand__init__.py: shared by the whole tree. The rootconftest.pyapplies its Harbor registry fixture by module name, so the move does not change which tests get it.Code owners
.github/CODEOWNERSon main has only/tests/for tests, so this PR does not change any code owners. With the proposed per-package entries, giving eachtests/unit_tests/<package>/the same group asnemo_gym/<package>/covers every moved file:tests/unit_tests/sandbox/followsnemo_gym/sandbox/.tests/unit_tests/cli/followsnemo_gym/cli/.tests/unit_tests/data/tasks/harbor/falls undertests/unit_tests/data/.tests/unit_tests/benchmarks/would take the same group as/benchmarks/and/resources_servers/.tests/unit_tests/tooling/and the flat files fall back to the/tests/default.Issue
Refs #5166, the package reorganization this completes for tests.
Validation
Run on this branch rebased onto main, against main itself, in one environment synced with the extras of the core and sandbox CI jobs:
pytest --collect-onlyovertests/: the collected test IDs match main's apart from the module path prefix. The only other differences are two parametrize IDs whose value is a moved path. The only test files left flat are the ones listed below.-m "not sandbox" --core-shard-index=N --core-num-shards=4) plus the sandbox job (-m sandbox) collect every test exactly once.pytest -n 16, sandbox-marked tests included, on main and on this branch: the same tests pass, skip and fail. The failures are specific to the local machine and also fail on main: Harbor run summaries and benchmark listing (hostname resolution) and Docker PTY. One more Docker PTY test failed once under-n 16on this branch and passes when run alone.tests/that refers totests/unit_tests(the Harbor environment server, the legacy single-turn environment, the Codex and simple agents, and the harness conformance scripts) passes. The miniswe agent's tests need its own dependencies, so for those only the moved helper's import was checked.pre-commit run --fileson every changed file, and the import-linter hook over the repository: the layer contract is kept.Rollout evidence
N/A. This PR moves tests and updates path references, and changes no environment or agent behavior.