Skip to content

feat(checkpoint): checkpoint evaluation runs from rollout collection - #3893

Open
ananthsub wants to merge 1 commit into
mainfrom
ananthsub/partial-ckpt-rollout-collection
Open

ananthsub wants to merge 1 commit into
mainfrom
ananthsub/partial-ckpt-rollout-collection

Conversation

@ananthsub

@ananthsub ananthsub commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

What changed and why

The rest of the stack lets a controller checkpoint Gym with rollouts in flight. The controller has to be the process that sends episodes: only it knows which episodes it will continue, and only it can send their replacements after a restore. A training framework such as NeMo RL plays that role for training. For an evaluation run it is rollout collection, which so far could only restart unfinished rows from their inputs (resume_from_cache). This PR makes rollout collection the controller of its own rows, so a long evaluation task survives a preemption without starting over.

With checkpoint_dir set (and the global checkpoint: block enabled):

  • Triggers.

    • SIGUSR1 checkpoints the rows in flight and continues.
    • SIGTERM checkpoints them and stops the run. Slurm can send it ahead of a preemption, for example with --signal=B:TERM@120.
    • checkpoint_every_s checkpoints on a timer, which covers crashes that give no warning.
    • The collector writes its process ID to checkpoint_dir/collector.pid, for sending the signals.
    • A signal or timer that fires during startup waits until the run has reset or restored its state, and a restore holds the checkpoint lock, so a checkpoint never overwrites the progress being recovered.
    • Rollout collection is the only owner of SIGTERM for the run. With dispatch_start_report_within_s also set, its handler prints the timing table first, then checkpoints and stops. The previous SIGUSR1 and SIGTERM handlers are restored when the run ends.
  • A checkpoint:

    1. stops starting new rows;
    2. prepares every participant;
    3. commits the rows whose /run has not replied, except restarts, rows some server cannot capture, which keep running and are listed in the manifest;
    4. writes the run's manifest, then publishes the checkpoint by pointing checkpoint_dir/LATEST at it;
    5. resumes, unless it is stopping.

    The checkpoint just published is always kept, plus the newest of the others, so a clock that steps back never deletes the checkpoint LATEST names.
    A checkpoint that does not prepare in time publishes nothing, and the run continues from its previous checkpoint.
    A checkpoint the timer started is finished before the run closes, and a checkpoint that is cancelled still resumes every participant.

  • Dispatch log. Every run keeps checkpoint_dir/dispatched.jsonl: its run ID, then one line for each attempt it sends, written just before the /run that sends it, plus the attempts a restore installs and the retires it carries over from a crashed run. The log is flushed to the kernel but never synced. A killed process leaves its lines behind, and a crashed host takes its Gym servers down with it, so nothing on it is left to retire.

  • Rows refused while a checkpoint is open. A row can pass the dispatch gate and still reach the environment server after prepare closed admission, for example while it waits for a connection. The server refuses it with 409 checkpoint_admission_closed before anything of the episode runs. Rollout collection sends the same attempt again after the resume; the row stays in flight and is not counted as a failure. Any other 409 is handled as before.

  • Restart. A run restarted with resume_from_cache=true reads the log and the latest checkpoint before dispatching anything.

    • If the run that wrote the log also published the checkpoint, the checkpoint's rows are restored and continue as their next attempt.
    • Every other unfinished row in the log, whether a restart or a row sent after the checkpoint, has its logged attempt retired everywhere and starts from its input one attempt later. A harness that outlived the crash, for example in a remote sandbox, cannot write into the rerun. Before this, such a row was sent again as the same attempt.
    • If a later run crashed before its own first checkpoint, it may have sent the checkpoint's rows again, so nothing is restored and those rows start over too. That is slower, but it never installs an attempt next to one that is still running.
    • Rows that were never sent keep their attempt and need no retire. Rows that finished are already in the output file, which rollout collection writes as each row completes.
    • If the restore fails, coordination has already retired that attempt everywhere, so those rows start from their input one attempt later.
    • The attempt index is excluded from metrics, so sending a row as a later attempt changes its capture keys, not its results.
    • A fresh run (resume_from_cache=false) forgets the checkpoints of an earlier run in the same directory, as it already does with the output file, and starts a new log.
  • Code.

    • The new nemo_gym/_checkpoint/collection.py holds the controller.
    • In rollout_collection.py: three config fields, a gate and an in-flight count around each row's /run, and the restore step after the cache load.
    • gym eval and the collection CLI print a message when a preemption signal stopped the run, and exit with a SIGTERM kill's status, 143, as main does for a run a SIGTERM interrupts without a checkpoint. A scheduler then never takes a stopped run for a finished one.
  • Cleanup.

    • A row whose /run got no reply is retired everywhere, in batches under the checkpoint lock, so a retire never lands inside a checkpoint. Otherwise state it left behind, such as a restored record nothing claimed, would be exported again by every later checkpoint. A reply, even a failure, means the environment server already ended the episode and freed what it held.
    • A retire that fails is kept and repeated before the next checkpoint and again when the run closes, so the servers that missed it still free the attempt.
    • Rollout collection forgets each rollout it retired once a later attempt of that row replies, and forgets the rest when the run closes, since nothing dispatches them again. Rows it restored are forgotten the same way, because a restore marks the attempts it replaces as retired on every participant. A forget that a server refuses with retire_incomplete, because it still holds a session of the rollout (for example one a protocol route started after the retire), is answered by retiring the rollout again at the attempt it retired and then forgetting it again, before the next checkpoint and when the run closes, so that session never holds up a later checkpoint. Every participant's refusal of the retired attempts is bounded by the run.
    • A run that stopped after a checkpoint sends no retire or forget, including for rows its shutdown cancels: its servers stay committed and would refuse both. The next run's restore retires every attempt the stopped run sent.
    • Rows restored from a checkpoint but not dispatched yet are continued by the next checkpoint, at the attempt they will run as. Without this, a crash between a restore and their dispatch restarted them from their input.
    • Publishing removes partial checkpoint directories left by a failed commit and temporary files left by a crash.

Without checkpoint_dir, rollout collection behaves exactly as before. The dispatch hook is only passed when it is set, so subclasses that override the internal dispatch method keep working.

How it works

Where this PR sits in the overall flow

The highlighted part is what this PR adds.

flowchart LR
  C["Rollout collection, this PR<br/>SIGTERM, SIGUSR1, or a timer"]
  CO["Coordination<br/>prepare, commit, restore, resume,<br/>retire, forget"]
  K["Control plane on every server<br/>phases, lease, storage,<br/>retire: refuse, stop, free;<br/>forget"]
  subgraph G["One participant per Gym server"]
    E["Environment server<br/>episode steps"]
    M["Policy model<br/>held responses, generation cuts"]
    A["Agent<br/>sessions parked at boundaries"]
    R["Resources server<br/>session state"]
  end
  W["Inference worker<br/>stages cut prefixes"]
  D[("Checkpoint directory<br/>records, then manifest")]
  L[("Capture ledger<br/>retire and delete from #3938, #3939")]
  C --> CO --> K
  K --> E & M & A & R
  M --> W
  M --> L
  G --> D
  classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937
  class C this
Loading

One checkpoint, a crash, and the restore, end to end:

sequenceDiagram
  participant C as Controller
  participant G as Gym participants
  participant D as Checkpoint directory
  C->>G: prepare, in order environment, model, agent, resources
  Note over G: admission closes, in-flight work parks at a boundary,<br/>undelivered model responses are held
  G-->>C: prepared, or blockers at the deadline
  C->>G: commit with the episodes the controller continues
  G->>D: each participant writes its records, then its manifest
  Note over G: restored state the commit's scope leaves out is deleted
  C->>C: publish the checkpoint with the controller's own state
  C->>G: resume, in order resources, agent, model, environment
  Note over C,G: crash - every Gym process dies
  C->>G: restore the checkpoint in fresh processes, all or nothing
  D-->>G: records installed under attempt N+1, attempt N's capture ledger retired
  C->>G: resume
  C->>G: /run as attempt N+1 continues each episode from its boundary
  Note over C,G: retiring an episode, only while no checkpoint is open
  C->>G: retire - environment, then agent, then model and resources
  Note over G: each server marks the attempt, stops its work, waits, frees its state, then replies
  C->>G: forget, once the rollout is done for good - same order
  Note over G: the mark refusing the retired attempt goes away
Loading

This PR

Checkpointing an evaluation run, and continuing it after the job restarts.

sequenceDiagram
  participant S as Slurm or an operator
  participant RC as Rollout collection
  participant G as Gym servers
  participant D as checkpoint_dir
  S->>RC: SIGTERM, SIGUSR1, or the timer fires
  RC->>RC: stop starting rows
  RC->>G: prepare
  RC->>G: commit the rows whose /run has not replied
  RC->>D: collection.json, then LATEST
  alt SIGTERM
    RC-->>S: exit, the job is requeued
  else SIGUSR1 or the timer
    RC->>G: resume, rows start again
  end
  Note over RC,G: restarted job with resume_from_cache=true
  RC->>D: read dispatched.jsonl and LATEST
  RC->>G: retire the logged attempt of every unfinished row it does not restore
  RC->>G: restore, then resume, if the log's run published LATEST
  RC->>G: continued rows as attempt N+1, other unfinished rows from their input
  Note over RC: restored rows not dispatched yet stay in the next checkpoint's scope
  Note over RC: a row whose /run got no reply is retired, between checkpoints
Loading

Where this sits in the stack

This PR is independent of the rest of the checkpoint chain: #3909 (several workers) has merged, and #4197 (fault tests) does not touch rollout collection. It builds on #4997, which has merged and gives each checkpoint concept one name before the feature is released, including the checkpoint_ prefix on every checkpoint error code. Everything else it builds on (#3882 to #3889) has merged.
Once #3903 merges, this PR also gets its two spans, gym.checkpoint.collection.checkpoint and gym.checkpoint.collection.restore; if this PR merges first, they come in a small follow-up instead.
#3909 added the end-to-end test helper reward_of() that this PR's collection tests use, so this PR uses that one instead of adding its own.

Relationship to the old stack (#2939 to #2946)

New. The old stack left the controller to the training framework and had no evaluation path.

Issue

No tracking issue exists. This came out of design discussion of the stack: how a single long evaluation task would be checkpointed without a training controller.

Validation

Run on this branch, on the current main, which includes #4997 and #3909:

  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_checkpoint_collection.py tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_cli_eval.py tests/unit_tests/test_cli_eval_submit.py: passed.
    test_cli_eval.py adds: a run that checkpointed and stopped exits with status 143; without the change it exits as a success.
  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_checkpoint_*.py: passed. test_checkpoint_collection.py covers the controller with coordination faked:
    • a checkpoint commits exactly the rows in flight and publishes them;
    • no row starts while a checkpoint is open;
    • a checkpoint that does not prepare publishes nothing and resumes;
    • a stop does not resume, and a stopped run sends no retire or forget, even for a row that fails during the stopping checkpoint or is cancelled at shutdown;
    • only two checkpoints are kept;
    • restore continues only unfinished rows, as their next attempt, and falls back to a later attempt when it fails;
    • after a crash, the rows the previous run sent after its checkpoint, and its restarts, are retired and sent one attempt later, while rows it never sent keep their attempt;
    • a checkpoint is not restored when a later run crashed before its own first checkpoint;
    • a retire is carried through a run that crashes before it sends the row again, and a half-written last line of the log is ignored;
    • a failed retire is repeated before the next checkpoint and at close;
    • a fresh run forgets old checkpoints;
    • a restored row is forgotten once its replacement replies, or when the run closes;
    • a forget refused with checkpoint_retire_incomplete is retired and forgotten again before the next checkpoint prepares, and when the run closes;
    • rows restored but not dispatched yet are continued by the next checkpoint;
    • rows whose /run got no reply are retired, and rows that replied are not;
    • publishing removes partial checkpoints and temporary files;
    • the timer checkpoints only while rows are in flight;
    • the timer needs checkpoint_dir;
    • a row refused with checkpoint_admission_closed because it reached the environment server after prepare runs again after the resume and finishes with its reward, with the failure sidecar off and on, using the real environment participant and control routes;
    • SIGTERM with the timing report on prints the table, checkpoints, publishes and stops, and the previous handler is restored;
    • closing during a timed checkpoint, and cancelling a checkpoint while it is prepared, leave every participant resumed;
    • a checkpoint signal during a resumed startup waits for the restore, so the earlier checkpoint's continued row is restored;
    • retention keeps the checkpoint LATEST names when the clock steps back.
  • The process-level rollout collection scenarios (SIGTERM on the flat-row wrapper and on the typed single_agent_turn request path, and SIGUSR1) pass in the full e2e suite at the top of the stack.
  • The full core unit suite (tests/unit_tests, eight processes, without test_opensandbox_compose.py, which needs the optional opensandbox package): passed except a sandbox retry test, which fails the same way on main in this development environment.
  • pre-commit run --files <files changed by this PR>: all hooks passed, and no hook modified a file.
  • Every commit carries a Signed-off-by line.

Rollout evidence

The process-level tests above run real Gym server processes against the fake inference backend. This PR also adds an opt-in real-model test (NEMO_GYM_CHECKPOINT_VLLM_URL) that kills the collector without warning after a checkpoint, while the Gym servers keep running, and checks that every row finishes exactly once after the resume.

  • Real model, on one workstation: Qwen/Qwen3-4B-Instruct-2507 served by vLLM 0.24.0 on one RTX 6000 Ada (--enable-auto-tool-choice --tool-call-parser hermes), driving real Gym server processes that the opt-in tests in test(checkpoint): add a process-level end-to-end suite driven by coordination #3889, feat(checkpoint): checkpoint evaluation runs from rollout collection #3893 and feat(checkpoint): resources, agent, and environment servers with several workers #3909 checkpoint mid-run, kill and restore:
    • Saved rollouts. Weather rollouts on the typed single_agent_turn request path, sent 20 ms apart until the first one finished. The checkpoint caught about 20 of 26 inside their agent turn. After every Gym process group was killed and the checkpoint restored into fresh processes, every continued rollout finished with reward 1.0, as did every rollout that finished before the checkpoint.
    • Restarts. Half the rollouts used a restart_only resources server. Prepare listed every restart still in flight and no saved rollout, and the commit left the restarts out. After the kill, the restarts were retired and started over from their input while the saved rollouts continued, and every rollout finished with reward 1.0.
    • Rollout collection. 48 rows, four at a time: a SIGUSR1 checkpoint, four more rows sent, then the collector killed with SIGKILL while the Gym servers kept running. The resumed run retired the four rows sent after the checkpoint and sent them again as attempt 1, and all 48 rows finished exactly once with reward 1.0.
    • The first two run with one and two policy workers and with one and two workers per server. Each scenario passed repeated runs.
    • NEMO_GYM_CHECKPOINT_E2E=1 NEMO_GYM_MULTIWORKER_E2E=1 NEMO_GYM_CHECKPOINT_VLLM_URL=http://127.0.0.1:8100/v1 NEMO_GYM_CHECKPOINT_VLLM_MODEL=Qwen/Qwen3-4B-Instruct-2507 RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/e2e/checkpoint -k real_model

Compatibility

  • New rollout collection config:
    • checkpoint_dir (default unset);
    • checkpoint_every_s (needs checkpoint_dir);
    • checkpoint_timeout_s (default 300).
  • checkpoint_dir also holds the run's dispatch log, dispatched.jsonl, which the next run replaces.
  • With checkpoint_dir set, rollout collection installs SIGTERM and SIGUSR1 handlers for the length of the run and restores the previous ones afterwards. The timing report then prints its table from that handler instead of installing its own.
  • With checkpoint_dir set, a /run refused with 409 checkpoint_admission_closed is sent again as the same attempt after the resume.
  • A run that checkpointed and stopped on SIGTERM exits with status 143, like a run a SIGTERM interrupts without a checkpoint.
  • Closing can wait up to checkpoint_timeout_s for a checkpoint that is already under way.
  • Restoring needs Gym servers that have not served sessions yet, which is the case after a job restart. Gym servers that keep running across collector restarts cannot be restored into; their rows then start from their input one attempt later.

@ananthsub ananthsub added feature New capabilities, enhancements, or enablement work area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters labels Oct 1, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 1, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ananthsub
ananthsub requested a review from zyzhou5 October 1, 2026 19:04
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-rollout-collection branch from b5ffb47 to 22a07dd Compare October 1, 2026 19:38
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-e2e branch from 5f8b5af to 87e1f0c Compare October 1, 2026 19:38
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-rollout-collection branch from 22a07dd to 324d650 Compare October 1, 2026 21:53
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-e2e branch from 87e1f0c to 207f588 Compare October 1, 2026 21:53
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-e2e branch from 207f588 to afee845 Compare October 1, 2026 22:05
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-rollout-collection branch from 324d650 to 29a2feb Compare October 1, 2026 22:05
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-rollout-collection branch from 29a2feb to d1bac74 Compare October 2, 2026 13:22
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-e2e branch from afee845 to 6a3c667 Compare October 2, 2026 13:22
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-e2e branch from 173032c to b63ade8 Compare October 10, 2026 05:09
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-rollout-collection branch from 54ffae4 to 4b775b8 Compare October 10, 2026 05:09
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-rollout-collection branch from 4b775b8 to e4f30fa Compare October 10, 2026 05:30
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-e2e branch from a014331 to 8575f2b Compare October 12, 2026 14:56
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-rollout-collection branch from e4f30fa to 1d706d5 Compare October 12, 2026 14:56

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Four correctness findings below, plus one non-blocking checkpoint-retention suggestion.

Validation: 562 focused existing tests passed, along with reduced reproductions and changed-file lint/format checks, at e24a8bc3dc4b. The checkpoint patch is unchanged after the rebase to 6b3f363347e4; I rechecked the intervening code changes and current comment locations. These reproductions exercise the collector and checkpoint-control components, not a real-model or multi-host workload. The current unit-test check passes.

Comment thread nemo_gym/rollout_collection.py Resolved
Comment thread nemo_gym/_checkpoint/collection.py Outdated
self._collection = collection
loop = asyncio.get_running_loop()
loop.add_signal_handler(signal.SIGUSR1, self._spawn, lambda: self.checkpoint())
loop.add_signal_handler(signal.SIGTERM, self._spawn, self._checkpoint_and_stop)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Compose the checkpoint and timing-report SIGTERM handlers

When checkpoint_dir and dispatch_start_report_within_s are both set, _run_from_config() subsequently calls _print_on_sigterm(), replacing this event-loop handler. That helper saves signal.getsignal(SIGTERM), which is asyncio's no-op stub rather than this callback. On SIGTERM it prints the timing table, removes the event-loop handler and re-sends the signal to that stub; no checkpoint or stop occurs. I reproduced the combination yielding no coordination calls and a still-running collection, while the same check without timing reports published and stopped. Could the two features share/compose signal ownership, with a regression test enabling both?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. With checkpoint_dir set, the checkpoint is the only owner of SIGTERM on the event loop.
The timing report no longer installs _print_on_sigterm(); it registers its table with the checkpointer instead.
On SIGTERM the handler prints the table first, because printing is quick and must not be lost if the checkpoint runs out the grace period, then checkpoints and stops.
start() now records the SIGUSR1 and SIGTERM handlers it replaces, and close() puts them back when collection ends.
Without checkpoint_dir, the timing report keeps its own handler as before.

Covered by test_sigterm_with_the_timing_report_on_checkpoints_stops_and_prints_the_table, which runs run_from_config() with both features on and sends SIGTERM while a row is in flight.
It checks that prepare and commit ran, the checkpoint was published, collection stopped with CollectionStopped, the table was printed before the checkpoint published, and the previous SIGTERM handler is back.
On the previous head it timed out with no coordination calls and collection still running.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the signal composition works at d9ac070, but the new regression test currently fails at line 800: it expects [prepare, commit] and receives [prepare, commit, retire, forget]. With #4029 now in the base, collector shutdown awaits cancellation of the pending row, which runs the cleanup hooks. I reproduced this in isolation; updating only that expected call list in a scratch copy lets all the remaining checks pass, including the published checkpoint, CollectionStopped, timing output order, and restored signal handler. Could you update the assertion to account for that cleanup? The unmodified checkpoint/collector/eval-command suite is 545 passed, 1 failed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, reproduced at d9ac070: with #4029 in the base, shutdown now cancels the pending row, and its after_dispatch(failed=True) retired the attempt and close() then forgot it.
Rather than accept those calls in the assertion, the run no longer sends them, so the test keeps expecting only prepare and commit.
After a stopping checkpoint the servers stay committed, and the control routes refuse a retire or a forget outside the idle and restored phases.
Real servers would have refused both, and the run would have logged that the rows' state stays on the servers.

Fixed in 6ed373f:

  • once a run has stopped, after_dispatch() records no failure, and the background retire and forget tasks return without sending;
  • close() of a stopped run skips its final retire and forget;
  • the next run's restore already retires every attempt the stopped run sent, so nothing is left unfenced.

The new path also exposed a hang in close().
asyncio.gather() over tasks that have all finished returns without yielding, so a background task whose done callback had not run yet stayed in the set and the loop never ended.
close() now removes the tasks it waited for itself.

Covered by test_a_stopped_run_sends_no_retire_or_forget_to_its_committed_servers, where one row is retired before the stopping checkpoint, one fails while the checkpoint holds the lock, and one is cancelled after it.
It checks that nothing follows prepare and commit, and that the commit continues only the row still in flight.
On the previous head it fails with a retire and a forget after the commit; with only the close() change reverted it hangs.
The checkpoint, collector and evaluation command suites pass, including the SIGTERM test you ran.

Comment thread nemo_gym/_checkpoint/collection.py Resolved
Comment thread nemo_gym/rollout_collection.py Resolved
Comment thread nemo_gym/_checkpoint/collection.py Resolved
@ananthsub

Copy link
Copy Markdown
Contributor Author

/ok to test 3fa6d23

@copy-pr-bot

copy-pr-bot Bot commented Oct 19, 2026

Copy link
Copy Markdown

/ok to test 3fa6d23

@ananthsub, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

Rollout collection checkpoints the rows in flight on a signal or a
timer, and continues them when the run resumes from its cache. Rows
whose /run got no reply are retired everywhere, and the rollouts it
retired are forgotten once the run dispatches nothing more.

Rows that some server cannot capture, such as a restart-only agent's,
are left out of the commit and recorded as restarts; they keep running
through the checkpoint. After a crash, the run retires each restart's
attempt everywhere, so a harness that outlived the crash cannot write
into the rerun, and starts the row over as its next attempt.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — eefd3d5c Deployed Oct 26, 2026 by copy-pr-bot[bot] via release / finalize / notify #7241
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants