Skip to content

feat(checkpoint): continue environment server episodes from their boundaries - #3885

Merged
ananthsub merged 2 commits into
mainfrom
ananthsub/partial-ckpt-environment
Oct 14, 2026
Merged

ananthsub merged 2 commits into
mainfrom
ananthsub/partial-ckpt-environment

Conversation

@ananthsub

@ananthsub ananthsub commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

What changed and why

Every environment server takes part in checkpoints once checkpointing is on. The environment server owns the episode protocol, so it records where each episode can continue.

  • An episode records a boundary after each completed step and runs each step in wait or replay mode. Prepare closes episode admission and is ready once every live episode is parked at a boundary or inside a replay step.
  • A restored boundary is handed to the replacement attempt's /run, which continues from it. The task must match the one recorded.
  • single_agent_turn runs seed, invoke agent, close agent, verify, and return as explicit stages with typed session handles. Seeding and the agent invocation replay; closing the agent waits; verification uses the mode the resources server reported on its seed reply, kept in the session handles so a restored episode still knows it. Once the agent's activation returns, the episode records a close_agent boundary carrying the agent's response, so a checkpoint that closes then parks the episode there instead of waiting on the close step, and a restore at that boundary closes the agent and verifies without invoking the agent again.
  • Every stage starts with its boundary, including close_agent and return. After a restore and resume, a later checkpoint can prepare before the replacement /run arrives, and the replacement is then admitted while admission is closed. A replacement restored at close_agent used to start closing the agent before it recorded a boundary, which made a prepared participant not ready; one restored at return ran final cleanup and called end(), so an episode could finish while Gym was prepared. Both boundaries now sit at the start of their stage, as the others do, so a replacement at any stage parks before doing any work and a commit exports the boundary it was restored from. For a first attempt each boundary is recorded at the same point, with the same state, as before.
  • A failed agent activation closes the agent session at once to keep its observations, as main does since fix: keep agent observations on failed episodes #4562. The close runs after the failed replay step, which parks with no boundary while a checkpoint is open, so it waits for resume and never runs while Gym is prepared.
  • A boundary's state can be given as a function, which is called only when checkpointing is on, so single_agent_turn builds no boundary payloads otherwise.
  • The single_agent_turn_legacy adapter checkpoints too, since its rows run the same protocol.
  • An unclaimed restored episode, one whose replacement has not started yet, is exported again by the next checkpoint while the controller's commit scope still continues it, so a second crash still continues it. Once the scope leaves it out, it is not exported, and the restored record is retired.
  • A replacement /run is refused with admission_closed until the controller resumes after the restore, and the restored record stays until a replacement actually starts. A retire of a replacement that had already started would wait for its final cleanup, whose calls wait for that same resume.
  • A retired attempt's late /run, such as a client retry, is refused with stale_attempt until the controller forgets the rollout. An episode with no boundary yet is exported with a null boundary and starts over from its input after a crash.
  • A /run of an attempt that is already running is refused, and the refused request owns nothing: it neither ends nor cleans up the live episode. An episode is untracked even if its final cleanup is cancelled.
  • A replay step that fails while a checkpoint is open clears the episode's boundary instead of blocking the checkpoint, so the checkpoint commits and the episode starts over from its input after a crash. The episode then parks at that cleared boundary until resume before the failure propagates. It stays out of the blockers and is exported with no boundary, but its final cleanup, its end(), and its /run reply wait for resume; otherwise it could finish and drop its record before the commit, and a commit that names it would export nothing for it. A retire still stops a parked episode. With no checkpoint open, the episode goes back to running as before.
  • Final cleanup, which closes the episode's agent and resources sessions, runs before /run replies on every path, and a retire never interrupts it. A retire cancels an episode only before cleanup starts, then waits until the episode has ended, so its reply means the sessions are closed. This matters because anyio's shielded scope does not stop a direct task.cancel(). A retire cut short by its deadline keeps the episode and its task digest until the episode has ended, so the next prepare reports it as a blocker instead of exporting it.
  • Restarts. An environment server that does not record boundaries (checkpoint_boundaries = False, the default) cannot continue its episodes, so each one is a restart: it never blocks prepare, is never exported, and is reported in prepare's restarts. single_agent_turn sets checkpoint_boundaries = True, and marks an episode as a restart when its resources or agent seed reply carries x-ng-checkpoint-restart, because a server it depends on cannot capture its part.
  • A commit exports only the live episodes inside its scope, as it already did for restored episodes. The controller leaves every participant's restarts out of the scope, so a live episode that another server reports as a restart must not be exported here either.
  • The participant answers the readiness wait's ready() from its running blocker count, so a prepare does not list every episode after each change.
  • Checkpointing refuses num_workers > 1 on environment servers, since episodes live in one process.

How it works

Where this PR sits in the overall flow

The highlighted part is what this PR adds.

flowchart LR
  C["Controller<br/>NeMo RL, or rollout collection"]
  CO["Coordination<br/>prepare, commit, restore, resume,<br/>retire, forget"]
  K["Control plane on every server<br/>phases, lease, storage,<br/>retire: refuse, stop, free;<br/>forget"]
  subgraph G["One participant per Gym server"]
    E["Environment server<br/>episode steps"]
    M["Policy model<br/>held responses, generation cuts"]
    A["Agent<br/>sessions parked at boundaries"]
    R["Resources server<br/>session state"]
  end
  W["Inference worker<br/>stages cut prefixes"]
  D[("Checkpoint directory<br/>records, then manifest")]
  L[("Capture ledger<br/>retire and delete from #3938, #3939")]
  C --> CO --> K
  K --> E & M & A & R
  M --> W
  M --> L
  G --> D
  classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937
  class E this
Loading

One checkpoint, a crash, and the restore, end to end:

sequenceDiagram
  participant C as Controller
  participant G as Gym participants
  participant D as Checkpoint directory
  C->>G: prepare, in order environment, model, agent, resources
  Note over G: admission closes, in-flight work parks at a boundary,<br/>undelivered model responses are held
  G-->>C: prepared, or blockers at the deadline
  C->>G: commit with the episodes the controller continues
  G->>D: each participant writes its records, then its manifest
  Note over G: restored state the commit's scope leaves out is deleted
  C->>C: publish the checkpoint with the controller's own state
  C->>G: resume, in order resources, agent, model, environment
  Note over C,G: crash - every Gym process dies
  C->>G: restore the checkpoint in fresh processes, all or nothing
  D-->>G: records installed under attempt N+1, attempt N's capture ledger retired
  C->>G: resume
  C->>G: /run as attempt N+1 continues each episode from its boundary
  Note over C,G: retiring an episode, only while no checkpoint is open
  C->>G: retire - environment, then agent, then model and resources
  Note over G: each server marks the attempt, stops its work, waits, frees its state, then replies
  C->>G: forget, once the rollout is done for good - same order
  Note over G: the mark refusing the retired attempt goes away
Loading

This PR

single_agent_turn records a boundary at the start of every stage. A checkpoint exports the latest boundary, and a restored attempt starts at its next step, parking there first if a checkpoint is open.

flowchart LR
  B1(["next: seed"]) --> S["seed the resources session"]
  S --> B2(["next: invoke_agent"]) --> I["invoke the agent<br/>which parks its own turn loop"]
  I --> BC(["next: close_agent"]) --> CA["close the agent session"]
  CA --> B3(["next: verify"]) --> V["verify<br/>wait or replay, per the resources server"]
  V --> B4(["next: return"]) --> R["reply to /run"]
Loading

Final cleanup closes the episode's sessions before /run replies. A retire never interrupts it: it cancels an episode only before cleanup starts, and then waits for the episode to end.

flowchart TD
  X{"retire arrives"} -->|"episode still in its protocol"| C["cancel the episode"] --> F
  X -->|"final cleanup already started"| F["wait"]
  F --> CL["final cleanup closes the agent and resources sessions"] --> D["retire replies"]
Loading

Where this sits in the stack

This is one PR in a stack of draft PRs that re-cut partial-rollout checkpointing onto environment servers. Each PR is one commit, and its base is the branch of the PR before it, so each diff shows only that PR's change.

The stack builds on the token-capture cleanup PRs #3938 (capture ledger retire and delete) and #3939 (complete-record retire and delete), which have merged. #3882 and #3883 have merged too, so #3884's base is main. The checkpoint stack uses those operations to free the ledgers of retired attempts and to clear a dead execution's capture files before a restore. The capture-ledger cache bound (#4268) and the striped lock files (#3937) are independent of the stack.

  1. feat(checkpoint): add the participant control plane, episode steps, and coordination #3882 (ananthsub/partial-ckpt-core): feat(checkpoint): add the participant control plane, episode steps, and coordination (merged)
  2. feat(checkpoint): make policy model servers checkpoint participants with generation cuts #3883 (ananthsub/partial-ckpt-policy-model): feat(checkpoint): make policy model servers checkpoint participants with generation cuts (merged)
  3. feat(token-capture): add worker staging helpers for generation cuts #3884 (ananthsub/partial-ckpt-model-worker-cuts): feat(token-capture): add worker staging helpers for generation cuts
  4. feat(checkpoint): continue environment server episodes from their boundaries #3885 (ananthsub/partial-ckpt-environment): feat(checkpoint): continue environment server episodes from their boundaries (this PR)
  5. feat(checkpoint): add the resources server participant with asynchronous session hooks #3886 (ananthsub/partial-ckpt-resources): feat(checkpoint): add the resources server participant with asynchronous session hooks
  6. feat(checkpoint): declare twenty training verifiers stateless with replayable verification #3887 (ananthsub/partial-ckpt-verifier-declarations): feat(checkpoint): declare five training verifiers stateless with replayable verification
  7. feat(checkpoint): add the agent session participant and Simple Agent continuation #3888 (ananthsub/partial-ckpt-agent): feat(checkpoint): add the agent session participant and Simple Agent continuation
  8. test(checkpoint): add a process-level end-to-end suite driven by coordination #3889 (ananthsub/partial-ckpt-e2e): test(checkpoint): add a process-level end-to-end suite driven by coordination
  9. feat(checkpoint): checkpoint evaluation runs from rollout collection #3893 (ananthsub/partial-ckpt-rollout-collection): feat(checkpoint): checkpoint evaluation runs from rollout collection
  10. feat(checkpoint): spans and metrics for partial-rollout checkpoints #3903 (ananthsub/partial-ckpt-telemetry): feat(checkpoint): spans and metrics for partial-rollout checkpoints
  11. feat(checkpoint): resources, agent, and environment servers with several workers #3909 (ananthsub/partial-ckpt-multi-worker): feat(checkpoint): resources, agent, and environment servers with several workers
  12. test(checkpoint): check checkpoint invariants over random sequences of operations #4197 (ananthsub/partial-ckpt-fault-tests): test(checkpoint): check checkpoint invariants over random sequences of operations

Partial-rollout checkpointing lets a training controller, such as NeMo RL, checkpoint Gym while rollouts are in flight and, after a crash, continue those rollouts from their last safe point instead of starting them over. Checkpointing is off by default; with the checkpoint: block unset, no server installs any checkpoint routes or behavior.

Relationship to the old stack (#2939 to #2946)

New compared with the old stack, which had no environment servers. It takes over the protocol side of #2942 (where an episode can continue), which the old stack reconstructed from agent and resources state after the fact.

Issue

No tracking issue exists for this re-cut. The design and the mapping from the old stack are described in this PR series, and the old stack's PRs (#2939 to #2946) carry the original discussion.

Validation

Run on this branch, on the current main:

  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_checkpoint_*.py: passed, including, each failing without its change:

    • a retire during an episode's final cleanup does not interrupt it;
    • a retire replies only after the episode and its cleanup have ended;
    • a commit whose scope leaves out a restored episode retires it;
    • a refused duplicate /run leaves the live episode tracked, and an episode whose cleanup is cancelled is still untracked;
    • a replay step that fails during a checkpoint lets the checkpoint commit with the episode exported with no boundary, the episode has not finished or replied before resume even with its cleanup running at once, and it starts over after a crash;
    • a retire stops an episode parked after its replay step raised;
    • a commit does not export live episodes outside its scope;
    • an agent that returns during a checkpoint parks at close_agent and stays ready, and a restore at close_agent closes and verifies without invoking the agent;
    • boundary payloads are not built with checkpointing off;
    • a replacement restored at close_agent or at return and admitted during a second checkpoint parks before its stage: readiness stays true, no close request and no end() happen before resume, and the commit exports the same boundary (without the fix, the agent close or the final resources close started and readiness was false);
    • a retire stops an episode parked after its replay step raised, waiting on the step and on the [None] boundary rather than on the task ending (the previous version passed without reaching the retire, and the new one fails if the failed step does not park or the retire skips the episode);
    • an agent failure during a checkpoint closes the agent session only after resume, and the failure carries the agent's observations;

    It also checks that an episode whose retire was cut short blocks the next prepare, is not committed, and is freed by a retried retire after resume, and that readiness is answered from the blocker count.

  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_environment_server.py: passed.

  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider environment_servers/single_agent_turn/tests: passed.

  • RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider environment_servers/single_agent_turn_legacy/tests: passed.

  • responses_api_agents/codex_agent/tests, responses_api_agents/openclaw_agent/tests, responses_api_agents/opencode_agent/tests, and responses_api_agents/pi_agent/tests, each run in its own server environment, created from that server's requirements.txt as gym env test does: passed, including the integration tests that collect rows through an environment server's /run. Their fake responses gain an empty headers mapping, because the environment server now reads the seed and agent response headers. One Pi outcome test, test_outcome_extension_uses_pinned_pi_context_classifier, needs a newer Node.js than the development machine's Node.js 20, and fails the same way on main.

  • The full core unit suite (tests/unit_tests, eight processes, without test_opensandbox_compose.py, which needs the optional opensandbox package): passed except a sandbox retry test, which fails the same way on main in this development environment.

  • pre-commit run --files <files changed by this PR>: all hooks passed, and no hook modified a file.

  • Every commit carries a Signed-off-by line.

Rollout evidence

  • Process-level evidence is in the top PR (an episode on the typed single_agent_turn request path continues from its last boundary; second crash before the replacement starts).
  • Real model: see the rollout evidence in test(checkpoint): add a process-level end-to-end suite driven by coordination #3889, whose opt-in tests drove this PR's code with Qwen/Qwen3-4B-Instruct-2507 on vLLM: saved rollouts continued after a kill and restore, restarts started over, and every rollout finished with reward 1.0.

Compatibility

  • Nothing changes unless the global checkpoint: block sets enabled: true.
  • With checkpointing on, an environment server with num_workers > 1 fails at startup.
  • A commit exports only the live episodes inside its episode_ids scope. A commit request without episode_ids still exports every live episode.

@ananthsub ananthsub added feature New capabilities, enhancements, or enablement work area:env-infra Shared environment framework, lifecycle, registry, scaffolding, and validation labels Oct 1, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 1, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ananthsub
ananthsub requested a review from zyzhou5 October 1, 2026 19:04
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-environment branch 2 times, most recently from 7c6475d to b9ac5f1 Compare October 1, 2026 21:53
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-environment branch from b9ac5f1 to 6c95786 Compare October 1, 2026 22:05
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-environment branch from 6c95786 to 1739ac6 Compare October 2, 2026 13:22
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-environment branch from 1739ac6 to 2812ca7 Compare October 2, 2026 19:43
@ananthsub
ananthsub force-pushed the ananthsub/partial-ckpt-environment branch from 2812ca7 to 5aa1530 Compare October 2, 2026 21:17
Comment thread nemo_gym/_checkpoint/steps.py Resolved
Comment thread nemo_gym/_checkpoint/environment.py Resolved
async with participant.step(episode_id, "replay"):
raise RuntimeError("the agent failed")

await participant.close_admission(CLOSE)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[TEST blocker] This test passes without exercising the retire path it names.

Admission is closed before episode() starts, so participant.begin() raises AdmissionClosedError. The polling loop exits because task.done() is true, retire() finds no tracked episode, and the final assertion passes while pytest only logs the unretrieved task exception.

Please start the task first and synchronize after it has entered the replay step; then close admission, trigger the failure, wait specifically until the exported boundary is [None], assert the task is still pending, retire it, and await/assert its cancellation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, it passed without reaching the retire path.
The test now starts the episode first and waits on an event set inside the replay step, then closes admission and lets the step fail.
It waits only for the exported boundary to become [None], checks that the episode is still pending, retires it, and checks that the episode was cancelled and is no longer tracked or exported.
To show it is not vacuous, I broke the code twice and restored it:

  • with a failed replay step that does not park, the test fails waiting for the [None] boundary;
  • with a retire that skips the episode, the test fails waiting for the episode to stop.
    The old version passed under both changes.
    Its neighbour, test_a_replay_step_that_raises_during_a_checkpoint_parks_until_resume_and_starts_over_after_a_crash, started its checkpoint after a fixed sleep and could also leave its wait on task.done(), so it now waits for the same event and for the [None] boundary alone.
    No other test in the file closes admission before its episode begins.

An inference worker that takes generation cuts stages the token prefix
of each cut call, and the replacement call continues it. This adds the
worker-side staging helpers, and validates a continued prefix against
the replacement's admission: staging keys, source call, digest,
lineage, the exact parent tokens, token count, and policy version.

The policy checkpoint coordinator also logs its workers by process ID:
an info line when one joins or stops cleanly, and a warning when one is
lost, saying whether an open checkpoint can no longer commit.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…ndaries

An environment server is a checkpoint participant. Each episode records
a boundary after every completed step. A checkpoint parks episodes at
their boundaries, or lets a replay step finish after it, and a restored
episode continues from its boundary as the next attempt. Retire stops
an episode, waits for its cleanup, and frees it.

An episode that uses a server which cannot capture its part, such as a
restart-only agent or resources server, is a restart: the seed reply
says so, and the episode never holds up a checkpoint, is never
exported, and starts over from its input after a crash. Episodes of a
protocol that records no boundaries are restarts from the start.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38488611212

This branch was previously deployed

1 inactive deployment
public — a0063f51 Deployed Oct 13, 2026 by copy-pr-bot[bot] via release / finalize / notify #5237
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:env-infra Shared environment framework, lifecycle, registry, scaffolding, and validation complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants