Repository navigation
feat(checkpoint): continue environment server episodes from their boundaries - #3885
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
7c6475d to
b9ac5f1
Compare
b9ac5f1 to
6c95786
Compare
6c95786 to
1739ac6
Compare
1739ac6 to
2812ca7
Compare
2812ca7 to
5aa1530
Compare
52f841a to
9dbbfe7
Compare
9dbbfe7 to
8c21514
Compare
8c21514 to
e2da953
Compare
| async with participant.step(episode_id, "replay"): | ||
| raise RuntimeError("the agent failed") | ||
|
|
||
| await participant.close_admission(CLOSE) |
There was a problem hiding this comment.
[TEST blocker] This test passes without exercising the retire path it names.
Admission is closed before episode() starts, so participant.begin() raises AdmissionClosedError. The polling loop exits because task.done() is true, retire() finds no tracked episode, and the final assertion passes while pytest only logs the unretrieved task exception.
Please start the task first and synchronize after it has entered the replay step; then close admission, trigger the failure, wait specifically until the exported boundary is [None], assert the task is still pending, retire it, and await/assert its cancellation.
There was a problem hiding this comment.
Agreed, it passed without reaching the retire path.
The test now starts the episode first and waits on an event set inside the replay step, then closes admission and lets the step fail.
It waits only for the exported boundary to become [None], checks that the episode is still pending, retires it, and checks that the episode was cancelled and is no longer tracked or exported.
To show it is not vacuous, I broke the code twice and restored it:
- with a failed replay step that does not park, the test fails waiting for the
[None]boundary; - with a retire that skips the episode, the test fails waiting for the episode to stop.
The old version passed under both changes.
Its neighbour,test_a_replay_step_that_raises_during_a_checkpoint_parks_until_resume_and_starts_over_after_a_crash, started its checkpoint after a fixed sleep and could also leave its wait ontask.done(), so it now waits for the same event and for the[None]boundary alone.
No other test in the file closes admission before its episode begins.
An inference worker that takes generation cuts stages the token prefix of each cut call, and the replacement call continues it. This adds the worker-side staging helpers, and validates a continued prefix against the replacement's admission: staging keys, source call, digest, lineage, the exact parent tokens, token count, and policy version. The policy checkpoint coordinator also logs its workers by process ID: an info line when one joins or stops cleanly, and a warning when one is lost, saying whether an open checkpoint can no longer commit. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…ndaries An environment server is a checkpoint participant. Each episode records a boundary after every completed step. A checkpoint parks episodes at their boundaries, or lets a replay step finish after it, and a restored episode continues from its boundary as the next attempt. Retire stops an episode, waits for its cleanup, and frees it. An episode that uses a server which cannot capture its part, such as a restart-only agent or resources server, is a restart: the seed reply says so, and the episode never holds up a checkpoint, is never exported, and starts over from its input after a crash. Episodes of a protocol that records no boundaries are restarts from the start. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38488611212 |
What changed and why
Every environment server takes part in checkpoints once checkpointing is on. The environment server owns the episode protocol, so it records where each episode can continue.
waitorreplaymode. Prepare closes episode admission and is ready once every live episode is parked at a boundary or inside a replay step./run, which continues from it. The task must match the one recorded.single_agent_turnruns seed, invoke agent, close agent, verify, and return as explicit stages with typed session handles. Seeding and the agent invocation replay; closing the agent waits; verification uses the mode the resources server reported on its seed reply, kept in the session handles so a restored episode still knows it. Once the agent's activation returns, the episode records aclose_agentboundary carrying the agent's response, so a checkpoint that closes then parks the episode there instead of waiting on the close step, and a restore at that boundary closes the agent and verifies without invoking the agent again.close_agentandreturn. After a restore and resume, a later checkpoint can prepare before the replacement/runarrives, and the replacement is then admitted while admission is closed. A replacement restored atclose_agentused to start closing the agent before it recorded a boundary, which made a prepared participant not ready; one restored atreturnran final cleanup and calledend(), so an episode could finish while Gym was prepared. Both boundaries now sit at the start of their stage, as the others do, so a replacement at any stage parks before doing any work and a commit exports the boundary it was restored from. For a first attempt each boundary is recorded at the same point, with the same state, as before.maindoes since fix: keep agent observations on failed episodes #4562. The close runs after the failed replay step, which parks with no boundary while a checkpoint is open, so it waits for resume and never runs while Gym is prepared.single_agent_turnbuilds no boundary payloads otherwise.single_agent_turn_legacyadapter checkpoints too, since its rows run the same protocol./runis refused withadmission_closeduntil the controller resumes after the restore, and the restored record stays until a replacement actually starts. A retire of a replacement that had already started would wait for its final cleanup, whose calls wait for that same resume./run, such as a client retry, is refused withstale_attemptuntil the controller forgets the rollout. An episode with no boundary yet is exported with a null boundary and starts over from its input after a crash./runof an attempt that is already running is refused, and the refused request owns nothing: it neither ends nor cleans up the live episode. An episode is untracked even if its final cleanup is cancelled.end(), and its/runreply wait for resume; otherwise it could finish and drop its record before the commit, and a commit that names it would export nothing for it. A retire still stops a parked episode. With no checkpoint open, the episode goes back to running as before./runreplies on every path, and a retire never interrupts it. A retire cancels an episode only before cleanup starts, then waits until the episode has ended, so its reply means the sessions are closed. This matters because anyio's shielded scope does not stop a directtask.cancel(). A retire cut short by its deadline keeps the episode and its task digest until the episode has ended, so the next prepare reports it as a blocker instead of exporting it.checkpoint_boundaries = False, the default) cannot continue its episodes, so each one is a restart: it never blocks prepare, is never exported, and is reported in prepare'srestarts.single_agent_turnsetscheckpoint_boundaries = True, and marks an episode as a restart when its resources or agent seed reply carriesx-ng-checkpoint-restart, because a server it depends on cannot capture its part.ready()from its running blocker count, so a prepare does not list every episode after each change.num_workers > 1on environment servers, since episodes live in one process.How it works
Where this PR sits in the overall flow
The highlighted part is what this PR adds.
flowchart LR C["Controller<br/>NeMo RL, or rollout collection"] CO["Coordination<br/>prepare, commit, restore, resume,<br/>retire, forget"] K["Control plane on every server<br/>phases, lease, storage,<br/>retire: refuse, stop, free;<br/>forget"] subgraph G["One participant per Gym server"] E["Environment server<br/>episode steps"] M["Policy model<br/>held responses, generation cuts"] A["Agent<br/>sessions parked at boundaries"] R["Resources server<br/>session state"] end W["Inference worker<br/>stages cut prefixes"] D[("Checkpoint directory<br/>records, then manifest")] L[("Capture ledger<br/>retire and delete from #3938, #3939")] C --> CO --> K K --> E & M & A & R M --> W M --> L G --> D classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937 class E thisOne checkpoint, a crash, and the restore, end to end:
This PR
single_agent_turnrecords a boundary at the start of every stage. A checkpoint exports the latest boundary, and a restored attempt starts at itsnextstep, parking there first if a checkpoint is open.Final cleanup closes the episode's sessions before
/runreplies. A retire never interrupts it: it cancels an episode only before cleanup starts, and then waits for the episode to end.flowchart TD X{"retire arrives"} -->|"episode still in its protocol"| C["cancel the episode"] --> F X -->|"final cleanup already started"| F["wait"] F --> CL["final cleanup closes the agent and resources sessions"] --> D["retire replies"]Where this sits in the stack
This is one PR in a stack of draft PRs that re-cut partial-rollout checkpointing onto environment servers. Each PR is one commit, and its base is the branch of the PR before it, so each diff shows only that PR's change.
The stack builds on the token-capture cleanup PRs #3938 (capture ledger
retireanddelete) and #3939 (complete-recordretireanddelete), which have merged. #3882 and #3883 have merged too, so #3884's base ismain. The checkpoint stack uses those operations to free the ledgers of retired attempts and to clear a dead execution's capture files before a restore. The capture-ledger cache bound (#4268) and the striped lock files (#3937) are independent of the stack.ananthsub/partial-ckpt-core): feat(checkpoint): add the participant control plane, episode steps, and coordination (merged)ananthsub/partial-ckpt-policy-model): feat(checkpoint): make policy model servers checkpoint participants with generation cuts (merged)ananthsub/partial-ckpt-model-worker-cuts): feat(token-capture): add worker staging helpers for generation cutsananthsub/partial-ckpt-environment): feat(checkpoint): continue environment server episodes from their boundaries (this PR)ananthsub/partial-ckpt-resources): feat(checkpoint): add the resources server participant with asynchronous session hooksananthsub/partial-ckpt-verifier-declarations): feat(checkpoint): declare five training verifiers stateless with replayable verificationananthsub/partial-ckpt-agent): feat(checkpoint): add the agent session participant and Simple Agent continuationananthsub/partial-ckpt-e2e): test(checkpoint): add a process-level end-to-end suite driven by coordinationananthsub/partial-ckpt-rollout-collection): feat(checkpoint): checkpoint evaluation runs from rollout collectionananthsub/partial-ckpt-telemetry): feat(checkpoint): spans and metrics for partial-rollout checkpointsananthsub/partial-ckpt-multi-worker): feat(checkpoint): resources, agent, and environment servers with several workersananthsub/partial-ckpt-fault-tests): test(checkpoint): check checkpoint invariants over random sequences of operationsPartial-rollout checkpointing lets a training controller, such as NeMo RL, checkpoint Gym while rollouts are in flight and, after a crash, continue those rollouts from their last safe point instead of starting them over. Checkpointing is off by default; with the
checkpoint:block unset, no server installs any checkpoint routes or behavior.Relationship to the old stack (#2939 to #2946)
New compared with the old stack, which had no environment servers. It takes over the protocol side of #2942 (where an episode can continue), which the old stack reconstructed from agent and resources state after the fact.
Issue
No tracking issue exists for this re-cut. The design and the mapping from the old stack are described in this PR series, and the old stack's PRs (#2939 to #2946) carry the original discussion.
Validation
Run on this branch, on the current
main:RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_checkpoint_*.py: passed, including, each failing without its change:/runleaves the live episode tracked, and an episode whose cleanup is cancelled is still untracked;close_agentand stays ready, and a restore atclose_agentcloses and verifies without invoking the agent;close_agentor atreturnand admitted during a second checkpoint parks before its stage: readiness stays true, no close request and noend()happen before resume, and the commit exports the same boundary (without the fix, the agent close or the final resources close started and readiness was false);[None]boundary rather than on the task ending (the previous version passed without reaching the retire, and the new one fails if the failed step does not park or the retire skips the episode);It also checks that an episode whose retire was cut short blocks the next prepare, is not committed, and is freed by a retried retire after resume, and that readiness is answered from the blocker count.
RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_environment_server.py: passed.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider environment_servers/single_agent_turn/tests: passed.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider environment_servers/single_agent_turn_legacy/tests: passed.responses_api_agents/codex_agent/tests,responses_api_agents/openclaw_agent/tests,responses_api_agents/opencode_agent/tests, andresponses_api_agents/pi_agent/tests, each run in its own server environment, created from that server'srequirements.txtasgym env testdoes: passed, including the integration tests that collect rows through an environment server's/run. Their fake responses gain an emptyheadersmapping, because the environment server now reads the seed and agent response headers. One Pi outcome test,test_outcome_extension_uses_pinned_pi_context_classifier, needs a newer Node.js than the development machine's Node.js 20, and fails the same way onmain.The full core unit suite (
tests/unit_tests, eight processes, withouttest_opensandbox_compose.py, which needs the optionalopensandboxpackage): passed except a sandbox retry test, which fails the same way onmainin this development environment.pre-commit run --files <files changed by this PR>: all hooks passed, and no hook modified a file.Every commit carries a
Signed-off-byline.Rollout evidence
single_agent_turnrequest path continues from its last boundary; second crash before the replacement starts).Qwen/Qwen3-4B-Instruct-2507on vLLM: saved rollouts continued after a kill and restore, restarts started over, and every rollout finished with reward 1.0.Compatibility
checkpoint:block setsenabled: true.num_workers > 1fails at startup.episode_idsscope. A commit request withoutepisode_idsstill exports every live episode.