Repository navigation
feat(checkpoint): make policy model servers checkpoint participants with generation cuts - #3883
Merged
Merged
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
This was referenced Oct 1, 2026
Merged
ananthsub
force-pushed
the
ananthsub/partial-ckpt-core
branch
from
October 1, 2026 19:38
0a24eb7 to
c2b5103
Compare
ananthsub
force-pushed
the
ananthsub/partial-ckpt-policy-model
branch
2 times, most recently
from
October 1, 2026 21:53
bf327a4 to
aeda6de
Compare
ananthsub
force-pushed
the
ananthsub/partial-ckpt-core
branch
from
October 1, 2026 21:53
c2b5103 to
9040654
Compare
This was referenced Oct 1, 2026
ananthsub
force-pushed
the
ananthsub/partial-ckpt-core
branch
from
October 1, 2026 22:05
9040654 to
846c721
Compare
ananthsub
force-pushed
the
ananthsub/partial-ckpt-policy-model
branch
from
October 1, 2026 22:05
aeda6de to
efc0332
Compare
ananthsub
force-pushed
the
ananthsub/partial-ckpt-core
branch
from
October 2, 2026 19:43
846c721 to
f4092dc
Compare
ananthsub
force-pushed
the
ananthsub/partial-ckpt-policy-model
branch
2 times, most recently
from
October 2, 2026 21:17
1a3143b to
461da5b
Compare
ananthsub
force-pushed
the
ananthsub/partial-ckpt-policy-model
branch
from
October 2, 2026 23:32
461da5b to
3e94e72
Compare
Contributor
|
See #3971 for MInf support |
ananthsub
force-pushed
the
ananthsub/partial-ckpt-policy-model
branch
from
October 6, 2026 19:28
3e94e72 to
2ceb080
Compare
ananthsub
force-pushed
the
ananthsub/partial-ckpt-policy-model
branch
from
October 10, 2026 05:30
807db7d to
e54fbdd
Compare
macandro96
reviewed
Oct 10, 2026
ananthsub
force-pushed
the
ananthsub/partial-ckpt-core
branch
from
October 12, 2026 14:56
963d2e0 to
f2475bc
Compare
ananthsub
force-pushed
the
ananthsub/partial-ckpt-policy-model
branch
from
October 12, 2026 14:56
e54fbdd to
8a9681a
Compare
macandro96
reviewed
Oct 12, 2026
ananthsub
force-pushed
the
ananthsub/partial-ckpt-policy-model
branch
from
October 12, 2026 22:55
8a9681a to
384b21d
Compare
ananthsub
force-pushed
the
ananthsub/partial-ckpt-core
branch
from
October 12, 2026 22:55
f2475bc to
082b250
Compare
7 tasks done
macandro96
reviewed
Oct 13, 2026
ananthsub
force-pushed
the
ananthsub/partial-ckpt-policy-model
branch
from
October 13, 2026 13:44
384b21d to
275dabd
Compare
macandro96
reviewed
Oct 13, 2026
macandro96
approved these changes
Oct 13, 2026
Contributor
Author
|
/ok to test 978380f |
…ith generation cuts A policy model server with checkpoint_policy becomes a participant. A checkpoint closes admission and holds every response not yet delivered. A call's capture-ledger row is published only when its response starts, so undelivered calls have no row and are left out; the agent makes them again after a restore. Model servers that an agent with token capture calls outside the checkpoint gate are refused at startup. With generation cuts, each inference worker cuts its in-flight calls at a durable token prefix that the replacement call continues, at a per-backend control URL when the backend serves cuts elsewhere. Restored ledger rows continue the lineage under the next attempt. Retire stops an attempt's calls and frees its ledgers, and restored attempts the controller no longer continues are deleted without a fence. A coordinator in the main process makes a server with several uvicorn workers one participant. Co-authored-by: Anish Mahishi <amahishi@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38473765817 |
This was referenced Oct 13, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
A policy model server with
checkpoint_policy: truebecomes a checkpoint participant. The checkpoint never waits for generations.admission_closed. A rollout that keeps running through the checkpoint, such as a restart, would have shown that refusal to the model or failed.stale_attemptat once instead of running after resume.stale_attempt. A waiting call leaves the tracking when it leaves the gate, so the tracking holds only calls still waiting.deferred_commit), and the gate writes it afterdeliver_responseand before the chunk is sent.model_call_id, but a second policy server sharing the ledger could not see them and exported their rows; and a call whose capture context attached after the commit's snapshot could land its row before the export read the ledger. Either way a checkpoint could hold the answer of a call the caller never received.checkpoint_generation_cuts: true). Prepare asks each inference worker to cut its in-flight calls at a durable token prefix. The replacement call carries the cut in its capture admission, and the worker continues the prefix instead of regenerating it. The wire models (GenerationCutInventory,GenerationCutReceipt,GenerationCutPrefixAck) keep the field layout NeMo RL already uses. A call with constrained decoding (tool_choiceother thanautoornone, a non-textresponse_format, guided or structured outputs) is never cut and never continues a cut, because a token prefix cannot restore its structured decoder. A cut gets half of the prepare time remaining, so a worker that hangs on its cut still leaves time for the stages after the model.num_workers > 1. It is still one participant to the controller: a coordinator in the main process owns the phases, lease, storage, and restored cuts, and each worker runs a gate (admission, held responses, its own cut round) that reports to the coordinator over a Unix socket. A worker that has not reported, or that disconnects during a checkpoint, blocks the checkpoint instead of weakening it. A worker can also disconnect after the commit checked readiness but before the coordinator collects the workers' snapshots; the rows of that worker's undelivered calls would then be exported as delivered. The commit therefore fails withexport_failedif a worker was lost or fewer workers answered than expected, and that failure is final for the checkpoint, because a later export could take a later cut. With one worker nothing changes.generation_cut_control_urllists one control URL perbase_url, for backends that serve cuts somewhere other than their OpenAI URL. It is refused alongsideendpoint_file, and a backend without an address fails its cut, which only costs regenerating the call./tmp/ng-<pid>-<random>/(nemo_gym/runtime_dir.py), under/tmpbecause AF_UNIX paths are limited to about 100 bytes. It used to be a socket file in the temporary directory that nothing ever removed./tmpbut not its process namespace, and a connection that times out counts as alive. Nothing is removed by age.staging_keys, the staged token-store keys the checkpoint still refers to, so the training framework knows which rows to keep. Cut requests use the reserved control connection pool, and cut failures are counted in readiness and logged.delete_restored), because the controller may still start the rollout over as that attempt. A started replacement is never touched, so its training rows are kept.policy_modelandpolicy_model_reasoning_offdo, but each tracks its restored attempts on its own. When the first call of a restored attempt reaches a server, before the call can write a row or be held, that server creates a shared "started" marker next to the attempt's ledger,<capture key>.lineage.started(mark_started()). The marker is created withO_CREAT|O_EXCLunder the rollout's lock. It is not synced to the directory: other servers only need to see it, and after a crash the next step is a restore, which deletes started markers with the other stale capture files. The commit deletes an unscoped restored attempt throughdelete_unstarted(), which checks for the marker under the same lock and keeps the ledger if any server sharing it has started the attempt. Without the marker, a server that never saw the attempt start would delete the rows another server wrote for it, including the rows of a call that is held and has no row yet.deleteoperations of feat(token-capture): retire and delete external-staging capture ledgers #3938 and feat(token-capture): retire and delete complete-record captures #3939: ledgers, token records, intents, incomplete markers, capture state, and retire fences.jsonencoder, as the checkpoint writer is.orjsonrefused integers beyond 64 bits and silently wrote NaN and infinities, such as a-inflogprob, asnull. A reply that still cannot be framed is answered with a typed error, so the caller does not wait until its deadline. A message's send and reply share one timeout, and a timeout or transport error isCoordinatorUnavailableError, so a cut claim against a stalled coordinator falls back to regenerating. A ticket's exit reports to the coordinator even if giving back its cut fails.retireof attempt N cancels the calls of attempt N and every earlier attempt, each only once, and waits for them, then retires those attempts' ledgers, so their files are freed and no late row recreates them. The policy gate refuses the retired attempt's late calls until the controller forgets the rollout. A retire skips calls without a rollout prefix, such as evaluation, judge, or health-probe calls, and a malformed rollout prefix is a 400invalid_rollout_id. With several workers, the coordinator marks the attempt on every worker before any of them stops its calls, relaysforget, and gives a worker that registers later the current marks.wrong_rollout, so the trainer dropped the row. Restore now stamps each carried-over ledger row with the capture key it was staged under (CallRecord.capture_key), and rebuild accepts the same or an earlier attempt of the rollout; anything else fails asforeign_capture_key. A chain of restores keeps each row's original stamp.staging_keys_by_episode: each committed episode's staged keys under its capture key, built from the same records as the flatstaging_keyslist. The controller keeps only the rows of episodes it continues, and a key's name does not always say whose it is: generation-cut keys name no rollout.PolicyGate.deliver_responselets a response start, holding it while a checkpoint is open; it wasrelease_response.StreamingResponsesendshttp.response.startbefore it reads its body, and the buffered chat-streaming path sends SSE keep-alive comments while the backend is still generating.deliver_response()on the headers, so prepare treated a call that had produced nothing as streaming and waited for it instead of cutting it. If the snapshot fence held the call's terminal token write, the call could not finish and prepare stalled.deliver_response()on the first body chunk that carries data, skipping empty chunks and, for an event stream,SSE_KEEPALIVE.SSE_KEEPALIVEmoved tonemo_gym/chat_streaming.py, so the middleware and the streaming path share one definition of a liveness frame.attempt, asEpisodeIddoes, and the continuation a re-issued call carries names its prefixprefix_token_countandprefix_digest, as the worker's acknowledgement does.Conflict note:
CaptureContextonmaingainedexternal_worker_response_seen(#3670); this PR addsadmission_hooknext to it, and both fields are kept.How it works
Where this PR sits in the overall flow
The highlighted part is what this PR adds.
flowchart LR C["Controller<br/>NeMo RL, or rollout collection"] CO["Coordination<br/>prepare, commit, restore, resume,<br/>retire, forget"] K["Control plane on every server<br/>phases, lease, storage,<br/>retire: refuse, stop, free;<br/>forget"] subgraph G["One participant per Gym server"] E["Environment server<br/>episode steps"] M["Policy model<br/>held responses, generation cuts"] A["Agent<br/>sessions parked at boundaries"] R["Resources server<br/>session state"] end W["Inference worker<br/>stages cut prefixes"] D[("Checkpoint directory<br/>records, then manifest")] L[("Capture ledger<br/>retire and delete from #3938, #3939")] C --> CO --> K K --> E & M & A & R M --> W M --> L G --> D classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937 class M thisOne checkpoint, a crash, and the restore, end to end:
This PR
A model call across a checkpoint and a crash.
With several uvicorn workers, one coordinator in the main process is the participant; every worker keeps its own gate and forwards control calls to the coordinator.
The capture ledger across checkpoints, restores and the training framework's cleanup.
Where this sits in the stack
This is one PR in a stack of draft PRs that re-cut partial-rollout checkpointing onto environment servers. Each PR is one commit, and its base is the branch of the PR before it, so each diff shows only that PR's change.
The stack builds on the token-capture cleanup PRs #3938 (capture ledger
retireanddelete) and #3939 (complete-recordretireanddelete), which have merged. #3882 has merged too, so #3883's base ismain. The checkpoint stack uses those operations to free the ledgers of retired attempts and to clear a dead execution's capture files before a restore. The capture-ledger cache bound (#4268) and the striped lock files (#3937) are independent of the stack.ananthsub/partial-ckpt-core): feat(checkpoint): add the participant control plane, episode steps, and coordination (merged)ananthsub/partial-ckpt-policy-model): feat(checkpoint): make policy model servers checkpoint participants with generation cuts (this PR)ananthsub/partial-ckpt-model-worker-cuts): feat(token-capture): add worker staging helpers for generation cutsananthsub/partial-ckpt-environment): feat(checkpoint): continue environment server episodes from their boundariesananthsub/partial-ckpt-resources): feat(checkpoint): add the resources server participant with asynchronous session hooksananthsub/partial-ckpt-verifier-declarations): feat(checkpoint): declare five training verifiers stateless with replayable verificationananthsub/partial-ckpt-agent): feat(checkpoint): add the agent session participant and Simple Agent continuationananthsub/partial-ckpt-e2e): test(checkpoint): add a process-level end-to-end suite driven by coordinationananthsub/partial-ckpt-rollout-collection): feat(checkpoint): checkpoint evaluation runs from rollout collectionananthsub/partial-ckpt-telemetry): feat(checkpoint): spans and metrics for partial-rollout checkpointsananthsub/partial-ckpt-multi-worker): feat(checkpoint): resources, agent, and environment servers with several workersananthsub/partial-ckpt-fault-tests): test(checkpoint): check checkpoint invariants over random sequences of operationsPartial-rollout checkpointing lets a training controller, such as NeMo RL, checkpoint Gym while rollouts are in flight and, after a crash, continue those rollouts from their last safe point instead of starting them over. Checkpointing is off by default; with the
checkpoint:block unset, no server installs any checkpoint routes or behavior.Relationship to the old stack (#2939 to #2946)
main's capture ledger plus rows re-imported under the next attempt; admission no longer waits for generations.nemo_gym._checkpoint.generation_cutinstead ofmodel_control_contracts.Issue
No tracking issue exists for this re-cut. The design and the mapping from the old stack are described in this PR series, and the old stack's PRs (#2939 to #2946) carry the original discussion.
Validation
Run on this branch, on the current
main, which includes #3882:RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_checkpoint_*.py: passed.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_token_capture_*.py tests/unit_tests/test_token_id_capture.py tests/unit_tests/test_base_responses_api_model.py: passed.RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider responses_api_models/vllm_model/tests: passed.capture ledger for roll-1-a1 already holds rows from another execution), with the files and fences of the target and later attempts deleted and look-alike rollouts untouched;stale_attemptand nothing is served;RAY_TMPDIR=/tmp .venv/bin/python -m pytest -q -p no:cacheprovider tests/unit_tests/test_runtime_dir.py: passed. A dead server's directory with closed sockets is removed; a live process's directory is kept; a directory whose socket accepts is kept even when its process looks dead; only runtime directory names are considered; and a process creates its directory once, with mode 0700, and removes it at exit.tests/unit_tests, eight processes, withouttest_opensandbox_compose.py, which needs the optionalopensandboxpackage): passed except a sandbox retry test, which fails the same way onmainin this development environment.pre-commit run --files <files changed by this PR>: all hooks passed, and no hook modified a file.Signed-off-byline.Rollout evidence
Real model: see the rollout evidence in test(checkpoint): add a process-level end-to-end suite driven by coordination #3889, whose opt-in tests drove this PR's code with
Qwen/Qwen3-4B-Instruct-2507on vLLM: saved rollouts continued after a kill and restore, restarts started over, and every rollout finished with reward 1.0. A real vLLM worker serving generation cuts is not covered: stock vLLM does not serve the cut route.Scale, on one workstation with every Gym server and a fake inference backend (weather rollouts through the typed
single_agent_turnrequest path, token capture and generation cuts on, each rollout's second model call held so most rollouts are in flight at the checkpoint; checkpoint, kill every Gym process group, restart, restore, run the replacement attempts). After the scale fixes in this PR, measured on the branch this stack was cut from, with the same commits:In the first 16,000-rollout run, the policy model logged 8 uvloop "File descriptor N is used by transport" errors and 16 requests failed with "ledger lineage resolution failed". The cause is a libuv accept race while clients disconnect, not checkpoint code; the ledger failed closed, clients retried, and no replacement failed. The rerun and every 8,000-rollout run had none.
Compatibility
checkpoint_policy(defaultfalse) makes the server a participant, andcheckpoint_generation_cuts(defaultfalse) turns on generation cuts. Both only take effect with the globalcheckpoint:block enabled.token_id_capture.generation_prefix_cuts_enabledkey (never onmain) switch tocheckpoint_generation_cuts.nemo_gym._checkpoint.generation_cut.CheckpointableLedgerprotocol gainsmark_started()anddelete_unstarted(), so a custom ledger used with a checkpoint participant must implement both.PolicyWorkerLinktakes aledger=argument, which a worker needs to mark restored attempts started.<capture key>.lineage.startedfile in the capture ledger directory. It is removed with the attempt's ledger.