Repository navigation
perf(checkpoint): defer rollout and trainer data-plane persistence - #5139
Open
macandro96 wants to merge 4 commits into
Open
macandro96 wants to merge 4 commits into
macandro96 wants to merge 4 commits into
Conversation
Contributor
Author
|
/ok to test ff6cd89 |
Contributor
Author
|
/ok to test e7b0f3a |
macandro96
force-pushed
the
amahishi/async-tq-rollout-snapshots-main
branch
from
November 10, 2026 01:07
e7b0f3a to
5586eb6
Compare
Contributor
Author
|
/ok to test 5586eb6 |
macandro96
force-pushed
the
amahishi/async-tq-rollout-snapshots-main
branch
from
November 10, 2026 23:27
5586eb6 to
1b29553
Compare
Contributor
Author
|
/ok to test 1b29553 |
Contributor
|
Thanks for the update. I wrote a CPU test for writers racing a SimpleStorage capture. It is on
|
macandro96
force-pushed
the
amahishi/async-tq-rollout-snapshots-main
branch
from
November 16, 2026 00:28
1b29553 to
da03aaa
Compare
Base automatically changed from
amahishi/async-simple-tq-snapshots-main
to
main
November 16, 2026 20:20
…resume Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Capture trainer data-plane views and controller metadata under the mutation barrier, then persist and release the retained view after canonical mutations resume. Wait for checkpoint I/O before model saves and publication; settle cancellation before releasing retained storage. Preserve synchronous fallback and existing trainer/Gym recovery semantics. Add tests for barrier release, detached cursor state, publication ordering, persistence/release failures, cancellation, and native fallback. Validation: 395 focused checkpoint/restore tests passed; Ruff and scoped Pyrefly checks passed. Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
…rier Split the barrier hold left after the TQ capture into the replay inventory check, rollout-recovery state build, its serialization, and the staging inventory check, and log each under timing/rollout_checkpoint. Register these and the TQ capture/persist/release phases for the OTel checkpoint duration series. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
macandro96
force-pushed
the
amahishi/async-tq-rollout-snapshots-main
branch
from
November 16, 2026 23:22
400c3a2 to
bb8bc1f
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
PR 2 of 2, based on #5137. Review this PR against the foundation branch; merge #5137 first, then retarget this PR to main.
Summary
RolloutSnapshotter: capture TQ and detach matching recovery metadata under the barrier, resume the environment/dispatch, then persist the retained storage view before publishing the snapshot.154945680eb85dc175bb50e6296eb3828b34e849): capture under the controller mutation barrier, then persist and release after the barrier opens, before model saves and publication. Canonical TQ writes can resume during persistence; the trainer save itself still awaits it.SnapshotCapturewrapper, with failure/cancellation cleanup.CapturedRolloutStateremains recovery metadata only. Rollout snapshot metrics split capture/persist/release timing.Scope
Changes relative to #5137 are limited to the snapshot coordinator, single-controller trainer-save integration, tests, and documentation. No Gym, prefix-cut, router-replay, writer-actor, or launcher dependency.
The deferred path requires
data_plane.simple.async_checkpoint: true, with SSD offload disabled. Trainer checkpoints additionally requirecheckpointing.save_data_plane: true. Unsupported adapters/configurations keep their synchronous behavior. Trainer checkpoints do not pause Gym or capture in-progress Gym sessions: the controller barrier protects recovery-owned rows, not every live staging write. Environment recovery still requires a rollout snapshot.The snapshot-save lock remains held through publication, so optimizer commits can still wait. This change moves TQ persistence outside mutation/environment pauses, not the whole trainer save into the background.
Barrier phase timing
400c3a252splits the rest of the rollout-snapshot barrier hold, beyond the TQ capture, into fourtiming/rollout_checkpointmetrics:replay_inventory_secondsrecovery_state_build_secondsrecovery_state_serialize_secondstorch.saveand sha256 of that statestaging_inventory_secondsA phase that does not run for the configuration logs 0. These four and
tq_capture_seconds,tq_persist_seconds, andtq_release_secondsare also registered asrl.checkpoint.durationphases for OTel/Lens. Together withexclusive_hold_secondsandgym_parked_seconds, they show which part of the pause to target next. The data-plane README lists them.Validation
471 CPU checkpoint/restore/dispatch-race/policy regression tests passed on this rebased stack, including rollout resume/barrier-release, cleanup, trainer capture/publication ordering, cancellation settlement, and synchronous fallback. The suite uses
--noconftestwith an explicitly bounded two-CPU, zero-GPU local Ray instance for an existing policy metrics test.The foundation's controller-byte follow-up also passed real Ray/ZMQ concurrent mutation/failure/fresh-actor restore tests and a nine-trial saved-metadata timing comparison in the production container; see #5137. Controller serialization remains inside capture, but now produces immutable bytes directly instead of recursively copying metadata first. Controller-byte and shard persistence happen after resume/barrier release.
The integration commits were rebased onto the lifecycle follow-up in #5137. An additional runtime safeguard stops checkpointing and retains the unpublished directory if capture release cannot be confirmed; scoped Ruff and Pyrefly checks passed for the updated runtime.
The earlier rollout integration has a separate 10-node GPU save/restart/restore smoke. This exact main-based port, trainer extension, and controller-byte follow-up have not had their own GPU smoke. The snapshot-save lock still protects the trainer anchor through publication, so optimizer commits can still wait.
Lifecycle review follow-up
Rebased onto #5137's context-manager, single-attempt persistence, and bounded RPC-wait changes.
1b295537fmakes failed cleanup fail closed: if release cannot be confirmed, raise an invariant error and retain unpublished files because remote RPCs may still be writing. The checkpoint pump treats this as fatal rather than retrying a new capture. A new regression verifies both retained files and no publication.The existing save lock is acquired before the mutation barrier and held through persist/release/publication, so snapshot N+1 cannot capture while N is still persisting. The one-view error remains an invariant, not scheduling flow control.
471 combined CPU checkpoint/restore/dispatch-race/policy regressions passed using the exact pinned Gym checkout and a bounded two-CPU, zero-GPU local Ray instance. The tests use
--noconftest. Scoped Ruff and Pyrefly passed; real Ray/ZMQ timeout/late-release/concurrent-mutation/restore validation passed on the foundation. No new GPU allocation was used.