Skip to content

perf(checkpoint): defer rollout and trainer data-plane persistence - #5139

Open
macandro96 wants to merge 4 commits into
mainfrom
amahishi/async-tq-rollout-snapshots-main
Open

macandro96 wants to merge 4 commits into
mainfrom
amahishi/async-tq-rollout-snapshots-main

Conversation

@macandro96

@macandro96 macandro96 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Stack

PR 2 of 2, based on #5137. Review this PR against the foundation branch; merge #5137 first, then retarget this PR to main.

Summary

  • Integrate the two-phase data-plane API into RolloutSnapshotter: capture TQ and detach matching recovery metadata under the barrier, resume the environment/dispatch, then persist the retained storage view before publishing the snapshot.
  • Extend the same retained-view path to full trainer checkpoints in a separate follow-up commit (154945680eb85dc175bb50e6296eb3828b34e849): capture under the controller mutation barrier, then persist and release after the barrier opens, before model saves and publication. Canonical TQ writes can resume during persistence; the trainer save itself still awaits it.
  • Keep temporary ownership in a separate SnapshotCapture wrapper, with failure/cancellation cleanup. CapturedRolloutState remains recovery metadata only. Rollout snapshot metrics split capture/persist/release timing.

Scope

Changes relative to #5137 are limited to the snapshot coordinator, single-controller trainer-save integration, tests, and documentation. No Gym, prefix-cut, router-replay, writer-actor, or launcher dependency.

The deferred path requires data_plane.simple.async_checkpoint: true, with SSD offload disabled. Trainer checkpoints additionally require checkpointing.save_data_plane: true. Unsupported adapters/configurations keep their synchronous behavior. Trainer checkpoints do not pause Gym or capture in-progress Gym sessions: the controller barrier protects recovery-owned rows, not every live staging write. Environment recovery still requires a rollout snapshot.

The snapshot-save lock remains held through publication, so optimizer commits can still wait. This change moves TQ persistence outside mutation/environment pauses, not the whole trainer save into the background.

Barrier phase timing

400c3a252 splits the rest of the rollout-snapshot barrier hold, beyond the TQ capture, into four timing/rollout_checkpoint metrics:

Metric Phase held under the barrier
replay_inventory_seconds Compare TQ replay keys with the replay index
recovery_state_build_seconds Copy and validate the rollout-recovery ledger
recovery_state_serialize_seconds torch.save and sha256 of that state
staging_inventory_seconds Compare TQ staging keys with recovery ownership

A phase that does not run for the configuration logs 0. These four and tq_capture_seconds, tq_persist_seconds, and tq_release_seconds are also registered as rl.checkpoint.duration phases for OTel/Lens. Together with exclusive_hold_seconds and gym_parked_seconds, they show which part of the pause to target next. The data-plane README lists them.

Validation

471 CPU checkpoint/restore/dispatch-race/policy regression tests passed on this rebased stack, including rollout resume/barrier-release, cleanup, trainer capture/publication ordering, cancellation settlement, and synchronous fallback. The suite uses --noconftest with an explicitly bounded two-CPU, zero-GPU local Ray instance for an existing policy metrics test.

The foundation's controller-byte follow-up also passed real Ray/ZMQ concurrent mutation/failure/fresh-actor restore tests and a nine-trial saved-metadata timing comparison in the production container; see #5137. Controller serialization remains inside capture, but now produces immutable bytes directly instead of recursively copying metadata first. Controller-byte and shard persistence happen after resume/barrier release.

The integration commits were rebased onto the lifecycle follow-up in #5137. An additional runtime safeguard stops checkpointing and retains the unpublished directory if capture release cannot be confirmed; scoped Ruff and Pyrefly checks passed for the updated runtime.

The earlier rollout integration has a separate 10-node GPU save/restart/restore smoke. This exact main-based port, trainer extension, and controller-byte follow-up have not had their own GPU smoke. The snapshot-save lock still protects the trainer anchor through publication, so optimizer commits can still wait.

Lifecycle review follow-up

Rebased onto #5137's context-manager, single-attempt persistence, and bounded RPC-wait changes. 1b295537f makes failed cleanup fail closed: if release cannot be confirmed, raise an invariant error and retain unpublished files because remote RPCs may still be writing. The checkpoint pump treats this as fatal rather than retrying a new capture. A new regression verifies both retained files and no publication.

The existing save lock is acquired before the mutation barrier and held through persist/release/publication, so snapshot N+1 cannot capture while N is still persisting. The one-view error remains an invariant, not scheduling flow control.

471 combined CPU checkpoint/restore/dispatch-race/policy regressions passed using the exact pinned Gym checkout and a bounded two-CPU, zero-GPU local Ray instance. The tests use --noconftest. Scoped Ruff and Pyrefly passed; real Ray/ZMQ timeout/late-release/concurrent-mutation/restore validation passed on the foundation. No new GPU allocation was used.

@copy-pr-bot

copy-pr-bot Bot commented Nov 9, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test ff6cd89

@macandro96
macandro96 added this pull request to stack #5140 November 9, 2026 20:06
@macandro96 macandro96 changed the title perf(checkpoint): persist rollout snapshots after environment resume perf(checkpoint): defer rollout and trainer data-plane persistence Nov 9, 2026
@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test e7b0f3a

@macandro96
macandro96 force-pushed the amahishi/async-tq-rollout-snapshots-main branch from e7b0f3a to 5586eb6 Compare November 10, 2026 01:07
@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test 5586eb6

@macandro96
macandro96 force-pushed the amahishi/async-tq-rollout-snapshots-main branch from 5586eb6 to 1b29553 Compare November 10, 2026 23:27
@macandro96 macandro96 added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Nov 10, 2026
@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test 1b29553

@ZhiyuLi-Nvidia

ZhiyuLi-Nvidia commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Thanks for the update. I wrote a CPU test for writers racing a SimpleStorage capture. It is on zhiyul/test/simple-capture-concurrent-writers (git cherry-pick 43b38767b). It runs real Ray/ZMQ actors and uses the real DataPlaneCheckpointBarrier: checkpoint() for capture, mutation(kind) for writers.

  • Unfenced: a clear between the controller RPC and the shard RPCs, with no barrier, leaves k0 listed but not served after restore.
  • Drained: a writer inside mutation() makes checkpoint() wait. The cut contains its clear.
  • Fenced: a writer that asks for mutation() during checkpoint() waits. The restore matches the cut.
  • Stress: 3 of 3 restores match an oracle while an async writer runs.

@macandro96
macandro96 force-pushed the amahishi/async-tq-rollout-snapshots-main branch from 1b29553 to da03aaa Compare November 16, 2026 00:28
Base automatically changed from amahishi/async-simple-tq-snapshots-main to main November 16, 2026 20:20
macandro96 and others added 4 commits November 16, 2026 15:13
…resume

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Capture trainer data-plane views and controller metadata under the mutation barrier, then persist and release the retained view after canonical mutations resume. Wait for checkpoint I/O before model saves and publication; settle cancellation before releasing retained storage.

Preserve synchronous fallback and existing trainer/Gym recovery semantics. Add tests for barrier release, detached cursor state, publication ordering, persistence/release failures, cancellation, and native fallback.

Validation: 395 focused checkpoint/restore tests passed; Ruff and scoped Pyrefly checks passed.
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
…rier

Split the barrier hold left after the TQ capture into the replay inventory
check, rollout-recovery state build, its serialization, and the staging
inventory check, and log each under timing/rollout_checkpoint. Register
these and the TQ capture/persist/release phases for the OTel checkpoint
duration series.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
@macandro96
macandro96 force-pushed the amahishi/async-tq-rollout-snapshots-main branch from 400c3a2 to bb8bc1f Compare November 16, 2026 23:22

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants