Skip to content

feat(data-plane): add two-phase SimpleStorage checkpoint capture - #5137

Merged
ZhiyuLi-Nvidia merged 6 commits into
mainfrom
amahishi/async-simple-tq-snapshots-main
Nov 16, 2026
Merged

ZhiyuLi-Nvidia merged 6 commits into
mainfrom
amahishi/async-simple-tq-snapshots-main

Conversation

@macandro96

@macandro96 macandro96 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Stack

PR 1 of 2, targeting main. Rollout-snapshot integration is #5139 and is intentionally not included here. Merge this foundation first, then retarget the integration PR to main.

Summary

  • Add an optional DataPlaneClient.capture_checkpoint API returning a temporary persist/release handle, plus cancellation-safe helpers and metrics-wrapper forwarding. Unsupported adapters return None for synchronous fallback.
  • Implement stable-view capture for opt-in in-memory SimpleStorage (data_plane.simple.async_checkpoint: false by default), preserving the native TQ checkpoint format while live PUT/CLEAR operations continue during persistence. Serialize controller metadata directly into immutable bytes under the shared request lock, then persist those bytes and retained storage shards in parallel. Include configuration, documentation, version guards, and storage/lifecycle tests.

Scope and limitations

No rollout coordinator or trainer-controller integration changes are included here. TQ controller metadata capture and CPU-only real Ray/ZMQ tests are included. The flag enables the capture API; it does not make existing save_checkpoint calls asynchronous. Production rollout-snapshot integration is in #5139.

Audited against the currently pinned TQ 0.1.11. SSD offload with async capture is rejected. The captured dictionaries retain the original payload references; cleared/replaced old values can temporarily increase host-memory use. Capture must run under the caller's global mutation barrier; the caller owns persistence, release and final publication.

Validation

  • 39 focused CPU unit/regression tests passed for capture lifecycle, native restore, controller patches, failure cleanup, mutation exclusion, and custom partition snapshot behavior. The rebased combined stack passes 471 checkpoint/restore/dispatch-race/policy regression tests.
  • Real Ray/ZMQ CPU integration passed on the main-based implementation and in the production container (Ray 2.58.0, Torch 2.13.0+cu130, TQ 0.1.11): 41 PUT/CLEAR mutations completed while persistence was held; fresh controller/storage actors restored the captured values, not subsequent replacements/clears; capture and persistence failures left no completion marker and did not damage the previous snapshot.
  • CPU replay of the same 82,172,086-byte controller checkpoint, three counterbalanced trials per method:
Controller capture method Median elapsed Median thread CPU
Native recursive copy, before serialization 8.28s 8.28s
Specialized metadata copy, before serialization (comparison only) 5.53s 5.53s
Direct immutable-byte capture, including serialization 0.64s 0.64s

All nine trials preserved serialized size and passed native restore/value checks. The specialized-copy comparator is not included in this PR. Byte capture avoids the copy entirely. Persistence measurements used local temporary files without fsync; these are not end-to-end GPU rollout-pause measurements.

Scoped Ruff 0.9.9 lint/format and Pyrefly 0.24.2 checks passed. Tests use --noconftest; the combined suite preinitializes a bounded two-CPU, zero-GPU local Ray instance for an existing policy metrics test. No GPU allocation was required for this follow-up.

Lifecycle review follow-up

7da2bb0f4 adds context-manager cleanup and a single-attempt persist() guard, including after failure; exclusive file creation remains unchanged. data_plane.simple.checkpoint_io_timeout_s defaults to 600 seconds and bounds each persist/release RPC-wait operation.

A wait timeout does not cancel remote writes. Their references remain on the handle; release cannot drop a captured view until every write settles. Release is retryable, and an unfinished release must complete before deleting files or starting another capture. No expiring lease is introduced. A completed remote write that itself raises TimeoutError is distinguished from an RPC wait expiring.

#5139 acquires its save lock before the mutation barrier and holds it through persist/release/publication. Its follow-up 1b295537f stops checkpointing and retains unpublished files if release cannot be confirmed, instead of deleting files while a late RPC may still be writing.

Validation on this revision: 39 focused tests and 471 combined CPU checkpoint/restore/dispatch-race/policy regressions passed. Real Ray/ZMQ CPU integration passed with 41 concurrent PUT/CLEAR mutations, fresh-actor restore, forced persist/release timeouts, and successful release retry after late RPC completion. Scoped Ruff and Pyrefly checks passed. These lifecycle changes have not had a new GPU smoke.

@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test 0d775e4

@copy-pr-bot

copy-pr-bot Bot commented Nov 9, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@macandro96
macandro96 force-pushed the amahishi/async-simple-tq-snapshots-main branch from 0d775e4 to e23ef83 Compare November 9, 2026 20:03
@macandro96 macandro96 changed the title perf(checkpoint): persist SimpleStorage rollout snapshots after dispatch resumes feat(data-plane): add two-phase SimpleStorage checkpoint capture Nov 9, 2026
@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test e23ef83

@macandro96
macandro96 added this pull request to stack #5140 November 9, 2026 20:06
@macandro96 macandro96 added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Nov 9, 2026
@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test d0044d6

ZhiyuLi-Nvidia pushed a commit that referenced this pull request Nov 10, 2026
…pture

Add capture_checkpoint support for the nixl backend, stacked on the
SimpleStorage two-phase capture from #5137. Today nixl returns None there,
so the caller falls back to the synchronous save.

- capture: save the controller, then hold every published blob on each unit
  (the hold save_shard already uses).
- persist: each unit writes unit-<id>.shard and .rows, then manifest.json,
  then metadata.json last. The files are the same as a synchronous nixl
  save, so restore needs no new format.
- release: drop the holds; idempotent.

After capture the caller may put new keys. It must not update or clear a key
of the cut until release(). A unit cannot refuse such a call, because TQ
marks the keys in the controller before the unit sees a clear. The unit
notes the call instead, and persist_shard raises, so no completion marker is
written.

The SSD tier works with no special code: a hold allocates nothing, and new
puts spill to disk while a capture is held.

- nixl_storage_unit: split save_shard into _take_holds, _write_shard and
  _drop_holds (sync behavior unchanged); add capture_shard, persist_shard
  and release_shard
- tq_nixl_checkpoint: add NixlCheckpointCapture and capture_nixl_checkpoint
- transfer_queue: capture_checkpoint returns the nixl capture when
  data_plane.nixl.async_checkpoint is true
- interfaces, 2 yaml files, README: the async_checkpoint flag (default false)
- tests: bare unit, fake actors, and real Ray and NIXL round trips, with and
  without the SSD tier

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Zhiyu Li <zhiyul@oci-aga-slurm-1-dm-02.cm.cluster>
ZhiyuLi-Nvidia pushed a commit that referenced this pull request Nov 10, 2026
…pture

Add capture_checkpoint support for the nixl backend, stacked on the
two-phase capture from #5137. Today nixl returns None there, so the caller
falls back to the synchronous save.

- capture: capture the controller with the SimpleStorage controller capture
  (tq_controller_checkpoint), then hold every published blob on each unit
  (the hold save_shard already uses). The controller is not saved inside
  the barrier.
- persist: the controller bytes and each unit's unit-<id>.shard and .rows
  are written in parallel, then manifest.json, then metadata.json last.
  The files are the same as a synchronous nixl save, so restore needs no new
  format.
- release: drop the controller bytes and the unit holds; idempotent.

After capture the caller may put new keys. It must not update or clear a key
of the cut until release(). A unit cannot refuse such a call, because TQ
marks the keys in the controller before the unit sees a clear. The unit
notes the call instead, and persist_shard raises, so no completion marker is
written.

The SSD tier works with no special code: a hold allocates nothing, and new
puts spill to disk while a capture is held.

- nixl_storage_unit: split save_shard into _take_holds, _write_shard and
  _drop_holds (sync behavior unchanged); add capture_shard, persist_shard
  and release_shard
- tq_nixl_checkpoint: add NixlCheckpointCapture and capture_nixl_checkpoint
- transfer_queue: capture_checkpoint returns the nixl capture when
  data_plane.nixl.async_checkpoint is true; the adapter patches the
  controller before tq.init
- interfaces, 2 yaml files, README: the async_checkpoint flag (default false)
- tests: bare unit, fake actors, and real Ray and NIXL round trips, with and
  without the SSD tier

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Zhiyu Li <zhiyul@oci-aga-slurm-1-dm-02.cm.cluster>
@ZhiyuLi-Nvidia

Copy link
Copy Markdown
Contributor

Proposal: support the nixl backend in capture_checkpoint

This PR adds the two-phase capture for SimpleStorage. With backend: nixl, capture_checkpoint returns None today, so the caller falls back to the synchronous save. I implemented the same API for nixl on top of the current head (d0044d63a). It can be picked up into this PR, or land right after it.

Code. Branch zhiyul/feat/nixl-async-capture, one commit (337bbc154) on top of d0044d63a. 11 files, about 270 source lines and 330 test lines.

How it works

capture  (caller's barrier)  capture the controller (tq_controller_checkpoint),
                             then hold every published blob on each unit
persist  (barrier open)      write controller_state.pkl and each unit's
                             unit-<id>.shard and .rows in parallel,
                             then manifest.json, then metadata.json last
release                      drop the controller bytes and the unit holds (idempotent)

persist writes the same files as the synchronous nixl save. Restore needs no new format, and it works onto a different number of units.

What changed

  • nixl_storage_unit.py: save_shard is split into _take_holds, _write_shard and _drop_holds. The sync save behaves as before. New unit methods: capture_shard, persist_shard, release_shard.
  • tq_nixl_checkpoint.py: NixlCheckpointCapture and capture_nixl_checkpoint. The controller side reuses SnapshotControllerMixin from this PR.
  • transfer_queue.py: capture_checkpoint returns the nixl capture when data_plane.nixl.async_checkpoint is true. In that case the adapter calls configure_controller_checkpoint() before tq.init, as it does for SimpleStorage.
  • interfaces.py, two yaml files, README: the async_checkpoint: false flag. It is opt-in.

Contract. After capture, the caller may put new keys. The caller must not update or clear a key of the cut until release().

The README here says a capture retains the old values by reference while live puts replace them and clears remove them. That fits SimpleStorage, which copies the dicts. For nixl I propose a stricter rule, for two reasons. The save_shard docstring calls a clear during a checkpoint outside TQ's contract. And a unit cannot refuse a clear: kv_clear calls async_clear_samples, which first marks the keys as pending deletion in the controller, then clears the storage units, then releases the controller metadata (TQ at the pinned commit, client.py L603-L610). A refusal at the unit would leave the keys marked but not cleared.

So the nixl unit notes an update or a clear of a key of the cut. persist_shard raises if that note is set when its final check runs, after the shard write. When it raises, no metadata.json is written. A call that arrives after that check is not detected, so the caller must hold off until release().

Which rule does #5139 enforce? I suggest the rule above for every backend. SimpleStorage tolerates a violation because it copies the dicts, but one rule for callers is simpler to keep correct.

SSD tier. It works with no special code. A hold allocates no slab space and copies no payload bytes, so a capture needs no extra slab space. New puts go to RAM first and spill to disk when the put area is full, as they do without a capture.

Cost. I compared the barrier time of tq.save_checkpoint with the capture, on this PR's head. I used 1 node, 3 units, and the shared filesystem. Each number is the median of 3 alternating runs, with the range in brackets.

Data save_checkpoint capture barrier persist (barrier open)
4 GiB (4096 keys of 1 MiB) 0.54 s (0.50 to 0.69) 0.013 s (0.013 to 0.014) 0.52 s
149 MiB (100k keys, 5 fields) 0.57 s (0.56 to 0.65) 0.066 s (0.065 to 0.071) 0.08 s

At 4 GiB the barrier is 42 times shorter, and capture plus persist takes about the same total time as the sync save (0.53 s against 0.54 s). At 100k keys capture plus persist takes 0.14 s against 0.57 s. Most of that gain comes from the controller capture in d0044d63a, not from the split into two phases. Both paths wrote shards of the same total size, to the nearest MiB, in every run.

Tests. Bare-unit tests cover the cut, new-key puts, and the failure on a clear or an update. Fake-actor tests cover the file layout and the cleanup. Real Ray and NIXL tests cover capture, a new key, persist and restore, with and without the SSD tier. I ran the data-plane test files and test_config_v2.py in a container with TQ 0.1.11. 153 tests passed, including the existing nixl and SimpleStorage tests. ruff and pyrefly report no errors.

Limits

  • I tested on one node. Units on different nodes writing to a shared filesystem are not tested. The sync save has the same shape.
  • I did not measure how a persist affects put latency on the SSD tier.
  • Nothing calls the nixl capture yet. It needs the caller in perf(checkpoint): defer rollout and trainer data-plane persistence #5139. The flag defaults to false.
  • _wait_all and the release logic copy tq_simple_checkpoint.py. A shared version in checkpoint.py would remove about 25 lines. I did not edit your files.

Ask. Please cherry-pick the commit, or apply the diff against this PR, if you want nixl covered here. If you prefer a separate PR after this one merges, tell me and I will open it.

@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test 7da2bb0

@macandro96

macandro96 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor Author

@ZhiyuLi-Nvidia Thanks for the implementation and benchmarks. The lifecycle suggestions are addressed in 7da2bb0, with details and validation in the two inline replies.

Regarding the NIXL proposal, I'd prefer a separate follow-up PR on top of this one, retaining the stable-view contract rather than requiring callers to prohibit clears.

In #5139, the mutation barrier and Gym pause end before persistence starts. Normal terminal/prefix staging generally creates new keys, but resumed processing can clear captured staging/replay keys. The existing save lock prevents overlapping captures; it does not block those live clears. SimpleStorage retains the old values independently, so those clears are supported.

Could we extend NIXL's existing blob holds so a clear removes the live row immediately, while physical reclamation waits until snapshot release? Persistence would use the captured row mapping and held blobs, not the current live index. The existing holds / _released_held machinery looks like a good starting point. With that guarantee validated, clears/updates would not need to poison the capture via _cut_disturbed.

One SSD race needs coverage: _stage() currently sets rec.disk = None and closes the file when a blob has disappeared from the live blob table, without consulting snapshot holds. If a clear happens while staging is in flight, the snapshot needs the file retained or its own descriptor before that close.

Comment thread nemo_rl/data_plane/README.md Resolved
@ZhiyuLi-Nvidia

Copy link
Copy Markdown
Contributor

Test coverage: gaps I found, from reading the test bodies

Ranked by value and risk:

  1. No real-actor run in CI. tests/functional/data_plane_simple_checkpoint_cpu.py runs real Ray actors, the bootstrap patch order on the real TQ classes, and real ZMQ. The diff shows no CI wiring for it, and the README says to run it by hand. The unit tests use an in-process fake cluster. They do cover the controller _handle_request wrapper.
  2. A release that fails during a cancelled or failed capture. test_cancelled_capture_releases_late_result and test_partial_capture_failure_releases_every_unit use a release that always succeeds. Release failures are tested only inside SimpleCheckpointCapture. They are not tested in the capture_checkpoint cancel path or in the except block of capture_simple_checkpoint, where a failed release is logged and swallowed. That can leave a retained view with no handle, and the next capture then fails with the one-view error.
  3. A hung capture RPC. Every timeout test passes timeout_s to persist or release. The capture waits have no timeout (see my inline comment on tq_simple_checkpoint.py:327). A bounded test can spy on the timeout argument of ray.get and _wait_all and fail at once when it is None, for both the controller and the shard waits. It needs no real hang.
  4. Concurrent persist() and release(). The _operation_lock contention path ("operation is still running") is never exercised.

Items 2 and 3 fit best with the fix for each. Item 4 is cheap and can go in now.

Comment thread nemo_rl/data_plane/checkpoint.py Resolved
@macandro96

Copy link
Copy Markdown
Contributor Author

@ZhiyuLi-Nvidia Thanks for the test coverage review. All four are in 65ba789:

  1. CI: tests/functional/data_plane_simple_checkpoint_cpu.sh wraps the script (under coverage) and runs in L1_Functional_Tests_SingleController_3, next to the TQ recovery tests. CPU-only, ~26 s.
  2. Release failure during a failed or cancelled capture: now raises CheckpointReleaseError (details in the inline thread), tested for capture failure, single cancellation and repeated cancellation.
  3. Hung capture: waits are bounded; the test spies on timeout and fails at once if it is None, for both the controller and the shard waits.
  4. Contention: test_release_during_persist_reports_running_operation calls release() while persist() holds the lock, expects "still running" with the views kept, and checks both complete afterwards.

@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test 65ba789

@macandro96
macandro96 force-pushed the amahishi/async-simple-tq-snapshots-main branch from 65ba789 to 46895f1 Compare November 14, 2026 00:30
@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test 46895f1

macandro96 and others added 4 commits November 15, 2026 16:20
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Serialize native controller metadata under its request lock without recursively copying every field. Persist controller bytes alongside retained SimpleStorage shards after the mutation cut, and settle all RPCs before release. Preserve the native restore layout and leave synchronous saves unchanged.

Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Address review on the two-phase SimpleStorage capture:

- Bound the controller and storage-unit capture waits with
  checkpoint_io_timeout_s. Capture runs under the caller's mutation
  barrier, so a frozen actor no longer stalls dispatch indefinitely.
- Add CheckpointReleaseError and a token on CheckpointCapture. A failed
  or cancelled capture whose cleanup release cannot be confirmed now
  raises it, naming the token, instead of only logging. The cancel path
  awaits the late release directly, so repeated cancellation cannot hide
  the failure.
- Run the real Ray/ZMQ functional test in
  L1_Functional_Tests_SingleController_3.
- Document host-memory sizing for persist: live data plus twice the
  unit's share of the cut (sync save: plus one copy).
- Test hung capture RPCs, release failures under single and repeated
  cancellation, and release contending with a running persist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
@macandro96
macandro96 force-pushed the amahishi/async-simple-tq-snapshots-main branch from 46895f1 to f46de05 Compare November 16, 2026 00:21
@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test f46de05

MetricsDataPlaneClient now forwards capture_checkpoint under its own
span, which test_every_emitted_span_name_is_documented requires
span-groups.md to name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Nov 16, 2026
@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test d59ae38

@ZhiyuLi-Nvidia ZhiyuLi-Nvidia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @macandro96.

LGTM!

@ZhiyuLi-Nvidia ZhiyuLi-Nvidia added the CI:L1 Run doctests, unit tests, and functional tests label Nov 16, 2026
@ZhiyuLi-Nvidia
ZhiyuLi-Nvidia merged commit ae3a80e into main Nov 16, 2026
167 of 169 checks passed
@ZhiyuLi-Nvidia
ZhiyuLi-Nvidia deleted the amahishi/async-simple-tq-snapshots-main branch November 16, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L1 Run doctests, unit tests, and functional tests CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants