Repository navigation
feat(data-plane): add two-phase SimpleStorage checkpoint capture - #5137
Conversation
|
/ok to test 0d775e4 |
0d775e4 to
e23ef83
Compare
|
/ok to test e23ef83 |
|
/ok to test d0044d6 |
…pture Add capture_checkpoint support for the nixl backend, stacked on the SimpleStorage two-phase capture from #5137. Today nixl returns None there, so the caller falls back to the synchronous save. - capture: save the controller, then hold every published blob on each unit (the hold save_shard already uses). - persist: each unit writes unit-<id>.shard and .rows, then manifest.json, then metadata.json last. The files are the same as a synchronous nixl save, so restore needs no new format. - release: drop the holds; idempotent. After capture the caller may put new keys. It must not update or clear a key of the cut until release(). A unit cannot refuse such a call, because TQ marks the keys in the controller before the unit sees a clear. The unit notes the call instead, and persist_shard raises, so no completion marker is written. The SSD tier works with no special code: a hold allocates nothing, and new puts spill to disk while a capture is held. - nixl_storage_unit: split save_shard into _take_holds, _write_shard and _drop_holds (sync behavior unchanged); add capture_shard, persist_shard and release_shard - tq_nixl_checkpoint: add NixlCheckpointCapture and capture_nixl_checkpoint - transfer_queue: capture_checkpoint returns the nixl capture when data_plane.nixl.async_checkpoint is true - interfaces, 2 yaml files, README: the async_checkpoint flag (default false) - tests: bare unit, fake actors, and real Ray and NIXL round trips, with and without the SSD tier Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@oci-aga-slurm-1-dm-02.cm.cluster>
…pture Add capture_checkpoint support for the nixl backend, stacked on the two-phase capture from #5137. Today nixl returns None there, so the caller falls back to the synchronous save. - capture: capture the controller with the SimpleStorage controller capture (tq_controller_checkpoint), then hold every published blob on each unit (the hold save_shard already uses). The controller is not saved inside the barrier. - persist: the controller bytes and each unit's unit-<id>.shard and .rows are written in parallel, then manifest.json, then metadata.json last. The files are the same as a synchronous nixl save, so restore needs no new format. - release: drop the controller bytes and the unit holds; idempotent. After capture the caller may put new keys. It must not update or clear a key of the cut until release(). A unit cannot refuse such a call, because TQ marks the keys in the controller before the unit sees a clear. The unit notes the call instead, and persist_shard raises, so no completion marker is written. The SSD tier works with no special code: a hold allocates nothing, and new puts spill to disk while a capture is held. - nixl_storage_unit: split save_shard into _take_holds, _write_shard and _drop_holds (sync behavior unchanged); add capture_shard, persist_shard and release_shard - tq_nixl_checkpoint: add NixlCheckpointCapture and capture_nixl_checkpoint - transfer_queue: capture_checkpoint returns the nixl capture when data_plane.nixl.async_checkpoint is true; the adapter patches the controller before tq.init - interfaces, 2 yaml files, README: the async_checkpoint flag (default false) - tests: bare unit, fake actors, and real Ray and NIXL round trips, with and without the SSD tier Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@oci-aga-slurm-1-dm-02.cm.cluster>
|
Proposal: support the nixl backend in This PR adds the two-phase capture for SimpleStorage. With Code. Branch How it works
What changed
Contract. After The README here says a capture retains the old values by reference while live puts replace them and clears remove them. That fits SimpleStorage, which copies the dicts. For nixl I propose a stricter rule, for two reasons. The So the nixl unit notes an update or a clear of a key of the cut. Which rule does #5139 enforce? I suggest the rule above for every backend. SimpleStorage tolerates a violation because it copies the dicts, but one rule for callers is simpler to keep correct. SSD tier. It works with no special code. A hold allocates no slab space and copies no payload bytes, so a capture needs no extra slab space. New puts go to RAM first and spill to disk when the put area is full, as they do without a capture. Cost. I compared the barrier time of
At 4 GiB the barrier is 42 times shorter, and capture plus persist takes about the same total time as the sync save (0.53 s against 0.54 s). At 100k keys capture plus persist takes 0.14 s against 0.57 s. Most of that gain comes from the controller capture in Tests. Bare-unit tests cover the cut, new-key puts, and the failure on a clear or an update. Fake-actor tests cover the file layout and the cleanup. Real Ray and NIXL tests cover capture, a new key, persist and restore, with and without the SSD tier. I ran the data-plane test files and Limits
Ask. Please cherry-pick the commit, or apply the diff against this PR, if you want nixl covered here. If you prefer a separate PR after this one merges, tell me and I will open it. |
|
/ok to test 7da2bb0 |
|
@ZhiyuLi-Nvidia Thanks for the implementation and benchmarks. The lifecycle suggestions are addressed in 7da2bb0, with details and validation in the two inline replies. Regarding the NIXL proposal, I'd prefer a separate follow-up PR on top of this one, retaining the stable-view contract rather than requiring callers to prohibit clears. In #5139, the mutation barrier and Gym pause end before persistence starts. Normal terminal/prefix staging generally creates new keys, but resumed processing can clear captured staging/replay keys. The existing save lock prevents overlapping captures; it does not block those live clears. SimpleStorage retains the old values independently, so those clears are supported. Could we extend NIXL's existing blob holds so a clear removes the live row immediately, while physical reclamation waits until snapshot release? Persistence would use the captured row mapping and held blobs, not the current live index. The existing One SSD race needs coverage: |
|
Test coverage: gaps I found, from reading the test bodies Ranked by value and risk:
Items 2 and 3 fit best with the fix for each. Item 4 is cheap and can go in now. |
|
@ZhiyuLi-Nvidia Thanks for the test coverage review. All four are in 65ba789:
|
|
/ok to test 65ba789 |
65ba789 to
46895f1
Compare
|
/ok to test 46895f1 |
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Serialize native controller metadata under its request lock without recursively copying every field. Persist controller bytes alongside retained SimpleStorage shards after the mutation cut, and settle all RPCs before release. Preserve the native restore layout and leave synchronous saves unchanged. Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Address review on the two-phase SimpleStorage capture: - Bound the controller and storage-unit capture waits with checkpoint_io_timeout_s. Capture runs under the caller's mutation barrier, so a frozen actor no longer stalls dispatch indefinitely. - Add CheckpointReleaseError and a token on CheckpointCapture. A failed or cancelled capture whose cleanup release cannot be confirmed now raises it, naming the token, instead of only logging. The cancel path awaits the late release directly, so repeated cancellation cannot hide the failure. - Run the real Ray/ZMQ functional test in L1_Functional_Tests_SingleController_3. - Document host-memory sizing for persist: live data plus twice the unit's share of the cut (sync save: plus one copy). - Test hung capture RPCs, release failures under single and repeated cancellation, and release contending with a running persist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
46895f1 to
f46de05
Compare
|
/ok to test f46de05 |
MetricsDataPlaneClient now forwards capture_checkpoint under its own span, which test_every_emitted_span_name_is_documented requires span-groups.md to name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
|
/ok to test d59ae38 |
ZhiyuLi-Nvidia
left a comment
There was a problem hiding this comment.
Thank you @macandro96.
LGTM!
Stack
PR 1 of 2, targeting main. Rollout-snapshot integration is #5139 and is intentionally not included here. Merge this foundation first, then retarget the integration PR to main.
Summary
DataPlaneClient.capture_checkpointAPI returning a temporary persist/release handle, plus cancellation-safe helpers and metrics-wrapper forwarding. Unsupported adapters returnNonefor synchronous fallback.data_plane.simple.async_checkpoint: falseby default), preserving the native TQ checkpoint format while live PUT/CLEAR operations continue during persistence. Serialize controller metadata directly into immutable bytes under the shared request lock, then persist those bytes and retained storage shards in parallel. Include configuration, documentation, version guards, and storage/lifecycle tests.Scope and limitations
No rollout coordinator or trainer-controller integration changes are included here. TQ controller metadata capture and CPU-only real Ray/ZMQ tests are included. The flag enables the capture API; it does not make existing
save_checkpointcalls asynchronous. Production rollout-snapshot integration is in #5139.Audited against the currently pinned TQ 0.1.11. SSD offload with async capture is rejected. The captured dictionaries retain the original payload references; cleared/replaced old values can temporarily increase host-memory use. Capture must run under the caller's global mutation barrier; the caller owns persistence, release and final publication.
Validation
All nine trials preserved serialized size and passed native restore/value checks. The specialized-copy comparator is not included in this PR. Byte capture avoids the copy entirely. Persistence measurements used local temporary files without fsync; these are not end-to-end GPU rollout-pause measurements.
Scoped Ruff 0.9.9 lint/format and Pyrefly 0.24.2 checks passed. Tests use
--noconftest; the combined suite preinitializes a bounded two-CPU, zero-GPU local Ray instance for an existing policy metrics test. No GPU allocation was required for this follow-up.Lifecycle review follow-up
7da2bb0f4adds context-manager cleanup and a single-attemptpersist()guard, including after failure; exclusive file creation remains unchanged.data_plane.simple.checkpoint_io_timeout_sdefaults to 600 seconds and bounds each persist/release RPC-wait operation.A wait timeout does not cancel remote writes. Their references remain on the handle; release cannot drop a captured view until every write settles. Release is retryable, and an unfinished release must complete before deleting files or starting another capture. No expiring lease is introduced. A completed remote write that itself raises
TimeoutErroris distinguished from an RPC wait expiring.#5139 acquires its save lock before the mutation barrier and holds it through persist/release/publication. Its follow-up
1b295537fstops checkpointing and retains unpublished files if release cannot be confirmed, instead of deleting files while a late RPC may still be writing.Validation on this revision: 39 focused tests and 471 combined CPU checkpoint/restore/dispatch-race/policy regressions passed. Real Ray/ZMQ CPU integration passed with 41 concurrent PUT/CLEAR mutations, fresh-actor restore, forced persist/release timeouts, and successful release retry after late RPC completion. Scoped Ruff and Pyrefly checks passed. These lifecycle changes have not had a new GPU smoke.