Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/observability/span-groups.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ One consequence worth knowing: registration is an *import side effect*, so a gro
| `advantage` | RL | `rl.<algo>.advantage_calculation` |
| `policy_update` | RL | `rl.<algo>.policy_training` (and `rl.ppo.value_training` for PPO) |
| `data_processing` | RL | `rl.<algo>.data_processing` |
| `data_plane` | RL | `rl.data_plane.<op>` — one span per transfer-queue operation from a batch-shaped caller. The eleven ops give `rl.data_plane.register`, `rl.data_plane.claim_meta`, `rl.data_plane.get_data`, `rl.data_plane.check_consumption_status`, `rl.data_plane.put`, `rl.data_plane.get`, `rl.data_plane.list_sample_ids`, `rl.data_plane.clear`, `rl.data_plane.save_checkpoint`, `rl.data_plane.load_checkpoint`, `rl.data_plane.close`. The nixl backend's checkpoint restore adds `rl.data_plane.restore_shards`, which covers reading the saved shards back into the store |
| `data_plane` | RL | `rl.data_plane.<op>` — one span per transfer-queue operation from a batch-shaped caller. The twelve ops give `rl.data_plane.register`, `rl.data_plane.claim_meta`, `rl.data_plane.get_data`, `rl.data_plane.check_consumption_status`, `rl.data_plane.put`, `rl.data_plane.get`, `rl.data_plane.list_sample_ids`, `rl.data_plane.clear`, `rl.data_plane.capture_checkpoint`, `rl.data_plane.save_checkpoint`, `rl.data_plane.load_checkpoint`, `rl.data_plane.close`. The nixl backend's checkpoint restore adds `rl.data_plane.restore_shards`, which covers reading the saved shards back into the store |
| `per_prompt` | RL | spans emitted once per prompt: `rl.sc.generate_and_push`, the rollout path's `rl.data_plane.put`, and `rl.gym.run_rollouts` when the single controller dispatches it. A cardinality axis rather than a phase — see [Per-prompt spans](#per-prompt-spans) |
| `efficiency` | RL | idle phases on async GRPO — driver-side `rl.idle.buffer_starvation`, `rl.idle.refit_bubble`, and collector-side `rl.idle.refit_event_wait`, `rl.idle.generation_limit_pause` |

Expand Down
2 changes: 2 additions & 0 deletions examples/configs/grpo_math_1B.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -724,6 +724,8 @@ data_plane:
# absent block means that backend's defaults (see SimpleStorageConfig /
# MooncakeCpuConfig in nemo_rl/data_plane/interfaces.py).
simple:
async_checkpoint: false # enable the optional stable-view capture API; RAM-only
checkpoint_io_timeout_s: 600.0 # RPC budget per retained-view persist/release
storage_capacity: 1000000 # max samples retained per partition
num_storage_units: ${mul:2, ${cluster.num_nodes}} # TQ wants >= 2 per node
storage_unit_placement: null # all, or cluster names like [inference]; null = TQ SPREAD
Expand Down
2 changes: 2 additions & 0 deletions examples/configs/ppo_math_1B_megatron_single_controller.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,8 @@ data_plane:
# absent block means that backend's defaults (see SimpleStorageConfig /
# MooncakeCpuConfig in nemo_rl/data_plane/interfaces.py).
simple:
async_checkpoint: false # enable the optional stable-view capture API; RAM-only
checkpoint_io_timeout_s: 600.0 # RPC budget per retained-view persist/release
storage_capacity: 1000000 # max samples retained per partition
num_storage_units: ${mul:2, ${cluster.num_nodes}} # TQ wants >= 2 per node
storage_unit_placement: null # all, or cluster names like [inference]; null = TQ SPREAD
Expand Down
70 changes: 70 additions & 0 deletions nemo_rl/data_plane/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -741,6 +741,76 @@ For the no-data-plane path use the legacy `nemo_rl.algorithms.grpo.grpo_train`;

---

## Two-phase data-plane checkpoints

| Key | Default | Behavior |
|---|---|---|
| `data_plane.simple.async_checkpoint` | `false` | Enable the optional stable-view capture API for in-memory SimpleStorage. |
| `data_plane.simple.checkpoint_io_timeout_s` | `600.0` | Positive, finite RPC wait budget per retained-view persist/release operation. Does not interrupt filesystem calls. |

`DataPlaneClient.capture_checkpoint` returns a temporary handle or `None` when
the adapter requests synchronous fallback. The caller must hold a global mutation
barrier while capturing. After capture returns, it may resume live traffic,
call `persist()`, and then `release()` after all writes settle, including on
failure or cancellation. Publication remains the caller's responsibility. If a
failed or cancelled capture cannot confirm its own release, it raises
`CheckpointReleaseError` naming the capture token, even past repeated
cancellation; the caller has no handle to retry with and must stop
checkpointing.

This API requires caller integration: setting the flag alone does not change
the existing synchronous `save_checkpoint` calls. Other backends return `None`.
SimpleStorage capture requires SSD offload disabled and writes the existing
native TQ checkpoint format, so restore does not need a new format.

Capture copies storage indexes, not tensor payloads. References retain the old
values while live puts replace them and clears remove them, until `release()`.
Persistence pickles each shard like the synchronous save, which holds a
serialized copy of the shard in the storage actor until the write ends. Size a
storage unit's host memory for live data plus twice its share of the cut: one
copy retained and one being serialized. The synchronous save needs live data
plus one copy. This contract
is audited against the pinned TQ 0.1.11 implementation; a dependency update needs
the same audit. Only one snapshot may be retained at a time.

Controller capture serializes the native checkpoint layout into immutable bytes
under a lock shared with the ZMQ request dispatcher. It avoids recursively
copying the metadata before serialization. Unknown partition subclasses retain
their native snapshot behavior. Controller and shard persistence run in parallel
after the caller resumes traffic; neither serializes live controller state or
returns checkpoint payloads through Ray. The controller bytes remain resident
until the capture handle is released. File creation is exclusive, and the
completion marker is written only after all persistence RPCs succeed.

Use the capture as a context manager, or always call `release()` in `finally`.
`persist()` is single-attempt, including after a partial failure. Reusing the
handle raises a clear error; release it, discard the failed directory, and
capture again. Exclusive file creation remains an overwrite guard.

```python
capture = client.capture_checkpoint(checkpoint_dir, metadata=metadata)
if capture is not None:
with capture:
capture.persist()
```

A timeout raises `TimeoutError` but does not cancel remote RPCs. The handle
retains their references, and `release()` waits for all persistence RPCs to
settle before releasing any view. If release also times out, keep the handle
and retry release, or terminate the job; do not delete/reuse the temporary
directory or start another capture until release succeeds. Views have no lease
expiry. The caller must wait for snapshot N's persistence and release before
taking snapshot N+1's mutation barrier; the one-view error is an invariant,
not scheduling flow control.

CPU-only real Ray/ZMQ validation, including concurrent PUT/CLEAR requests,
failure cleanup and restore into fresh actors, runs in
`L1_Functional_Tests_SingleController_3`. To run it locally:

```bash
bash tests/functional/data_plane_simple_checkpoint_cpu.sh
```

## Async path (proposed)

The data-plane interface covers both sync and async, but the **sync trainer uses only half of it**. The task-mediated half (`claim_meta` / `get_data` / `check_consumption_status`) is reserved for the async trainer, which is not yet wired into production.
Expand Down
137 changes: 137 additions & 0 deletions nemo_rl/data_plane/adapters/tq_controller_checkpoint.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Immutable controller checkpoint bytes for pinned TransferQueue 0.1.11.

Capture serializes the native checkpoint layout under a lock shared with the
controller's ZMQ request dispatcher. Persistence writes only the retained bytes
without excluding live requests. Callers enforce a global data-plane mutation
cut across controller and storage capture. No metadata payload crosses Ray.
"""

from __future__ import annotations

import logging
import pickle
import threading
import time
from typing import Any

import ray
import transfer_queue.interface as interface
from transfer_queue.controller import DataPartitionStatus

from nemo_rl.data_plane.adapters.tq_controller_patch import assert_stock_tq_version

LOGGER = logging.getLogger(__name__)


class SnapshotControllerMixin:
"""Keep one immutable metadata cut while live request traffic continues."""

def __init__(self, *args: Any, **kwargs: Any) -> None:
self._controller_snapshot_lock = threading.RLock()
self._controller_snapshot_views: dict[str, bytes] = {}
super().__init__(*args, **kwargs)

def _handle_request(self, request_msg: Any, monitor: Any) -> Any:
with self._controller_snapshot_lock:
return super()._handle_request(request_msg, monitor)

def load_checkpoint(self, path: str) -> None:
with self._controller_snapshot_lock:
if self._controller_snapshot_views:
raise RuntimeError(
"Cannot restore while a controller snapshot is retained"
)
super().load_checkpoint(path)

def nrl_capture_snapshot(self, token: str) -> str:
"""Serialize live metadata under the request lock; return only its ID."""
started, cpu_started = time.monotonic(), time.thread_time()
with self._controller_snapshot_lock:
acquired = time.monotonic()
if self._controller_snapshot_views:
raise RuntimeError("Only one controller snapshot may be retained")
state = {
"controller_id": self.controller_id,
"partitions": {
pid: partition
if type(partition) is DataPartitionStatus
else partition.to_snapshot()
for pid, partition in self.partitions.items()
},
"index_manager": {
"partition_to_indexes": dict(
self.index_manager.partition_to_indexes
),
"reusable_indexes": list(self.index_manager.reusable_indexes),
"global_index_counter": self.index_manager.global_index_counter,
"allocated_indexes": self.index_manager.allocated_indexes,
},
"sampler": self.sampler.save_checkpoint(),
}
layout_finished = time.monotonic()
payload = pickle.dumps(state, protocol=pickle.HIGHEST_PROTOCOL)
serialized = time.monotonic()
self._controller_snapshot_views[token] = payload
LOGGER.info(
"controller_snapshot captured token=%s lock_wait_s=%.6f layout_s=%.6f "
"serialize_s=%.6f thread_cpu_s=%.6f bytes=%d total_s=%.6f",
token,
acquired - started,
layout_finished - acquired,
serialized - layout_finished,
time.thread_time() - cpu_started,
len(payload),
time.monotonic() - started,
)
return self.controller_id

def nrl_persist_snapshot(self, token: str, path: str) -> int:
"""Write captured bytes without holding the live-request lock."""
with self._controller_snapshot_lock:
payload = self._controller_snapshot_views[token]
started, cpu_started = time.monotonic(), time.thread_time()
with open(path, "xb") as output:
output.write(payload)
LOGGER.info(
"controller_snapshot persisted token=%s seconds=%.6f thread_cpu_s=%.6f bytes=%d",
token,
time.monotonic() - started,
time.thread_time() - cpu_started,
len(payload),
)
return len(payload)

def nrl_release_snapshot(self, token: str) -> None:
"""Idempotently release the immutable payload after writes settle."""
with self._controller_snapshot_lock:
payload = self._controller_snapshot_views.pop(token, None)
del payload


def configure_controller_checkpoint() -> None:
"""Wrap the controller before initialization, retaining existing patches."""
assert_stock_tq_version()
actor_class = interface.TransferQueueController
base = actor_class.__ray_metadata__.modified_class
if issubclass(base, SnapshotControllerMixin):
return

class SnapshotController(SnapshotControllerMixin, base): # pragma: no cover
pass

interface.TransferQueueController = ray.remote(**actor_class._default_options)(
SnapshotController
)
Loading
Loading