Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/guides/single-controller.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,7 @@ If one shard replica fails restore, every replica conservatively retires its rep

If no rollout snapshot follows the newest trainer checkpoint, SC restores that checkpoint on its own: completed groups come back from TQ and unfinished turn- and prefix-level rollouts restart from their tasks. A rollout snapshot without its `gym_checkpoint.json` is refused.

The `prefix` target needs everything the `turn` target needs (listed below), and does not yet support multimodal capture, router replay, or `policy.generation.top_logprobs > 0`. Each cut must finish within `token_capture.control_timeout_s`; a call that is not cut in time restarts from its last saved turn. Some calls are never cut and also restart from their last saved turn:
The `prefix` target needs everything the `turn` target needs (listed below), and does not yet support router replay or `policy.generation.top_logprobs > 0`. Each cut must finish within `token_capture.control_timeout_s`; a call that is not cut in time restarts from its last saved turn. Some calls are never cut and also restart from their last saved turn:

- Calls using stop strings, multi-token `bad_words`, or a per-request `seed` while sampling, because a resumed call could not apply them to the tokens it already decoded.
- Calls with structured output (a non-text `response_format`, `guided_*` or `structured_outputs`) or a `tool_choice` other than `auto` or `none`, which NeMo Gym leaves uncut.
Expand Down Expand Up @@ -450,6 +450,7 @@ The SC path is still under active development. Feature gaps are tracked in [issu
RL hands the owned tensors (`imgs`, `imgs_sizes`, and optional `num_frames`) to Gym's `complete_call_from_response` as opaque attachments, and the TQ sink writes them in the same `put` as the token columns, so `staged` coordinates acknowledge tokens and pixels together and a failed write is `capture_failed` at call time. Tensors keep their native shapes and dtypes on the wire; two per-row flags (`media_present`, `media_has_frames`) mark which rows carry pixels and whether they are video. Media-enabled staging partitions carry these columns on every row; text-only runs register and read none of them.
vLLM pixels are rearranged losslessly into packed patches; the finalizer reads the presence flags with the base columns, then issues one batched read of the tensor columns for the terminal-chain calls that carry media, and publishes `pixel_values`, `imgs_sizes`, and `num_frames` for the existing Megatron learner without resampling or normalizing the media again.
Only newly introduced occurrences are staged. vLLM-specific `media_spans` extras retain placeholder positions and token hashes for multi-turn prefix replacement, including video's timestamp-separated visual-token spans.
With `rollout_recovery.target_level: prefix`, the first durable prefix row for a model call carries that call's new media metadata and a SHA-256 digest of each packed tensor, not the pixels; later token chunks carry no media. The prefix digest covers that metadata, so a restore checks it with the prefix, then reprocesses the reissued request, checks its media against those tensor digests, and stages the reprocessed tensors in the canonical terminal row. A prefix that fails its digest check makes the call regenerate from its turn boundary, like any unusable prefix; reprocessed media that does not match the digests is rejected with `retained_media_changed`. The original image or video must therefore still be resolvable by vLLM when the request is reissued: this path does not provide source-free media prefill if the original media has disappeared.
Capture requests use vLLM's `skip_mm_cache=True` path to obtain concrete processor tensors. vLLM can still reuse its processor-only cache; this does not guarantee fresh preprocessing. vLLM tiles images from a dummy prompt, so the request text never changes geometry, but the images of one request share the token budget: adding an image under a tight budget can re-tile a retained one, and a warm processor-only cache can keep a geometry a fresh processor would not reproduce. If that changes retained geometry or placeholder tokens, the worker rejects the continuation before inference with HTTP 400 and error code `retained_media_changed` (other capture-time validation failures use `media_capture_rejected`). Gym's current exception middleware wraps the upstream error in HTTP 500, retaining the code in the response body. RL then classifies it as an infrastructure `GymTransportError`, eligible for the configured prompt retry policy; it is not a dedicated terminal rejection.
Text-call rows carry sentinels in each column's own dtype, because TransferQueue keeps one dtype per field across live rows.
Media-enabled rows also store `media_metadata_digest`, a SHA-256 checksum of the small route-less extras JSON. Sources check it before exposing media descriptors, without reading pixels or routed-expert tensors. Missing or changed checksums reject the row. This detects accidental metadata corruption; it is independent of Gym's receipt and combined `extras_digest` commitment.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,35 @@
# generation, a Megatron learner, and vLLM media token capture enabled.
#
# Functional-test config (not a nightly recipe, so it lives beside the other
# Gym smoke configs rather than under examples/configs/recipes/):
# tests/functional/nemotron_omni_clevr_vllm_token_capture_single_controller_1n2g.sh
# supplies the data paths, the 1 trainer + 1 generator split, and the step
# budget. Token capture requires the Gym rollout path, so the native CLEVR
# Gym smoke configs rather than under examples/configs/recipes/). It pins the
# 1n2g split both Omni CLEVR vLLM tests use: one GPU trains the frozen-decoder
# policy, one hosts async vLLM. The tests
# (tests/functional/nemotron_omni_clevr_vllm_*_single_controller_1n2g.sh)
# supply the data paths, sequence lengths and step budget. Token capture
# requires the Gym rollout path, so the native CLEVR
# dataset/environment of the parent recipe are swapped for Gym rows here.
# Prefix caching stays off (hybrid-Mamba align cache mode asserts chunked
# prefill) and the parent's mm_processor_cache_gb=0 keeps concrete pixels
# available to the capture host.
defaults: ../configs/recipes/vlm/vlm_grpo-nemotron-omni-30ba3b-clevr-8n4g-megatron-single-controller-async.v1.yaml
cluster:
num_nodes: 1
gpus_per_node: 2
segment_size: 1
grpo:
val_period: 0
val_at_start: false
val_at_end: false
async_rl:
generation_fleet_health:
refit_timeout_s: null
data_plane:
simple:
num_storage_units: 2
logger:
tensorboard_enabled: true
wandb_enabled: false
monitor_gpus: false
data:
_override_: true
max_input_seq_length: null
Expand Down Expand Up @@ -52,10 +72,39 @@ env:
simple_agent:
max_steps: 1
policy:
train_micro_batch_size: 1
megatron_cfg:
tensor_model_parallel_size: 1
expert_model_parallel_size: 1
expert_tensor_parallel_size: 1
context_parallel_size: 1
sequence_parallel: true
activation_checkpointing: true
bias_dropout_fusion: false
freeze_config:
freeze_language_model: true
optimizer:
optimizer_cpu_offload: false
optimizer_offload_fraction: 0.0
params_dtype: bfloat16
main_grads_dtype: bfloat16
main_params_dtype: float16
exp_avg_dtype: bfloat16
exp_avg_sq_dtype: bfloat16
store_param_remainders: false
generation:
backend: vllm
refit_transport: null
colocated:
enabled: false
resources:
num_nodes: 1
gpus_per_node: 1
vllm_cfg:
tensor_parallel_size: 1
expert_parallel_size: 1
pipeline_parallel_size: 1
gpu_memory_utilization: 0.6
async_engine: true
expose_http_server: true
logprobs_mode: raw_logprobs
Expand Down
4 changes: 0 additions & 4 deletions nemo_rl/algorithms/single_controller_utils/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -1347,10 +1347,6 @@ def setup_single_controller(
)
if generation_prefix_cuts_enabled:
# The vLLM-only check ran above with the other turn/prefix checks.
if capture_media:
raise NotImplementedError(
"generation-prefix recovery does not yet support multimodal capture"
)
if router_replay_enabled(master_config.policy):
raise NotImplementedError(
"generation-prefix recovery does not yet support router replay"
Expand Down
60 changes: 59 additions & 1 deletion nemo_rl/data/captured_media.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@

import hashlib
import json
from collections.abc import Mapping
from dataclasses import asdict, dataclass, replace
from typing import TYPE_CHECKING, Any, Literal

Expand Down Expand Up @@ -138,10 +139,67 @@ def verify_tokens(self, tokens: list[int], *, origin: int) -> None:

@dataclass(frozen=True)
class CapturedMedia:
"""New placeholder metadata and owned tensors for the common staging sink."""
"""New placeholder metadata and owned tensors for the common staging sink.

``tensor_digests`` fingerprints ``tensors`` (see ``media_tensor_digests``)
when generation-prefix cuts are on. A media bundle restored from a prefix
row carries the staged digests and no tensors.
"""

items: tuple[CapturedMediaItem, ...]
tensors: dict[str, torch.Tensor] | None
tensor_digests: dict[str, dict[str, Any]] | None = None


# Generation-prefix extras key for the call's tensor digests. Prefix rows store
# these instead of the pixels: a restore reprocesses the media anyway and only
# has to prove it produced the bytes the interrupted call staged.
MEDIA_TENSOR_DIGESTS_FIELD = "media_tensor_digests"


def media_tensor_digests(
tensors: Mapping[str, torch.Tensor] | None,
) -> dict[str, dict[str, Any]]:
"""Fingerprint each media tensor by its dtype, shape, and raw bytes."""
digests: dict[str, dict[str, Any]] = {}
for name, value in sorted((tensors or {}).items()):
tensor = value.detach().cpu().contiguous()
raw = tensor.reshape(-1).view(torch.uint8).numpy()
digests[name] = {
"dtype": str(tensor.dtype),
"shape": list(tensor.shape),
"sha256": hashlib.sha256(raw).hexdigest(),
}
return digests


def verify_restored_media(captured: CapturedMedia, restored: CapturedMedia) -> None:
"""Require reprocessed media to match what an interrupted call staged.

Args:
captured: The media just reprocessed from the resumed request.
restored: The items and tensor digests read from the call's first
generation-prefix row.

Raises:
MediaCaptureRejected: ``retained_media_changed`` when the items or
any tensor's dtype, shape, or bytes differ.
"""
current = captured.tensor_digests or {}
staged = restored.tensor_digests or {}
if captured.items != restored.items or current != staged:
# Name what differs, so a rejection can be triaged from the log.
changed = sorted(
name
for name in current.keys() | staged.keys()
if current.get(name) != staged.get(name)
)
raise MediaCaptureRejected(
"Restored generation media differs from reprocessed input "
f"(items equal: {captured.items == restored.items}; "
f"changed tensors: {changed})",
code="retained_media_changed",
)


def verified_media_items(call: FetchedStagedCall) -> tuple[CapturedMediaItem, ...]:
Expand Down
8 changes: 7 additions & 1 deletion nemo_rl/models/generation/prefix_recovery/parts.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,10 @@
from typing import TYPE_CHECKING, Any

from nemo_rl.models.generation.prefix_recovery.progress import ProgressReader
from nemo_rl.models.generation.prefix_recovery.state import CaptureRegistry
from nemo_rl.models.generation.prefix_recovery.state import (
CaptureRegistry,
_RequestCaptureState,
)

if TYPE_CHECKING:
from nemo_rl.data_plane.tq_token_sink import TQTokenSink, TQTokenSource
Expand Down Expand Up @@ -49,3 +52,6 @@ class PrefixRecoveryParts:
# Bound on one cut, measured from its arrival; None waits indefinitely.
control_timeout_s: float | None
progress_reader: ProgressReader
# Extras for a call's first prefix row, from the media its capture state
# holds; None for a call without media.
prefix_row_extras: Callable[[_RequestCaptureState], dict[str, Any] | None]
19 changes: 15 additions & 4 deletions nemo_rl/models/generation/prefix_recovery/recovery.py
Original file line number Diff line number Diff line change
Expand Up @@ -519,6 +519,7 @@ def _cut_active(
prompt_token_ids=state.prompt_token_ids,
generated_token_ids=generated_token_ids,
generated_logprobs=frozen.generated_logprobs,
extras=self._parts.prefix_row_extras(state),
)
if frozen.chunk_token_ids:
chunk_record = (
Expand Down Expand Up @@ -670,9 +671,19 @@ def _rebuild_prefix(self, admission: Any, prefix_token_ids: list[int]) -> Any:
continuation = admission.generation_cut
# Gym requires at least one staging key per continuation.
staging_keys = list(continuation.staging_keys)
snapshots = self._parts.staging_source.fetch(staging_keys)
if len(snapshots) != len(staging_keys):
# The finalization read also returns each row's extras, which carry a
# media call's spans and tensor digests; plain ``fetch`` does not.
chunks = self._parts.staging_source.fetch_for_finalization(staging_keys)
if len(chunks) != len(staging_keys):
raise RuntimeError("generation-cut fetch did not return every staged chunk")
snapshots = [chunk.snapshot for chunk in chunks]
# A media call's spans and tensor digests ride the first chunk's extras,
# and the advertised prefix digest covers them.
if any(chunk.extras is not None for chunk in chunks[1:]):
raise RuntimeError(
"only the first generation-cut chunk may carry call extras"
)
extras = chunks[0].extras
weight_versions = [snapshot.weight_version for snapshot in snapshots]
if weight_versions != sorted(weight_versions):
raise RuntimeError(
Expand Down Expand Up @@ -732,8 +743,8 @@ def _rebuild_prefix(self, admission: Any, prefix_token_ids: list[int]) -> Any:
token_ids_delta=token_ids_delta,
token_mask_delta=token_mask_delta,
generation_log_probs_delta=generation_logprobs_delta,
extras=None,
extras_digest=compute_extras_digest(None),
extras=extras,
extras_digest=compute_extras_digest(extras),
chain_hash=compute_chain_hash(admission.parent_chain_hash, token_ids_delta),
cumulative_hash=hash_token_ids(prefix_token_ids + token_ids_delta),
)
Expand Down
5 changes: 2 additions & 3 deletions nemo_rl/models/generation/prefix_recovery/state.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,9 +109,8 @@ def __init__(
self._call = call
self._prompt_token_ids = list(prompt_token_ids)
# Backend-owned opaque state that must survive until the terminal row
# is staged. vLLM uses this for its processed-media snapshot; prefix
# recovery is setup-guarded to text-only, but ordinary terminal capture
# still shares this lifecycle container.
# is staged. vLLM uses this for its processed-media snapshot; the first
# generation-prefix row carries its tensor digests across recovery.
self._media = media
self._resumed_generation_token_ids = list(resumed_generation_token_ids)
self._lock = threading.Lock()
Expand Down
Loading
Loading