Skip to content

feat(checkpoint): recover multimodal generation prefixes - #4583

Merged
terrykong merged 2 commits into
mainfrom
amahishi/prefix-recovery-multimodal
Nov 14, 2026
Merged

terrykong merged 2 commits into
mainfrom
amahishi/prefix-recovery-multimodal

Conversation

@macandro96

@macandro96 macandro96 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Adds multimodal support to vLLM prefix-level rollout recovery. A checkpoint can now preserve an in-flight image or video generation prefix, record a SHA-256 digest of each of its media tensors, verify the reprocessed request against those digests, and continue decoding. Prefix rows never store pixels; the canonical terminal row stages them once, as before.

This is a stacked PR on top of #4507; the reviewable change is the seven commits on amahishi/prefix-recovery-multimodal. The stack is ordered correctness first, optimization last: #4507 → this PR → #4612 (router replay across prefixes) → #4508 (batched prefix TQ I/O).

Design

  • Fingerprints each media tensor once, when the call is admitted (off the event loop): SHA-256 over its name, dtype, shape, and bytes.
  • Stores media_spans and those tensor digests in the digest-covered extras of the first durable prefix row for a model call, with no pixel attachments, so a checkpoint cut writes no media under its deadline.
  • Keeps later prefix chunks token-only and references the first row through the existing ordered staging-key chain.
  • On recovery, reads the prefix rows (spans and digests) without any tensor read, reprocesses the reissued multimodal request, and verifies its media items and tensor digests (verify_restored_media, which names the mismatch: items, tensor set, dtype/shape, or bytes).
  • Stages the reprocessed tensors, now proven byte-identical, in the canonical terminal row.
  • Preserves the existing buffer-swap, deadline, and terminal-completion race behavior from the parent stack.
  • Fails closed when media metadata/tensors are missing or changed. A cut before media preprocessing has completed is reported as durable_failure, so that model call is redispatched instead of restoring an incomplete prefix.

Checkpoint and recovery flow

flowchart TD
    A["Gym / agent sends request plus original media reference"] --> B["vLLM resolves and preprocesses media"]
    B --> C["Decode while token capture buffers token deltas"]
    C --> D{"Checkpoint cuts this active model call?"}
    D -- "No" --> E["Stage canonical terminal row with tokens and media"]
    D -- "Yes" --> F["Atomically swap the active token buffer"]
    F --> G{"First durable prefix row for this call?"}
    G -- "Yes" --> H["Stage tokens, media_spans, and media tensor digests in TQ"]
    G -- "No" --> I["Stage a token-only prefix chunk in TQ"]
    H --> J["Publish the coordinated TQ plus Gym / SC checkpoint"]
    I --> J
    J --> K["Process restarts and loads the checkpoint"]
    K --> L["Gym reissues the call with original media and the staging-key chain"]
    L --> M["Generation worker fetches the prefix rows (tokens, spans, digests) from TQ"]
    M --> N["vLLM reprocesses the original media"]
    N --> O{"Prompt tokens, media metadata, and tensor digests match?"}
    O -- "No" --> P["Fail closed; redispatch the model call from a safe boundary"]
    O -- "Yes" --> Q["Prefill the durable token prefix with verified media"]
    Q --> R["Decode the remaining suffix"]
    R --> S["Stage the canonical terminal row with the reprocessed media tensors"]
    S --> T["Clear obsolete prefix keys only after terminal staging succeeds"]
Loading

Constraints and assumptions

Constraints

  • Prefix-level recovery remains vLLM-only. Megatron inference is not enabled by this PR.
  • Router replay with prefix-level recovery remains unsupported and retains its existing setup guard.
  • The original image or video must still be resolvable when Gym reissues the request. Prefix rows hold only digests of the packed tensors, so this PR does not provide source-free media prefill.
  • This does not checkpoint vLLM's KV cache or media-processor cache. Recovery reprocesses the media and prefills the durable token prefix.
  • A cut that arrives before media preprocessing and capture setup complete cannot save a media prefix. It returns durable_failure, and the call is redispatched from the preceding safe boundary.
  • Only the first durable prefix row for a model call may carry media metadata and tensor digests. Later rows are token-only; missing, malformed, or conflicting media metadata fails closed.
  • Missing or corrupt prefix-row metadata, changed media, or any metadata or digest mismatch rejects prefix recovery (retained_media_changed) rather than continuing with uncertain state.
  • The configured data-plane checkpoint backend must restore tensor payload rows, not controller metadata alone.

Assumptions

  • Restore uses a compatible model, tokenizer, renderer, media processor, and capture schema; the parent checkpoint stack enforces the broader configuration identity.
  • Reprocessing the same source media is deterministic enough to reproduce the exact captured metadata and tensor bytes. The implementation intentionally requires byte equality, checked by SHA-256 digest.
  • The ordered staging-key chain is preserved, and its first media-bearing row remains live until a canonical terminal row has been staged successfully.
  • The parent stack publishes Gym state, SC recovery metadata, and the corresponding TQ snapshot as one coordinated recovery point.
  • Gym/agent/environment state recovery is provided by the parent Gym v2 checkpoint protocol. This PR only adds multimodal generation-prefix payload handling.

Issues

None.

Usage

The existing prefix-recovery configuration now also accepts supported vLLM multimodal workloads:

token_capture:
  enabled: true

rollout_recovery:
  target_level: prefix

The generation backend must be vLLM and media capture must be supported by the configured VLM processor.

Testing

Added or extended unit coverage for:

  • digest-only first prefix rows and later token-only chunks, for one and two cuts;
  • the image and video (num_frames) cut/restore round trip through a real TQ prefix row, with no tensor read on restore;
  • the prefix digest written at the cut matching the one restore rebuilds (text, image with one cut, image with two cuts);
  • each restore rejection: missing, malformed, or mismatched spans and digests; item, tensor-set, dtype/shape, and byte mismatches; extras after the first chunk; a reprocessed prompt that differs from the durable prefix;
  • terminal completion racing with an active prefix cut, terminal staging failure, and obsolete-prefix cleanup failure;
  • checkpointing before media preprocessing completes;
  • a recovery request whose source media disappeared failing before any TQ read or begin_call;
  • setup accepting multimodal prefix recovery.

The shared prefix-recovery parity checks now find a restored call by counting back from its row's end, so positions the trainer masked inside the call (top-k/top-p -inf, #4154) no longer break them; the Workplace parity test uses the same fix.

Unit tests pass locally on Linux at the head commit:

  • --nemo-gym-only (test_captured_media.py, test_vllm_token_capture_hosting.py): 149 passed;
  • tests/unit/environments/test_gym_prefix_recovery_parity.py: 26 passed;
  • tests/unit/single_controller/test_setup.py: 173 passed.

Functional: tests/functional/nemotron_omni_clevr_vllm_prefix_recovery_single_controller_1n2g.sh cuts active image generations at a checkpoint, hard-kills, restores, and requires each restored call to verify its image, finish once, and train on its exact restored prefix. It is registered in L1_Functional_Tests_GB200_Vllm_Omni_Single_Controller.sh (full mode). It needs Gym's string_match to declare checkpoint_mode = "stateless" (NVIDIA-NeMo/Gym#3887), which the stack's Gym pin includes. It passed locally on H100 with 3 GPUs, TP2 training, and gpu_memory_utilization=0.9 passed as overrides (4 cuts across both image groups restored and trained, media_row_rate 1). The committed defaults (2 GPUs, TP1, 0.6) match the sibling Omni token-capture L1 test and have not yet run on GB200.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

@macandro96 macandro96 added the CI:L1 Run doctests, unit tests, and functional tests label Oct 8, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Oct 8, 2026
@macandro96
macandro96 requested a review from terrykong October 8, 2026 23:42
@macandro96
macandro96 force-pushed the amahishi/prefix-recovery-multimodal branch from 6cd7225 to 01ef7df Compare October 12, 2026 18:30
@macandro96
macandro96 changed the base branch from amahishi/ckpt-v2-prefix-batched-io to amahishi/ckpt-v2-prefix-recovery October 12, 2026 18:30
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 01ef7df (PR #4583 from amahishi/prefix-recovery-multimodal)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of amahishi/ckpt-v2-prefix-batched-io branch (fast-forward)

All submodule changes look good! ✨

1 similar comment
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 01ef7df (PR #4583 from amahishi/prefix-recovery-multimodal)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of amahishi/ckpt-v2-prefix-batched-io branch (fast-forward)

All submodule changes look good! ✨

@macandro96
macandro96 force-pushed the amahishi/prefix-recovery-multimodal branch from 01ef7df to 0235e32 Compare October 12, 2026 18:38
@macandro96
macandro96 added this pull request to stack #4685 October 12, 2026 23:16
@macandro96
macandro96 removed this pull request from stack #4685 October 12, 2026 23:30
@macandro96
macandro96 added this pull request to stack #4686 October 12, 2026 23:32
@macandro96
macandro96 force-pushed the amahishi/prefix-recovery-multimodal branch from 0235e32 to 0bf5789 Compare October 15, 2026 02:17
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 0bf5789 (PR #4583 from amahishi/prefix-recovery-multimodal)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

@terrykong terrykong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR extends vLLM prefix-level rollout recovery to multimodal (Omni image/video) calls. At a checkpoint cut, the first prefix row of a model call stores its media_spans and packed media tensors in TQ. On restore, the worker reprocesses the original media, checks it against what was saved, and continues decoding. The PR also removes the setup guards that blocked multimodal prefix recovery.

Thanks for this. Good parts:

  • MEDIA_SPANS_FIELD is imported from Gym where it is read, with no copied string.
  • fetch_generation_cut is built on the existing fetch_for_finalization and fetch_media helpers instead of new TQ reads.
  • The fail-closed design is the right call, and the "Constraints and assumptions" section in the description makes the limits easy to check.

Size. This PR's own source change is small (about 190 lines). The comment on return restored would cut about 55 of them, and the comment on the functional script would move about 500 test lines to the PR that can actually run them. The stack is already over 20k lines, so keeping each step tight matters. Later PRs in the stack (#4612, #4508) rewrite parts of this PR's new fetch and staging code, so landing the hash change and the getattr cleanup here first avoids carrying them up the stack.

CI. Only the lint and metadata checks have run on this head (0bf5789). No unit or functional CI has run yet.

Question for the author (no action required): does the Omni chat template render the same text on every call? A media restore now re-renders the first-turn prompt and fails closed if it differs, so a template that is not stable would turn every media restore into a redispatch.

Visual explainer: https://terrykong.github.io/gh-pages-poc/terryk/pr-4583-text-vs-media.html
Visual explainer: https://terrykong.github.io/gh-pages-poc/terryk/pr-4583-size-and-pixel-copy.html

Generated by Claude Code

Comment thread nemo_rl/data_plane/tq_token_sink.py Resolved
Comment thread tests/unit/data/test_captured_media.py Resolved
@macandro96
macandro96 force-pushed the amahishi/prefix-recovery-multimodal branch from 0bf5789 to 746b304 Compare October 18, 2026 22:01
@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test 746b304

@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test 8574280

@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test 13d42ba

@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test a4a4644

@macandro96
macandro96 force-pushed the amahishi/prefix-recovery-multimodal branch from 3821aa8 to 25ae296 Compare November 11, 2026 00:01
@terrykong
terrykong force-pushed the amahishi/prefix-recovery-multimodal branch from 25ae296 to 9191b8d Compare November 11, 2026 00:02
@macandro96
macandro96 removed this pull request from stack #5057 November 12, 2026 15:29
@macandro96
macandro96 added this pull request to stack #5181 November 12, 2026 15:29
Base automatically changed from amahishi/ckpt-v2-prefix-recovery to main November 12, 2026 21:16
@terrykong
terrykong force-pushed the amahishi/prefix-recovery-multimodal branch from 9191b8d to 4253178 Compare November 12, 2026 21:16
@terrykong
terrykong enabled stacked auto-merge (squash) November 12, 2026 23:01
@terrykong
terrykong disabled stacked auto-merge November 12, 2026 23:01
@terrykong

Copy link
Copy Markdown
Collaborator

@aroshanghias-nvd could you also take a pass of this PR?

@terrykong
terrykong enabled stacked auto-merge (squash) November 12, 2026 23:21
@terrykong
terrykong disabled stacked auto-merge November 12, 2026 23:22

@terrykong terrykong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround on round 1: all 9 comments (C1–C9) are fixed, and storing SHA-256 digests in the prefix rows instead of pixels removed the extra pixel write from every checkpoint. The design is right. This round is about size, since the stack is already past 20k lines.

This PR adds ~300 source lines and ~1,400 test lines (unit and functional). The comments below would remove roughly 300 lines without changing behavior (about 110 source, 200 test), and the two missing tests add back about 45:

  1. (C10) Turn verify_restored_media into one equality check.
  2. (C11) Read restored media from the verified prefix snapshot, and keep resolve_restore's return type; drop the chunk pass-through and its shape guards.
  3. (C12) Drop _verify_resumed_media_prompt; the item compare already covers image placement.
  4. (C16, C17, C18) Reuse existing test helpers, and move shared functional-test settings into the test-only YAML.

Two missing tests: (C14) nothing runs the handler's restored-media branch, and (C15) nothing restores an image call and then cuts it again.

Merge order with #4979 (Gym through Dynamo): we'd like this PR to merge first. #4979 moves the chat handler's restore steps into a shared start_capture that the Dynamo worker also runs, and its rebase then moves this PR's restored-media branch there, giving Dynamo image restore with no extra code. C11 (return type, one worker method) and C14 (a test that catches a dropped branch) keep that move small and safe; #4979 has the matching note for its side.

Before merge: please trigger CI on this head (/ok to test 425317889212cb4a5e1252030797d83982ae4fc8; the last full run was on 19051ca, before the rebase), and link one GB200 run of the new L1 Omni prefix-recovery test with the committed 2-GPU / TP1 / 0.6 settings. So far only a 3-GPU H100 run with different settings has passed.

Visual explainer: https://terrykong.github.io/gh-pages-poc/terryk/pr-4583-v2-text-vs-media.html
Visual explainer: https://terrykong.github.io/gh-pages-poc/terryk/pr-4583-v2-what-can-shrink.html
Generated by Claude Code

Comment thread nemo_rl/data/captured_media.py Resolved
Comment thread tests/unit/data/test_captured_media.py Resolved
@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test d678db0

macandro96 and others added 2 commits November 13, 2026 13:13
Lift the setup refusal of generation-prefix recovery with media capture.

A media call's first durable prefix row carries its media spans and a
SHA-256 digest of each packed media tensor in its extras, not the
pixels; later chunks carry no media. The restored prefix digest covers
those extras. On restore the worker reads the items and digests from
that row, reprocesses the reissued request's media, checks it against
them (retained_media_changed on a mismatch), and stages the reprocessed
tensors in the canonical terminal row. A row whose media metadata
cannot be read makes the call regenerate from its turn boundary.

Adds an Omni CLEVR multimodal prefix-recovery functional test, unit
coverage of media cuts, races and restores, and shares the prefix
recovery functional phases (_gym_prefix_recovery_phases.sh).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
- verify_restored_media is one comparison of items and tensor digests that
  names the changed tensors; the dead fallback and None branches are gone.
- resolve_restore returns the verified snapshot again, as on main; the worker
  reads a restored call's media from that snapshot after the digest check, so
  _ResolvedGenerationCut, the restore_media hook, _fetch_chunks and the media
  row guards go away. The handler's restored-media steps sit behind
  _capture_restored_media.
- Drop _verify_resumed_media_prompt: vLLM prefills the saved prefix, and equal
  media items already pin every placeholder.
- Tests: cover the handler's restored-media branch and a restored image call
  cut again; fold duplicated helpers, sinks and setup patching; fix the
  missing-media handler case, which lacked the sampling fields the handler
  checks.
- The two Omni CLEVR vLLM tests take their fixed topology from their YAML,
  and the prefix-recovery tests share restore_selected_snapshot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
@macandro96
macandro96 force-pushed the amahishi/prefix-recovery-multimodal branch from d678db0 to ce47ea4 Compare November 13, 2026 21:32
@macandro96

Copy link
Copy Markdown
Contributor Author

Rebased onto main to resolve the docs conflict (#5211 unwrapped single-controller.md). The review changes the replies above cite as d678db08e are now in ce47ea4bc. That commit also fixes the two CI failures from the last run:

  • The C12 cleanup had also deleted HTTP_SERVER_PROTECTED_PATH_PREFIXES, which sat right after _verify_resumed_media_prompt. It is restored, which fixes the API-key tests.
  • test_omni_capture_setup_rejects_video_pruning's prefix-cuts case now gives the fake worker the HTTP client and progress reader that setup requires.

@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test ce47ea4

@macandro96

Copy link
Copy Markdown
Contributor Author

/ok to test ce47ea4

@terrykong
terrykong enabled stacked auto-merge (squash) November 13, 2026 22:14
@terrykong
terrykong merged commit bb9ed78 into main Nov 14, 2026
166 of 168 checks passed
@terrykong
terrykong deleted the amahishi/prefix-recovery-multimodal branch November 14, 2026 00:03
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 14, 2026
…mo-gym

main gained NVIDIA-NeMo#4583, which lets a call resumed from a checkpoint cut carry
media. Three files conflicted:

- nemo_rl/models/generation/vllm/vllm_worker_async.py: NVIDIA-NeMo#4583 edits the chat
  handler this branch moves to chat_server.py and prompt_and_capture.py. This
  branch's side is kept, plus the `replace` import NVIDIA-NeMo#4583's
  _capture_request_media needs.
- tests/functional/grpo_async_gym_single_controller_prefix_recovery.sh: main
  moves stop_phase1 and the exit cleanup into
  _gym_prefix_recovery_phases.sh. main's side is kept, and the Dynamo orphan
  cleanup this branch had in stop_phase1 moves into the shared helper; a test
  that runs behind Dynamo names phase one's log in PHASE1_RUN_LOG.
- tests/unit/data/test_captured_media.py: main's side (its chat_request
  helper).

NVIDIA-NeMo#4583's handler change is re-applied where the handler now lives, as agreed
on the review (C92): start_capture calls the worker's
_capture_restored_media for a restored call, with the prepared prompt's
splice, which is None for a call without a prefix (so the method's
annotation now allows None). The Dynamo worker runs the same start_capture,
so a restored image call is checked and staged there too.

Two cases in test_dynamo_vllm_adapter_capture.py cover that path: a restored
image call through the engine adapter stages its reprocessed pixels, or is
refused with retained_media_changed before it begins or generates; and a cut
writes a media call's tensor digests into its first prefix row.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
DanialTaheri added a commit that referenced this pull request Nov 14, 2026
Brings in the 10 commits on main since the PR branched, including
#4583 (multimodal generation-prefix recovery). The only conflict was
the import lists in test_vllm_token_capture_hosting.py; both sides are
kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Danial Mohseni Taheri <49656670+DanialTaheri@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants