Repository navigation
feat(checkpoint): recover multimodal generation prefixes - #4583
Conversation
6cd7225 to
01ef7df
Compare
1 similar comment
01ef7df to
0235e32
Compare
0235e32 to
0bf5789
Compare
terrykong
left a comment
There was a problem hiding this comment.
This PR extends vLLM prefix-level rollout recovery to multimodal (Omni image/video) calls. At a checkpoint cut, the first prefix row of a model call stores its media_spans and packed media tensors in TQ. On restore, the worker reprocesses the original media, checks it against what was saved, and continues decoding. The PR also removes the setup guards that blocked multimodal prefix recovery.
Thanks for this. Good parts:
MEDIA_SPANS_FIELDis imported from Gym where it is read, with no copied string.fetch_generation_cutis built on the existingfetch_for_finalizationandfetch_mediahelpers instead of new TQ reads.- The fail-closed design is the right call, and the "Constraints and assumptions" section in the description makes the limits easy to check.
Size. This PR's own source change is small (about 190 lines). The comment on return restored would cut about 55 of them, and the comment on the functional script would move about 500 test lines to the PR that can actually run them. The stack is already over 20k lines, so keeping each step tight matters. Later PRs in the stack (#4612, #4508) rewrite parts of this PR's new fetch and staging code, so landing the hash change and the getattr cleanup here first avoids carrying them up the stack.
CI. Only the lint and metadata checks have run on this head (0bf5789). No unit or functional CI has run yet.
Question for the author (no action required): does the Omni chat template render the same text on every call? A media restore now re-renders the first-turn prompt and fails closed if it differs, so a template that is not stable would turn every media restore into a redispatch.
Visual explainer: https://terrykong.github.io/gh-pages-poc/terryk/pr-4583-text-vs-media.html
Visual explainer: https://terrykong.github.io/gh-pages-poc/terryk/pr-4583-size-and-pixel-copy.html
Generated by Claude Code
0bf5789 to
746b304
Compare
|
/ok to test 746b304 |
|
/ok to test 8574280 |
|
/ok to test 13d42ba |
|
/ok to test a4a4644 |
3821aa8 to
25ae296
Compare
25ae296 to
9191b8d
Compare
9191b8d to
4253178
Compare
|
@aroshanghias-nvd could you also take a pass of this PR? |
terrykong
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround on round 1: all 9 comments (C1–C9) are fixed, and storing SHA-256 digests in the prefix rows instead of pixels removed the extra pixel write from every checkpoint. The design is right. This round is about size, since the stack is already past 20k lines.
This PR adds ~300 source lines and ~1,400 test lines (unit and functional). The comments below would remove roughly 300 lines without changing behavior (about 110 source, 200 test), and the two missing tests add back about 45:
- (C10) Turn
verify_restored_mediainto one equality check. - (C11) Read restored media from the verified prefix snapshot, and keep
resolve_restore's return type; drop the chunk pass-through and its shape guards. - (C12) Drop
_verify_resumed_media_prompt; the item compare already covers image placement. - (C16, C17, C18) Reuse existing test helpers, and move shared functional-test settings into the test-only YAML.
Two missing tests: (C14) nothing runs the handler's restored-media branch, and (C15) nothing restores an image call and then cuts it again.
Merge order with #4979 (Gym through Dynamo): we'd like this PR to merge first. #4979 moves the chat handler's restore steps into a shared start_capture that the Dynamo worker also runs, and its rebase then moves this PR's restored-media branch there, giving Dynamo image restore with no extra code. C11 (return type, one worker method) and C14 (a test that catches a dropped branch) keep that move small and safe; #4979 has the matching note for its side.
Before merge: please trigger CI on this head (/ok to test 425317889212cb4a5e1252030797d83982ae4fc8; the last full run was on 19051ca, before the rebase), and link one GB200 run of the new L1 Omni prefix-recovery test with the committed 2-GPU / TP1 / 0.6 settings. So far only a 3-GPU H100 run with different settings has passed.
Visual explainer: https://terrykong.github.io/gh-pages-poc/terryk/pr-4583-v2-text-vs-media.html
Visual explainer: https://terrykong.github.io/gh-pages-poc/terryk/pr-4583-v2-what-can-shrink.html
Generated by Claude Code
4253178 to
d678db0
Compare
|
/ok to test d678db0 |
Lift the setup refusal of generation-prefix recovery with media capture. A media call's first durable prefix row carries its media spans and a SHA-256 digest of each packed media tensor in its extras, not the pixels; later chunks carry no media. The restored prefix digest covers those extras. On restore the worker reads the items and digests from that row, reprocesses the reissued request's media, checks it against them (retained_media_changed on a mismatch), and stages the reprocessed tensors in the canonical terminal row. A row whose media metadata cannot be read makes the call regenerate from its turn boundary. Adds an Omni CLEVR multimodal prefix-recovery functional test, unit coverage of media cuts, races and restores, and shares the prefix recovery functional phases (_gym_prefix_recovery_phases.sh). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
- verify_restored_media is one comparison of items and tensor digests that names the changed tensors; the dead fallback and None branches are gone. - resolve_restore returns the verified snapshot again, as on main; the worker reads a restored call's media from that snapshot after the digest check, so _ResolvedGenerationCut, the restore_media hook, _fetch_chunks and the media row guards go away. The handler's restored-media steps sit behind _capture_restored_media. - Drop _verify_resumed_media_prompt: vLLM prefills the saved prefix, and equal media items already pin every placeholder. - Tests: cover the handler's restored-media branch and a restored image call cut again; fold duplicated helpers, sinks and setup patching; fix the missing-media handler case, which lacked the sampling fields the handler checks. - The two Omni CLEVR vLLM tests take their fixed topology from their YAML, and the prefix-recovery tests share restore_selected_snapshot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
d678db0 to
ce47ea4
Compare
|
Rebased onto
|
|
/ok to test ce47ea4 |
|
/ok to test ce47ea4 |
…mo-gym main gained NVIDIA-NeMo#4583, which lets a call resumed from a checkpoint cut carry media. Three files conflicted: - nemo_rl/models/generation/vllm/vllm_worker_async.py: NVIDIA-NeMo#4583 edits the chat handler this branch moves to chat_server.py and prompt_and_capture.py. This branch's side is kept, plus the `replace` import NVIDIA-NeMo#4583's _capture_request_media needs. - tests/functional/grpo_async_gym_single_controller_prefix_recovery.sh: main moves stop_phase1 and the exit cleanup into _gym_prefix_recovery_phases.sh. main's side is kept, and the Dynamo orphan cleanup this branch had in stop_phase1 moves into the shared helper; a test that runs behind Dynamo names phase one's log in PHASE1_RUN_LOG. - tests/unit/data/test_captured_media.py: main's side (its chat_request helper). NVIDIA-NeMo#4583's handler change is re-applied where the handler now lives, as agreed on the review (C92): start_capture calls the worker's _capture_restored_media for a restored call, with the prepared prompt's splice, which is None for a call without a prefix (so the method's annotation now allows None). The Dynamo worker runs the same start_capture, so a restored image call is checked and staged there too. Two cases in test_dynamo_vllm_adapter_capture.py cover that path: a restored image call through the engine adapter stages its reprocessed pixels, or is refused with retained_media_changed before it begins or generates; and a cut writes a media call's tensor digests into its first prefix row. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Brings in the 10 commits on main since the PR branched, including #4583 (multimodal generation-prefix recovery). The only conflict was the import lists in test_vllm_token_capture_hosting.py; both sides are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Danial Mohseni Taheri <49656670+DanialTaheri@users.noreply.github.com>
What does this PR do ?
Adds multimodal support to vLLM prefix-level rollout recovery. A checkpoint can now preserve an in-flight image or video generation prefix, record a SHA-256 digest of each of its media tensors, verify the reprocessed request against those digests, and continue decoding. Prefix rows never store pixels; the canonical terminal row stages them once, as before.
This is a stacked PR on top of #4507; the reviewable change is the seven commits on
amahishi/prefix-recovery-multimodal. The stack is ordered correctness first, optimization last: #4507 → this PR → #4612 (router replay across prefixes) → #4508 (batched prefix TQ I/O).Design
media_spansand those tensor digests in the digest-covered extras of the first durable prefix row for a model call, with no pixel attachments, so a checkpoint cut writes no media under its deadline.verify_restored_media, which names the mismatch: items, tensor set, dtype/shape, or bytes).durable_failure, so that model call is redispatched instead of restoring an incomplete prefix.Checkpoint and recovery flow
flowchart TD A["Gym / agent sends request plus original media reference"] --> B["vLLM resolves and preprocesses media"] B --> C["Decode while token capture buffers token deltas"] C --> D{"Checkpoint cuts this active model call?"} D -- "No" --> E["Stage canonical terminal row with tokens and media"] D -- "Yes" --> F["Atomically swap the active token buffer"] F --> G{"First durable prefix row for this call?"} G -- "Yes" --> H["Stage tokens, media_spans, and media tensor digests in TQ"] G -- "No" --> I["Stage a token-only prefix chunk in TQ"] H --> J["Publish the coordinated TQ plus Gym / SC checkpoint"] I --> J J --> K["Process restarts and loads the checkpoint"] K --> L["Gym reissues the call with original media and the staging-key chain"] L --> M["Generation worker fetches the prefix rows (tokens, spans, digests) from TQ"] M --> N["vLLM reprocesses the original media"] N --> O{"Prompt tokens, media metadata, and tensor digests match?"} O -- "No" --> P["Fail closed; redispatch the model call from a safe boundary"] O -- "Yes" --> Q["Prefill the durable token prefix with verified media"] Q --> R["Decode the remaining suffix"] R --> S["Stage the canonical terminal row with the reprocessed media tensors"] S --> T["Clear obsolete prefix keys only after terminal staging succeeds"]Constraints and assumptions
Constraints
durable_failure, and the call is redispatched from the preceding safe boundary.retained_media_changed) rather than continuing with uncertain state.Assumptions
Issues
None.
Usage
The existing prefix-recovery configuration now also accepts supported vLLM multimodal workloads:
The generation backend must be vLLM and media capture must be supported by the configured VLM processor.
Testing
Added or extended unit coverage for:
num_frames) cut/restore round trip through a real TQ prefix row, with no tensor read on restore;begin_call;The shared prefix-recovery parity checks now find a restored call by counting back from its row's end, so positions the trainer masked inside the call (top-k/top-p
-inf, #4154) no longer break them; the Workplace parity test uses the same fix.Unit tests pass locally on Linux at the head commit:
--nemo-gym-only(test_captured_media.py,test_vllm_token_capture_hosting.py): 149 passed;tests/unit/environments/test_gym_prefix_recovery_parity.py: 26 passed;tests/unit/single_controller/test_setup.py: 173 passed.Functional:
tests/functional/nemotron_omni_clevr_vllm_prefix_recovery_single_controller_1n2g.shcuts active image generations at a checkpoint, hard-kills, restores, and requires each restored call to verify its image, finish once, and train on its exact restored prefix. It is registered inL1_Functional_Tests_GB200_Vllm_Omni_Single_Controller.sh(full mode). It needs Gym'sstring_matchto declarecheckpoint_mode = "stateless"(NVIDIA-NeMo/Gym#3887), which the stack's Gym pin includes. It passed locally on H100 with 3 GPUs, TP2 training, andgpu_memory_utilization=0.9passed as overrides (4 cuts across both image groups restored and trained,media_row_rate1). The committed defaults (2 GPUs, TP1, 0.6) match the sibling Omni token-capture L1 test and have not yet run on GB200.Before your PR is "Ready for review"
Pre checks:
Additional Information
13d42ba2eaf118982aa6364e45482e729f44ee51