Repository navigation
fix(vllm): harden speculative draft refit lifecycle - #3744
Conversation
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
seonjinn
left a comment
There was a problem hiding this comment.
Contributor self-review (/review-pr-team checklist)
Reviewed exact head ef32181adfa88b6d11426ea3d0fe1b8622e930b7 against current main 8863ebeff172fc3e1d14ab6cdbd0af626f477c01. The branch is mergeable, has one CI:L1 label, and has no unresolved review threads.
Findings
- Resolved before posting: FP8 KV-cache post-processing failures occurred outside the fail-closed refit lifecycle guard. A worker that had already applied target weights could therefore report the post-processing error without marking itself unusable, allowing a later refit attempt. Head commit
ef32181amoves the post-processing step under the lifecycle guard and adds a regression test proving that the next refit is rejected. - No actionable findings remain in the reviewed diff.
Verification
- OCI-HSG gate job
6406866:COMPLETED (0:0)on the exact reviewed head, using vLLM 0.25.1 and OpenAI 2.25.0. - Pyrefly: 0 errors.
- Speculator runtime suite: 11 passed.
- vLLM backend suite: 37 passed.
- Policy IPC/lifecycle suite: 18 passed.
- Local focused regression checks: 11 passed; final lifecycle pair: 2 passed.
- GitHub DCO, semantic-title, secrets, copyright, and submodule checks are green.
Scope and residual risk
The focused GPU gate validates the vLLM 0.25.1 refit/lifecycle paths changed by this PR. End-to-end training performance/convergence and distributed trainer CP > 1 remain outside this review's validation scope.
Independent reviewer-agent capacity was unavailable because PR9-PR11 work was already running in parallel, so this is a degraded direct full-diff audit rather than an independently adversarial review wave.
Generated by Codex contributor self-review
seonjinn
left a comment
There was a problem hiding this comment.
Supplemental self-review (the earlier one on this PR stands; this pass ran the PR9 lifecycle checklist -- repeated refit, fail-closed windows, stale-state reuse, error-contract consistency -- specifically against surfaces it did not cover).
Verified clean beyond the two inline findings: speculator_runtime.py holds no cross-refit mutable state that survives a failure; _refit_unusable_reason has no reset path (poisoning is permanent until worker restart, which is the intended semantics); a second successful refit starts from clean state; and worker restart reinitializes without partial-state carryover.
Note for the stack: #3750 builds on this head and touches the same transport -- it should rebase once the follow-up commit lands.
Signed-off-by: seonjinn <sna@nvidia.com>
|
/ok to test 8a03385 |
Signed-off-by: seonjinn <sna@nvidia.com>
|
/ok to test c31cead |
|
CI triage: the L0_Unit_Tests_Vllm_1-3 failures on 8a03385 were a pre-existing signature drift this PR's first real CI run surfaced — |
Signed-off-by: seonjinn <sna@nvidia.com>
|
/ok to test 4a7f8ca |
Signed-off-by: seonjinn <sna@nvidia.com>
|
/ok to test 52e2553 |
|
Second signature-drift in the same subclass, same root cause as the finalize fix: the base |
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
|
/nvskills-ci |
- Share SPECULATOR_TYPES and find_draft_model between speculator_runtime, vllm_backend and draft_utils. - Always raise SpeculatorRuntimeError from refit handlers; report a watchdog abort as RefitAborted. - Reject co-trained draft manifests on checkpoint-engine and sparse-delta. - Detect draft-carrying speculative configs by method and K in the nccl_reshard guard. - Add tests for draft owner skip, incomplete coverage and finalize configs. Signed-off-by: seonjinn <sna@nvidia.com>
|
/ok to test d939dea |
…t-main-fixes4 Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
The runtime adapter derives ownership from rank_in_group/world_size, so the patched PP group must carry them. DFlash/DSpark only run at PP=1, so the non-owner case moves to eagle3. Signed-off-by: seonjinn <sna@nvidia.com>
|
/ok to test 89d8986 |
Signed-off-by: seonjinn <sna@nvidia.com>
|
/ok to test f4617c7 |
Main's watchdog source test requires the hold inside a literal `with RefitAbortWatchdog(` block. Move the draft error translation inside that block; the post-exit fired check still reports a late abort. Signed-off-by: seonjinn <sna@nvidia.com>
|
/ok to test 5dfe9eb |
…t-main Signed-off-by: seonjinn <sna@nvidia.com>
|
/ok to test 9caf100 |
Signed-off-by: seonjinn <sna@nvidia.com> # Conflicts: # nemo_rl/models/generation/vllm/vllm_backend.py
|
/ok to test f4efb4c |
There was a problem hiding this comment.
Please rebase on main and resolve the refit-lifecycle conflict before merging. Main's FP8 preparation, added by #2517, restores checkpoint-format scale storage before loading weights. Preserve that preparation together with this PR's failure tracking and draft finalization, then rerun the focused refit/FP8 tests and CI with the CI:L1 label on the resolved head.
No additional inline correctness findings at f4efb4c. All 16 existing review threads remain addressed. The merge conflict is an integration requirement; the resolved implementation still needs review and validation.
Generated by Codex using review-pr-team.
Signed-off-by: seonjinn <sna@nvidia.com> # Conflicts: # nemo_rl/models/generation/vllm/vllm_backend.py
|
/ok to test 9823b9f |
yuki-97
left a comment
There was a problem hiding this comment.
Thanks for resolving the previous round of comments.
Scope: the PR at 9823b9f, checked against every earlier review thread and the merge request in the last review, plus a scan of the changed files for new issues.
Checked:
- Merge with main:
prepare_fp8_linear_refitstill runs beforeyield finalize, now inside the failure tracking, and is safe to run again on a retry. - Refit recovery: a retry after a failed refit only warns, and a completed lifecycle clears
_refit_unusable_reason. - Draft rejection on native transports is live: checkpoint-engine
init_communicatorbuilds the manifest, so the draft check fires before any weight loads. - Draft finalization passes
speculative_config.draft_model_config, the config vLLM 0.29.0 loads the drafter with.
Remaining comments are on:
- unused fields in
speculator_runtime.py
nit
- duplicated refit-recovery test
- long comment in
nccl_reshard_utils.py
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
|
/ok to test 913342c |
What does this PR do ?
Hardens the vLLM speculative-draft refit lifecycle.
Supported contract
Issues
No issue is closed by this PR.
Usage
Existing Eagle3, DFlash, and DSpark generation configurations use the common refit lifecycle automatically.
Before your PR is "Ready for review"
Pre checks:
Prior exact-head GPU evidence
6406866: PASS (COMPLETED 0:0) on the predecessor exact head, covering 11 speculator-runtime, 37 vLLM refit-lifecycle, and 18 policy IPC/lifecycle tests.Validation / Remaining
8a03385f2320ed3dd1677a30a52856f54556da3f