Skip to content

fix(vllm): harden speculative draft refit lifecycle - #3744

Merged
yuki-97 merged 39 commits into
NVIDIA-NeMo:mainfrom
seonjinn:seonjinn/pr9-v3-latest-main
Nov 3, 2026
Merged

yuki-97 merged 39 commits into
NVIDIA-NeMo:mainfrom
seonjinn:seonjinn/pr9-v3-latest-main

Conversation

@seonjinn

@seonjinn seonjinn commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Hardens the vLLM speculative-draft refit lifecycle.

  • Resolves Eagle3, DFlash, and DSpark capabilities through one typed interface.
  • Verifies exact target/draft weight coverage before completing refit.
  • Fails closed on partial transfer, bulk receive, or FP8 KV-cache post-processing failures.
  • Releases IPC resources deterministically and preserves ordered draft manifests.

Supported contract

Issues

No issue is closed by this PR.

Usage

Existing Eagle3, DFlash, and DSpark generation configurations use the common refit lifecycle automatically.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Prior exact-head GPU evidence

  • OCI-HSG 6406866: PASS (COMPLETED 0:0) on the predecessor exact head, covering 11 speculator-runtime, 37 vLLM refit-lifecycle, and 18 policy IPC/lifecycle tests.

Validation / Remaining

  • Exact head: 8a03385f2320ed3dd1677a30a52856f54556da3f
  • All completed GitHub checks pass; the H100 container build is still running.
  • This is a correctness change; no throughput improvement is claimed.

Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn
seonjinn requested review from a team as code owners August 21, 2026 05:17
@seonjinn seonjinn added the CI:L1 Run doctests, unit tests, and functional tests label Aug 21, 2026
@copy-pr-bot

copy-pr-bot Bot commented Aug 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>

@seonjinn seonjinn left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Contributor self-review (/review-pr-team checklist)

Reviewed exact head ef32181adfa88b6d11426ea3d0fe1b8622e930b7 against current main 8863ebeff172fc3e1d14ab6cdbd0af626f477c01. The branch is mergeable, has one CI:L1 label, and has no unresolved review threads.

Findings

  • Resolved before posting: FP8 KV-cache post-processing failures occurred outside the fail-closed refit lifecycle guard. A worker that had already applied target weights could therefore report the post-processing error without marking itself unusable, allowing a later refit attempt. Head commit ef32181a moves the post-processing step under the lifecycle guard and adds a regression test proving that the next refit is rejected.
  • No actionable findings remain in the reviewed diff.

Verification

  • OCI-HSG gate job 6406866: COMPLETED (0:0) on the exact reviewed head, using vLLM 0.25.1 and OpenAI 2.25.0.
  • Pyrefly: 0 errors.
  • Speculator runtime suite: 11 passed.
  • vLLM backend suite: 37 passed.
  • Policy IPC/lifecycle suite: 18 passed.
  • Local focused regression checks: 11 passed; final lifecycle pair: 2 passed.
  • GitHub DCO, semantic-title, secrets, copyright, and submodule checks are green.

Scope and residual risk

The focused GPU gate validates the vLLM 0.25.1 refit/lifecycle paths changed by this PR. End-to-end training performance/convergence and distributed trainer CP > 1 remain outside this review's validation scope.

Independent reviewer-agent capacity was unavailable because PR9-PR11 work was already running in parallel, so this is a degraded direct full-diff audit rather than an independently adversarial review wave.

Generated by Codex contributor self-review

@seonjinn seonjinn left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Supplemental self-review (the earlier one on this PR stands; this pass ran the PR9 lifecycle checklist -- repeated refit, fail-closed windows, stale-state reuse, error-contract consistency -- specifically against surfaces it did not cover).

Verified clean beyond the two inline findings: speculator_runtime.py holds no cross-refit mutable state that survives a failure; _refit_unusable_reason has no reset path (poisoning is permanent until worker restart, which is the intended semantics); a second successful refit starts from clean state; and worker restart reinitializes without partial-state carryover.

Note for the stack: #3750 builds on this head and touches the same transport -- it should rebase once the follow-up commit lands.

@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test 8a03385

@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test c31cead

@seonjinn

Copy link
Copy Markdown
Contributor Author

CI triage: the L0_Unit_Tests_Vllm_1-3 failures on 8a03385 were a pre-existing signature drift this PR's first real CI run surfaced — VllmQuantInternalWorkerExtension._weight_update_lifecycle's inner finalize() took zero arguments while the base lifecycle callers pass finalize(finalize_draft), so every real-quant refit test hit TypeError. Fixed in the new head: the override now matches the signature and fails loudly if draft finalization is ever requested on the real-quant path (unsupported there).

Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test 4a7f8ca

Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test 52e2553

@seonjinn

Copy link
Copy Markdown
Contributor Author

Second signature-drift in the same subclass, same root cause as the finalize fix: the base _load_weights gained a coverage keyword this PR, but VllmQuantInternalWorkerExtension._load_weights (and three test stubs) kept the old signature -- every real-quant IPC/sync refit threw TypeError and reported 507 missing keys. Fixed on the new head: the override accepts coverage and records the full delivered name set (real-quant deliberately filters some scale entries out of the actual load, but they are handled); stubs updated. Audited the subclass for further drift against the base's changed surfaces -- none remaining.

Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn

Copy link
Copy Markdown
Contributor Author

/nvskills-ci

- Share SPECULATOR_TYPES and find_draft_model between speculator_runtime,
  vllm_backend and draft_utils.
- Always raise SpeculatorRuntimeError from refit handlers; report a
  watchdog abort as RefitAborted.
- Reject co-trained draft manifests on checkpoint-engine and sparse-delta.
- Detect draft-carrying speculative configs by method and K in the
  nccl_reshard guard.
- Add tests for draft owner skip, incomplete coverage and finalize configs.

Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test d939dea

…t-main-fixes4

Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
The runtime adapter derives ownership from rank_in_group/world_size, so
the patched PP group must carry them. DFlash/DSpark only run at PP=1,
so the non-owner case moves to eagle3.

Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test 89d8986

Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test f4617c7

Main's watchdog source test requires the hold inside a literal
`with RefitAbortWatchdog(` block. Move the draft error translation inside
that block; the post-exit fired check still reports a late abort.

Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test 5dfe9eb

…t-main

Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test 9caf100

copy-pr-bot Bot pushed a commit that referenced this pull request Oct 22, 2026
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>

# Conflicts:
#	nemo_rl/models/generation/vllm/vllm_backend.py
@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test f4efb4c

yuekaizhang
yuekaizhang previously approved these changes Oct 28, 2026

@yuekaizhang yuekaizhang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the draft co-training refit/transport stack across #3723/#3726/#3744/#3750/#3757. All findings from this review are resolved (see individual threads), CI is green. Approving.

@sharonyu-115 sharonyu-115 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rebase on main and resolve the refit-lifecycle conflict before merging. Main's FP8 preparation, added by #2517, restores checkpoint-format scale storage before loading weights. Preserve that preparation together with this PR's failure tracking and draft finalization, then rerun the focused refit/FP8 tests and CI with the CI:L1 label on the resolved head.

No additional inline correctness findings at f4efb4c. All 16 existing review threads remain addressed. The merge conflict is an integration requirement; the resolved implementation still needs review and validation.

Generated by Codex using review-pr-team.

Signed-off-by: seonjinn <sna@nvidia.com>

# Conflicts:
#	nemo_rl/models/generation/vllm/vllm_backend.py
@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test 9823b9f

@yuki-97 yuki-97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for resolving the previous round of comments.

Scope: the PR at 9823b9f, checked against every earlier review thread and the merge request in the last review, plus a scan of the changed files for new issues.

Checked:

  • Merge with main: prepare_fp8_linear_refit still runs before yield finalize, now inside the failure tracking, and is safe to run again on a retry.
  • Refit recovery: a retry after a failed refit only warns, and a completed lifecycle clears _refit_unusable_reason.
  • Draft rejection on native transports is live: checkpoint-engine init_communicator builds the manifest, so the draft check fires before any weight loads.
  • Draft finalization passes speculative_config.draft_model_config, the config vLLM 0.29.0 loads the drafter with.

Remaining comments are on:

  • unused fields in speculator_runtime.py

nit

  • duplicated refit-recovery test
  • long comment in nccl_reshard_utils.py

Signed-off-by: seonjinn <sna@nvidia.com>
@yuki-97

yuki-97 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

/ok to test 913342c

@yuki-97
yuki-97 enabled auto-merge (squash) November 3, 2026 06:26
@yuki-97
yuki-97 merged commit db03c2c into NVIDIA-NeMo:main Nov 3, 2026
90 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants