Skip to content

[Feat] speculative: Support synchronous STANDALONE PD handoff - #47751

Open
andrewzyxing wants to merge 23 commits into
sgl-project:mainfrom
andrewzyxing:feat/standalone-pd-seed-handoff
Open

andrewzyxing wants to merge 23 commits into
sgl-project:mainfrom
andrewzyxing:feat/standalone-pd-seed-handoff

Conversation

@andrewzyxing

@andrewzyxing andrewzyxing commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

What changes

STANDALONE prefill produces a draft seed, but PD previously serialized it only with EAGLE hidden states and had no STANDALONE decode factory. Decode can now resume from the transferred top-k probabilities/indices without target hidden states, using the existing target/draft KV transport.

  • Separate top-k metadata from hidden-state metadata; clear reused rows and keep the existing wire layout.
  • Rebuild the first decode draft input with hidden_states=None.
  • Reject malformed producer/consumer seeds through request-local abort and resource cleanup, without admitting the affected request to drafting.
  • Add startup checks, configuration guidance and reproducible CPU/manual GPU regressions.

Initial support contract

Both workers must use STANDALONE with matching target/draft assets, full vocabulary/token IDs, compatible KV layouts and speculative settings. This does not add a peer model-identity handshake.

Startup checks require synchronous execution, TP/PP/DP/CP sizes of one, top-k1 and early allocation, and reject the explicit host/cache/staging options outside this initial scope. Ordinary PD, EAGLE PD and monolithic STANDALONE retain their existing configuration behavior. TP is restricted because the new post-transfer seed guard is request-local and does not implement distributed abort consensus.

The GPU validation configuration is NVIDIA H20, BF16/FA3, page size 1, CUDA graphs disabled, one GPU with separate P/D processes, and Mooncake TCP. STANDALONE uses Qwen3-4B/Qwen3-0.6B; the real trained EAGLE3 regression uses Qwen3-8B/AngelSlim-Qwen3-8B_eagle3.

Validation

  • CPU, final public head: 36 tests + 62 subtests passed, no skips/xfails. Five complete registered files cover dispatch, serialization/reuse, malformed seeds, request/KV/metadata cleanup and startup configuration. They are automatically discovered by base-a-test-cpu.
  • GitHub Lint: passed on the final head.
  • Documentation: Mintlify build and broken-links checks passed; existing unrelated syntax warnings remain.
  • GPU: four complete manual regressions passed: STANDALONE handoff, request-local seed rejection and recovery, ordinary PD, and trained EAGLE3 PD. All 36 POSTs reached a terminal result (35 HTTP 200; one expected HTTP 500 for the injected bad seed), with no automatic retries. All 264 successful decode output tokens exactly matched ordinary target decoding. The recovery run naturally reused request, KV and metadata slots after rejecting the bad request. Ordinary PD kept speculative metadata clear; EAGLE3 retained identical top-k/hidden fingerprints across the handoff and executed its trained draft network. The 700-token shared-path cases transferred chunks of 512 + 188 tokens. All engine roles passed cleanup, and the GPU deployment was scaled to zero with no remaining instances.

GPU checks used a native build integrating this branch through 6011505 and community main af633e1. The nine inspected PD/speculative production owners and manual tests matched that public source. The final 24e3ce1 delta only switches four imports to their defining modules (including one existing attention-test lint error); the sole production change is the CaptureHiddenMode import path. AST equivalence and actual alias-object identity were checked, and the complete CPU suite was rerun on the final public head. Internal image/source provenance is recorded separately; this is not a full upstream CI or performance result.

Commands and requirements: manual regression guide.

Follow-up scope

GPU overlap/FutureMap scheduling, multi-rank consensus, cross-host/RDMA, host/staging/HiCache recovery, additional model families and performance remain separate work. Recovery covers one producer seed corrupted before a successful transfer and serial good requests afterward; it does not claim arbitrary transport corruption or partial-transfer recovery. Valid terminal handoffs are CPU-covered; missing/malformed seeds have no terminal exemption.

Related roadmap: #43699.

Upstream sync and CPU regression (2026-11-09)

Merged main@76dafba55a5e without rewriting branch history. Followed the current device-helper import and request-retirement interfaces while preserving the seed-handoff and cleanup assertions. Previously reported GPU results describe their recorded revision; this sync was validated on CPU. On 6116dea334bb5f9d4abd7359ce5a3d9a6ca610a2, the complete relevant CPU suite passed: 36 tests + 62 subtests, with no skips or retries.


CI States

Latest PR Test (Base): ❌ Run #43149932430
Latest PR Test (Extra): ❌ Run #43149932412
Latest PR Test (AMD ROCm 10): ❌ Run #43149932193

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@github-actions github-actions Bot added the npu label Nov 2, 2026
@mintlify

mintlify Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit f84ddf0.

Approve preview

@github-actions github-actions Bot added documentation Improvements or additions to documentation speculative-decoding labels Nov 5, 2026
@andrewzyxing andrewzyxing changed the title [Feat] speculative: Wire STANDALONE PD draft seeds without hidden states [Feat] speculative: Support synchronous STANDALONE PD handoff Nov 5, 2026
@andrewzyxing
andrewzyxing marked this pull request as ready for review November 5, 2026 05:33

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@andrewzyxing

andrewzyxing commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

The initial synchronous, single-rank implementation is ready for review. This now includes startup scope checks, configuration documentation, and the complete handoff/recovery tests.

Validation on the final head: 36 CPU tests + 62 subtests passed. Four complete native-H20 GPU regressions passed (STANDALONE handoff/recovery, ordinary PD and trained EAGLE3 PD); 264 decode tokens exactly matched ordinary target decoding. The one injected bad-seed request was rejected and its resources were naturally reused. The GPU allocation has been released. Source correspondence and the final canonical-import-only delta are detailed in the PR body.

Could a maintainer please trigger the standard CPU CI for this head (/run-cpu-ci)? All five registered files are auto-discovered by base-a-test-cpu. The current draft-gate failures skipped product tests; the author account cannot grant CI permissions. Broader multi-rank, overlap and RDMA coverage remains follow-up work.

Claude's automatic review is disabled for fork PRs. Could a repository maintainer also comment @claude review to trigger the one-time review?

@jlee5814

jlee5814 commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Ran the five registered files: 36 passed. Four mutations against the row zeroing, seed check, overlap guard and receiver cleanup each went red. LGTM.

Three of the test files inherit unittest.TestCase; test/registered/unit/README.md asks for CustomTestCase.

@github-actions

This comment has been minimized.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation npu speculative-decoding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants