Repository navigation
[Feat] speculative: Support synchronous STANDALONE PD handoff - #47751
andrewzyxing wants to merge 23 commits into
Conversation
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
|
The initial synchronous, single-rank implementation is ready for review. This now includes startup scope checks, configuration documentation, and the complete handoff/recovery tests. Validation on the final head: 36 CPU tests + 62 subtests passed. Four complete native-H20 GPU regressions passed (STANDALONE handoff/recovery, ordinary PD and trained EAGLE3 PD); 264 decode tokens exactly matched ordinary target decoding. The one injected bad-seed request was rejected and its resources were naturally reused. The GPU allocation has been released. Source correspondence and the final canonical-import-only delta are detailed in the PR body. Could a maintainer please trigger the standard CPU CI for this head ( Claude's automatic review is disabled for fork PRs. Could a repository maintainer also comment |
|
Ran the five registered files: 36 passed. Four mutations against the row zeroing, seed check, overlap guard and receiver cleanup each went red. LGTM. Three of the test files inherit |
What changes
STANDALONE prefill produces a draft seed, but PD previously serialized it only with EAGLE hidden states and had no STANDALONE decode factory. Decode can now resume from the transferred top-k probabilities/indices without target hidden states, using the existing target/draft KV transport.
hidden_states=None.Initial support contract
Both workers must use STANDALONE with matching target/draft assets, full vocabulary/token IDs, compatible KV layouts and speculative settings. This does not add a peer model-identity handshake.
Startup checks require synchronous execution, TP/PP/DP/CP sizes of one, top-k1 and
earlyallocation, and reject the explicit host/cache/staging options outside this initial scope. Ordinary PD, EAGLE PD and monolithic STANDALONE retain their existing configuration behavior. TP is restricted because the new post-transfer seed guard is request-local and does not implement distributed abort consensus.The GPU validation configuration is NVIDIA H20, BF16/FA3, page size 1, CUDA graphs disabled, one GPU with separate P/D processes, and Mooncake TCP. STANDALONE uses Qwen3-4B/Qwen3-0.6B; the real trained EAGLE3 regression uses Qwen3-8B/AngelSlim-Qwen3-8B_eagle3.
Validation
base-a-test-cpu.GPU checks used a native build integrating this branch through
6011505and community mainaf633e1. The nine inspected PD/speculative production owners and manual tests matched that public source. The final24e3ce1delta only switches four imports to their defining modules (including one existing attention-test lint error); the sole production change is theCaptureHiddenModeimport path. AST equivalence and actual alias-object identity were checked, and the complete CPU suite was rerun on the final public head. Internal image/source provenance is recorded separately; this is not a full upstream CI or performance result.Commands and requirements: manual regression guide.
Follow-up scope
GPU overlap/FutureMap scheduling, multi-rank consensus, cross-host/RDMA, host/staging/HiCache recovery, additional model families and performance remain separate work. Recovery covers one producer seed corrupted before a successful transfer and serial good requests afterward; it does not claim arbitrary transport corruption or partial-transfer recovery. Valid terminal handoffs are CPU-covered; missing/malformed seeds have no terminal exemption.
Related roadmap: #43699.
Upstream sync and CPU regression (2026-11-09)
Merged
main@76dafba55a5ewithout rewriting branch history. Followed the current device-helper import and request-retirement interfaces while preserving the seed-handoff and cleanup assertions. Previously reported GPU results describe their recorded revision; this sync was validated on CPU. On6116dea334bb5f9d4abd7359ce5a3d9a6ca610a2, the complete relevant CPU suite passed: 36 tests + 62 subtests, with no skips or retries.CI States
Latest PR Test (Base): ❌ Run #43149932430
Latest PR Test (Extra): ❌ Run #43149932412
Latest PR Test (AMD ROCm 10): ❌ Run #43149932193
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: