Repository navigation
[Perf] TurboQuant: run spec-decode verify batches as decodes with FULL cudagraphs - #53410
giannisanni wants to merge 2 commits into
Conversation
TurboQuantMetadataBuilder declared cudagraph support UNIFORM_BATCH while also declaring supports_spec_as_decode=False. With speculative decoding enabled, the runner classifies uniform verify batches (query_len of 1 + num_speculative_tokens) as uniform decode by shape alone and FULL graph captures the attention forward. That forward routes spec-shaped batches through the per-request Python prefill loop, which iterates over CPU-resident metadata: capture bakes in the dummy metadata from build_for_cudagraph_capture (seq_lens filled with 1, so cached_len goes negative and the synthetic seq_lens slice is empty), and every replayed verify step returns zeros or reads out of bounds instead of attending. Symptoms: silent repetition collapse with num_speculative_tokens > 1, illegal memory access with num_speculative_tokens == 1. Eager mode and non-speculative graph mode are unaffected, which is why the bug only appears for the cudagraph + spec-decode combination. Declare UNIFORM_SINGLE_TOKEN_DECODE instead so spec-shaped batches fall back to piecewise cudagraphs and run the attention path eagerly with real metadata. Pure decode batches (query_len 1) keep FULL capture. Fixes vllm-project#52475 Signed-off-by: giannisanni <115853836+giannisanni@users.noreply.github.com>
…L cudagraphs Follow-up to the correctness fix for vllm-project#52475: that change stopped FULL cudagraph capture of spec-decode verify batches because the TurboQuant prefill path that served them is not capturable, which costs decode throughput under MTP (every verify step pays eager kernel-launch overhead plus a per-request Python attention path). This teaches the backend spec-as-decode: the builder declares supports_spec_as_decode=True so verify batches (query_len up to 1 + num_speculative_tokens) classify as decodes, and expands their per-request metadata to per-token synthetic single-token decode rows (seq_len = seq_lens[r] - q_r + 1 + offset over the request's block table, reproducing causal masking inside the chunk, whose K/V are already stored to the TQ cache by do_kv_cache_update). The expansion runs once per step in the builder with GPU-only ops into persistent buffers, so the decode kernels run unchanged with B = num_decode_tokens and remain FULL-cudagraph capturable; replays observe fresh values through the capture-time buffer addresses. The unused tail of the seq_lens buffer is zeroed each build so padded rows of captured batches are no-ops. The numerics are identical to the existing continuation-decode fast path in _prefill_attention (which remains for genuine chunked prefill): this is the same synthetic-seq_lens construction, moved from per-layer CPU ints to per-step GPU tensors. Decode workspace reservations are sized in tokens instead of requests to cover verify batches, and UNIFORM_BATCH cudagraph support is restored (sound now that the decode path serves spec batches). Signed-off-by: giannisanni <115853836+giannisanni@users.noreply.github.com>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
Independent reproduction and test of this PR on an Ampere card, hybrid GDN + NVFP4 target Setup: NVIDIA A40 (sm86, 46 GB), vLLM v0.33.0 release wheel (torch 2.14 cu130), Stock v0.33.0, TurboQuant + MTP-2 (and MTP-1): greedy output diverges from plain decode within the first 10 characters on a short prompt, the tool call comes back empty, long-context recall 0/8 at every length, acceptance ~55-80% (looks healthy). Same with or without prefix caching, with or without the drafter compiled. fp8 KV + MTP-2 on the same build is correct (recall 8/8, routing gate identical to plain). Root cause we see on this build: #53406 alone on our case: correctness fixed, speed lost. Long-context recall 8/8 at all four lengths, tool-routing gate 31/44 (plain 31-33/44), greedy divergence only at near-ties. Decode 84 / 274 / 683 tok/s at 1 / 4 / 32 streams vs plain 133 / 348 / 1,199, i.e. slower than no speculation (verify batches now run uncaptured); acceptance 41% per draft token; 20.2 sessions at 128k vs 27.8 plain. #53406 + #53410 on our case: (the PR patch already includes #53406 as its first commit, so it must be applied alone) correctness kept, speed back. Recall 8/8 at all lengths; greedy output identical to plain on the 20k prompt, near-tie divergences only on the short and tool prompts. Decode MTP-2: 176 / 453 / 1,256 tok/s (+32% / +30% / +5% over plain), acceptance 57%, gate 32/44, 20.0 sessions. MTP-1: 164 / 433 / 1,391 (+23% / +24% / +16%), acceptance 72%, gate 33/44, 21.6 sessions. Under mixed load (3 x 60k prefills) the mean common prefix with the single-stream plain output was 0.23 / 0.54 / 0.74 (short / tool / 20k) for MTP-2 and 0.12 / 0.60 / 0.59 for MTP-1, against 0.17 / 0.42 / 0.97 for plain against itself (plain is batch-nondeterministic on this hardware); the 20k figure is lower for both spec variants, 6 trials each, worth a second look but not the stock-build failure. All spec runs used Happy to run further variants on this hardware (sm86, Marlin W4A16 paths, hybrid GDN) if useful for review. |
Purpose
Performance follow-up to #53406 (the correctness fix for #52475). That fix stopped FULL cudagraph capture of spec-decode verify batches because the TurboQuant path serving them (the per-request Python prefill loop) is not graph-capturable. Correct, but expensive: with MTP enabled, every verify step runs the attention eagerly with per-layer Python iteration and CPU-resident metadata, and measured decode throughput dropped well below the fp8 KV baseline.
This PR teaches the backend spec-as-decode so verify batches run as decode batches and FULL cudagraphs come back:
supports_spec_as_decode=True, soreorder_batch_thresholdbecomes1 + num_speculative_tokensand verify batches classify as decodes.seq_len = seq_lens[r] - q_r + 1 + (t - query_start_loc[r])over request r's block table. This reproduces causal masking inside the chunk, whose K/V are already stored to the TQ cache bydo_kv_cache_update. The construction is numerically identical to the existing continuation-decode fast path in_prefill_attention; it is the same synthetic-seq_lens idea moved from per-layer CPU ints to once-per-step GPU tensors.searchsorted, gathers,index_select) into persistent buffers, so the decode kernels run unchanged withB = num_decode_tokensand stay FULL-cudagraph capturable. Replays observe fresh values through the capture-time buffer addresses; the unused tail of the seq_lens buffer is zeroed each build so padded rows of captured batches are no-ops.UNIFORM_BATCHcudagraph support is restored (sound now that the decode path serves spec batches)._prefill_attentionand_continuation_prefillremain for genuine chunked prefill.Not a duplicate: no open PR implements spec-as-decode for TurboQuant; the open TurboQuant PRs (#53060, #52745, #53231, #51082, #50248, #49798, #49465, #40858) touch decode kernel optimization, cache-shape resolution, or platform enablement. Stacked on #53406 (first commit); the second commit is this change.
Test Plan
Hardware: 2x RTX 5060 Ti (sm120), TP=2, Qwen3.8-27B NVFP4 (hybrid GDN, head_dim 256, GQA 6), greedy sampling.
--max-model-len 262144 --kv-cache-dtype turboquant_4bit_nc, MTP K=3, vision tower loaded; needle retrieval at 12k and 60k token prompts; decode throughput before/after.pytest tests/quantization/test_turboquant.py(mock config in the workspace-reservation tests gainsspeculative_config/compilation_config, plus a new test asserting the token-scaled decode workspace).Test Result
--enforce-eagercontrol ("Rome." / a well-formed<think>block and essay).turboquant_4bit_nc): needle found at 12k and 60k, finish reasonstop, draft acceptance in the mid-70s percent.test_turboquant.py: 111 passed, 17 skipped.AI assistance was used to develop and test this change and draft this PR; I reviewed every changed line and ran the tests above.