Repository navigation
[feat](vllm-atom k3): support DSpark speculative decoding under DCP - #2033
Merged
Merged
Conversation
Lifts the MLA-DSpark DCP config guard, localizes the draft's KV slots and seq lens to the DCP rank, routes causal multi-token decode through aiter's round-robin (cprr) kernel, and restores the KDA reorder_batch_threshold that vLLM otherwise clamps to 1 under DCP. TP8 + DCP8 + N=7 measures gsm8k 5-shot 0.9538, matching the no-DCP baseline (0.9538), with ~52% draft acceptance. Needs --gpu-memory-utilization 0.75: FULL cudagraph capture OOMs on a 9.35 GiB scratch alloc at 0.85. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
gpu-memory-utilization drops 0.93 -> 0.75 because it has to: DCP8 makes every rank compute all 128 MLA heads over its 1/8 KV shard, and with DSpark's 8-token queries the FULL cudagraph capture needs ~10 GiB of scratch on top of a 38 GiB graph pool. At 0.93 capture deadlocks, at 0.85 it OOMs on a 9.35 GiB alloc. Measured at 0.75: gsm8k 5-shot 0.9538 / 0.9530, matching the no-DCP baseline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
valarLip
previously approved these changes
Aug 26, 2026
valarLip
reviewed
Aug 26, 2026
| draft_model_path: "Inferact/Kimi-K3-DSpark" | ||
| client_command: "lm_eval --model local-completions --model_args model=${MODEL_PATH},base_url=http://127.0.0.1:${VLLM_PORT}/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,trust_remote_code=True --tasks gsm8k --num_fewshot ${LM_EVAL_NUM_FEWSHOT} --output_path ${OUTPUT_PATH}" | ||
| extra_args: "--trust-remote-code --language-model-only --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 16384 --max-num-seqs 64 --max-num-batched-tokens 16384 --gpu-memory-utilization 0.93 --block-size 128 --enable-prefix-caching --mamba-cache-mode align --compilation-config '{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}' --additional-config '{\"online_quant_config\":{\"global_quant_config\":\"ptpc_fp8\",\"exclude_layer\":[\"lm_head\",\"model.embed_tokens\",\"*self_attn.[qkv]_conv1d*\",\"*block_sparse_moe.experts*\",\"*block_sparse_moe.routed_expert_*\",\"*vision_tower*\",\"*mm_projector*\"]}}' --speculative-config '{\"method\":\"dspark\",\"model\":\"/models/Inferact/Kimi-K3-DSpark\",\"num_speculative_tokens\":7}'" | ||
| extra_args: "--trust-remote-code --language-model-only --tensor-parallel-size 8 --decode-context-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 16384 --max-num-seqs 64 --max-num-batched-tokens 16384 --gpu-memory-utilization 0.75 --block-size 128 --enable-prefix-caching --mamba-cache-mode align --compilation-config '{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}' --additional-config '{\"online_quant_config\":{\"global_quant_config\":\"ptpc_fp8\",\"exclude_layer\":[\"lm_head\",\"model.embed_tokens\",\"*self_attn.[qkv]_conv1d*\",\"*block_sparse_moe.experts*\",\"*block_sparse_moe.routed_expert_*\",\"*vision_tower*\",\"*mm_projector*\"]}}' --speculative-config '{\"method\":\"dspark\",\"model\":\"/models/Inferact/Kimi-K3-DSpark\",\"num_speculative_tokens\":7}'" |
Collaborator
There was a problem hiding this comment.
why --gpu-memory-utilization 0.75
Contributor
Author
There was a problem hiding this comment.
DCP 下 persistent_num_heads 会按 dcp_size 放大,每个 rank 承担全部 head(16 → 128,8×),DSpark 让 query 长度从 1 变成 8,而 packed_qo_len = 8 × 128 = 1024 跨过了 tiling 阈值,使 max_qo_tiles_per_batch 从 1 跳到 8,reduce_partial_map 又涨 8 倍。合起来相对无 DCP 的普通 decode 是几百倍。
native ATOM最大也只能开到0.85,vllm-atom如果减小graph capture size到64也能开到0.85。整体上vllm比atom的显存消耗还要高一点。
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PerryZhang01
pushed a commit
that referenced
this pull request
Sep 3, 2026
`Accuracy (Kimi-K3 DSpark DCP8 TP8)` cannot start. The engine aborts
while sizing the KV cache:
INFO gpu_worker.py:563] Available KV cache memory: -5.07 GiB
ValueError: No available memory for the cache blocks.
Try increasing `gpu_memory_utilization`
The job is red with no score at all, which reads like an accuracy or
spec-decode regression and is neither.
#2033 moved this entry to DCP8 and set `--gpu-memory-utilization` from
0.93 to 0.7. Kimi-K3 is ~187 GiB of weights per GPU at TP8; on a 288 GiB
MI355X a 0.7 budget is 201.6 GiB, which does not leave a KV cache behind
after weights and activations. The threshold to merely reach zero is
0.7 + 5.07/288 ~= 0.718, so 0.7 is not marginally low -- it is on the
wrong side of the window.
The window is narrow in both directions, which is why 0.7 looks like a
safe choice and is not: FULL cudagraph capture takes its scratch OUTSIDE
the vLLM budget, so a HIGHER utilization starves capture (0.85 dies with
`OutOfMemoryError` in `mla_decode_fwd`; 0.93 leaves so little headroom
that one rank stalls in allocation while the others leave the capture
context and its MoE all_reduce deadlocks). 0.75 is the value that
satisfies both ends.
Measured on 8x MI355X through the CI path itself
(`.github/scripts/atom_oot_test.sh accuracy ci`, this matrix entry), with
every GPU verified idle (<1 GiB) before each launch:
gpu-mem KV cache FULL capture outcome
0.70 -5.07 GiB never reached engine abort (reproduced twice,
bit-identical)
0.75 +9.33 GiB 48/48 ok serves; gsm8k 5-shot 0.9454,
DSpark acceptance 0.514
The +14.40 GiB delta is exactly 0.05 x 287.98 GiB, confirming the budget
is the only variable in play.
Acceptance 0.514 matches the ~52% recorded for DCP8 + DSpark N=7, so the
draft head is healthy. Note the run still lands under the 0.95 accuracy
threshold; that is a separate question from being able to start at all,
and is being characterized separately -- this commit only restores a
server that can allocate a KV cache.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gyohuangxin
pushed a commit
that referenced
this pull request
Sep 3, 2026
* Change atom-plugin CI runner to spur-runner-mi355x-8gpu * Move plugin CI logs onto shared_nfs so Spur runners can write them. Co-authored-by: Cursor <cursoragent@cursor.com> * Write plugin CI logs under the GitHub runner workspace. Co-authored-by: Cursor <cursoragent@cursor.com> * Run vLLM and SGLang plugin CI directly on the GPU runner. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(v4): keep the profile forward off the decode running_tokens branch Every vllm-atom DeepSeek-V4 server has died at startup since #2042. All four workers abort inside `determine_available_memory` -> `profile_run`: AttributeError: 'types.SimpleNamespace' object has no attribute 'max_seqlen_q' atom/plugin/vllm/deepseek_v4_bridge.py:2164 atom/utils/forward_context.py:296 running_tokens_from_bs `force_dummy` builds `attn_metadata` as a bare namespace carrying only `state` and `in_hipgraph` (#1839). That is deliberate: the real builder allocates and copies tensors, which HIP forbids under stream capture. The namespace has therefore never had a sequence length. #2042 replaced `graph_bs=batch_size` at this call site with `running_tokens_from_bs(...)`, which dereferences `attn_metadata.max_seqlen_q` on its decode branch. `force_dummy` also forces `is_prefill=False`, so the profile forward takes that branch unconditionally -- it cannot not crash. Pass the `None` the helper already documents as "count only", which restores exactly the `batch_size` the replaced expression produced. The fix goes at the call site, where `force_dummy` is known, rather than loosening the helper: a real decode metadata that is missing `max_seqlen_q` should still fail loudly instead of silently under-padding the MoE all_gather. Not a new-model or spec-decode issue -- it reproduces on plain `DeepSeek-V4-Flash TP4`, the PR-CI accuracy job, and on the nightly's `DeepSeek-V4-Pro TP8`, which shares this bridge. Verified on 4x MI355X via the CI path itself (`.github/scripts/atom_oot_test.sh accuracy ci`, matrix entry "DeepSeek-V4-Flash TP4"): the job goes from dying at startup with no score at all to gsm8k 20-shot flexible 0.9568 / strict 0.9568 against a 0.93 threshold, in 9 minutes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ci): give Kimi-K3 DCP8 enough budget to allocate a KV cache `Accuracy (Kimi-K3 DSpark DCP8 TP8)` cannot start. The engine aborts while sizing the KV cache: INFO gpu_worker.py:563] Available KV cache memory: -5.07 GiB ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization` The job is red with no score at all, which reads like an accuracy or spec-decode regression and is neither. #2033 moved this entry to DCP8 and set `--gpu-memory-utilization` from 0.93 to 0.7. Kimi-K3 is ~187 GiB of weights per GPU at TP8; on a 288 GiB MI355X a 0.7 budget is 201.6 GiB, which does not leave a KV cache behind after weights and activations. The threshold to merely reach zero is 0.7 + 5.07/288 ~= 0.718, so 0.7 is not marginally low -- it is on the wrong side of the window. The window is narrow in both directions, which is why 0.7 looks like a safe choice and is not: FULL cudagraph capture takes its scratch OUTSIDE the vLLM budget, so a HIGHER utilization starves capture (0.85 dies with `OutOfMemoryError` in `mla_decode_fwd`; 0.93 leaves so little headroom that one rank stalls in allocation while the others leave the capture context and its MoE all_reduce deadlocks). 0.75 is the value that satisfies both ends. Measured on 8x MI355X through the CI path itself (`.github/scripts/atom_oot_test.sh accuracy ci`, this matrix entry), with every GPU verified idle (<1 GiB) before each launch: gpu-mem KV cache FULL capture outcome 0.70 -5.07 GiB never reached engine abort (reproduced twice, bit-identical) 0.75 +9.33 GiB 48/48 ok serves; gsm8k 5-shot 0.9454, DSpark acceptance 0.514 The +14.40 GiB delta is exactly 0.05 x 287.98 GiB, confirming the budget is the only variable in play. Acceptance 0.514 matches the ~52% recorded for DCP8 + DSpark N=7, so the draft head is healthy. Note the run still lands under the 0.95 accuracy threshold; that is a separate question from being able to start at all, and is being characterized separately -- this commit only restores a server that can allocate a KV cache. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(vllm): restore the GDN FULL-cudagraph decode metadata compaction vLLM pads a FULL-cudagraph decode batch out to a captured graph size and leaves the padded request rows in the GDN metadata. ATOM's GDN decode kernel is request-indexed, so it walks those rows and folds them into ssm_state. #1985 deleted the pass that compacted them away, to fix Qwen3.5-35B-A3B block-FP8 -- but it scoped only its MHA/MoE half to 35B-A3B and dropped the GDN pass for every GDN model. Kimi-K3 kept an identical copy in its own KDA builder (AtomKimiK3KDAMetadataBuilder._adapt_full_graph_decode_metadata); Qwen3.5-397B was left with nothing. That is what took Accuracy (Qwen3.5-397B-A17B-MXFP4 TP4) from the 0.85-0.86 it held on the nightly dashboard through 2026-08-24 down under its 0.83 gate, on every branch that has run it since. Put the pass back and give it the same 35B-A3B exemption the rest of #1985 uses, so that fix is untouched. Measured on 4x gfx950 with the runner's own image assembly (rocm/atom-dev:vllm-latest + the aiter nightly wheel CI resolves + this branch), gsm8k 3-shot x1319 at the harness's FULL_AND_PIECEWISE: as CI runs it today 0.7817 this change 0.8544 whole #1985 GDN revert (reference) 0.8431 cudagraph_mode=PIECEWISE (reference) 0.8355 The 35B-A3B exemption is covered by unit tests but not by an end-to-end 35B block-FP8 run; that path keeps today's behavior either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(vllm): drop the model scope from the GDN metadata compaction The previous commit kept the pass off for Qwen3.5-35B-A3B, mirroring how #1985 scoped its MHA/MoE half. That guard was cargo-culted: a metadata pass has no business knowing which model it is running under, and the reason #1985 gave for removing the pass does not survive checking. Why the pass is unconditionally needed. vLLM means to mark the padded rows itself: non_spec_state_indices_tensor[num_decodes:].fill_(NULL_BLOCK_ID) but num_decodes comes from split_decodes_and_prefills, whose fast path if max_query_len <= decode_threshold and ...: return num_reqs, 0, num_tokens, 0 fires for any pure-decode batch (max_query_len == 1) and returns num_decodes = num_reqs, i.e. the *padded* request count under FULL-graph replay. The fill therefore starts at num_decodes == batch_size and is a no-op: the padded slots keep whatever block_table_tensor[:, 0] held, and ATOM's request-indexed GDN decode kernel folds them into ssm_state. That depends only on the decode batch being padded -- not on the model, its quantization, or TP. It is the same reason Kimi-K3 carries the pass in its own KDA builder. Why #1985's reason does not hold. It removed the pass because it "rewrote that metadata using num_actual_tokens (token-padded graph size) and shrunk num_decodes". That was true of the original version (2255001) and was fixed two versions earlier, in 441476e, which switched to num_reqs with a comment saying exactly why. Qwen3.5-397B-A17B-MXFP4 TP4, gsm8k 3-shot x1319, 4x gfx950, runner image assembly, FULL_AND_PIECEWISE: 0.8423 (scoped variant measured 0.8544; the two are within one stderr of each other, and both clear the 0.83 gate against 0.7817 before the fix). The 35B-A3B block-FP8 path is now covered by the pass again. I could not score it end to end -- this box has the checkpoint but no validated vLLM plugin recipe for it, and my attempt floored at 0.05-0.07 under both FULL and PIECEWISE, so it measured nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(vllm): keep the MTP position-0 mask off the layer signature Every vLLM-plugin MTP run has been dying in profile_run since the fused FP8 prologue landed: TypeError: DeepSeekMultiTokenPredictorLayer.forward() takes from 5 to 6 positional arguments but 7 were given The plugin installed its position-0 input-embedding mask by rebinding each draft layer's forward to a wrapper that re-called the original with a hardcoded five-argument list. When the fused prologue gave the layer a sixth argument (eh_input_quant), the wrapper kept advertising the old arity -- and functools.wraps kept the original qualname, so the error pointed at a model signature that was in fact correct. The wrapper could not have worked on the fused path anyway: there the embedding never materializes as a tensor the layer receives, so there is nothing for a layer-level wrapper to mask. Move the mask into the predictors, which own the embedding lookup on both paths and cannot drift from their own layers. On the unfused path it is the same torch.where as before. On the fused path the masked rows are fed the sentinel id -1: the prologue kernel already emits a zero embedding for any id outside [0, vocab_size), and RMSNorm(0) is 0, so the result is bit-identical to zeroing inputs_embeds -- without materializing the [num_tokens, hidden] tensor the fusion exists to avoid. The plugin now flips a flag instead of rewriting a signature, and raises if an arch in _MTP_MASK_INPUT_ARCH has no predictor to flip it on, rather than silently serving an unmasked draft. The regression test lives in tests/ rather than tests/plugin/ because run_unit_tests.sh skips the latter. Fixes the GLM-5.2-MXFP4-MTP TP4 plugin accuracy job. * ci: download plugin CI draft models like the target Kimi-K3 DSpark failed because speculative-config pointed at a missing local draft path. Cache the draft the same way as the target and rewrite extra_args to the resolved directory. Co-authored-by: Cursor <cursoragent@cursor.com> * ci: SIGKILL leftover host/KFD VRAM holders after plugin docker cleanup Plugin HIP smoke was failing on spur runners when VRAM stayed allocated by host PIDs after leftover containers were already gone. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: perzhang <perzhang@amd.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: PerryZhang01 <Perry.Zhang@amd.com> Co-authored-by: Junyan Yang <junyyang@crs-m2m-cpu-spur-v2-009.us-east2-a.compute.internal> Co-authored-by: Junyan Yang <junyyang@crs-m2m-cpu-spur-012.us-east2-a.compute.internal>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
This PR supports dcp + dspark kimi k3 on vllm-atom backend
Technical Details
Two runtime patches against vLLM
dspark_dcp_patch.pyAttention metadata
metadata.pyTest Result
flexible-extrac:0.9538