Skip to content

[vLLM-ATOM] Support DSpark speculative decoding for Kimi-K3 - #1789

Closed
whx-sjtu wants to merge 5 commits into
mainfrom
hexwang/k3_dspark
Closed

whx-sjtu wants to merge 5 commits into
mainfrom
hexwang/k3_dspark

Conversation

@whx-sjtu

@whx-sjtu whx-sjtu commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Enables DSpark block-drafting speculative decoding for Kimi-K3 under the ATOM vLLM plugin backend, with the standalone Inferact/Kimi-K3-DSpark draft checkpoint. Validated at TP8 on MI355 with full CUDA graphs and asynchronous scheduling.

The draft is a standalone MLA model rather than an in-checkpoint MTP head, which is what most of this diff is about:

  • Causal block drafting. vLLM's DSpark speculator drafts non-causally, so every block position attends to the whole block. ATOM's mla_decode_fwd masks tail-aligned-causally whatever it is asked for, so that upper triangle would have to be merged in by log-sum-exp. spec_decode_patch.py drafts causally instead for the K3DSparkModel architecture only, matching ATOM's native DSpark — one drafting semantics across both engines and no merge to build. This is a scope decision, not a kernel limitation: the persistent decode kernel does return a usable LSE on gfx950, which is what [feat][DCP] Switch DCP decode to persistent PA + make DCP compatible with MTP #1746 just put DCP's cross-rank merge on, so non-causal is a buildable follow-up. No other model's drafting is touched.
  • Standalone draft plumbing. SpeculativeConfig rewrites every non-Qwen3 DSpark draft into a DeepSeek-V4 one, which makes the Kimi-K3 draft config unresolvable; the patch keeps its own architecture. model_wrapper.py generalizes the eagle3-specific standalone-draft handling to cover DSpark, and stops exposing get_mtp_target_hidden_states on non-MTP models, which MRV2 probes with a bare hasattr.
  • Separate bf16 draft KV pool. The draft gets its own KV cache group, pinned to bf16 regardless of the target's --kv-cache-dtype fp8.
  • Persistent MLA work-descriptor sizing. The descriptors were sized for a 1-token decode but filled at the real decode width. They are now sized for the worst case over every width split_decodes_and_prefills can route to _build_decode, i.e. 1..reorder_batch_threshold. The sizes are not monotonic in query length, so this takes a per-slot maximum rather than the value at the largest width. Costs no extra memory: the descriptor shapes saturate well below the threshold.
  • Padded decode slots. A full decode graph replays at its captured request count, so vLLM pads the batch with slots carrying no query rows and no sequence. Those empty slots left the persistent kernel waiting on partials nothing produces; they now get one page of KV and their share of query rows.
  • DFlash slot-mapping units. BlockTables stores block ids in kernel blocks, but the DFlash speculator converts them back to slots with the manager's block size. Those agree for vLLM's own backends; ATOM's MLA pages by the token, so the draft wrote past its pool. Wrapped for the set_attn call only, and only when the two sizes actually differ.

Validation

Full 1319-example GSM8K, 5-shot, over the served endpoint at 64 concurrency, TP8, fp8 KV, FULL_AND_PIECEWISE CUDA Graph, async scheduling on:

exact_match
flexible-extract 0.9545 ± 0.0057
strict-match 0.9538 ± 0.0058

The target alone scores 0.9553 ± 0.0057 under the same harness and concurrency (see the Kimi-K3 recipe), so drafting is accuracy neutral as expected for verify-based speculative decoding.

Acceptance over that run: mean accepted length 3.777 tokens per forward, acceptance rate 39.7%, over 36,119 verify forwards. Per-position acceptance rate is 0.923 / 0.774 / 0.602 / 0.442 / 0.034 / 0.002 / 0.000 — causal drafting means the last three of the seven block positions contribute about 1% of accepted tokens between them, which is the cost of not needing the LSE merge.

Also verified that prompts of every length from 1 to 20 tokens serve correctly. Prompts of 10 to 15 tokens are the interesting ones: parallel_drafting widens reorder_batch_threshold to 1 + 2 * num_speculative_tokens = 15, so their prefill is classified as a decode and reaches _build_decode. An accuracy sweep never covers that window.

Rebased onto main over #1746 (DCP persistent PA), which touched the same two attention files. The work-descriptor sizing now uses that PR's persistent_num_heads inside this PR's per-query-length worst-case loop, and its DCP-on-gfx942 persistent opt-out runs before this PR's capacity assertion. Every number above was measured against the rebased tree, and the length sweep and smoke were re-run on it too. Acceptance held at 3.77-3.79 across four separate runs spanning the pre- and post-#1746 trees.

Test plan

  • Full GSM8K 1319 examples over vllm serve, 64 concurrency, fresh server
  • Prompt length sweep 1..20 tokens against the served endpoint
  • Deterministic smoke from the recipe (Question: What is 17 + 25? Answer: returns 42)
  • Torch profile at 8k input / 100 output / 16 concurrency with stacks, both models visible under their own MLA kernels (aiter::mla_a8w8_* target, aiter::mla_a16w16_* draft)
  • Reviewer: confirm no regression for DeepSeek-V4 MTP and MiniMax-M3 EAGLE3, which share the patched speculative-decoding paths

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 1789 --add-label <label>

whx-sjtu and others added 3 commits August 4, 2026 12:34
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
…tion

Re-measure GSM8K over `vllm serve` with the same lm_eval harness the Kimi-K3
recipe uses, so the DSpark number and the target-only number are comparable
rather than one offline and one served. Add per-position acceptance, which is
what makes the causal-drafting trade-off legible, and document torch profiling,
which moved to a server-side --profiler-config flag.

Co-authored-by: Cursor <cursoragent@cursor.com>
The rationale claimed no aiter decode kernel returns a usable LSE, and pinned
that on the persistent kernel leaving final_lse at FLT_MAX. That is backwards.
Offline comparison against a torch reference showed the persistent kernel's LSE
is correct across batch sizes, and #1746 has since switched DCP decode onto it
precisely because gfx950 emits a per-token LSE for the cross-rank merge. The
kernel that cannot be used is the split-KV one, whose stage2 reduce leaves
final_lse untouched for a request whose valid split count collapses.

So causal drafting is a scope decision -- one drafting semantics shared with
ATOM's native DSpark, no merge to build -- not a kernel limitation, and
non-causal is a buildable follow-up. Say that instead.

Co-authored-by: Cursor <cursoragent@cursor.com>
@whx-sjtu
whx-sjtu force-pushed the hexwang/k3_dspark branch from 6d76192 to 53a2859 Compare August 4, 2026 12:35
@zufayu
zufayu requested a review from valarLip August 5, 2026 01:39
Numbers now come from a run against the rebased tree rather than the one taken
before #1746 (DCP persistent PA) was integrated: 0.9545 flexible / 0.9538
strict, acceptance 3.777. All inside the error bar of the earlier 0.9522, as
expected for a change that only re-sizes work descriptors.

Also note the TimeoutError trap. A run whose requests hit the local-completions
client timeout read 0.9424 while its acceptance figures stayed normal -- the
retries re-generate under a different batch composition, and greedy chain of
thought is sensitive enough to flip on that. Worth checking before anyone
reports a regression from these numbers.

Co-authored-by: Cursor <cursoragent@cursor.com>
@whx-sjtu
whx-sjtu force-pushed the hexwang/k3_dspark branch from 53a2859 to db5629b Compare August 5, 2026 02:43
CI pipes ruff into reviewdog with -filter-mode=diff_context and
-fail-on-error=true, so a finding anywhere the diff reaches fails the job --
including findings older than this branch.

Two of the five were self-inflicted: an earlier `ruff --select I001 --fix` run
reordered an import block this branch had no reason to touch, which dragged the
pre-existing PLR0402 and BLE001 on those lines into the diff. Restore main's
ordering and they leave the diff with it.

The rest are this branch's own: drop two `setattr` calls with constant names in
favour of plain assignment, and narrow the standalone-draft import guard from
`Exception` to `ImportError`, which is the only thing two `from ... import`
lines raise and the failure the warning already describes.

Co-authored-by: Cursor <cursoragent@cursor.com>
@whx-sjtu

Copy link
Copy Markdown
Contributor Author

Superseded by #1899, which brings this to the vLLM 0.26.1 line (targets vllm0.26.1, the #1839 upgrade branch) with the DSpark optimizations from #1876 already in place. Closing in favour of that one.

@whx-sjtu whx-sjtu closed this Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant