Repository navigation
Conversation
Contributor
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
…tion Re-measure GSM8K over `vllm serve` with the same lm_eval harness the Kimi-K3 recipe uses, so the DSpark number and the target-only number are comparable rather than one offline and one served. Add per-position acceptance, which is what makes the causal-drafting trade-off legible, and document torch profiling, which moved to a server-side --profiler-config flag. Co-authored-by: Cursor <cursoragent@cursor.com>
The rationale claimed no aiter decode kernel returns a usable LSE, and pinned that on the persistent kernel leaving final_lse at FLT_MAX. That is backwards. Offline comparison against a torch reference showed the persistent kernel's LSE is correct across batch sizes, and #1746 has since switched DCP decode onto it precisely because gfx950 emits a per-token LSE for the cross-rank merge. The kernel that cannot be used is the split-KV one, whose stage2 reduce leaves final_lse untouched for a request whose valid split count collapses. So causal drafting is a scope decision -- one drafting semantics shared with ATOM's native DSpark, no merge to build -- not a kernel limitation, and non-causal is a buildable follow-up. Say that instead. Co-authored-by: Cursor <cursoragent@cursor.com>
whx-sjtu
force-pushed
the
hexwang/k3_dspark
branch
from
August 4, 2026 12:35
6d76192 to
53a2859
Compare
Numbers now come from a run against the rebased tree rather than the one taken before #1746 (DCP persistent PA) was integrated: 0.9545 flexible / 0.9538 strict, acceptance 3.777. All inside the error bar of the earlier 0.9522, as expected for a change that only re-sizes work descriptors. Also note the TimeoutError trap. A run whose requests hit the local-completions client timeout read 0.9424 while its acceptance figures stayed normal -- the retries re-generate under a different batch composition, and greedy chain of thought is sensitive enough to flip on that. Worth checking before anyone reports a regression from these numbers. Co-authored-by: Cursor <cursoragent@cursor.com>
whx-sjtu
force-pushed
the
hexwang/k3_dspark
branch
from
August 5, 2026 02:43
53a2859 to
db5629b
Compare
CI pipes ruff into reviewdog with -filter-mode=diff_context and -fail-on-error=true, so a finding anywhere the diff reaches fails the job -- including findings older than this branch. Two of the five were self-inflicted: an earlier `ruff --select I001 --fix` run reordered an import block this branch had no reason to touch, which dragged the pre-existing PLR0402 and BLE001 on those lines into the diff. Restore main's ordering and they leave the diff with it. The rest are this branch's own: drop two `setattr` calls with constant names in favour of plain assignment, and narrow the standalone-draft import guard from `Exception` to `ImportError`, which is the only thing two `from ... import` lines raise and the failure the warning already describes. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enables DSpark block-drafting speculative decoding for Kimi-K3 under the ATOM vLLM plugin backend, with the standalone Inferact/Kimi-K3-DSpark draft checkpoint. Validated at TP8 on MI355 with full CUDA graphs and asynchronous scheduling.
The draft is a standalone MLA model rather than an in-checkpoint MTP head, which is what most of this diff is about:
mla_decode_fwdmasks tail-aligned-causally whatever it is asked for, so that upper triangle would have to be merged in by log-sum-exp.spec_decode_patch.pydrafts causally instead for theK3DSparkModelarchitecture only, matching ATOM's native DSpark — one drafting semantics across both engines and no merge to build. This is a scope decision, not a kernel limitation: the persistent decode kernel does return a usable LSE on gfx950, which is what [feat][DCP] Switch DCP decode to persistent PA + make DCP compatible with MTP #1746 just put DCP's cross-rank merge on, so non-causal is a buildable follow-up. No other model's drafting is touched.SpeculativeConfigrewrites every non-Qwen3 DSpark draft into a DeepSeek-V4 one, which makes the Kimi-K3 draft config unresolvable; the patch keeps its own architecture.model_wrapper.pygeneralizes the eagle3-specific standalone-draft handling to cover DSpark, and stops exposingget_mtp_target_hidden_stateson non-MTP models, which MRV2 probes with a barehasattr.--kv-cache-dtype fp8.split_decodes_and_prefillscan route to_build_decode, i.e.1..reorder_batch_threshold. The sizes are not monotonic in query length, so this takes a per-slot maximum rather than the value at the largest width. Costs no extra memory: the descriptor shapes saturate well below the threshold.BlockTablesstores block ids in kernel blocks, but the DFlash speculator converts them back to slots with the manager's block size. Those agree for vLLM's own backends; ATOM's MLA pages by the token, so the draft wrote past its pool. Wrapped for theset_attncall only, and only when the two sizes actually differ.Validation
Full 1319-example GSM8K, 5-shot, over the served endpoint at 64 concurrency, TP8, fp8 KV,
FULL_AND_PIECEWISECUDA Graph, async scheduling on:The target alone scores 0.9553 ± 0.0057 under the same harness and concurrency (see the Kimi-K3 recipe), so drafting is accuracy neutral as expected for verify-based speculative decoding.
Acceptance over that run: mean accepted length 3.777 tokens per forward, acceptance rate 39.7%, over 36,119 verify forwards. Per-position acceptance rate is
0.923 / 0.774 / 0.602 / 0.442 / 0.034 / 0.002 / 0.000— causal drafting means the last three of the seven block positions contribute about 1% of accepted tokens between them, which is the cost of not needing the LSE merge.Also verified that prompts of every length from 1 to 20 tokens serve correctly. Prompts of 10 to 15 tokens are the interesting ones:
parallel_draftingwidensreorder_batch_thresholdto1 + 2 * num_speculative_tokens = 15, so their prefill is classified as a decode and reaches_build_decode. An accuracy sweep never covers that window.Rebased onto
mainover #1746 (DCP persistent PA), which touched the same two attention files. The work-descriptor sizing now uses that PR'spersistent_num_headsinside this PR's per-query-length worst-case loop, and its DCP-on-gfx942 persistent opt-out runs before this PR's capacity assertion. Every number above was measured against the rebased tree, and the length sweep and smoke were re-run on it too. Acceptance held at 3.77-3.79 across four separate runs spanning the pre- and post-#1746 trees.Test plan
vllm serve, 64 concurrency, fresh serverQuestion: What is 17 + 25? Answer:returns42)aiter::mla_a8w8_*target,aiter::mla_a16w16_*draft)