Repository navigation
feat(k3): Kimi-K3 vLLM plugin vision + DSpark draft support - #1910
Conversation
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
574f4ed to
0cc72e5
Compare
6742a7c to
b3ffe3e
Compare
|
@valarLip, @whx-sjtu, @zejunchen-zejun, |
|
@zejunchen-zejun, |
58445f2 to
1b937ea
Compare
|
Reviewed at Checked and cleared, so they don't get raised again:
1. Root cause: a correctly-sized immutable buffer was replaced by a smaller mutable one, and the old one was left in place as dead code
self._packed_start_loc = torch.arange(
self.vllm_config.scheduler_config.max_num_seqs + 1, dtype=torch.int32, device=device)
The new builder hook uses a different buffer: # kda_backend.py:73
query_start_loc_buf = self.non_spec_query_start_loc[: batch_size + 1]
The old helper is now dead: Two mechanisms for one field. Delete one — and if the one that survives is the builder-side staging, it needs the scheduler-capacity bound the other one had. 1a. The overflow is silent, and it relocates a CUDA-graph buffer
I measured what that actually does, rather than assuming: This is worth stating precisely because it is easy to describe wrongly: the buffer is not left holding uninitialized Reachable with 1b. The staging also runs where upstream allocates freshUpstream allocates a new Two Separately, on the full-CUDA-graph pure-decode path the staging's 2. Deleting
|
16cad94 to
e317056
Compare
Keep main's later K3/DSpark/core fixes; overlay only the still-unique plugin vision, DSpark draft, MLA/metadata, and recipe changes. Co-authored-by: Cursor <cursoragent@cursor.com>
The multimodal plugin classes in kimi_k3.py referenced symbols without importing them, which made Ruff fail in pre-checks CI. Co-authored-by: Cursor <cursoragent@cursor.com>
385af20 to
ff37588
Compare
Thanks for the double pass. The three cleared items stay as-is ( #1 Fixed. Dropped #2 Fixed. Restored #3 Fixed. Quant exclude mapping is applied after the HF→ATOM mapper and includes the #4 Partial, intentional for this PR. We keep vLLM’s #4a Device/dtype #4b K3 restores compressed-tensors → skip ( #5 Fixed. Initial #6 Fixed. #7 Fixed. Native #8 Partial. Tautological asserts are gone; real checks added. The multimodal-config guard lives in #9 Mostly fixed. Recipe no longer uses #10 Partial. Class renamed to Happy to take another look at 4a if you still want a full |
Comment 1 — Shared MLA attention changes are out of scope for a Kimi-K3 PRThis PR is titled "Kimi-K3 vLLM plugin vision + DSpark draft support", but it also changes three
Blast radius: every plugin-mode MLA model — DeepSeek-V3 / V3.2, GLM-5.x, Kimi-K2 / K2.5 — plus To be fair about the risk: the trims look correct, and on the V1 path they are no-ops Two other things about this hunk:
Ask: please either (a) split these three files into their own PR — they read like a Comment 2 — Extraction looks lossy: a no-op hunk plus a description that does not match the diffThe PR body describes this as a rebased extraction of the remaining delta from A no-op hunk in block_size = vllm_config.cache_config.block_size
return MLAAttentionSpec(
block_size=block_size,
...
)This extracts a local variable and changes nothing. A hunk like this is normally the residue of a Description claims that are not in the diff:
Ask: please rewrite the Motivation / Technical Details to describe exactly what this diff does, (Related: @valarLip's two inline threads are anchored on Comment 3 — The user-visible promise changed, but there is no test result and one load-bearing hook lost its only guardThis PR flips Kimi-K3 from "text-only" to "multimodal" in the recipe, removes Also note the recipe keeps the Separately, one behavioral guard was removed with nothing replacing it in tests. The old
That explicit hook is gone; the path is now |
Adds DSpark (`--speculative-config '{"method":"dspark"}'`) as a draft method for
DeepSeek-V4 in vLLM plugin mode, where ATOM owns the model and vLLM owns the
scheduler and KV cache manager.
Model
- `DeepseekV4Model.forward` optionally returns per-layer auxiliary hidden states
for EAGLE3/DSpark/DFlash drafts, averaged over the mHC dimension to match the
target the drafter was trained against. `DeepseekV4ForCausalLM` PCP-all-gathers
each aux tensor outside the compile boundary.
- The aux branch sits inside the `@support_torch_compile` region, so
`aux_hidden_state_layers` is baked into the graph. From `--level 2` up the
custom dispatcher replays code object 0 without evaluating guards, so it must
be set at load time and never mutated after; noted at the top of the file.
KV cache
- DSpark's draft needs a second KV group (sliding-window MLA, block 64) beside
the V4 proxy's block 128. `dspark_draft_kv_patch` registers the spec lazily and
exposes it for pickle via PEP-562 `__getattr__`.
- Prefix caching: V4's SWA ring is not carried by vLLM's block cache, so every
hit is rolled back by `max(win_with_spec, index_topk)` tokens and the tail
re-forwarded. Converted per group, since the two groups differ in block size.
The `index_topk` term is an empirical workaround for a sparse-indexer defect
that also reproduces with prefix caching off; flagged as a KNOWN ISSUE.
Correctness fixes found along the way
- `VocabParallelEmbedding.forward` uses `replicated_embedding` instead of
`F.embedding` on the `tp_size == 1` branch. `replicated_embedding` already
masks the -1 spec placeholder async scheduling can emit, and its comment
states VocabParallelEmbedding routes through the masked op -- true only of
the `tp_size > 1` branch. Single-GPU still reached a raw gather, which reads
the row before the table and GPU-faults.
- Async scheduling removes the sampled-token sync, so the pinned staging buffers
can tear. Gated with a depth-1 `torch.cuda.Event`.
- Idle DP ranks reach `build()` via `execute_dummy_batch()` with
`seq_len == query_len`; synthesized a decode context for that case.
- Piecewise cudagraph modes are demoted on the plugin path, a demotion vLLM
skips when it compiles nothing.
Request ids are passed to ATOM metadata builders through a thread-local instead
of a per-step `block_table[:, 0]` D2H copy, covering both of vLLM's unrelated
`GPUModelRunner` classes (V2 is forced for dspark).
Tests: 4 new/extended vLLM plugin test files; `tests/plugin` 201 passed. The 34
sglang failures are pre-existing and unrelated. black + ruff clean.
Cherry-pick onto main (174 commits later); three files conflicted:
- `deepseek_v4.py`: main grew `HCState.res_preshuffle` and threads it into
`hc_post`. The new aux-layer `hc_post` calls take it too -- the flag is
invariant across layers, and omitting it would reconstruct an aux residual
from the gfx1250 shuffled layout as if it were the ordinary one.
- `deepseek_v4_bridge.py`: main's req-id passthrough landed separately, so only
the comment wording conflicted. On the inline-build fallback main since
decided the opposite of this commit -- drop the allocator for throwaway slots
rather than feed it live req_ids -- so main's behavior is kept and the
now-dead `_req_ids` lookup dropped. The commit's `_warn_inline_v4_build`
diagnostic is kept ahead of it, which is still the anomaly it reports.
- `test_vllm_kimi_k3.py`: hunk dropped. It flipped `dcp_persistent_supported`
to False in `test_dense_mla_decode_pads_gathered_dcp_heads`; ROCm#1910 since
renamed that test to `..._keeps_supported_persistent_gathered_head_width`
and fixed the same inconsistency the other way -- keep True, assert 96/pad 0
-- which is what this commit's own comment says persistent does. git matched
the stale hunk onto the renamed sibling and broke it.
Tests: `tests/plugin` (sglang excluded, unrelated import errors) 262 passed,
47 failed -- byte-identical failure set to origin/main's baseline, +34 passing.
black/ruff not installed in this environment, so neither was run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds DSpark (`--speculative-config '{"method":"dspark"}'`) as a draft method for
DeepSeek-V4 in vLLM plugin mode, where ATOM owns the model and vLLM owns the
scheduler and KV cache manager.
Model
- `DeepseekV4Model.forward` optionally returns per-layer auxiliary hidden states
for EAGLE3/DSpark/DFlash drafts, averaged over the mHC dimension to match the
target the drafter was trained against. `DeepseekV4ForCausalLM` PCP-all-gathers
each aux tensor outside the compile boundary.
- The aux branch sits inside the `@support_torch_compile` region, so
`aux_hidden_state_layers` is baked into the graph. From `--level 2` up the
custom dispatcher replays code object 0 without evaluating guards, so it must
be set at load time and never mutated after; noted at the top of the file.
KV cache
- DSpark's draft needs a second KV group (sliding-window MLA, block 64) beside
the V4 proxy's block 128. `dspark_draft_kv_patch` registers the spec lazily and
exposes it for pickle via PEP-562 `__getattr__`.
- Prefix caching: V4's SWA ring is not carried by vLLM's block cache, so every
hit is rolled back by `max(win_with_spec, index_topk)` tokens and the tail
re-forwarded. Converted per group, since the two groups differ in block size.
The `index_topk` term is an empirical workaround for a sparse-indexer defect
that also reproduces with prefix caching off; flagged as a KNOWN ISSUE.
Correctness fixes found along the way
- `VocabParallelEmbedding.forward` uses `replicated_embedding` instead of
`F.embedding` on the `tp_size == 1` branch. `replicated_embedding` already
masks the -1 spec placeholder async scheduling can emit, and its comment
states VocabParallelEmbedding routes through the masked op -- true only of
the `tp_size > 1` branch. Single-GPU still reached a raw gather, which reads
the row before the table and GPU-faults.
- Async scheduling removes the sampled-token sync, so the pinned staging buffers
can tear. Gated with a depth-1 `torch.cuda.Event`.
- Idle DP ranks reach `build()` via `execute_dummy_batch()` with
`seq_len == query_len`; synthesized a decode context for that case.
- Piecewise cudagraph modes are demoted on the plugin path, a demotion vLLM
skips when it compiles nothing.
Request ids are passed to ATOM metadata builders through a thread-local instead
of a per-step `block_table[:, 0]` D2H copy, covering both of vLLM's unrelated
`GPUModelRunner` classes (V2 is forced for dspark).
Tests: 4 new/extended vLLM plugin test files; `tests/plugin` 201 passed. The 34
sglang failures are pre-existing and unrelated. black + ruff clean.
Cherry-pick onto main (174 commits later); three files conflicted:
- `deepseek_v4.py`: main grew `HCState.res_preshuffle` and threads it into
`hc_post`. The new aux-layer `hc_post` calls take it too -- the flag is
invariant across layers, and omitting it would reconstruct an aux residual
from the gfx1250 shuffled layout as if it were the ordinary one.
- `deepseek_v4_bridge.py`: main's req-id passthrough landed separately, so only
the comment wording conflicted. On the inline-build fallback main since
decided the opposite of this commit -- drop the allocator for throwaway slots
rather than feed it live req_ids -- so main's behavior is kept and the
now-dead `_req_ids` lookup dropped. The commit's `_warn_inline_v4_build`
diagnostic is kept ahead of it, which is still the anomaly it reports.
- `test_vllm_kimi_k3.py`: hunk dropped. It flipped `dcp_persistent_supported`
to False in `test_dense_mla_decode_pads_gathered_dcp_heads`; ROCm#1910 since
renamed that test to `..._keeps_supported_persistent_gathered_head_width`
and fixed the same inconsistency the other way -- keep True, assert 96/pad 0
-- which is what this commit's own comment says persistent does. git matched
the stale hunk onto the renamed sibling and broke it.
Tests: `tests/plugin` (sglang excluded, unrelated import errors) 262 passed,
47 failed -- byte-identical failure set to origin/main's baseline, +34 passing.
black/ruff not installed in this environment, so neither was run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds DSpark (`--speculative-config '{"method":"dspark"}'`) as a draft method for
DeepSeek-V4 in vLLM plugin mode, where ATOM owns the model and vLLM owns the
scheduler and KV cache manager.
Model
- `DeepseekV4Model.forward` optionally returns per-layer auxiliary hidden states
for EAGLE3/DSpark/DFlash drafts, averaged over the mHC dimension to match the
target the drafter was trained against. `DeepseekV4ForCausalLM` PCP-all-gathers
each aux tensor outside the compile boundary.
- The aux branch sits inside the `@support_torch_compile` region, so
`aux_hidden_state_layers` is baked into the graph. From `--level 2` up the
custom dispatcher replays code object 0 without evaluating guards, so it must
be set at load time and never mutated after; noted at the top of the file.
KV cache
- DSpark's draft needs a second KV group (sliding-window MLA, block 64) beside
the V4 proxy's block 128. `dspark_draft_kv_patch` registers the spec lazily and
exposes it for pickle via PEP-562 `__getattr__`.
- Prefix caching: V4's SWA ring is not carried by vLLM's block cache, so every
hit is rolled back by `max(win_with_spec, index_topk)` tokens and the tail
re-forwarded. Converted per group, since the two groups differ in block size.
The `index_topk` term is an empirical workaround for a sparse-indexer defect
that also reproduces with prefix caching off; flagged as a KNOWN ISSUE.
Correctness fixes found along the way
- `VocabParallelEmbedding.forward` uses `replicated_embedding` instead of
`F.embedding` on the `tp_size == 1` branch. `replicated_embedding` already
masks the -1 spec placeholder async scheduling can emit, and its comment
states VocabParallelEmbedding routes through the masked op -- true only of
the `tp_size > 1` branch. Single-GPU still reached a raw gather, which reads
the row before the table and GPU-faults.
- Async scheduling removes the sampled-token sync, so the pinned staging buffers
can tear. Gated with a depth-1 `torch.cuda.Event`.
- Idle DP ranks reach `build()` via `execute_dummy_batch()` with
`seq_len == query_len`; synthesized a decode context for that case.
- Piecewise cudagraph modes are demoted on the plugin path, a demotion vLLM
skips when it compiles nothing.
Request ids are passed to ATOM metadata builders through a thread-local instead
of a per-step `block_table[:, 0]` D2H copy, covering both of vLLM's unrelated
`GPUModelRunner` classes (V2 is forced for dspark).
Tests: 4 new/extended vLLM plugin test files; `tests/plugin` 201 passed. The 34
sglang failures are pre-existing and unrelated. black + ruff clean.
Cherry-pick onto main (174 commits later); three files conflicted:
- `deepseek_v4.py`: main grew `HCState.res_preshuffle` and threads it into
`hc_post`. The new aux-layer `hc_post` calls take it too -- the flag is
invariant across layers, and omitting it would reconstruct an aux residual
from the gfx1250 shuffled layout as if it were the ordinary one.
- `deepseek_v4_bridge.py`: main's req-id passthrough landed separately, so only
the comment wording conflicted. On the inline-build fallback main since
decided the opposite of this commit -- drop the allocator for throwaway slots
rather than feed it live req_ids -- so main's behavior is kept and the
now-dead `_req_ids` lookup dropped. The commit's `_warn_inline_v4_build`
diagnostic is kept ahead of it, which is still the anomaly it reports.
- `test_vllm_kimi_k3.py`: hunk dropped. It flipped `dcp_persistent_supported`
to False in `test_dense_mla_decode_pads_gathered_dcp_heads`; ROCm#1910 since
renamed that test to `..._keeps_supported_persistent_gathered_head_width`
and fixed the same inconsistency the other way -- keep True, assert 96/pad 0
-- which is what this commit's own comment says persistent does. git matched
the stale hunk onto the renamed sibling and broke it.
Tests: `tests/plugin` (sglang excluded, unrelated import errors) 262 passed,
47 failed -- byte-identical failure set to origin/main's baseline, +34 passing.
black/ruff not installed in this environment, so neither was run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds DSpark (`--speculative-config '{"method":"dspark"}'`) as a draft method for
DeepSeek-V4 in vLLM plugin mode, where ATOM owns the model and vLLM owns the
scheduler and KV cache manager.
Model
- `DeepseekV4Model.forward` optionally returns per-layer auxiliary hidden states
for EAGLE3/DSpark/DFlash drafts, averaged over the mHC dimension to match the
target the drafter was trained against. `DeepseekV4ForCausalLM` PCP-all-gathers
each aux tensor outside the compile boundary.
- The aux branch sits inside the `@support_torch_compile` region, so
`aux_hidden_state_layers` is baked into the graph. From `--level 2` up the
custom dispatcher replays code object 0 without evaluating guards, so it must
be set at load time and never mutated after; noted at the top of the file.
KV cache
- DSpark's draft needs a second KV group (sliding-window MLA, block 64) beside
the V4 proxy's block 128. `dspark_draft_kv_patch` registers the spec lazily and
exposes it for pickle via PEP-562 `__getattr__`.
- Prefix caching: V4's SWA ring is not carried by vLLM's block cache, so every
hit is rolled back by `max(win_with_spec, index_topk)` tokens and the tail
re-forwarded. Converted per group, since the two groups differ in block size.
The `index_topk` term is an empirical workaround for a sparse-indexer defect
that also reproduces with prefix caching off; flagged as a KNOWN ISSUE.
Correctness fixes found along the way
- `VocabParallelEmbedding.forward` uses `replicated_embedding` instead of
`F.embedding` on the `tp_size == 1` branch. `replicated_embedding` already
masks the -1 spec placeholder async scheduling can emit, and its comment
states VocabParallelEmbedding routes through the masked op -- true only of
the `tp_size > 1` branch. Single-GPU still reached a raw gather, which reads
the row before the table and GPU-faults.
- Async scheduling removes the sampled-token sync, so the pinned staging buffers
can tear. Gated with a depth-1 `torch.cuda.Event`.
- Idle DP ranks reach `build()` via `execute_dummy_batch()` with
`seq_len == query_len`; synthesized a decode context for that case.
- Piecewise cudagraph modes are demoted on the plugin path, a demotion vLLM
skips when it compiles nothing.
Request ids are passed to ATOM metadata builders through a thread-local instead
of a per-step `block_table[:, 0]` D2H copy, covering both of vLLM's unrelated
`GPUModelRunner` classes (V2 is forced for dspark).
Tests: 4 new/extended vLLM plugin test files; `tests/plugin` 201 passed. The 34
sglang failures are pre-existing and unrelated. black + ruff clean.
Cherry-pick onto main (174 commits later); three files conflicted:
- `deepseek_v4.py`: main grew `HCState.res_preshuffle` and threads it into
`hc_post`. The new aux-layer `hc_post` calls take it too -- the flag is
invariant across layers, and omitting it would reconstruct an aux residual
from the gfx1250 shuffled layout as if it were the ordinary one.
- `deepseek_v4_bridge.py`: main's req-id passthrough landed separately, so only
the comment wording conflicted. On the inline-build fallback main since
decided the opposite of this commit -- drop the allocator for throwaway slots
rather than feed it live req_ids -- so main's behavior is kept and the
now-dead `_req_ids` lookup dropped. The commit's `_warn_inline_v4_build`
diagnostic is kept ahead of it, which is still the anomaly it reports.
- `test_vllm_kimi_k3.py`: hunk dropped. It flipped `dcp_persistent_supported`
to False in `test_dense_mla_decode_pads_gathered_dcp_heads`; ROCm#1910 since
renamed that test to `..._keeps_supported_persistent_gathered_head_width`
and fixed the same inconsistency the other way -- keep True, assert 96/pad 0
-- which is what this commit's own comment says persistent does. git matched
the stale hunk onto the renamed sibling and broke it.
Tests: `tests/plugin` (sglang excluded, unrelated import errors) 262 passed,
47 failed -- byte-identical failure set to origin/main's baseline, +34 passing.
black/ruff not installed in this environment, so neither was run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Motivation
Land the remaining Kimi-K3 vLLM OOT work from
hexwang/k3_dspark_026_visiononto currentmainas a clean, single-commit PR:Earlier pieces of that branch already merged via squash PRs (#1752, #1782, #1806, #1876, #1820, etc.). This PR keeps
main's later versions of those shared files and overlays only the still-unique delta.Technical Details
K3DSparkModel→atom.plugin.vllm.models.kimi_k3_dspark:KimiK3DSparkVllmkimi_k3.pywith segmented KDA forward, aux hidden-state collection, and multimodal outer/inner classes; addkimi_k3to_PLUGIN_SUPPORTED_MULTIMODAL_MODELSrecipes/atom_vllm/Kimi-K3.md(prefix caching + DSpark speculative launch) and addrecipes/atom_vllm/KVV.mdplus vision design/plan docstests/plugin/test_vllm_kimi_k3.pySource branch (dirty history, 75 behind / 37 ahead):
hexwang/k3_dspark_026_vision. This PR is a rebased extraction of the remaining tip delta.Test Plan
black .andruff check .python -m pytest tests/(non-GPU unit tests, includingtests/plugin/test_vllm_kimi_k3.py)@support_torch_compiledecorated model files were modified (nativekimi_k3.pyuntouched)recipes/atom_vllm/Kimi-K3.mdInferact/Kimi-K3-DSparkandnum_speculative_tokens=7Test Result
Pending — draft PR for review of the extracted delta. Draft PRs do not trigger heavy CI.
Submission Checklist