Skip to content

feat(k3): Kimi-K3 vLLM plugin vision + DSpark draft support - #1910

Merged
sajandhy merged 8 commits into
ROCm:mainfrom
sajandhy:hexwang/k3_dspark_026_vision_rebased
Sep 20, 2026
Merged

sajandhy merged 8 commits into
ROCm:mainfrom
sajandhy:hexwang/k3_dspark_026_vision_rebased

Conversation

@sajandhy

@sajandhy sajandhy commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Note: This PR is being opened on behalf of others. The implementation is primarily from whx-sjtu / hexwang (xiaowang990929@gmail.com), ganyi (ygan@amd.com), Guanbao Yu (Guanbao.Yu@amd.com), and zejchen (zejchen@amd.com) on branch hexwang/k3_dspark_026_vision; I am submitting a cleaned rebased extraction to the upstream repo from my fork.

Land the remaining Kimi-K3 vLLM OOT work from hexwang/k3_dspark_026_vision onto current main as a clean, single-commit PR:

  • Multimodal / vision path for Kimi-K3 in the vLLM plugin
  • DSpark draft model registration and auxiliary hidden-state taps for speculative decoding
  • MLA / metadata / model-wrapper plumbing needed for mixed KDA batches and draft KV alignment
  • Recipe and design/plan docs for the plugin launch and DSpark speculative config

Earlier pieces of that branch already merged via squash PRs (#1752, #1782, #1806, #1876, #1820, etc.). This PR keeps main's later versions of those shared files and overlays only the still-unique delta.

Technical Details

  • Register K3DSparkModel → atom.plugin.vllm.models.kimi_k3_dspark:KimiK3DSparkVllm
  • Extend plugin kimi_k3.py with segmented KDA forward, aux hidden-state collection, and multimodal outer/inner classes; add kimi_k3 to _PLUGIN_SUPPORTED_MULTIMODAL_MODELS
  • Attention / metadata / model-wrapper updates for multi-token draft queries and dtype-aligned draft KV pages
  • Update recipes/atom_vllm/Kimi-K3.md (prefix caching + DSpark speculative launch) and add recipes/atom_vllm/KVV.md plus vision design/plan docs
  • Expand tests/plugin/test_vllm_kimi_k3.py

Source branch (dirty history, 75 behind / 37 ahead): hexwang/k3_dspark_026_vision. This PR is a rebased extraction of the remaining tip delta.

Test Plan

  • black . and ruff check .
  • python -m pytest tests/ (non-GPU unit tests, including tests/plugin/test_vllm_kimi_k3.py)
  • Confirm no @support_torch_compile decorated model files were modified (native kimi_k3.py untouched)
  • vLLM plugin smoke: text-only Kimi-K3 serve per recipes/atom_vllm/Kimi-K3.md
  • Multimodal / vision path smoke (when image assets available)
  • DSpark speculative decode smoke with Inferact/Kimi-K3-DSpark and num_speculative_tokens=7

Test Result

Pending — draft PR for review of the extracted delta. Draft PRs do not trigger heavy CI.

Submission Checklist

@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 1910 --add-label <label>

@sajandhy
sajandhy force-pushed the hexwang/k3_dspark_026_vision_rebased branch from 574f4ed to 0cc72e5 Compare August 17, 2026 03:57
@sajandhy
sajandhy marked this pull request as ready for review August 18, 2026 03:40
@sajandhy
sajandhy requested a review from valarLip August 18, 2026 06:01
@sajandhy
sajandhy force-pushed the hexwang/k3_dspark_026_vision_rebased branch from 6742a7c to b3ffe3e Compare August 19, 2026 15:40
Comment thread atom/models/kimi_k3_dspark.py Resolved
@zufayu
zufayu requested a review from yhl-amd August 20, 2026 00:52
@sajandhy
sajandhy requested a review from valarLip August 20, 2026 20:19
@sajandhy

Copy link
Copy Markdown
Contributor Author

@valarLip, @whx-sjtu, @zejunchen-zejun,
Please complete the review.
Thanks
Satya

@sajandhy

sajandhy commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

@zejunchen-zejun,
Please take a look at this PR.
Thank you

@sajandhy
sajandhy force-pushed the hexwang/k3_dspark_026_vision_rebased branch from 58445f2 to 1b937ea Compare September 6, 2026 16:32
@valarLip

Copy link
Copy Markdown
Collaborator

Reviewed at 16cad94a9, twice, with the second pass checking the first.

Checked and cleared, so they don't get raised again:

  • state_indices[real_num_decodes:].fill_(PAD_SLOT_ID) diverging from upstream's NULL_BLOCK_ID is not newly armed by deleting the early return. That return fired when real_num_decodes == metadata.num_decodes, and the enclosing guard has already established num_prefills == 0 and num_spec_decodes == 0, so on a pure-decode batch num_decodes == num_reqs == batch_size and the filled slice is empty in exactly that case. The -1 vs 0 semantic difference remains latent, as it was before.
  • hf_to_atom_mapper's prefix rule is correct for the real checkpoint — /data/Kimi-K3 does store language_model.model.*, and there is no top-level lm_head. key.
  • self.config.media_placeholder_token_id without a default matches upstream amd/model.py:136 exactly.

1. Root cause: a correctly-sized immutable buffer was replaced by a smaller mutable one, and the old one was left in place as dead code

kimi_k3.py:303 already solves this problem, and its docstring says so:

self._packed_start_loc = torch.arange(
    self.vllm_config.scheduler_config.max_num_seqs + 1, dtype=torch.int32, device=device)

Allocated at scheduler capacity and only sliced, so a captured graph keeps pointing at the same memory.

The new builder hook uses a different buffer:

# kda_backend.py:73
query_start_loc_buf = self.non_spec_query_start_loc[: batch_size + 1]

self.non_spec_query_start_loc is torch.empty(decode_cudagraph_max_bs + 1) (/app/vllm/vllm/v1/attention/backends/gdn_attn.py:156), and decode_cudagraph_max_bs = min(max_num_seqs * (num_spec + 1), max_cudagraph_capture_size) (:117-124). So the replacement is min(max_num_seqs·k, max_cudagraph_capture_size) + 1 where the original was max_num_seqs + 1 — strictly smaller whenever the capture size clamps — and it is rewritten every step where the original was written once.

The old helper is now dead: kimi_k3.py:289-293 calls it only if kda_metadata.non_spec_query_start_loc is None, and the new hook sets that field on every num_decodes > 0 build. Its docstring ("vLLM leaves non_spec_query_start_loc unset there") is now false.

Two mechanisms for one field. Delete one — and if the one that survives is the builder-side staging, it needs the scheduler-capacity bound the other one had.

1a. The overflow is silent, and it relocates a CUDA-graph buffer

_stage_packed_decode_query_start_loc has neither guard its sibling four lines below carries — no use_full_cuda_graph gate, and no batch_size > self.decode_cudagraph_max_bs check (_adapt_full_graph_decode_metadata:105). When num_decodes exceeds the buffer, the [: num_decodes + 1] slice clamps and torch.arange(out=...) gets a short out.

I measured what that actually does, rather than assuming:

storage ptr   before/after: 258541504 -> 413869568   MOVED
storage bytes before/after: 2052 -> 2804
one UserWarning, no exception

This is worth stating precisely because it is easy to describe wrongly: the buffer is not left holding uninitialized torch.empty — the arange values do land, and buf[512] == 512 afterwards. What happens is that torch.arange(out=<short view>) reallocates the underlying storage, so the persistent buffer moves to a new address. That buffer exists specifically for CUDA-graph capture, and a captured graph holds the old address. Every replay after the move reads freed memory.

Reachable with --max-num-seqs 1024 and a capture max of 512: any step with more than 512 decodes.

1b. The staging also runs where upstream allocates fresh

Upstream allocates a new torch.zeros for non_spec_query_start_loc on every non-cudagraph build (kda_metadata.py:543) and reuses the persistent self.non_spec_query_start_loc only inside the full-cudagraph branch. The new helper hands out the shared slice unconditionally, with no use_full_cuda_graph gate.

Two build() calls against the same builder in one forward — TBO/DBO micro-batching, or a spec step that builds twice — make the second overwrite the first's storage, so ubatch 0 runs with ubatch 1's offsets. On the eager and piecewise paths it also burns an arange + fill_ per step where upstream touches nothing.

Separately, on the full-CUDA-graph pure-decode path the staging's arange + fill_ are provably discarded four lines later: _adapt_full_graph_decode_metadata writes the identical slice with its compact cumsum copy. Two wasted launches per decode step.

2. Deleting if real_num_decodes == metadata.num_decodes: return

-        if real_num_decodes == metadata.num_decodes:
-            return

No comment explains it, and it has three consequences.

Correctness. _adapt_full_graph_decode_metadata now runs on the ordinary unpadded decode step, and it calls mamba_get_block_table_tensor(...) unconditionally, ignoring mamba_aligned_state_indices entirely. Upstream refuses to re-derive state indices from the block table in align cache mode — kda_metadata.py:381-393 uses the precomputed field and its fallback carries assert not use_v2_model_runner, "Aligned Mamba state indices must be precomputed". Run Kimi-K3 with --mamba-cache-mode align on the V2 runner and every decode step overwrites non_spec_state_indices_tensor with slots upstream considers invalid in that mode: requests read and write the wrong recurrent-state slot. Latent only because the mode defaults to none.

Performance. block_table_tensor[real_decode_mask, 0] (:123) is boolean-mask advanced indexing on a CUDA tensor, which lowers to nonzero() and must synchronize the stream to size its output — a hipStreamSynchronize now on the host critical path every step, which is what CPU run-ahead is. Add two .item() calls, a CPU torch.zeros + cumsum, two pageable H2D copies, and mamba_get_block_table_tensor itself (6 kernels and 4 allocations in align mode). In the case the guard used to cover, every value computed equals what upstream already produced — pure overhead, no correctness gain.

Divergence from the twin. gdn_backend.py:89 still has the identical guard, and AtomGDNAttentionMetadataBuilder's class docstring at :46 states "Kimi-K3 runs the same pass from its own builder (AtomKimiK3KDAMetadataBuilder._adapt_full_graph_decode_metadata)". That sentence is now false. Either both need the change or neither does, and the docstring asserting parity will mislead whoever next touches either file.

3. quant_exclude_name_mapping is applied on top of the new hf_to_atom_mapper, and its values are missing a root

Both renames now run, in sequence, on quant_config.exclude_layers: model_wrapper.py:462 applies hf_to_atom_mapper via remap_layer_name, then :479 applies quant_exclude_name_mapping via apply_exclude_name_mapping — an unanchored str.replace (config.py:525). The previously-registered class had no hf_to_atom_mapper and an identity exclude mapping, so both steps were no-ops; this PR makes both mutate.

Simulated:

input     : language_model.model.layers.0.mlp
after #1  : language_model.language_model.model.layers.0.mlp
after #2  : language_model.language_model.language_model.model.layers.0.mlp

The substring language_model.model. reappears at offset 15 after the first pass, so the second pass fires again.

Independently of that, the mapping's values omit the model. root that real module paths carry. The inner class is constructed with the default prefix="model", and _matches_exclude (config.py:515) is anchored (== or startswith(ignore + ".")):

real module : model.language_model.language_model.model.layers.0.mlp
value       : language_model.language_model.model.
prefix match: False

The sibling gets both right — kimi_k25.py:194 is "language_model.model.": "model.language_model.model.", with disjoint mapper keys and the model. root.

Consequence: layers the quantized checkpoint marks high-precision are built quantized — a missing-scale load error, or silent garbage logits. Latent on the shipping /data/Kimi-K3 only because that checkpoint's quantization_config.ignore is entirely re: regexes.

And the new test pins the defect: tests/plugin/test_vllm_kimi_k3.py asserts value.startswith("language_model.language_model."), which is derived from the constant rather than from the module tree — so the correct value would fail the test added to protect it.

4. The plugin builds a second Kimi-K3 vision stack instead of reusing ATOM's, and loses its PP gating

atom/models/kimi_k3.py:1915 already ships KimiK3ForConditionalGeneration(KimiK3ForCausalLM) with a native MoonViT3d tower (atom/models/kimi_k3_vl.py, selected by model_runner.py:123). The new plugin class wires vLLM's K2.5 ViT instead, so the same architecture now has two divergent vision implementations, weight-name expectations and quant paths depending on which mode it runs in.

The native one also carries pipeline-parallel gating the plugin path drops:

self.has_vision_tower = get_pp_group().is_first_rank
if not self.has_vision_tower:
    self.vision_tower = PPMissingLayer()
    self.mm_projector = PPMissingLayer()
    self.skip_weight_prefixes = list(self.vision_weight_prefixes)
    self.media_placeholder_token_id = None
    return

With pp>1 every rank allocates a full ViT and the loader places tower weights on ranks that should not own them.

4a. nn.Module.__init__(self) skips upstream's device/dtype placement and the quant hookup

Upstream's __init__ (/app/vllm/vllm/models/kimi_k3/amd/model.py:99-123) does self.vision_tower.to(device=self.device, dtype=model_config.dtype) — with a deliberate branch that casts an unquantized tower's dtype and an quantized one's device only — and sets self.hidden_size / self.device. None of that runs. The inherited _parse_and_validate_media_input (:166-167) then takes its target dtype from next(self.vision_tower.parameters()).dtype, so whatever dtype the tower happened to be constructed in becomes the vision path's dtype with no reconciliation against model_config.dtype. Normally masked by vLLM's set_default_torch_dtype wrapper, but the branch upstream deliberately does not cast is now indistinguishable from the one it does. Only a real image request can catch this.

Separately, SupportsQuant.__new__._find_quant_config scans constructor arguments for a vLLM VllmConfig or QuantizationConfig. The new signature takes atom_config: Config, so it finds neither, _maybe_apply_model_mapping early-returns, and apply_vllm_mapper / packed_modules_mapping.update never run.

4b. _maybe_ignore_quant_config drops an unconditional guard and reaches check_contains through an elif chain

Upstream returns None whenever quant_config is a CompressedTensorsConfig, independent of any ignore list (amd/model.py:135-140). The override replaces that with an exclude-list lookup, and /data/Kimi-K3 is a compressed-tensors checkpoint — so the tower is now protected only because that checkpoint's ignore list happens to contain re:.*vision_tower.*. An unconditional guard became a checkpoint-dependent one.

The lookup itself is also incomplete: config.py:501-512 handles re: first, then */? globs, and only then elif check_contains. A checkpoint expressing the exclusion as vision_tower* or *.vision_tower.* never reaches the branch this relies on — fnmatch("vision_tower", "*.vision_tower.*") is False — so MoonViT3dPretrainedModel is built with quantized linears.

Also a Liskov violation: the signature changed from (self, quant_config) to (self, quant_config, exclude_layers, layer_name), so any inherited upstream path calling it with one argument raises TypeError. And it reaches into the private QuantizationConfig._matches_exclude from a call site, duplicating a loop the public _is_excluded already has. It is byte-identical to kimi_k25.py:269, so both need the fix.

5. The model and lm_head properties silently re-route the inner-model chain and swallow writes

_INNER_MODEL_ATTRS = ("model", "language_model") (model_wrapper.py:583) probes model first, and the new model property returns self.language_model.model. So the chain jumps from the plugin class straight to KimiLinearModel, skipping KimiLinearForCausalLM — the node that actually owns lm_head. It only still resolves because the same commit added the lm_head property one level up. Delete or rename either property, or look for any attribute that lives on KimiLinearForCausalLM, and first_holder() silently binds nothing.

Worse, lm_head is in _WEIGHT_SHARED_ATTRS (model_wrapper.py:579), and the draft-path hook writes to it. A read-only property on an nn.Module does not refuse that write. Measured:

setattr: no error
in _modules : True
read returns the NEW one?  False
read returns the OLD one?  True     <- write silently discarded

nn.Module.__setattr__ places the module in _modules without consulting the class-level data descriptor, and the descriptor then wins on read. Reachable when K3 is served with --speculative-config method eagle3 or mtp (the hook is gated on is_mtp or is_eagle3, not on the model): vLLM believes it tied the target head into the draft, the drafter scores against an unshared head, and acceptance rate and accuracy drop with no error.

This also falsifies the model property's own stated rationale — "A property, so it stays out of named_modules" — since the write does land in _modules.

Install both as object.__setattr__ aliases in __init__, or give them setters so a write either lands or raises.

6. Fix-then-sweep: slot_mapping was trimmed in one file and not its sibling

This PR adds, with a comment naming the hazard precisely:

# layer_mla.py:1035
# The cache kernels launch one block per slot and index q/kv by that block,
# so slot_mapping must not outrun the tensors sliced above.
slot_mapping = attn_metadata.slot_mapping[:num_actual_toks]

and applies it in two places, including sparse_meta.slot_mapping[:num_actual_toks]. But layer_sparse_mla.py:305 is still bare:

slot_mapping = indexer_meta.slot_mapping

fed straight to indexer_k_quant_and_cache / indexer_qk_rope_quant_and_cache alongside q_input/k/weights sized from hidden_states. On the V2 runner, where slot_mapping is padded to the CUDA-graph token width while num_actual_tokens stays unpadded, the kernel walks past the tensors it was given. Worth confirming minimax_m3_attnetion.py:418,425 too, which pass main_metadata.slot_mapping / index_metadata.slot_mapping untrimmed.

7. write_context_kv's new slot_mapping parameter has no caller, and two docstrings in this PR contradict each other about it

Full sweep of the callers:

atom/models/dspark_draft.py:66               layer.write_context_kv(ctx_hidden, positions)          2 args
atom/spec_decode/dspark_proposer.py:479      self.model.write_context_kv(main_hidden_all, positions) 2 args
atom/plugin/vllm/models/kimi_k3_dspark.py:54 layer.self_attn.write_context_kv(...)                  bypasses this method

The new docstring at kimi_k3_dspark.py:556 says "the vLLM plugin passes per-layer slots explicitly". The plugin's own docstring at plugin/vllm/models/kimi_k3_dspark.py:47 says the opposite, and explains why:

Enters at the attention module because the decoder layer's entry point reads its slots off ATOM's native forward context.

The base contract in dspark_draft.py:43-46 also still says "A layer reads its own addressing ... off the live forward context rather than taking it as an argument", and was not updated. tests/test_dspark.py:783 pins the 2-arg contract, and the sibling deepseek_v4_dspark.py:672 did not get the parameter — so the two implementations behind one documented base contract now have divergent signatures.

8. Tests: 93 new lines CI never runs, several that cannot fail

.github/scripts/run_unit_tests.sh:23 carries --ignore=tests/plugin, so none of the new tests execute in CI — including the registry-sync test, which is the only guard against model_wrapper.py and register.py drifting apart. Three of them also import vllm, which CI does not have, and _run_without_test_stubs asserts returncode == 0 with no skip.

Within the file:

  • assert hasattr(KimiK3ForConditionalGeneration_, "load_weights") is trivially true — upstream defines it — so it passes with this PR's override deleted, the opposite of its comment.
  • The hasattr loop over the three mamba classmethods asserts the existence of methods defined ten lines below.
  • assert packed / assert excl assert that literal dicts are non-empty.
  • The exclude-mapping assertion encodes the missing-model.-root defect as the expected invariant (see update the link to the repo #3).

Only test_kimi_k3_is_plugin_supported_multimodal is pure atom.config and could move to a CI-visible file. For the rest, please post a manual pytest tests/plugin/test_vllm_kimi_k3.py result on the PR, and keep only assertions that go red on a revert run.

9. Docs

git diff origin/main...HEAD --stat shows zero files under docs/, though the PR body says otherwise.

  • docs/vllm_plugin_backend_guide.md:188 still maps KimiK3ForConditionalGeneration to atom.plugin.vllm.models.kimi_k3.KimiK3ForCausalLM after the registry was repointed to KimiK3ForConditionalGeneration_, and :192-195 still advertises text-only requirements.
  • recipes/atom_vllm/Kimi-K3.md:3 still says "serves the text-only Kimi-K3 backbone", :35 still passes --language-model-only, and :145 still says the vision tower and projector are skipped. Anyone following the only committed recipe launches with vision disabled and concludes the feature does not work.
  • The PR body also claims it registers K3DSparkModel (already on main; register.py:42 unchanged) and adds vision design/plan docs (absent).

The one recipe file that was added, recipes/atom_vllm/KVV.md, is a scratch note: the acronym is expanded nowhere, grep -rn KVV . returns only the file itself, it hardcodes ~/Kimi-Vendor-Verifier and /workspace/shared/data/amd_int/models/Kimi-K3 eight times, ## Server points at an unnamed recipe, and line 77 reads "BEAM and Judge are low priority tests. Let's ignore them for now." Nothing references it.

10. Smaller items

  • Naming. KimiK3ForCausalLMVllm (kimi_k3.py:633) is now a multimodal ForConditionalGeneration wrapper registered with MULTIMODAL_REGISTRY. Both siblings were renamed at exactly this transition (KimiK25ForConditionalGeneration, Qwen3_5ForConditionalGeneration).
  • Duplication. _maybe_ignore_quant_config is byte-identical to kimi_k25.py:269; the __init__ vision half is ~40 lines shared with kimi_k25.py and qwen3_5.py; the inner load_weights is the fourth copy of the same load_model_in_plugin_mode(..., prefix="model.") call, and K2.5's four-line comment explaining why prefix="model." was dropped from this copy, so the constraint is now undocumented at one of four sites.
  • Scope. ~25 hunks of Optional[X] → X | None across atom/model_ops/fla_ops/* and atom/rollout/async_engine.py are unrelated to Kimi-K3, enforced by no lint rule (pyproject.toml has no [tool.ruff] section), and incomplete — each touched signature leaves the adjacent initial_state: torch.Tensor = None alone.

@sajandhy
sajandhy force-pushed the hexwang/k3_dspark_026_vision_rebased branch from 16cad94 to e317056 Compare September 11, 2026 17:30
whx-sjtu and others added 6 commits September 16, 2026 16:56
Keep main's later K3/DSpark/core fixes; overlay only the still-unique
plugin vision, DSpark draft, MLA/metadata, and recipe changes.

Co-authored-by: Cursor <cursoragent@cursor.com>
The multimodal plugin classes in kimi_k3.py referenced symbols without
importing them, which made Ruff fail in pre-checks CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sajandhy
sajandhy force-pushed the hexwang/k3_dspark_026_vision_rebased branch from 385af20 to ff37588 Compare September 16, 2026 21:56
@sajandhy

Copy link
Copy Markdown
Contributor Author
self._packed_start_loc =

Thanks for the double pass. The three cleared items stay as-is (PAD_SLOT_ID vs NULL_BLOCK_ID, mapper vs /data/Kimi-K3, no default for media_placeholder_token_id). How we addressed #1–#10:

#1 Fixed. Dropped _stage_packed_decode_query_start_loc. Eager/piecewise still uses a persistent _packed_decode_start_loc sized to max_num_seqs + 1. Full-graph compact stays in _adapt_full_graph_decode_metadata, gated on use_full_cuda_graph.

#2 Fixed. Restored if real_num_decodes == metadata.num_decodes: return so unpadded decode matches GDN and does not rewrite Mamba slots or add a host sync every step.

#3 Fixed. Quant exclude mapping is applied after the HF→ATOM mapper and includes the model. root. The test that pinned the old (wrong) strings is gone.

#4 Partial, intentional for this PR. We keep vLLM’s MoonViT3d processor path rather than ATOM’s native tower, so plugin and native still diverge there. Non-first PP ranks now get PPMissingLayer() for vision_tower / mm_projector instead of allocating a full ViT.

#4a Device/dtype .to(...) matches upstream, and vllm_config= is passed through. __init__ still starts with nn.Module.__init__(self) rather than vLLM’s super(), so SupportsQuant.__new__ may not run the full mapper path. We can follow up if that still bites weight load.

#4b K3 restores compressed-tensors → skip (None), keeps a 1-arg-compatible _maybe_ignore_quant_config, and checks more name forms (model. + child). K2.5’s twin helper is unchanged on purpose; not folding that into this PR.

#5 Fixed. Initial model / lm_head aliases use object.__setattr__; later writes go through __setattr__ into language_model. Covered by a unit test.

#6 Fixed. slot_mapping is trimmed to live tokens in layer_sparse_mla.py and the MiniMax sibling, matching layer_mla.py.

#7 Fixed. Native write_context_kv is 2-arg again (ctx_hidden, positions). The plugin still passes slots into attention; docs match that split.

#8 Partial. Tautological asserts are gone; real checks added. The multimodal-config guard lives in tests/test_kimi_k3_plugin_config.py (CI). Broader plugin tests remain under tests/plugin (--ignore in CI); we are not flipping that in this PR.

#9 Mostly fixed. Recipe no longer uses --language-model-only; scratch KVV.md is gone; the plugin guide table lists KimiK3ForConditionalGenerationVllm. One leftover: docs/vllm_plugin_backend_guide.md still says “text-only requirements” on the K3 recipe line — will drop that wording.

#10 Partial. Class renamed to KimiK3ForConditionalGenerationVllm. Unrelated Optional → X | None churn was reverted. Vision / _maybe_ignore_quant_config is still duplicated with K2.5; same as 4b, not de-duping here.

Happy to take another look at 4a if you still want a full SupportsQuant super() init in this PR.

@zejunchen-zejun

Copy link
Copy Markdown
Collaborator

Comment 1 — Shared MLA attention changes are out of scope for a Kimi-K3 PR

This PR is titled "Kimi-K3 vLLM plugin vision + DSpark draft support", but it also changes three
attention files that no Kimi-K3 user is alone on:

  • atom/plugin/vllm/attention/layer_mla.py — forward_impl and forward_impl_sparse now trim
    slot_mapping to num_actual_toks
  • atom/plugin/vllm/attention/layer_sparse_mla.py — sparse_attn_indexer_plugin_mode trims to
    q_input.shape[0]
  • atom/plugin/vllm/attention/minimax_m3_attnetion.py — _insert_qkv_and_index trims both
    slot_mapping and index_slot_mapping to num_tokens

Blast radius: every plugin-mode MLA model — DeepSeek-V3 / V3.2, GLM-5.x, Kimi-K2 / K2.5 — plus
MiniMax-M3. None of them appears anywhere in the Test Plan.

To be fair about the risk: the trims look correct, and on the V1 path they are no-ops
(len(slot_mapping) == num_actual_tokens, so the slice changes nothing). They also do not add a
new cudagraph hazard, because the immediately preceding lines already do the same dynamic slice
(q = q[:num_actual_toks]). But these are Tier-2 shared files: if num_actual_tokens is ever not
the live-token count on some backend, the result is fewer KV rows written, which shows up as a
silent accuracy drop, not a crash.

Two other things about this hunk:

  • layer_sparse_mla.py slices by q_input.shape[0] while layer_mla.py slices by
    num_actual_toks. Please make them consistent, or explain why the sparse path must use the
    padded-tensor width instead of the metadata's live-token count.
  • The minimax_m3_attnetion.py diff is 23+/25- of which most is an unrelated
    Optional[X] → X | None modernization across the whole file. Please drop that churn and keep
    only the two slot_mapping lines.

Ask: please either (a) split these three files into their own PR — they read like a
"model runner V2 slot_mapping padding" fix and deserve to be reviewed and landed as one — or
(b) keep them here and add a smoke/accuracy run for at least one DeepSeek-family plugin model and
one sparse-MLA model (GLM-5.2 or DSv3.2) to the Test Result section.


Comment 2 — Extraction looks lossy: a no-op hunk plus a description that does not match the diff

The PR body describes this as a rebased extraction of the remaining delta from
hexwang/k3_dspark_026_vision. A few signs suggest the extraction dropped things:

A no-op hunk in layer_mla.py:

block_size = vllm_config.cache_config.block_size
return MLAAttentionSpec(
    block_size=block_size,
    ...
)

This extracts a local variable and changes nothing. A hunk like this is normally the residue of a
real change that got lost in the rebase — and the PR body does claim "dtype-aligned draft KV
pages", which is not visible anywhere in the diff. Please confirm whether the real change was
dropped, or revert this hunk.

Description claims that are not in the diff:

  • "add recipes/atom_vllm/KVV.md plus vision design/plan docs" — no such files in the diff
  • "Register K3DSparkModel → ...kimi_k3_dspark:KimiK3DSparkVllm" — that line is context; it is
    already on main
  • "segmented KDA forward, aux hidden-state collection" — not visible;
    set_aux_hidden_state_layers / get_eagle3_aux_hidden_state_layers are pre-existing

Ask: please rewrite the Motivation / Technical Details to describe exactly what this diff does,
and state explicitly what was intentionally left out of the extraction. As a reviewer I currently
cannot tell an intentional omission from an accident. The title should probably drop "DSpark draft
support" too, since atom/plugin/vllm/models/kimi_k3_dspark.py is not touched here.

(Related: @valarLip's two inline threads are anchored on atom/models/kimi_k3_dspark.py, which is
no longer part of this diff, and the 2026-09-09 request to check
atom/plugin/vllm/models/kimi_k3_dspark.py#L84 refers to code that is not in this PR either.
Please either resolve those threads or note that the code moved out of this PR.)


Comment 3 — The user-visible promise changed, but there is no test result and one load-bearing hook lost its only guard

This PR flips Kimi-K3 from "text-only" to "multimodal" in the recipe, removes
--language-model-only from the validated launch command, and rewrites the Current scope section
to say image inputs are supported. Meanwhile Test Result says "Pending — draft PR", all six
Test Plan boxes are unchecked, and the PR is no longer a draft.

Also note the recipe keeps the 86.1% / 86.1% DSpark acceptance numbers, which were measured with
the old text-only launch. Loading the vision tower changes the memory footprint, so those numbers
need either a re-run or an explicit note that they predate the vision path.

Separately, one behavioral guard was removed with nothing replacing it in tests. The old
KimiK3ForCausalLMVllm.get_language_model() carried this docstring:

DSpark's loader binds the target's embedding and head into the draft through this hook; Kimi-K3
keeps both one level in, under language_model, so without it the loader would look for them
on this wrapper and silently leave the draft with neither
.

That explicit hook is gone; the path is now
ATOMForConditionalGeneration.get_language_model() → self.model.get_language_model() (inherited
from upstream vLLMKimiK3), plus the new KimiK3ForCausalLM.embed_input_ids for vLLM's
SupportsMultiModal discovery. That may well be correct — but the four new unit tests assert MRO,
placeholder string, weights mapper and quant mapping, and none of them asserts that
get_language_model() actually returns the ATOM KimiK3ForCausalLM
. The failure mode the
deleted docstring warns about is silent, so it will not show up in the text-only smoke either.

@sajandhy
sajandhy merged commit bd892e5 into ROCm:main Sep 20, 2026
54 of 57 checks passed
peizhang56 added a commit to peizhang56/ATOM that referenced this pull request Oct 3, 2026
Adds DSpark (`--speculative-config '{"method":"dspark"}'`) as a draft method for
DeepSeek-V4 in vLLM plugin mode, where ATOM owns the model and vLLM owns the
scheduler and KV cache manager.

Model
- `DeepseekV4Model.forward` optionally returns per-layer auxiliary hidden states
  for EAGLE3/DSpark/DFlash drafts, averaged over the mHC dimension to match the
  target the drafter was trained against. `DeepseekV4ForCausalLM` PCP-all-gathers
  each aux tensor outside the compile boundary.
- The aux branch sits inside the `@support_torch_compile` region, so
  `aux_hidden_state_layers` is baked into the graph. From `--level 2` up the
  custom dispatcher replays code object 0 without evaluating guards, so it must
  be set at load time and never mutated after; noted at the top of the file.

KV cache
- DSpark's draft needs a second KV group (sliding-window MLA, block 64) beside
  the V4 proxy's block 128. `dspark_draft_kv_patch` registers the spec lazily and
  exposes it for pickle via PEP-562 `__getattr__`.
- Prefix caching: V4's SWA ring is not carried by vLLM's block cache, so every
  hit is rolled back by `max(win_with_spec, index_topk)` tokens and the tail
  re-forwarded. Converted per group, since the two groups differ in block size.
  The `index_topk` term is an empirical workaround for a sparse-indexer defect
  that also reproduces with prefix caching off; flagged as a KNOWN ISSUE.

Correctness fixes found along the way
- `VocabParallelEmbedding.forward` uses `replicated_embedding` instead of
  `F.embedding` on the `tp_size == 1` branch. `replicated_embedding` already
  masks the -1 spec placeholder async scheduling can emit, and its comment
  states VocabParallelEmbedding routes through the masked op -- true only of
  the `tp_size > 1` branch. Single-GPU still reached a raw gather, which reads
  the row before the table and GPU-faults.
- Async scheduling removes the sampled-token sync, so the pinned staging buffers
  can tear. Gated with a depth-1 `torch.cuda.Event`.
- Idle DP ranks reach `build()` via `execute_dummy_batch()` with
  `seq_len == query_len`; synthesized a decode context for that case.
- Piecewise cudagraph modes are demoted on the plugin path, a demotion vLLM
  skips when it compiles nothing.

Request ids are passed to ATOM metadata builders through a thread-local instead
of a per-step `block_table[:, 0]` D2H copy, covering both of vLLM's unrelated
`GPUModelRunner` classes (V2 is forced for dspark).

Tests: 4 new/extended vLLM plugin test files; `tests/plugin` 201 passed. The 34
sglang failures are pre-existing and unrelated. black + ruff clean.

Cherry-pick onto main (174 commits later); three files conflicted:

- `deepseek_v4.py`: main grew `HCState.res_preshuffle` and threads it into
  `hc_post`. The new aux-layer `hc_post` calls take it too -- the flag is
  invariant across layers, and omitting it would reconstruct an aux residual
  from the gfx1250 shuffled layout as if it were the ordinary one.
- `deepseek_v4_bridge.py`: main's req-id passthrough landed separately, so only
  the comment wording conflicted. On the inline-build fallback main since
  decided the opposite of this commit -- drop the allocator for throwaway slots
  rather than feed it live req_ids -- so main's behavior is kept and the
  now-dead `_req_ids` lookup dropped. The commit's `_warn_inline_v4_build`
  diagnostic is kept ahead of it, which is still the anomaly it reports.
- `test_vllm_kimi_k3.py`: hunk dropped. It flipped `dcp_persistent_supported`
  to False in `test_dense_mla_decode_pads_gathered_dcp_heads`; ROCm#1910 since
  renamed that test to `..._keeps_supported_persistent_gathered_head_width`
  and fixed the same inconsistency the other way -- keep True, assert 96/pad 0
  -- which is what this commit's own comment says persistent does. git matched
  the stale hunk onto the renamed sibling and broke it.

Tests: `tests/plugin` (sglang excluded, unrelated import errors) 262 passed,
47 failed -- byte-identical failure set to origin/main's baseline, +34 passing.
black/ruff not installed in this environment, so neither was run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
peizhang56 added a commit to peizhang56/ATOM that referenced this pull request Oct 3, 2026
Adds DSpark (`--speculative-config '{"method":"dspark"}'`) as a draft method for
DeepSeek-V4 in vLLM plugin mode, where ATOM owns the model and vLLM owns the
scheduler and KV cache manager.

Model
- `DeepseekV4Model.forward` optionally returns per-layer auxiliary hidden states
  for EAGLE3/DSpark/DFlash drafts, averaged over the mHC dimension to match the
  target the drafter was trained against. `DeepseekV4ForCausalLM` PCP-all-gathers
  each aux tensor outside the compile boundary.
- The aux branch sits inside the `@support_torch_compile` region, so
  `aux_hidden_state_layers` is baked into the graph. From `--level 2` up the
  custom dispatcher replays code object 0 without evaluating guards, so it must
  be set at load time and never mutated after; noted at the top of the file.

KV cache
- DSpark's draft needs a second KV group (sliding-window MLA, block 64) beside
  the V4 proxy's block 128. `dspark_draft_kv_patch` registers the spec lazily and
  exposes it for pickle via PEP-562 `__getattr__`.
- Prefix caching: V4's SWA ring is not carried by vLLM's block cache, so every
  hit is rolled back by `max(win_with_spec, index_topk)` tokens and the tail
  re-forwarded. Converted per group, since the two groups differ in block size.
  The `index_topk` term is an empirical workaround for a sparse-indexer defect
  that also reproduces with prefix caching off; flagged as a KNOWN ISSUE.

Correctness fixes found along the way
- `VocabParallelEmbedding.forward` uses `replicated_embedding` instead of
  `F.embedding` on the `tp_size == 1` branch. `replicated_embedding` already
  masks the -1 spec placeholder async scheduling can emit, and its comment
  states VocabParallelEmbedding routes through the masked op -- true only of
  the `tp_size > 1` branch. Single-GPU still reached a raw gather, which reads
  the row before the table and GPU-faults.
- Async scheduling removes the sampled-token sync, so the pinned staging buffers
  can tear. Gated with a depth-1 `torch.cuda.Event`.
- Idle DP ranks reach `build()` via `execute_dummy_batch()` with
  `seq_len == query_len`; synthesized a decode context for that case.
- Piecewise cudagraph modes are demoted on the plugin path, a demotion vLLM
  skips when it compiles nothing.

Request ids are passed to ATOM metadata builders through a thread-local instead
of a per-step `block_table[:, 0]` D2H copy, covering both of vLLM's unrelated
`GPUModelRunner` classes (V2 is forced for dspark).

Tests: 4 new/extended vLLM plugin test files; `tests/plugin` 201 passed. The 34
sglang failures are pre-existing and unrelated. black + ruff clean.

Cherry-pick onto main (174 commits later); three files conflicted:

- `deepseek_v4.py`: main grew `HCState.res_preshuffle` and threads it into
  `hc_post`. The new aux-layer `hc_post` calls take it too -- the flag is
  invariant across layers, and omitting it would reconstruct an aux residual
  from the gfx1250 shuffled layout as if it were the ordinary one.
- `deepseek_v4_bridge.py`: main's req-id passthrough landed separately, so only
  the comment wording conflicted. On the inline-build fallback main since
  decided the opposite of this commit -- drop the allocator for throwaway slots
  rather than feed it live req_ids -- so main's behavior is kept and the
  now-dead `_req_ids` lookup dropped. The commit's `_warn_inline_v4_build`
  diagnostic is kept ahead of it, which is still the anomaly it reports.
- `test_vllm_kimi_k3.py`: hunk dropped. It flipped `dcp_persistent_supported`
  to False in `test_dense_mla_decode_pads_gathered_dcp_heads`; ROCm#1910 since
  renamed that test to `..._keeps_supported_persistent_gathered_head_width`
  and fixed the same inconsistency the other way -- keep True, assert 96/pad 0
  -- which is what this commit's own comment says persistent does. git matched
  the stale hunk onto the renamed sibling and broke it.

Tests: `tests/plugin` (sglang excluded, unrelated import errors) 262 passed,
47 failed -- byte-identical failure set to origin/main's baseline, +34 passing.
black/ruff not installed in this environment, so neither was run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
peizhang56 added a commit to peizhang56/ATOM that referenced this pull request Oct 5, 2026
Adds DSpark (`--speculative-config '{"method":"dspark"}'`) as a draft method for
DeepSeek-V4 in vLLM plugin mode, where ATOM owns the model and vLLM owns the
scheduler and KV cache manager.

Model
- `DeepseekV4Model.forward` optionally returns per-layer auxiliary hidden states
  for EAGLE3/DSpark/DFlash drafts, averaged over the mHC dimension to match the
  target the drafter was trained against. `DeepseekV4ForCausalLM` PCP-all-gathers
  each aux tensor outside the compile boundary.
- The aux branch sits inside the `@support_torch_compile` region, so
  `aux_hidden_state_layers` is baked into the graph. From `--level 2` up the
  custom dispatcher replays code object 0 without evaluating guards, so it must
  be set at load time and never mutated after; noted at the top of the file.

KV cache
- DSpark's draft needs a second KV group (sliding-window MLA, block 64) beside
  the V4 proxy's block 128. `dspark_draft_kv_patch` registers the spec lazily and
  exposes it for pickle via PEP-562 `__getattr__`.
- Prefix caching: V4's SWA ring is not carried by vLLM's block cache, so every
  hit is rolled back by `max(win_with_spec, index_topk)` tokens and the tail
  re-forwarded. Converted per group, since the two groups differ in block size.
  The `index_topk` term is an empirical workaround for a sparse-indexer defect
  that also reproduces with prefix caching off; flagged as a KNOWN ISSUE.

Correctness fixes found along the way
- `VocabParallelEmbedding.forward` uses `replicated_embedding` instead of
  `F.embedding` on the `tp_size == 1` branch. `replicated_embedding` already
  masks the -1 spec placeholder async scheduling can emit, and its comment
  states VocabParallelEmbedding routes through the masked op -- true only of
  the `tp_size > 1` branch. Single-GPU still reached a raw gather, which reads
  the row before the table and GPU-faults.
- Async scheduling removes the sampled-token sync, so the pinned staging buffers
  can tear. Gated with a depth-1 `torch.cuda.Event`.
- Idle DP ranks reach `build()` via `execute_dummy_batch()` with
  `seq_len == query_len`; synthesized a decode context for that case.
- Piecewise cudagraph modes are demoted on the plugin path, a demotion vLLM
  skips when it compiles nothing.

Request ids are passed to ATOM metadata builders through a thread-local instead
of a per-step `block_table[:, 0]` D2H copy, covering both of vLLM's unrelated
`GPUModelRunner` classes (V2 is forced for dspark).

Tests: 4 new/extended vLLM plugin test files; `tests/plugin` 201 passed. The 34
sglang failures are pre-existing and unrelated. black + ruff clean.

Cherry-pick onto main (174 commits later); three files conflicted:

- `deepseek_v4.py`: main grew `HCState.res_preshuffle` and threads it into
  `hc_post`. The new aux-layer `hc_post` calls take it too -- the flag is
  invariant across layers, and omitting it would reconstruct an aux residual
  from the gfx1250 shuffled layout as if it were the ordinary one.
- `deepseek_v4_bridge.py`: main's req-id passthrough landed separately, so only
  the comment wording conflicted. On the inline-build fallback main since
  decided the opposite of this commit -- drop the allocator for throwaway slots
  rather than feed it live req_ids -- so main's behavior is kept and the
  now-dead `_req_ids` lookup dropped. The commit's `_warn_inline_v4_build`
  diagnostic is kept ahead of it, which is still the anomaly it reports.
- `test_vllm_kimi_k3.py`: hunk dropped. It flipped `dcp_persistent_supported`
  to False in `test_dense_mla_decode_pads_gathered_dcp_heads`; ROCm#1910 since
  renamed that test to `..._keeps_supported_persistent_gathered_head_width`
  and fixed the same inconsistency the other way -- keep True, assert 96/pad 0
  -- which is what this commit's own comment says persistent does. git matched
  the stale hunk onto the renamed sibling and broke it.

Tests: `tests/plugin` (sglang excluded, unrelated import errors) 262 passed,
47 failed -- byte-identical failure set to origin/main's baseline, +34 passing.
black/ruff not installed in this environment, so neither was run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
peizhang56 added a commit to peizhang56/ATOM that referenced this pull request Oct 5, 2026
Adds DSpark (`--speculative-config '{"method":"dspark"}'`) as a draft method for
DeepSeek-V4 in vLLM plugin mode, where ATOM owns the model and vLLM owns the
scheduler and KV cache manager.

Model
- `DeepseekV4Model.forward` optionally returns per-layer auxiliary hidden states
  for EAGLE3/DSpark/DFlash drafts, averaged over the mHC dimension to match the
  target the drafter was trained against. `DeepseekV4ForCausalLM` PCP-all-gathers
  each aux tensor outside the compile boundary.
- The aux branch sits inside the `@support_torch_compile` region, so
  `aux_hidden_state_layers` is baked into the graph. From `--level 2` up the
  custom dispatcher replays code object 0 without evaluating guards, so it must
  be set at load time and never mutated after; noted at the top of the file.

KV cache
- DSpark's draft needs a second KV group (sliding-window MLA, block 64) beside
  the V4 proxy's block 128. `dspark_draft_kv_patch` registers the spec lazily and
  exposes it for pickle via PEP-562 `__getattr__`.
- Prefix caching: V4's SWA ring is not carried by vLLM's block cache, so every
  hit is rolled back by `max(win_with_spec, index_topk)` tokens and the tail
  re-forwarded. Converted per group, since the two groups differ in block size.
  The `index_topk` term is an empirical workaround for a sparse-indexer defect
  that also reproduces with prefix caching off; flagged as a KNOWN ISSUE.

Correctness fixes found along the way
- `VocabParallelEmbedding.forward` uses `replicated_embedding` instead of
  `F.embedding` on the `tp_size == 1` branch. `replicated_embedding` already
  masks the -1 spec placeholder async scheduling can emit, and its comment
  states VocabParallelEmbedding routes through the masked op -- true only of
  the `tp_size > 1` branch. Single-GPU still reached a raw gather, which reads
  the row before the table and GPU-faults.
- Async scheduling removes the sampled-token sync, so the pinned staging buffers
  can tear. Gated with a depth-1 `torch.cuda.Event`.
- Idle DP ranks reach `build()` via `execute_dummy_batch()` with
  `seq_len == query_len`; synthesized a decode context for that case.
- Piecewise cudagraph modes are demoted on the plugin path, a demotion vLLM
  skips when it compiles nothing.

Request ids are passed to ATOM metadata builders through a thread-local instead
of a per-step `block_table[:, 0]` D2H copy, covering both of vLLM's unrelated
`GPUModelRunner` classes (V2 is forced for dspark).

Tests: 4 new/extended vLLM plugin test files; `tests/plugin` 201 passed. The 34
sglang failures are pre-existing and unrelated. black + ruff clean.

Cherry-pick onto main (174 commits later); three files conflicted:

- `deepseek_v4.py`: main grew `HCState.res_preshuffle` and threads it into
  `hc_post`. The new aux-layer `hc_post` calls take it too -- the flag is
  invariant across layers, and omitting it would reconstruct an aux residual
  from the gfx1250 shuffled layout as if it were the ordinary one.
- `deepseek_v4_bridge.py`: main's req-id passthrough landed separately, so only
  the comment wording conflicted. On the inline-build fallback main since
  decided the opposite of this commit -- drop the allocator for throwaway slots
  rather than feed it live req_ids -- so main's behavior is kept and the
  now-dead `_req_ids` lookup dropped. The commit's `_warn_inline_v4_build`
  diagnostic is kept ahead of it, which is still the anomaly it reports.
- `test_vllm_kimi_k3.py`: hunk dropped. It flipped `dcp_persistent_supported`
  to False in `test_dense_mla_decode_pads_gathered_dcp_heads`; ROCm#1910 since
  renamed that test to `..._keeps_supported_persistent_gathered_head_width`
  and fixed the same inconsistency the other way -- keep True, assert 96/pad 0
  -- which is what this commit's own comment says persistent does. git matched
  the stale hunk onto the renamed sibling and broke it.

Tests: `tests/plugin` (sglang excluded, unrelated import errors) 262 passed,
47 failed -- byte-identical failure set to origin/main's baseline, +34 passing.
black/ruff not installed in this environment, so neither was run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants