Repository navigation
[ATOM Plugin CI] Fix the K3 DSpark MLA head-width, its breakable-cudagraph eager break, and the vision modules' meta-device move - #2160
Conversation
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
…ture size The `Accuracy (Kimi-K3 DSpark DCP8 TP8)` cell has been failing on every branch and on both runner pools (spur and crusoe). It is a genuine out-of-memory of the vLLM process itself, not GPU memory held by another job: the preflight log shows the leftover container being killed and "GPU memory released after 8s" before the server starts. Per GPU (287.98 GiB total) the cell was asking for: weights 189.65 GiB non-torch + activation peak ~16.8 GiB KV cache (--gpu-memory-utilization 0.75) 9.54 GiB cudagraph private pools (51 sizes, max 512) 45.71 GiB aiter MLA split-KV workspace @ 512 tokens 31.54 GiB The last two are allocated outside the budget vLLM profiles, so the run died in compile_or_warm_up_model -> _forward_decode -> aiter mla_decode_fwd while allocating the 31.54 GiB `logits` workspace with only 22.79 GiB free. On the crusoe pool the same cell does not even get that far and reports "Available KV cache memory: -4.86 GiB". That workspace is sized by the decode token count, so capping cudagraph capture at 256 tokens (32 seqs x 8 spec tokens) roughly halves it and also shrinks the capture-size list, while --gpu-memory-utilization 0.8 raises the KV cache from 9.54 GiB to ~23.7 GiB. FULL cudagraphs stay enabled; decode batches above 32 sequences fall back to piecewise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s KV cache
The first attempt capped the capture bound at 256 and raised
--gpu-memory-utilization to 0.8. The bound helped, the utilization hurt, and
the cell still OOMed in the same place:
capture 512 / util 0.75 capture 256 / util 0.8
pool 45.71 GiB 35.60 GiB
MLA workspace 31.54 GiB 24.53 GiB
KV cache 9.54 GiB 23.93 GiB
free 22.79 GiB 14.07 GiB
Two things the numbers say. The cudagraph pool costs about 1 GiB per captured
size (45.71/51 and 35.60/35), so the count of sizes is the lever, not the
bound. And the aiter MLA split-KV workspace scales sublinearly with the bound
(31.5 -> 24.5 GiB for a halved bound), so lowering it alone converges slowly.
Meanwhile the extra 14.4 GiB of KV bought by util 0.8 went straight out of the
headroom those two need: KV was never the constraint here -- the cell ran fine
with 9.54 GiB, which is 13x concurrency at this max-model-len.
So capture 6 sizes instead of 35 and put utilization back to 0.75. Estimated
per GPU: ~190 weights + ~17 non-torch/activation + ~10 KV + ~6 pool + ~20
workspace, which leaves real headroom rather than the 14 GiB that was not
enough for a 24.5 GiB allocation.
FULL_AND_PIECEWISE stays on. Every captured size is a multiple of
num_spec+1 = 8, as spec decode requires; decode batches above 24 sequences
fall back to piecewise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… serves `AiterMlaMetadataBuilderForVllm` took its query-head count from `config.model_config.hf_config`, which always describes the target model. A DSpark draft owns its own KV cache group and therefore its own builder, so on Kimi-K3 the draft's persistent work descriptors were planned for the target's 96 query heads while the draft's MLA decode dispatches at 64. Under DCP those two gathered widths differ (96 against 64), and a plan that describes a different kernel than the one that runs leaves that kernel indexing partials nobody planned. Read the width off the layers this builder was handed instead -- as ATOM native's DSparkProposer already does, and for the stated reason: "read it rather than recomputing, so the work descriptors planned here describe the kernel that will actually run" -- and pass `min_query_heads` to `mla_dcp_kernel_num_heads` alongside it, which is the other half of what the layer padded its queries to. Verified on 8xMI355X with Kimi-K3 DSpark DCP8 TP8: the draft's decode now plans and dispatches at 64 heads (probed q_heads == persistent_num_heads) and the target's stays at 96, unchanged. This alone does not turn the Accuracy (Kimi-K3 DSpark DCP8 TP8) cell green -- that cell still dies earlier, in the target's round-robin decode on a warmup batch. The first commit's premise for that failure (an OOM answerable by cudagraph capture sizes and --gpu-memory-utilization) is disproven and its workflow edit is reverted here; the PR description carries the measurements. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Kimi-K3 is on vLLM's breakable-cudagraph path (config/vllm.py auto-enables it for KimiK3ForConditionalGeneration), where there is no FX split: one stream capture swallows the whole forward and only ops carrying @eager_break_during_capture end a segment and run eagerly per replay. A layer that does not declare the break is captured whole, and every replay indexes the KV cache with the block tables, sequence lengths and DCP work descriptors frozen at capture -- an illegal access as soon as a later batch is shaped differently. atom_vllm_mla_attention never declared it, so all 24 of K3's full-attention layers were captured. Measured on 8xMI355X, capture itself finished and the first decode after it died in warmup with an illegal access. The KDA mixer in the same plugin already breaks out for the same reason -- the two halves of one omission. Also take `output` from the caller: the decorator requires an in-place buffer, and a tensor allocated inside the op lands at a fresh address each replay while the captured segments downstream keep reading the address they saw at capture. Segment counts confirm the mechanism rather than merely tracking the fix. K3 has 93 layers, 24 full-attention and 69 KDA; eager_breaks went 69 -> 93, i.e. exactly one break per attention layer. Predicted before measuring. before eager_breaks=69 died, illegal access in warmup decode (280s) after eager_breaks=93 ready (301s), full gsm8k 0.953 / 0.9484 Full decode graphs are untouched -- the decorator passes the call through when the runtime mode is FULL. Threshold 0.94, not 0.95: this cell measures 0.9484 on one of two full runs under the fix, so 0.95 has no margin. Reference points on the same box and harness: NONE-mode 0.9492, and the non-DCP Kimi-K3 DSpark TP8 cell 0.9469.
38ee416 to
0f86867
Compare
The vision tower and multimodal projector are built under the meta-device init that the plugin's weight loader materialises later, so the `.to()` calls added with the K3 vision support moved parameters that have no data yet and every worker died during model load: NotImplementedError: Cannot copy out of meta tensor; no data! atom/plugin/vllm/models/kimi_k3.py:606 in __init__ Kimi-K2.5 builds the same two modules and never calls `.to()` on them (atom/plugin/vllm/models/kimi_k25.py), which is the shape this path expects. Drop the calls here for the same reason. This is load-time only and hits every K3 plugin run, including the language-model-only accuracy cell, because `has_vision_tower` keys off the PP rank rather than the language-model-only flag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Accuracy run artefacts —
|
Three defects in the plugin's Kimi-K3 path — two in MLA, one in the vision module construction — plus a threshold line. With them, the
Accuracy (Kimi-K3 DSpark DCP8 TP8)cell passes: 0.9598 on the cell's own configuration, 1319/1319, no fault. It has been red since it was added in #2033.What changed in this revision
The fused-Markov-argmax fix that earlier revisions of this PR carried (
atom/model_ops/dspark_markov_sample.pyand its test) is gone from this branch, for two independent reasons:cfdfa279b) rewrote that reduce and clamps the sentinel to 0, so the crash-safety property is on main without this PR.torch.cuda.MemPoolinstead of inheriting a freed intermediate's address inside a cudagraph capture region — which is the origin of the garbage draft logits that earlier revisions of this PR localised to layer 0'sgate_up_projfp8 GEMM but could not explain. The run below uses the CI's own latest-main aiter wheel (7d2f6a51a, 11 commits after that merge) and shows no NaN-driven fault.Dropping those two files also removed this PR's merge conflict with main.
1. MLA metadata builder used the target's head width —
metadata.pyAiterMlaMetadataBuilderForVllmread its query-head count frommodel_config.hf_config, which always describes the target. A DSpark draft owns its own KV cache group and its own builder, so the draft planned persistent work descriptors for K3's 96 heads while its MLA decode dispatched at 64 — the kernel then indexes DCP partials nobody planned.Now read off the layers the builder was handed (as ATOM native's
DSparkProposerdoes), withmin_query_headspassed through tomla_dcp_kernel_num_heads. Verified: draft plans and dispatches at 64, target stays 96.2. MLA was missing the breakable-cudagraph eager break —
ops.py,layer_mla.pyK3 runs on vLLM's breakable cudagraph path (
VLLM_USE_BREAKABLE_CUDAGRAPH, compilation modeNONE, no FX splitting), where only ops carrying@eager_break_during_capturere-execute eagerly on replay. Upstream applies it to every attention entry point —unified_mla_attention_with_output,unified_attention_with_output,kimi_k3/nvidia/mla.py::_attention, the KDA mixers — and K3's AMD path has nomla.pyupstream, because the plugin'satom_vllm_mla_attentionis what stands in for it. It was the one such op without the decorator.The op now takes
outputfrom the caller (mutates_args=["output"]) instead of allocating it — the decorator's documented requirement, since a tensor allocated inside would land at a new address each replay — and delegates to a decorated helper that re-fetches metadata via_get_layer_contextinside the break.Effect is limited to PIECEWISE batches. The decorator short-circuits when
get_forward_context().cudagraph_runtime_mode == CUDAGraphMode.FULL, so underFULL_AND_PIECEWISEthe uniform-decode batches are unaffected by this change.graphs=70, eager_breaks=69(KDA only)graphs=94, eager_breaks=93(one per layer)93 = 69 KDA + 24 MLA, predicted before the run. Both rows use the same launch arguments, so the comparison holds; the capture counts come from a breakable-cudagraph debug run, not from the accuracy run below.
3. The K3 vision modules were moved off the meta device at build time —
kimi_k3.pyIndependent of MLA, and currently red for everyone on main: K3 plugin runs die during model load, all eight workers at once.
The vision tower and the multimodal projector are built under the meta-device init that the weight loader materialises later, so
.to()on them moves parameters that have no data yet. Kimi-K2.5 builds the same two modules from the same upstream classes and never calls.to()on them (atom/plugin/vllm/models/kimi_k25.py:236-255) — that is the shape this path expects. The calls are dropped here for the same reason.This hits the accuracy cell even though it launches with
--language-model-only, becausehas_vision_towerkeys offget_pp_group().is_first_rankrather than that flag; the vision modules are built regardless.Introduced by #1910 (
bd892e5a3), which landed after this branch's previous green run, which is why it appears only now.Attribution caveat. That this is independent of the MLA changes rests on two source-level facts — the failure is raised before
self.language_model = ...on line 613, and the three.to()calls arrive with #1910 — not on a stock-main control arm, which was not run.4. Threshold 0.95 -> 0.94
The gate reads
exact_match,flexible-extract(.github/scripts/atom_oot_test.sh:466).This revision, the cell's own launch arguments, latest-main aiter (
7d2f6a51a):f989d7381Why the gate moves down even though this run clears 0.95 comfortably: the cell has been observed to produce a fully correct run — all 1319 questions answered, zero faults — that scores 0.9484, i.e. below the current gate. A 0.95 threshold therefore does not merely sit close to this cell's run-to-run spread; it has been seen to fail a correct build. (That observation comes from repeat runs of an earlier revision of this PR, on an older aiter; those tables have been removed from this description at review request, so the number is quoted here as the reason for the change rather than presented as evidence for this revision.)
0.94 clears that lowest observation by 0.0084 and still catches any regression worth catching. If a later run comes in under 0.94 the honest response is more data, not another nudge; the neighbouring cell above already runs at 0.93.
Why runs vary at all: lm_eval decodes greedily and speculative drafts are verified by the target, so every run measures the same model — draft-side differences move throughput, not answers. What is left is the target's own run-to-run numerics.
For reference, ATOM native (TP8 + DCP8 + 7 spec tokens) measures 0.9500 — 0.95 would sit exactly on the value the reference implementation produces.
Limit on the evidence shown here: n = 1 for this revision. Whether ROCm/aiter#5430 also narrows the spread is untested.
Test results
Full run artefacts and provenance:
lm_evalgsm8k, 5-shot, 1319 questions, concurrency 64, 8xMI355X, the cell's own launch arguments (TP8 + DCP8, 7 speculative tokens,FULL_AND_PIECEWISE, default capture buckets).f989d7381, clean tree, basee4f5ff2d7amd_aiter-0.1.1.dev1+g7d2f6a51a(CI latest-main wheel;aiter/utility/graph_alloc.pycarries theMemPoolpath from ROCm/aiter#5430)0.28.1.dev0+g2cf0a6915.d20260916.rocm724/2.10.0+rocm7.2.4.lw.git3d3aa833Application startup completeHSA_STATUS_ERROR/Memory access fault/EngineDeadErrorall absent from the server log)Two caveats on this run, both stated so nobody reads more into the number than it carries:
f989d7381); the diff is byte-identical, but the measured tree was the patched worktree rather than this commit.