Repository navigation
perf(spec-decode): warm every draft forward at a batch the target captured - #2042
Merged
Merged
Conversation
`prepare_mtp_decode` rebuilt every buffer to `bs`, the count of sequences
carrying a real request. A drafter may run a WIDER batch than that, to land on
a shape the target already captured, and then every slice came up short: the
attention builder rebuilt kv_indptr, block_tables and slot_mapping for the real
rows only, and the fabricated ones kept whatever the previous, larger step had
left behind.
The row count is `positions.shape[-1]` -- the LAST axis, because MRoPE keeps the
token axis there and every other layout is one-dimensional, where the two agree.
`bs` stays the real sequence count, and the base carries the contract once so a
backend that never distinguishes the two simply never reads it.
DeepSeek-V4 additionally:
* marks the pad tail of `batch_id_per_token` with `-1`, the sentinel the verify
forward already uses. `write_v4_paged_decode_indices` returns on `bid < 0`
before it loads a ring slot, so a padded draft costs nothing for the rows it
invented instead of repeating a real one's gather -- at bs=65 padded to 128
that is half the batch. It writes into the buffer the verify forward
publishes on that field, not a private one: a captured draft graph bakes
that address, and republishing a different tensor would leave the replay
reading the verify map.
* publishes the compressor ring slots at the padded `bs`, which the ubatch
path 200 lines down already does. A consumer that runs the padded batch can
then slice to it instead of asking the drafter to stage a copy.
…tured aiter's flydsl builds one hgemm per tile config, in-process and chosen from the row count, so a batch first seen mid-serve stalls that step -- and every restart pays it again. The draft never landed on the shapes the target had captured, so it paid this on almost every distinct decode batch. A drafter now DECLARES its forwards as `DraftGraph`s. At the end of CUDAGraph capture the runner runs each one once per `graph_bs`; at serve time the drafter rounds its batch up to that same list, so warmed and reachable are one set by construction rather than two that drift. `ATOM_DRAFT_CUDAGRAPH` (default on) also captures each warmed shape, so the pass replays instead of relaunching every kernel. Measured, V4-Pro tp4 MTP3: acceptance 66.04% and GSM8K 0.9545, against a 64.09 -- 64.20% / 0.9522 -- 0.9545 baseline. On V4-Flash-DSpark tp4 the LM head inside the recording took draft kernel launches 30 -> 0 per pass and draft wall time 915.8 -> 118.9 us. Three capabilities, each needing strictly more than the last: warm needs nothing; pad needs staged inputs and a promise that fabricated rows are inert; capture needs pad, since a graph is one shape. A flavor that cannot answer one of them declares nothing rather than declaring a pass that claims to be warmable and is not -- Kimi-K3 drafts through a separate model, and MRoPE keeps its token axis last, so neither declares. `DraftGraph` lives apart from `Drafter` so it stays importable without a GPU AITER build: its invariants are pure torch, and CI has no aiter. Everything speaks the same three words as the code around it -- `bs` real, `pad_bs` what runs, `graph_bs` what was captured. Trace labels become `propose_eagle[...]` / `propose_dspark[...]`, both carrying `bs=<real>/<pad>` and a trailing ` graph` on a replay, so "did the draft get into a graph" is a grep. `tools/analyze_trace_summary.py` follows the rename.
…r's cwd One 8-rank fault writes ~283GB of them and fills the disk. They land next to the server's working directory, which is the repo, while `rm -rf ./gpucore.*` cleared wherever the launcher happened to be invoked from. `HSA_ENABLE_COREDUMP=0` does not prevent them (measured), so clearing is the only thing that works.
Review of the draft-graph passes turned up one place a padded draft could
corrupt another request, and several shapes that existed only because two
flavors had each grown their own answer to the same question.
A pass now runs at `forward_mode.effective_bs` -- the count `model_runner`
built the attention metadata with (model_runner.py:2618) -- or at its own
batch, and nothing else. That replaces `_step_pad_bs` (eagle),
`_block_graph_bs` + `_self_chosen_graph_bs` (DSpark) and
`DraftGraph.graph_bs_for` with one `DraftGraph.target_pad_bs`:
* Off the cudagraph path nothing is padded at all. Every eager branch of
`ForwardMode.decide` returns `effective_bs == scheduled_bs`, so a pad row
there reads per-sequence metadata nobody wrote this step. DSpark used to
pad anyway, on the strength of owning a graph at that batch.
* `effective_bs`, not `context.graph_bs`: the two agree except where
`_dspark_ragged_moe_graph_bs` overwrites graph_bs with a token count.
Reading effective_bs makes that unreachable rather than guarded against,
and the guard could not have been shared -- under DP-uniform decode
`padded_scheduled_bs` is a cross-rank max, so it would have switched
padding off exactly where it pays.
* The answer is a warmed shape by construction, since `decide` picks it out
of `graph_bs`, which is the list the startup sweep warms.
DeepSeek-V4's fabricated draft rows no longer scatter KV. Their ring slot is
the `0` that `prepare_decode` fills the padded tail with, and 0 is a real
position -- `physical_slot` is `slot_positions - 1 - group`, so it is the last
group, which the pool hands out as concurrency approaches max_num_seqs. The
fused SWA write gates on `bid >= 0`, so `DSparkIndexBuffers` now carries a `-1`
tail, restored from the shared arange each step so a sentinel cannot outlive
the row that earned it. `dspark_fp8_indices` had stated the opposite invariant
-- "the draft runs at scheduled_bs, not the padded effective_bs" -- which
padding had quietly made false.
Also:
* `warmup_draft_graphs` moves onto `Drafter`, beside `_declare_draft_graphs`
and `_build_draft_graphs`, so a pass's whole lifetime lives in one place.
The runner hands over a bound capture builder instead of making the drafter
probe the backend's signature a second time.
* One arange over the batch axis, on `CommonAttentionBuilder`. It replaces
`Drafter.arrange_bs` and `DeepseekV4AttentionMetadataBuilder._v4_row_ids`,
and serves DSpark's token -> request map as well.
* `_enter_decode_metadata` returns what `attn_metadata` does not already
carry: two values rather than five, so no buffer has a second name.
* `DraftGraph.label` -- the `bs=<real>/<pad>` + trailing ` graph` convention
was written out by hand in both flavors, so one grep answering both was a
coincidence rather than a property.
* `DraftGraph.stage` checks its contract before it copies rather than after;
`DraftGraph._runner` goes with its last reader; and five comments still
described the rounding this removes.
Measured on this branch, each arm against the same arm before the change:
V4-Flash-DSpark tp2 dspark k=5, acceptance 65.07--65.41% across four runs of
150k draft tokens, GSM8K 0.9522--0.9553; V4-Pro tp4 mtp k=3 (EagleProposer,
the flavor V4-Flash cannot exercise because its config carries dspark_*),
acceptance 66.14 / 66.25%, GSM8K 0.9462 / 0.9545. No faults.
A review pass over 298792f found two things it removed without meaning to, both reachable only from code that same commit wrote. `target_pad_bs` lost `graph_bs_for`'s empty-batch check. A pad row is a copy of the last real row, so a zero-row batch has nothing to copy: `stage` reaches `src[-1]` and raises IndexError. The case is a rank kept alive purely to reach the draft's collectives, which `target_pad_bs`'s own docstring contemplates. `DSparkIndexBuffers.batch_ids` went from a filled arange to `torch.empty`, on the argument that `mask_pad_tail` writes both halves. It does, but only from `propose`, and the startup sweep runs the block once per warmed batch before any step calls it. The damage is small -- the scatter reads `bid` as a liveness gate (`if dst_row < 0 or dst_row >= pool_rows or bid < 0: return`) and the row it writes comes from a separately bounds-checked `dest_rows`, so garbage only drops a random subset of the warm -- but a buffer whose contract gives `-1` a meaning should not start out undefined. Filled as `arange(n) // draft`, which is what the map is. Also: * The draft capture passes `capture_error_mode="thread_local"`. Global mode invalidates a capture when any thread makes an unsafe HIP call, and the NCCL watchdog polls `hipEventQuery` every ~100ms; `cuda_graph.py` reasoned that out for its own capture. The same word belongs on `model_runner.py`'s whole-graph capture and on the TBO one, which is a separate change with its own validation. * `prepare_mtp_decode` claimed in a comment to write "THE buffer the verify fwd publishes on this field". Under TBO the verify fwd publishes the ubatch-prefixed buffer, so that was false -- and taking whatever `attn_metadata` holds would be capture-UNSAFE there, since it alternates. The fixed name is right; the comment now says why it is. * Three claims about the uninitialized map were written from a report rather than from the kernel, and overstated it as an out-of-bounds write and a captured garbage read. Corrected, and the surrounding comments cut back.
…row count `prepare_mtp_decode` sizes what it rebuilds by `positions.shape[-1]`, the count of rows the draft actually runs. Two backends carry structures derived from those buffers and neither rebuilt them, which stayed invisible while the draft always ran at `scheduled_bs`. Triton MLA never overrode `prepare_mtp_decode` at all. Its `triton_block_table` / `triton_attn_logits` / `triton_lse` are cut in `prepare_decode` at `scheduled_bs`, and `attention_mla` enters that branch on `hasattr(attn_metadata, "triton_block_table")` -- so a draft reusing the target's metadata always lands there, handing the Triton decode kernel `pad_bs` q rows against outputs sized for fewer. The cut moves into one `_cut_decode_buffers` that both entry points call, since "only one path rebuilt them" is exactly the defect. That also settles a disagreement the padding did not cause: the base rebuilds `kv_indptr` for the mid-step and `context_lens` is bumped, so the verify forward's dense block table was already a token stale for the draft. It costs a zero plus a densify per mid-step, which this path did not pay before. MiniMax-M3's sparse decode metadata is built in `prepare_decode` from `context_lens[:scheduled_bs]`, and `attention_mha` picks it up off the shared metadata with a plain `getattr`. `prepare_mtp_decode` now rebuilds it at `pad_bs`, with this step's `slot_mapping` rather than the verify forward's, and `max_query_len=1` -- not the `max_seqlen_q` argument, which carries the verify width on the `only_update` path. Neither path runs on this box: Triton MLA needs `ATOM_FORCE_ATTN_TRITON=1` or a model that dispatches to it, and the sparse one needs MiniMax-M3 with an eagle3 draft. Both are read from the source, not measured.
Contributor
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
… work `DraftGraph._replays` gated on `context.is_dummy_run`. That field is per-rank: one step has the rank holding sequences running real while the rest run dummies purely to reach the collectives. Gating on it made the real rank replay a recorded collective while the others issued it eagerly, and all of them waited forever -- V4-Flash-DSpark tp8 with --enable-dp-attention hung 8/8 on the first real decode, 120s for a 32-token request. Every term this predicate reads has to be one the whole DP group agrees on, which `use_cudagraph` and the batch are and `is_dummy_run` is not. Replaying is safe on a dummy anyway: `warmup` asserts the recording was made on a real context. That axis runs through the whole shape vocabulary, so it is unified here rather than left to be re-derived per consumer: * `ForwardMode` keeps `scheduled_bs` (what this rank was handed) and `running_bs` (what the GPU is made to run, DP-unified). `effective_bs` is deleted -- across all five branches of `decide` it was already `running_bs` where the tensors carry padding and `scheduled_bs` everywhere else. * `Context.graph_bs` splits into `running_bs` (sequences) and `running_tokens` (rows). It held tokens on prefill and sequences on decode, with `pad_for_all_gather` multiplying one back up; a DSpark ragged step is neither, so both are stored. * `context.batch_size` becomes `scheduled_bs`, and `pad_bs` / `padded_bs` / `max_batch_size` / `original_hidden_size` collapse into the pair. `runner.graph_bs`, the capture ladder, moves to `Config.capture_sizes` -- it was computed identically in two places and read from neither. * A draft states its own pair before it runs (`Drafter._publish_draft_shape`), so MoE needs no draft case: the context it reuses describes the verified tokens, not the draft width. * `pad_for_all_gather` and `build_run_label` take both heights from the context and assert against the tensor, rather than inferring one from it and accepting `None` for a state that cannot occur. V4-Flash-DSpark tp8 + DPA + PIECEWISE ragged: the request that hung now answers in 0.8s, GSM8K 1319 reads 0.9545/0.9553 with seven runs spanning 0.948-0.954, and the new MoE height assert never fires through warmup or a full eval.
… methods Six tests in this file had been erroring on `'_StubBuilder' object has no attribute '_indexer_page_pools'` since `_page_unit_regions` grew that call: the stubs borrowed the real methods one by one, and the list did not follow. Nothing was wrong with the shipped code -- `_indexer_page_pools` is on the real builder, and the same method is covered green in `test_deepseek_v4_transfer_regions.py` against a real one. The doubles had simply stopped standing for anything, quietly, which is the worse half of the failure. Subclassing the builder and skipping only its construction removes the list. The six pass with no new fake state added -- the `model_runner` and `_indexer_fp4` that `_indexer_page_pools` reads were already there, so all that was ever missing was reach. Checked the other way too: a call to a method that exists nowhere still fails all six loudly, so they are responsive rather than merely green. Full unit suite: 4249 passed, 0 failed.
None of these are new -- all five are on `main` too. CI runs reviewdog
with `-filter-mode=diff_context`, so editing a line makes its neighbours'
pre-existing violations into review comments and turns the check red;
running ruff over whole files answers a different question and says
nothing about what CI will do.
atom/config.py SIM102, next to the removed
`_set_cudagraph_sizes()` call
model_runner.py RUF059 x2, `arange` / `max_tokens`
unpacked and never read
run_labels.py UP045 + UP037 on the `batch` annotation
the signature reorder touched
v4_kernels/compress_plan.py UP006 `Tuple`, next to a rename
`/app/logs_claude/check_ruff_diff_context.py` intersects ruff's output
with the diff's hunk ranges, which is the question CI actually asks; it
reports zero for this branch. It also caught the one violation this
commit did introduce -- moving `Iterable` off `typing` needs
`collections.abc`, and then the import block needs re-sorting.
`_attach_v4_per_fwd_meta` derived the per-token grid as `running_bs * max_q_len`, but `prepare_decode` pads `positions` to `batch.dynamic_num_tokens_pad` when a flat replay bucket exists (`:2509-2521`). Two rules, computed independently, with nothing tying them together -- so every per-token tensor built off that grid could end up a different height from the `positions` its consumers index by. `_refresh_fp4_ragged_windows` is where they meet: it slices `n_committed_per_token` by `positions_gpu.shape[0]`, which raised "The size of tensor a (12) must match the size of tensor b (6)". Pass the padded count in from the caller that computed it. Name the two parameters after the counts `model_runner` and `Context` already use -- `scheduled_tokens` is what this rank was handed, `running_tokens` is the DP-unified padded shape (`forward_context.py:305-313`, and the `running_bs >= scheduled_bs` invariant at `model_runner.py:2639`) -- so the old `total_tokens` no longer reads as if it were the grid height. Decode now sizes the grid exactly to `positions`. The ubatch and capture paths pass no `running_tokens` and keep the rectangle, which already covers their `positions` (capture clamps `num_tokens_pad` below `bs * max_q_len` at `:4224`); those two are left for a follow-up that gives them the same single source. Signed-off-by: ZhangLirong-amd <Lirong.Zhang@amd.com>
PerryZhang01
pushed a commit
that referenced
this pull request
Sep 3, 2026
Every vllm-atom DeepSeek-V4 server has died at startup since #2042. All four workers abort inside `determine_available_memory` -> `profile_run`: AttributeError: 'types.SimpleNamespace' object has no attribute 'max_seqlen_q' atom/plugin/vllm/deepseek_v4_bridge.py:2164 atom/utils/forward_context.py:296 running_tokens_from_bs `force_dummy` builds `attn_metadata` as a bare namespace carrying only `state` and `in_hipgraph` (#1839). That is deliberate: the real builder allocates and copies tensors, which HIP forbids under stream capture. The namespace has therefore never had a sequence length. #2042 replaced `graph_bs=batch_size` at this call site with `running_tokens_from_bs(...)`, which dereferences `attn_metadata.max_seqlen_q` on its decode branch. `force_dummy` also forces `is_prefill=False`, so the profile forward takes that branch unconditionally -- it cannot not crash. Pass the `None` the helper already documents as "count only", which restores exactly the `batch_size` the replaced expression produced. The fix goes at the call site, where `force_dummy` is known, rather than loosening the helper: a real decode metadata that is missing `max_seqlen_q` should still fail loudly instead of silently under-padding the MoE all_gather. Not a new-model or spec-decode issue -- it reproduces on plain `DeepSeek-V4-Flash TP4`, the PR-CI accuracy job, and on the nightly's `DeepSeek-V4-Pro TP8`, which shares this bridge. Verified on 4x MI355X via the CI path itself (`.github/scripts/atom_oot_test.sh accuracy ci`, matrix entry "DeepSeek-V4-Flash TP4"): the job goes from dying at startup with no score at all to gsm8k 20-shot flexible 0.9568 / strict 0.9568 against a 0.93 threshold, in 9 minutes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gyohuangxin
pushed a commit
that referenced
this pull request
Sep 3, 2026
* Change atom-plugin CI runner to spur-runner-mi355x-8gpu * Move plugin CI logs onto shared_nfs so Spur runners can write them. Co-authored-by: Cursor <cursoragent@cursor.com> * Write plugin CI logs under the GitHub runner workspace. Co-authored-by: Cursor <cursoragent@cursor.com> * Run vLLM and SGLang plugin CI directly on the GPU runner. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(v4): keep the profile forward off the decode running_tokens branch Every vllm-atom DeepSeek-V4 server has died at startup since #2042. All four workers abort inside `determine_available_memory` -> `profile_run`: AttributeError: 'types.SimpleNamespace' object has no attribute 'max_seqlen_q' atom/plugin/vllm/deepseek_v4_bridge.py:2164 atom/utils/forward_context.py:296 running_tokens_from_bs `force_dummy` builds `attn_metadata` as a bare namespace carrying only `state` and `in_hipgraph` (#1839). That is deliberate: the real builder allocates and copies tensors, which HIP forbids under stream capture. The namespace has therefore never had a sequence length. #2042 replaced `graph_bs=batch_size` at this call site with `running_tokens_from_bs(...)`, which dereferences `attn_metadata.max_seqlen_q` on its decode branch. `force_dummy` also forces `is_prefill=False`, so the profile forward takes that branch unconditionally -- it cannot not crash. Pass the `None` the helper already documents as "count only", which restores exactly the `batch_size` the replaced expression produced. The fix goes at the call site, where `force_dummy` is known, rather than loosening the helper: a real decode metadata that is missing `max_seqlen_q` should still fail loudly instead of silently under-padding the MoE all_gather. Not a new-model or spec-decode issue -- it reproduces on plain `DeepSeek-V4-Flash TP4`, the PR-CI accuracy job, and on the nightly's `DeepSeek-V4-Pro TP8`, which shares this bridge. Verified on 4x MI355X via the CI path itself (`.github/scripts/atom_oot_test.sh accuracy ci`, matrix entry "DeepSeek-V4-Flash TP4"): the job goes from dying at startup with no score at all to gsm8k 20-shot flexible 0.9568 / strict 0.9568 against a 0.93 threshold, in 9 minutes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ci): give Kimi-K3 DCP8 enough budget to allocate a KV cache `Accuracy (Kimi-K3 DSpark DCP8 TP8)` cannot start. The engine aborts while sizing the KV cache: INFO gpu_worker.py:563] Available KV cache memory: -5.07 GiB ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization` The job is red with no score at all, which reads like an accuracy or spec-decode regression and is neither. #2033 moved this entry to DCP8 and set `--gpu-memory-utilization` from 0.93 to 0.7. Kimi-K3 is ~187 GiB of weights per GPU at TP8; on a 288 GiB MI355X a 0.7 budget is 201.6 GiB, which does not leave a KV cache behind after weights and activations. The threshold to merely reach zero is 0.7 + 5.07/288 ~= 0.718, so 0.7 is not marginally low -- it is on the wrong side of the window. The window is narrow in both directions, which is why 0.7 looks like a safe choice and is not: FULL cudagraph capture takes its scratch OUTSIDE the vLLM budget, so a HIGHER utilization starves capture (0.85 dies with `OutOfMemoryError` in `mla_decode_fwd`; 0.93 leaves so little headroom that one rank stalls in allocation while the others leave the capture context and its MoE all_reduce deadlocks). 0.75 is the value that satisfies both ends. Measured on 8x MI355X through the CI path itself (`.github/scripts/atom_oot_test.sh accuracy ci`, this matrix entry), with every GPU verified idle (<1 GiB) before each launch: gpu-mem KV cache FULL capture outcome 0.70 -5.07 GiB never reached engine abort (reproduced twice, bit-identical) 0.75 +9.33 GiB 48/48 ok serves; gsm8k 5-shot 0.9454, DSpark acceptance 0.514 The +14.40 GiB delta is exactly 0.05 x 287.98 GiB, confirming the budget is the only variable in play. Acceptance 0.514 matches the ~52% recorded for DCP8 + DSpark N=7, so the draft head is healthy. Note the run still lands under the 0.95 accuracy threshold; that is a separate question from being able to start at all, and is being characterized separately -- this commit only restores a server that can allocate a KV cache. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(vllm): restore the GDN FULL-cudagraph decode metadata compaction vLLM pads a FULL-cudagraph decode batch out to a captured graph size and leaves the padded request rows in the GDN metadata. ATOM's GDN decode kernel is request-indexed, so it walks those rows and folds them into ssm_state. #1985 deleted the pass that compacted them away, to fix Qwen3.5-35B-A3B block-FP8 -- but it scoped only its MHA/MoE half to 35B-A3B and dropped the GDN pass for every GDN model. Kimi-K3 kept an identical copy in its own KDA builder (AtomKimiK3KDAMetadataBuilder._adapt_full_graph_decode_metadata); Qwen3.5-397B was left with nothing. That is what took Accuracy (Qwen3.5-397B-A17B-MXFP4 TP4) from the 0.85-0.86 it held on the nightly dashboard through 2026-08-24 down under its 0.83 gate, on every branch that has run it since. Put the pass back and give it the same 35B-A3B exemption the rest of #1985 uses, so that fix is untouched. Measured on 4x gfx950 with the runner's own image assembly (rocm/atom-dev:vllm-latest + the aiter nightly wheel CI resolves + this branch), gsm8k 3-shot x1319 at the harness's FULL_AND_PIECEWISE: as CI runs it today 0.7817 this change 0.8544 whole #1985 GDN revert (reference) 0.8431 cudagraph_mode=PIECEWISE (reference) 0.8355 The 35B-A3B exemption is covered by unit tests but not by an end-to-end 35B block-FP8 run; that path keeps today's behavior either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(vllm): drop the model scope from the GDN metadata compaction The previous commit kept the pass off for Qwen3.5-35B-A3B, mirroring how #1985 scoped its MHA/MoE half. That guard was cargo-culted: a metadata pass has no business knowing which model it is running under, and the reason #1985 gave for removing the pass does not survive checking. Why the pass is unconditionally needed. vLLM means to mark the padded rows itself: non_spec_state_indices_tensor[num_decodes:].fill_(NULL_BLOCK_ID) but num_decodes comes from split_decodes_and_prefills, whose fast path if max_query_len <= decode_threshold and ...: return num_reqs, 0, num_tokens, 0 fires for any pure-decode batch (max_query_len == 1) and returns num_decodes = num_reqs, i.e. the *padded* request count under FULL-graph replay. The fill therefore starts at num_decodes == batch_size and is a no-op: the padded slots keep whatever block_table_tensor[:, 0] held, and ATOM's request-indexed GDN decode kernel folds them into ssm_state. That depends only on the decode batch being padded -- not on the model, its quantization, or TP. It is the same reason Kimi-K3 carries the pass in its own KDA builder. Why #1985's reason does not hold. It removed the pass because it "rewrote that metadata using num_actual_tokens (token-padded graph size) and shrunk num_decodes". That was true of the original version (2255001) and was fixed two versions earlier, in 441476e, which switched to num_reqs with a comment saying exactly why. Qwen3.5-397B-A17B-MXFP4 TP4, gsm8k 3-shot x1319, 4x gfx950, runner image assembly, FULL_AND_PIECEWISE: 0.8423 (scoped variant measured 0.8544; the two are within one stderr of each other, and both clear the 0.83 gate against 0.7817 before the fix). The 35B-A3B block-FP8 path is now covered by the pass again. I could not score it end to end -- this box has the checkpoint but no validated vLLM plugin recipe for it, and my attempt floored at 0.05-0.07 under both FULL and PIECEWISE, so it measured nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(vllm): keep the MTP position-0 mask off the layer signature Every vLLM-plugin MTP run has been dying in profile_run since the fused FP8 prologue landed: TypeError: DeepSeekMultiTokenPredictorLayer.forward() takes from 5 to 6 positional arguments but 7 were given The plugin installed its position-0 input-embedding mask by rebinding each draft layer's forward to a wrapper that re-called the original with a hardcoded five-argument list. When the fused prologue gave the layer a sixth argument (eh_input_quant), the wrapper kept advertising the old arity -- and functools.wraps kept the original qualname, so the error pointed at a model signature that was in fact correct. The wrapper could not have worked on the fused path anyway: there the embedding never materializes as a tensor the layer receives, so there is nothing for a layer-level wrapper to mask. Move the mask into the predictors, which own the embedding lookup on both paths and cannot drift from their own layers. On the unfused path it is the same torch.where as before. On the fused path the masked rows are fed the sentinel id -1: the prologue kernel already emits a zero embedding for any id outside [0, vocab_size), and RMSNorm(0) is 0, so the result is bit-identical to zeroing inputs_embeds -- without materializing the [num_tokens, hidden] tensor the fusion exists to avoid. The plugin now flips a flag instead of rewriting a signature, and raises if an arch in _MTP_MASK_INPUT_ARCH has no predictor to flip it on, rather than silently serving an unmasked draft. The regression test lives in tests/ rather than tests/plugin/ because run_unit_tests.sh skips the latter. Fixes the GLM-5.2-MXFP4-MTP TP4 plugin accuracy job. * ci: download plugin CI draft models like the target Kimi-K3 DSpark failed because speculative-config pointed at a missing local draft path. Cache the draft the same way as the target and rewrite extra_args to the resolved directory. Co-authored-by: Cursor <cursoragent@cursor.com> * ci: SIGKILL leftover host/KFD VRAM holders after plugin docker cleanup Plugin HIP smoke was failing on spur runners when VRAM stayed allocated by host PIDs after leftover containers were already gone. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: perzhang <perzhang@amd.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: PerryZhang01 <Perry.Zhang@amd.com> Co-authored-by: Junyan Yang <junyyang@crs-m2m-cpu-spur-v2-009.us-east2-a.compute.internal> Co-authored-by: Junyan Yang <junyyang@crs-m2m-cpu-spur-012.us-east2-a.compute.internal>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
aiter's flydsl builds one hgemm per tile config, in-process and chosen from the
row count, so a batch first seen mid-serve stalls that step — and every restart
pays it again. The draft never landed on the shapes the target had captured, so
it paid this on almost every distinct decode batch.
A drafter now declares its forwards as
DraftGraphs. At the end of CUDAGraphcapture the runner runs each one once per
graph_bs; at serve time a pass runsat the batch the target just ran, which
ForwardMode.decidepicks out of thatsame list — so "warmed" and "reachable" are one set by construction rather than
two that drift.
ATOM_DRAFT_CUDAGRAPH(default on) also records each warmedshape, so a pass replays instead of relaunching every kernel.
Three capabilities, each needing strictly more than the last: warm needs
nothing; pad needs staged inputs and a promise that fabricated rows are
inert; capture needs pad, since a graph is one shape. A flavor that cannot
answer one of them declares nothing rather than declaring a pass that claims to
be warmable and is not — Kimi-K3 drafts through a separate model, and MRoPE
keeps its token axis last, so neither declares.
Measured
--method dspark -n 5--method mtp -n 3(EagleProposer)Four DSpark runs and two eagle runs, each over 126k–150k draft tokens; every arm
sits on its pre-change baseline. On V4-Flash-DSpark tp4, putting the LM head
inside the recording took draft kernel launches 30 → 0 per pass and draft wall
time 915.8 → 118.9 µs;
hipModuleLoadDataper rank went 6 → 0 on the16k/20/50c reproducer.
Startup cost is one log line:
Draft passes warmed at [1, 2, ..., 256] in 0.6s.Correctness fixes carried along
prepare_mtp_decodeis sized by rows, not by sequence count. The rowcount is
positions.shape[-1]— the LAST axis, because MRoPE keeps the tokenaxis there and every other layout is one-dimensional, where the two agree.
bsstays the real sequence count. All four backends and the base contractfollow this.
batch_id_per_tokengets-1over thefabricated rows — the sentinel the verify forward already uses — so the index
writer and
csa_translate_packskip them. At bs=65 padded to 128 that is halfthe batch.
prepare_decodefills the padded tail ofstate_slot_outwith0, and0is a real position (
physical_slotisslot_positions - 1 - group, so it isthe last group, which the pool hands out as concurrency approaches
max_num_seqs). The fused SWA write gates onbid >= 0, soDSparkIndexBuffersnow carries a-1tail.metadata at the drafter's row count. Neither did, which was invisible while
the draft always ran at
scheduled_bs.CommonAttentionBuilder, replacingDrafter.arrange_bsandDeepseekV4AttentionMetadataBuilder._v4_row_ids.Trace labels
propose_eagle[i/k tok= bs=<real>/<pad> (graph)]andpropose_dspark[bs=<real>/<pad> T= (graph)], both from oneDraftGraph.label— so "did the draft get into a graph" is a single grep acrossflavors.
tools/analyze_trace_summary.pyfollows the rename.Test plan
bash .github/scripts/run_unit_tests.sh— 4234 passed, 85 skipped. The 6failures in
tests/test_v4_checkpoint_slot_copy.pyreproduce on the mergebase with none of these changes applied.
tests/test_draft_graph.py— 18 cases with no aiter and no GPU, so CIactually runs them.
DraftGraphwas extracted fromdrafter.pyfor this:which batch a pass may run at, the tail-repeat, and the pad contract are
pure torch.
blackclean;ruffidentical per file to the merge base.(need
ATOM_FORCE_ATTN_TRITON=1/ M3 with an eagle3 draft), and anythingunder
data_parallel_size > 1. Those are read from the source.Known and deliberately not addressed here
dp_uniform_decodeat its capture default, so acaptured draft replays the uniform all-gather rather than the
all_gatherv_refresh_dp_metadataselects. Every rank replays the same graph at the samebatch, so the collective stays consistent; the variable-length path is simply
dead for a captured draft.
DraftGraph._cuda_graphsis invisible toModelRunner's graph reset and tothe rollout sleep/wake release, so draft graphs keep pinning draft weights
across a sleep.
capture_cudagraphis skipped entirely under--enforce-eager, and it is theonly caller of the warm — so the JIT saving is absent exactly where graphs
cannot be used.