Skip to content

perf(spec-decode): warm every draft forward at a batch the target captured - #2042

Merged
valarLip merged 10 commits into
mainfrom
refactor/draft-graph
Aug 27, 2026
Merged

valarLip merged 10 commits into
mainfrom
refactor/draft-graph

Conversation

@valarLip

@valarLip valarLip commented Aug 26, 2026 •

Copy link
Copy Markdown
Collaborator

What

aiter's flydsl builds one hgemm per tile config, in-process and chosen from the
row count, so a batch first seen mid-serve stalls that step — and every restart
pays it again. The draft never landed on the shapes the target had captured, so
it paid this on almost every distinct decode batch.

A drafter now declares its forwards as DraftGraphs. At the end of CUDAGraph
capture the runner runs each one once per graph_bs; at serve time a pass runs
at the batch the target just ran, which ForwardMode.decide picks out of that
same list — so "warmed" and "reachable" are one set by construction rather than
two that drift. ATOM_DRAFT_CUDAGRAPH (default on) also records each warmed
shape, so a pass replays instead of relaunching every kernel.

Three capabilities, each needing strictly more than the last: warm needs
nothing; pad needs staged inputs and a promise that fabricated rows are
inert; capture needs pad, since a graph is one shape. A flavor that cannot
answer one of them declares nothing rather than declaring a pass that claims to
be warmable and is not — Kimi-K3 drafts through a separate model, and MRoPE
keeps its token axis last, so neither declares.

Measured

Arm GSM8K flexible Acceptance toks/fwd
V4-Flash-DSpark tp2, --method dspark -n 5 0.9522 – 0.9553 65.07 – 65.41% 4.25 – 4.27
V4-Pro tp4, --method mtp -n 3 (EagleProposer) 0.9462 / 0.9545 66.14 / 66.25% 2.98 / 2.99

Four DSpark runs and two eagle runs, each over 126k–150k draft tokens; every arm
sits on its pre-change baseline. On V4-Flash-DSpark tp4, putting the LM head
inside the recording took draft kernel launches 30 → 0 per pass and draft wall
time 915.8 → 118.9 µs; hipModuleLoadData per rank went 6 → 0 on the
16k/20/50c reproducer.

Startup cost is one log line: Draft passes warmed at [1, 2, ..., 256] in 0.6s.

Correctness fixes carried along

  • prepare_mtp_decode is sized by rows, not by sequence count. The row
    count is positions.shape[-1] — the LAST axis, because MRoPE keeps the token
    axis there and every other layout is one-dimensional, where the two agree.
    bs stays the real sequence count. All four backends and the base contract
    follow this.
  • DeepSeek-V4 marks its pad tail. batch_id_per_token gets -1 over the
    fabricated rows — the sentinel the verify forward already uses — so the index
    writer and csa_translate_pack skip them. At bs=65 padded to 128 that is half
    the batch.
  • A padded DSpark block no longer scatters KV through a live ring slot.
    prepare_decode fills the padded tail of state_slot_out with 0, and 0
    is a real position (physical_slot is slot_positions - 1 - group, so it is
    the last group, which the pool hands out as concurrency approaches
    max_num_seqs). The fused SWA write gates on bid >= 0, so
    DSparkIndexBuffers now carries a -1 tail.
  • Triton MLA and the MiniMax-M3 sparse path rebuild their derived decode
    metadata at the drafter's row count.
    Neither did, which was invisible while
    the draft always ran at scheduled_bs.
  • One arange over the batch axis, on CommonAttentionBuilder, replacing
    Drafter.arrange_bs and DeepseekV4AttentionMetadataBuilder._v4_row_ids.

Trace labels

propose_eagle[i/k tok= bs=<real>/<pad> (graph)] and
propose_dspark[bs=<real>/<pad> T= (graph)], both from one
DraftGraph.label — so "did the draft get into a graph" is a single grep across
flavors. tools/analyze_trace_summary.py follows the rename.

Test plan

  • bash .github/scripts/run_unit_tests.sh — 4234 passed, 85 skipped. The 6
    failures in tests/test_v4_checkpoint_slot_copy.py reproduce on the merge
    base with none of these changes applied.
  • Every commit passes the suite standalone (bisect-safe).
  • tests/test_draft_graph.py — 18 cases with no aiter and no GPU, so CI
    actually runs them. DraftGraph was extracted from drafter.py for this:
    which batch a pass may run at, the tail-repeat, and the pad contract are
    pure torch.
  • black clean; ruff identical per file to the merge base.
  • GSM8K + acceptance on both drafter flavors, table above.
  • Not exercised on this box: the Triton MLA and MiniMax-M3 sparse fixes
    (need ATOM_FORCE_ATTN_TRITON=1 / M3 with an eagle3 draft), and anything
    under data_parallel_size > 1. Those are read from the source.

Known and deliberately not addressed here

  • Under DP the warm runs with dp_uniform_decode at its capture default, so a
    captured draft replays the uniform all-gather rather than the all_gatherv
    _refresh_dp_metadata selects. Every rank replays the same graph at the same
    batch, so the collective stays consistent; the variable-length path is simply
    dead for a captured draft.
  • DraftGraph._cuda_graphs is invisible to ModelRunner's graph reset and to
    the rollout sleep/wake release, so draft graphs keep pinning draft weights
    across a sleep.
  • capture_cudagraph is skipped entirely under --enforce-eager, and it is the
    only caller of the warm — so the JIT saving is absent exactly where graphs
    cannot be used.

`prepare_mtp_decode` rebuilt every buffer to `bs`, the count of sequences
carrying a real request. A drafter may run a WIDER batch than that, to land on
a shape the target already captured, and then every slice came up short: the
attention builder rebuilt kv_indptr, block_tables and slot_mapping for the real
rows only, and the fabricated ones kept whatever the previous, larger step had
left behind.

The row count is `positions.shape[-1]` -- the LAST axis, because MRoPE keeps the
token axis there and every other layout is one-dimensional, where the two agree.
`bs` stays the real sequence count, and the base carries the contract once so a
backend that never distinguishes the two simply never reads it.

DeepSeek-V4 additionally:

  * marks the pad tail of `batch_id_per_token` with `-1`, the sentinel the verify
    forward already uses. `write_v4_paged_decode_indices` returns on `bid < 0`
    before it loads a ring slot, so a padded draft costs nothing for the rows it
    invented instead of repeating a real one's gather -- at bs=65 padded to 128
    that is half the batch. It writes into the buffer the verify forward
    publishes on that field, not a private one: a captured draft graph bakes
    that address, and republishing a different tensor would leave the replay
    reading the verify map.

  * publishes the compressor ring slots at the padded `bs`, which the ubatch
    path 200 lines down already does. A consumer that runs the padded batch can
    then slice to it instead of asking the drafter to stage a copy.
…tured

aiter's flydsl builds one hgemm per tile config, in-process and chosen from the
row count, so a batch first seen mid-serve stalls that step -- and every restart
pays it again. The draft never landed on the shapes the target had captured, so
it paid this on almost every distinct decode batch.

A drafter now DECLARES its forwards as `DraftGraph`s. At the end of CUDAGraph
capture the runner runs each one once per `graph_bs`; at serve time the drafter
rounds its batch up to that same list, so warmed and reachable are one set by
construction rather than two that drift. `ATOM_DRAFT_CUDAGRAPH` (default on)
also captures each warmed shape, so the pass replays instead of relaunching
every kernel.

Measured, V4-Pro tp4 MTP3: acceptance 66.04% and GSM8K 0.9545, against a 64.09
-- 64.20% / 0.9522 -- 0.9545 baseline. On V4-Flash-DSpark tp4 the LM head inside
the recording took draft kernel launches 30 -> 0 per pass and draft wall time
915.8 -> 118.9 us.

Three capabilities, each needing strictly more than the last: warm needs
nothing; pad needs staged inputs and a promise that fabricated rows are inert;
capture needs pad, since a graph is one shape. A flavor that cannot answer one
of them declares nothing rather than declaring a pass that claims to be warmable
and is not -- Kimi-K3 drafts through a separate model, and MRoPE keeps its token
axis last, so neither declares.

`DraftGraph` lives apart from `Drafter` so it stays importable without a GPU
AITER build: its invariants are pure torch, and CI has no aiter. Everything
speaks the same three words as the code around it -- `bs` real, `pad_bs` what
runs, `graph_bs` what was captured.

Trace labels become `propose_eagle[...]` / `propose_dspark[...]`, both carrying
`bs=<real>/<pad>` and a trailing ` graph` on a replay, so "did the draft get
into a graph" is a grep. `tools/analyze_trace_summary.py` follows the rename.
…r's cwd

One 8-rank fault writes ~283GB of them and fills the disk. They land next to the
server's working directory, which is the repo, while `rm -rf ./gpucore.*` cleared
wherever the launcher happened to be invoked from. `HSA_ENABLE_COREDUMP=0` does
not prevent them (measured), so clearing is the only thing that works.
Review of the draft-graph passes turned up one place a padded draft could
corrupt another request, and several shapes that existed only because two
flavors had each grown their own answer to the same question.

A pass now runs at `forward_mode.effective_bs` -- the count `model_runner`
built the attention metadata with (model_runner.py:2618) -- or at its own
batch, and nothing else. That replaces `_step_pad_bs` (eagle),
`_block_graph_bs` + `_self_chosen_graph_bs` (DSpark) and
`DraftGraph.graph_bs_for` with one `DraftGraph.target_pad_bs`:

  * Off the cudagraph path nothing is padded at all. Every eager branch of
    `ForwardMode.decide` returns `effective_bs == scheduled_bs`, so a pad row
    there reads per-sequence metadata nobody wrote this step. DSpark used to
    pad anyway, on the strength of owning a graph at that batch.
  * `effective_bs`, not `context.graph_bs`: the two agree except where
    `_dspark_ragged_moe_graph_bs` overwrites graph_bs with a token count.
    Reading effective_bs makes that unreachable rather than guarded against,
    and the guard could not have been shared -- under DP-uniform decode
    `padded_scheduled_bs` is a cross-rank max, so it would have switched
    padding off exactly where it pays.
  * The answer is a warmed shape by construction, since `decide` picks it out
    of `graph_bs`, which is the list the startup sweep warms.

DeepSeek-V4's fabricated draft rows no longer scatter KV. Their ring slot is
the `0` that `prepare_decode` fills the padded tail with, and 0 is a real
position -- `physical_slot` is `slot_positions - 1 - group`, so it is the last
group, which the pool hands out as concurrency approaches max_num_seqs. The
fused SWA write gates on `bid >= 0`, so `DSparkIndexBuffers` now carries a `-1`
tail, restored from the shared arange each step so a sentinel cannot outlive
the row that earned it. `dspark_fp8_indices` had stated the opposite invariant
-- "the draft runs at scheduled_bs, not the padded effective_bs" -- which
padding had quietly made false.

Also:

  * `warmup_draft_graphs` moves onto `Drafter`, beside `_declare_draft_graphs`
    and `_build_draft_graphs`, so a pass's whole lifetime lives in one place.
    The runner hands over a bound capture builder instead of making the drafter
    probe the backend's signature a second time.
  * One arange over the batch axis, on `CommonAttentionBuilder`. It replaces
    `Drafter.arrange_bs` and `DeepseekV4AttentionMetadataBuilder._v4_row_ids`,
    and serves DSpark's token -> request map as well.
  * `_enter_decode_metadata` returns what `attn_metadata` does not already
    carry: two values rather than five, so no buffer has a second name.
  * `DraftGraph.label` -- the `bs=<real>/<pad>` + trailing ` graph` convention
    was written out by hand in both flavors, so one grep answering both was a
    coincidence rather than a property.
  * `DraftGraph.stage` checks its contract before it copies rather than after;
    `DraftGraph._runner` goes with its last reader; and five comments still
    described the rounding this removes.

Measured on this branch, each arm against the same arm before the change:
V4-Flash-DSpark tp2 dspark k=5, acceptance 65.07--65.41% across four runs of
150k draft tokens, GSM8K 0.9522--0.9553; V4-Pro tp4 mtp k=3 (EagleProposer,
the flavor V4-Flash cannot exercise because its config carries dspark_*),
acceptance 66.14 / 66.25%, GSM8K 0.9462 / 0.9545. No faults.
A review pass over 298792f found two things it removed without meaning to,
both reachable only from code that same commit wrote.

`target_pad_bs` lost `graph_bs_for`'s empty-batch check. A pad row is a copy of
the last real row, so a zero-row batch has nothing to copy: `stage` reaches
`src[-1]` and raises IndexError. The case is a rank kept alive purely to reach
the draft's collectives, which `target_pad_bs`'s own docstring contemplates.

`DSparkIndexBuffers.batch_ids` went from a filled arange to `torch.empty`, on
the argument that `mask_pad_tail` writes both halves. It does, but only from
`propose`, and the startup sweep runs the block once per warmed batch before
any step calls it. The damage is small -- the scatter reads `bid` as a liveness
gate (`if dst_row < 0 or dst_row >= pool_rows or bid < 0: return`) and the row
it writes comes from a separately bounds-checked `dest_rows`, so garbage only
drops a random subset of the warm -- but a buffer whose contract gives `-1` a
meaning should not start out undefined. Filled as `arange(n) // draft`, which
is what the map is.

Also:

  * The draft capture passes `capture_error_mode="thread_local"`. Global mode
    invalidates a capture when any thread makes an unsafe HIP call, and the
    NCCL watchdog polls `hipEventQuery` every ~100ms; `cuda_graph.py` reasoned
    that out for its own capture. The same word belongs on `model_runner.py`'s
    whole-graph capture and on the TBO one, which is a separate change with its
    own validation.
  * `prepare_mtp_decode` claimed in a comment to write "THE buffer the verify
    fwd publishes on this field". Under TBO the verify fwd publishes the
    ubatch-prefixed buffer, so that was false -- and taking whatever
    `attn_metadata` holds would be capture-UNSAFE there, since it alternates.
    The fixed name is right; the comment now says why it is.
  * Three claims about the uninitialized map were written from a report rather
    than from the kernel, and overstated it as an out-of-bounds write and a
    captured garbage read. Corrected, and the surrounding comments cut back.
…row count

`prepare_mtp_decode` sizes what it rebuilds by `positions.shape[-1]`, the count
of rows the draft actually runs. Two backends carry structures derived from
those buffers and neither rebuilt them, which stayed invisible while the draft
always ran at `scheduled_bs`.

Triton MLA never overrode `prepare_mtp_decode` at all. Its `triton_block_table`
/ `triton_attn_logits` / `triton_lse` are cut in `prepare_decode` at
`scheduled_bs`, and `attention_mla` enters that branch on
`hasattr(attn_metadata, "triton_block_table")` -- so a draft reusing the
target's metadata always lands there, handing the Triton decode kernel `pad_bs`
q rows against outputs sized for fewer. The cut moves into one
`_cut_decode_buffers` that both entry points call, since "only one path rebuilt
them" is exactly the defect.

That also settles a disagreement the padding did not cause: the base rebuilds
`kv_indptr` for the mid-step and `context_lens` is bumped, so the verify
forward's dense block table was already a token stale for the draft. It costs a
zero plus a densify per mid-step, which this path did not pay before.

MiniMax-M3's sparse decode metadata is built in `prepare_decode` from
`context_lens[:scheduled_bs]`, and `attention_mha` picks it up off the shared
metadata with a plain `getattr`. `prepare_mtp_decode` now rebuilds it at
`pad_bs`, with this step's `slot_mapping` rather than the verify forward's, and
`max_query_len=1` -- not the `max_seqlen_q` argument, which carries the verify
width on the `only_update` path.

Neither path runs on this box: Triton MLA needs `ATOM_FORCE_ATTN_TRITON=1` or a
model that dispatches to it, and the sparse one needs MiniMax-M3 with an eagle3
draft. Both are read from the source, not measured.
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2042 --add-label <label>

@zufayu
zufayu requested a review from yitingw1 August 27, 2026 01:23
valarLip and others added 4 commits August 27, 2026 04:02
… work

`DraftGraph._replays` gated on `context.is_dummy_run`. That field is
per-rank: one step has the rank holding sequences running real while the
rest run dummies purely to reach the collectives. Gating on it made the
real rank replay a recorded collective while the others issued it
eagerly, and all of them waited forever -- V4-Flash-DSpark tp8 with
--enable-dp-attention hung 8/8 on the first real decode, 120s for a
32-token request. Every term this predicate reads has to be one the
whole DP group agrees on, which `use_cudagraph` and the batch are and
`is_dummy_run` is not. Replaying is safe on a dummy anyway: `warmup`
asserts the recording was made on a real context.

That axis runs through the whole shape vocabulary, so it is unified here
rather than left to be re-derived per consumer:

* `ForwardMode` keeps `scheduled_bs` (what this rank was handed) and
  `running_bs` (what the GPU is made to run, DP-unified). `effective_bs`
  is deleted -- across all five branches of `decide` it was already
  `running_bs` where the tensors carry padding and `scheduled_bs`
  everywhere else.
* `Context.graph_bs` splits into `running_bs` (sequences) and
  `running_tokens` (rows). It held tokens on prefill and sequences on
  decode, with `pad_for_all_gather` multiplying one back up; a DSpark
  ragged step is neither, so both are stored.
* `context.batch_size` becomes `scheduled_bs`, and `pad_bs` /
  `padded_bs` / `max_batch_size` / `original_hidden_size` collapse into
  the pair. `runner.graph_bs`, the capture ladder, moves to
  `Config.capture_sizes` -- it was computed identically in two places
  and read from neither.
* A draft states its own pair before it runs
  (`Drafter._publish_draft_shape`), so MoE needs no draft case: the
  context it reuses describes the verified tokens, not the draft width.
* `pad_for_all_gather` and `build_run_label` take both heights from the
  context and assert against the tensor, rather than inferring one from
  it and accepting `None` for a state that cannot occur.

V4-Flash-DSpark tp8 + DPA + PIECEWISE ragged: the request that hung now
answers in 0.8s, GSM8K 1319 reads 0.9545/0.9553 with seven runs spanning
0.948-0.954, and the new MoE height assert never fires through warmup or
a full eval.
… methods

Six tests in this file had been erroring on
`'_StubBuilder' object has no attribute '_indexer_page_pools'` since
`_page_unit_regions` grew that call: the stubs borrowed the real methods
one by one, and the list did not follow. Nothing was wrong with the
shipped code -- `_indexer_page_pools` is on the real builder, and the
same method is covered green in `test_deepseek_v4_transfer_regions.py`
against a real one. The doubles had simply stopped standing for
anything, quietly, which is the worse half of the failure.

Subclassing the builder and skipping only its construction removes the
list. The six pass with no new fake state added -- the `model_runner`
and `_indexer_fp4` that `_indexer_page_pools` reads were already there,
so all that was ever missing was reach. Checked the other way too: a
call to a method that exists nowhere still fails all six loudly, so they
are responsive rather than merely green.

Full unit suite: 4249 passed, 0 failed.
None of these are new -- all five are on `main` too. CI runs reviewdog
with `-filter-mode=diff_context`, so editing a line makes its neighbours'
pre-existing violations into review comments and turns the check red;
running ruff over whole files answers a different question and says
nothing about what CI will do.

  atom/config.py                   SIM102, next to the removed
                                   `_set_cudagraph_sizes()` call
  model_runner.py                  RUF059 x2, `arange` / `max_tokens`
                                   unpacked and never read
  run_labels.py                    UP045 + UP037 on the `batch` annotation
                                   the signature reorder touched
  v4_kernels/compress_plan.py      UP006 `Tuple`, next to a rename

`/app/logs_claude/check_ruff_diff_context.py` intersects ruff's output
with the diff's hunk ranges, which is the question CI actually asks; it
reports zero for this branch. It also caught the one violation this
commit did introduce -- moving `Iterable` off `typing` needs
`collections.abc`, and then the import block needs re-sorting.
`_attach_v4_per_fwd_meta` derived the per-token grid as
`running_bs * max_q_len`, but `prepare_decode` pads `positions` to
`batch.dynamic_num_tokens_pad` when a flat replay bucket exists
(`:2509-2521`). Two rules, computed independently, with nothing tying them
together -- so every per-token tensor built off that grid could end up a
different height from the `positions` its consumers index by.

`_refresh_fp4_ragged_windows` is where they meet: it slices
`n_committed_per_token` by `positions_gpu.shape[0]`, which raised
"The size of tensor a (12) must match the size of tensor b (6)".

Pass the padded count in from the caller that computed it. Name the two
parameters after the counts `model_runner` and `Context` already use --
`scheduled_tokens` is what this rank was handed, `running_tokens` is the
DP-unified padded shape (`forward_context.py:305-313`, and the
`running_bs >= scheduled_bs` invariant at `model_runner.py:2639`) -- so the
old `total_tokens` no longer reads as if it were the grid height.

Decode now sizes the grid exactly to `positions`. The ubatch and capture
paths pass no `running_tokens` and keep the rectangle, which already covers
their `positions` (capture clamps `num_tokens_pad` below `bs * max_q_len`
at `:4224`); those two are left for a follow-up that gives them the same
single source.

Signed-off-by: ZhangLirong-amd <Lirong.Zhang@amd.com>
@valarLip
valarLip merged commit e7758fd into main Aug 27, 2026
31 of 32 checks passed
@valarLip
valarLip deleted the refactor/draft-graph branch August 27, 2026 08:27
PerryZhang01 pushed a commit that referenced this pull request Sep 3, 2026
Every vllm-atom DeepSeek-V4 server has died at startup since #2042. All
four workers abort inside `determine_available_memory` -> `profile_run`:

    AttributeError: 'types.SimpleNamespace' object has no attribute
    'max_seqlen_q'
      atom/plugin/vllm/deepseek_v4_bridge.py:2164
      atom/utils/forward_context.py:296  running_tokens_from_bs

`force_dummy` builds `attn_metadata` as a bare namespace carrying only
`state` and `in_hipgraph` (#1839). That is deliberate: the real builder
allocates and copies tensors, which HIP forbids under stream capture.
The namespace has therefore never had a sequence length.

#2042 replaced `graph_bs=batch_size` at this call site with
`running_tokens_from_bs(...)`, which dereferences
`attn_metadata.max_seqlen_q` on its decode branch. `force_dummy` also
forces `is_prefill=False`, so the profile forward takes that branch
unconditionally -- it cannot not crash.

Pass the `None` the helper already documents as "count only", which
restores exactly the `batch_size` the replaced expression produced. The
fix goes at the call site, where `force_dummy` is known, rather than
loosening the helper: a real decode metadata that is missing
`max_seqlen_q` should still fail loudly instead of silently under-padding
the MoE all_gather.

Not a new-model or spec-decode issue -- it reproduces on plain
`DeepSeek-V4-Flash TP4`, the PR-CI accuracy job, and on the nightly's
`DeepSeek-V4-Pro TP8`, which shares this bridge.

Verified on 4x MI355X via the CI path itself
(`.github/scripts/atom_oot_test.sh accuracy ci`, matrix entry
"DeepSeek-V4-Flash TP4"): the job goes from dying at startup with no
score at all to gsm8k 20-shot flexible 0.9568 / strict 0.9568 against a
0.93 threshold, in 9 minutes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gyohuangxin pushed a commit that referenced this pull request Sep 3, 2026
* Change atom-plugin CI runner to spur-runner-mi355x-8gpu

* Move plugin CI logs onto shared_nfs so Spur runners can write them.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Write plugin CI logs under the GitHub runner workspace.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Run vLLM and SGLang plugin CI directly on the GPU runner.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(v4): keep the profile forward off the decode running_tokens branch

Every vllm-atom DeepSeek-V4 server has died at startup since #2042. All
four workers abort inside `determine_available_memory` -> `profile_run`:

    AttributeError: 'types.SimpleNamespace' object has no attribute
    'max_seqlen_q'
      atom/plugin/vllm/deepseek_v4_bridge.py:2164
      atom/utils/forward_context.py:296  running_tokens_from_bs

`force_dummy` builds `attn_metadata` as a bare namespace carrying only
`state` and `in_hipgraph` (#1839). That is deliberate: the real builder
allocates and copies tensors, which HIP forbids under stream capture.
The namespace has therefore never had a sequence length.

#2042 replaced `graph_bs=batch_size` at this call site with
`running_tokens_from_bs(...)`, which dereferences
`attn_metadata.max_seqlen_q` on its decode branch. `force_dummy` also
forces `is_prefill=False`, so the profile forward takes that branch
unconditionally -- it cannot not crash.

Pass the `None` the helper already documents as "count only", which
restores exactly the `batch_size` the replaced expression produced. The
fix goes at the call site, where `force_dummy` is known, rather than
loosening the helper: a real decode metadata that is missing
`max_seqlen_q` should still fail loudly instead of silently under-padding
the MoE all_gather.

Not a new-model or spec-decode issue -- it reproduces on plain
`DeepSeek-V4-Flash TP4`, the PR-CI accuracy job, and on the nightly's
`DeepSeek-V4-Pro TP8`, which shares this bridge.

Verified on 4x MI355X via the CI path itself
(`.github/scripts/atom_oot_test.sh accuracy ci`, matrix entry
"DeepSeek-V4-Flash TP4"): the job goes from dying at startup with no
score at all to gsm8k 20-shot flexible 0.9568 / strict 0.9568 against a
0.93 threshold, in 9 minutes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(ci): give Kimi-K3 DCP8 enough budget to allocate a KV cache

`Accuracy (Kimi-K3 DSpark DCP8 TP8)` cannot start. The engine aborts
while sizing the KV cache:

    INFO gpu_worker.py:563] Available KV cache memory: -5.07 GiB
    ValueError: No available memory for the cache blocks.
                Try increasing `gpu_memory_utilization`

The job is red with no score at all, which reads like an accuracy or
spec-decode regression and is neither.

#2033 moved this entry to DCP8 and set `--gpu-memory-utilization` from
0.93 to 0.7. Kimi-K3 is ~187 GiB of weights per GPU at TP8; on a 288 GiB
MI355X a 0.7 budget is 201.6 GiB, which does not leave a KV cache behind
after weights and activations. The threshold to merely reach zero is
0.7 + 5.07/288 ~= 0.718, so 0.7 is not marginally low -- it is on the
wrong side of the window.

The window is narrow in both directions, which is why 0.7 looks like a
safe choice and is not: FULL cudagraph capture takes its scratch OUTSIDE
the vLLM budget, so a HIGHER utilization starves capture (0.85 dies with
`OutOfMemoryError` in `mla_decode_fwd`; 0.93 leaves so little headroom
that one rank stalls in allocation while the others leave the capture
context and its MoE all_reduce deadlocks). 0.75 is the value that
satisfies both ends.

Measured on 8x MI355X through the CI path itself
(`.github/scripts/atom_oot_test.sh accuracy ci`, this matrix entry), with
every GPU verified idle (<1 GiB) before each launch:

    gpu-mem   KV cache      FULL capture   outcome
    0.70      -5.07 GiB     never reached  engine abort (reproduced twice,
                                           bit-identical)
    0.75      +9.33 GiB     48/48 ok       serves; gsm8k 5-shot 0.9454,
                                           DSpark acceptance 0.514

The +14.40 GiB delta is exactly 0.05 x 287.98 GiB, confirming the budget
is the only variable in play.

Acceptance 0.514 matches the ~52% recorded for DCP8 + DSpark N=7, so the
draft head is healthy. Note the run still lands under the 0.95 accuracy
threshold; that is a separate question from being able to start at all,
and is being characterized separately -- this commit only restores a
server that can allocate a KV cache.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(vllm): restore the GDN FULL-cudagraph decode metadata compaction

vLLM pads a FULL-cudagraph decode batch out to a captured graph size and
leaves the padded request rows in the GDN metadata. ATOM's GDN decode
kernel is request-indexed, so it walks those rows and folds them into
ssm_state. #1985 deleted the pass that compacted them away, to fix
Qwen3.5-35B-A3B block-FP8 -- but it scoped only its MHA/MoE half to
35B-A3B and dropped the GDN pass for every GDN model. Kimi-K3 kept an
identical copy in its own KDA builder
(AtomKimiK3KDAMetadataBuilder._adapt_full_graph_decode_metadata);
Qwen3.5-397B was left with nothing.

That is what took Accuracy (Qwen3.5-397B-A17B-MXFP4 TP4) from the
0.85-0.86 it held on the nightly dashboard through 2026-08-24 down under
its 0.83 gate, on every branch that has run it since.

Put the pass back and give it the same 35B-A3B exemption the rest of
#1985 uses, so that fix is untouched. Measured on 4x gfx950 with the
runner's own image assembly (rocm/atom-dev:vllm-latest + the aiter
nightly wheel CI resolves + this branch), gsm8k 3-shot x1319 at the
harness's FULL_AND_PIECEWISE:

  as CI runs it today                     0.7817
  this change                             0.8544
  whole #1985 GDN revert (reference)      0.8431
  cudagraph_mode=PIECEWISE (reference)    0.8355

The 35B-A3B exemption is covered by unit tests but not by an end-to-end
35B block-FP8 run; that path keeps today's behavior either way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(vllm): drop the model scope from the GDN metadata compaction

The previous commit kept the pass off for Qwen3.5-35B-A3B, mirroring how
#1985 scoped its MHA/MoE half. That guard was cargo-culted: a metadata
pass has no business knowing which model it is running under, and the
reason #1985 gave for removing the pass does not survive checking.

Why the pass is unconditionally needed. vLLM means to mark the padded
rows itself:

    non_spec_state_indices_tensor[num_decodes:].fill_(NULL_BLOCK_ID)

but num_decodes comes from split_decodes_and_prefills, whose fast path

    if max_query_len <= decode_threshold and ...:
        return num_reqs, 0, num_tokens, 0

fires for any pure-decode batch (max_query_len == 1) and returns
num_decodes = num_reqs, i.e. the *padded* request count under FULL-graph
replay. The fill therefore starts at num_decodes == batch_size and is a
no-op: the padded slots keep whatever block_table_tensor[:, 0] held, and
ATOM's request-indexed GDN decode kernel folds them into ssm_state. That
depends only on the decode batch being padded -- not on the model, its
quantization, or TP. It is the same reason Kimi-K3 carries the pass in
its own KDA builder.

Why #1985's reason does not hold. It removed the pass because it
"rewrote that metadata using num_actual_tokens (token-padded graph size)
and shrunk num_decodes". That was true of the original version
(2255001) and was fixed two versions earlier, in 441476e, which
switched to num_reqs with a comment saying exactly why.

Qwen3.5-397B-A17B-MXFP4 TP4, gsm8k 3-shot x1319, 4x gfx950, runner image
assembly, FULL_AND_PIECEWISE: 0.8423 (scoped variant measured 0.8544;
the two are within one stderr of each other, and both clear the 0.83
gate against 0.7817 before the fix).

The 35B-A3B block-FP8 path is now covered by the pass again. I could not
score it end to end -- this box has the checkpoint but no validated vLLM
plugin recipe for it, and my attempt floored at 0.05-0.07 under both
FULL and PIECEWISE, so it measured nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(vllm): keep the MTP position-0 mask off the layer signature

Every vLLM-plugin MTP run has been dying in profile_run since the fused
FP8 prologue landed:

    TypeError: DeepSeekMultiTokenPredictorLayer.forward() takes from 5 to
    6 positional arguments but 7 were given

The plugin installed its position-0 input-embedding mask by rebinding
each draft layer's forward to a wrapper that re-called the original with
a hardcoded five-argument list. When the fused prologue gave the layer a
sixth argument (eh_input_quant), the wrapper kept advertising the old
arity -- and functools.wraps kept the original qualname, so the error
pointed at a model signature that was in fact correct.

The wrapper could not have worked on the fused path anyway: there the
embedding never materializes as a tensor the layer receives, so there is
nothing for a layer-level wrapper to mask.

Move the mask into the predictors, which own the embedding lookup on
both paths and cannot drift from their own layers. On the unfused path
it is the same torch.where as before. On the fused path the masked rows
are fed the sentinel id -1: the prologue kernel already emits a zero
embedding for any id outside [0, vocab_size), and RMSNorm(0) is 0, so
the result is bit-identical to zeroing inputs_embeds -- without
materializing the [num_tokens, hidden] tensor the fusion exists to
avoid.

The plugin now flips a flag instead of rewriting a signature, and raises
if an arch in _MTP_MASK_INPUT_ARCH has no predictor to flip it on,
rather than silently serving an unmasked draft.

The regression test lives in tests/ rather than tests/plugin/ because
run_unit_tests.sh skips the latter.

Fixes the GLM-5.2-MXFP4-MTP TP4 plugin accuracy job.

* ci: download plugin CI draft models like the target

Kimi-K3 DSpark failed because speculative-config pointed at a missing local draft path. Cache the draft the same way as the target and rewrite extra_args to the resolved directory.

Co-authored-by: Cursor <cursoragent@cursor.com>

* ci: SIGKILL leftover host/KFD VRAM holders after plugin docker cleanup

Plugin HIP smoke was failing on spur runners when VRAM stayed allocated by host PIDs after leftover containers were already gone.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: perzhang <perzhang@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: PerryZhang01 <Perry.Zhang@amd.com>
Co-authored-by: Junyan Yang <junyyang@crs-m2m-cpu-spur-v2-009.us-east2-a.compute.internal>
Co-authored-by: Junyan Yang <junyyang@crs-m2m-cpu-spur-012.us-east2-a.compute.internal>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants