Skip to content

[MiniMax-M3] Route the paged decode to aiter's FlyDSL kernel with work planner - #2366

Merged
valarLip merged 13 commits into
mainfrom
flydsl-pa-decode-routing
Sep 23, 2026
Merged

valarLip merged 13 commits into
mainfrom
flydsl-pa-decode-routing

Conversation

@yitingw1

@yitingw1 yitingw1 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

1. Motivation

gluon's paged decode gives every request in a batch the same number of KV splits. On an agentic trace a decode batch is wildly uneven — measured over 30,400 steps at conc 32, the step-internal max/min context ratio is p50 2.49, p90 22.2, max 1610, and 31.2% of steps sit above 4× — so one long request owns the critical path while the short ones idle.

aiter #4332 (FlyDSL paged attention) plus #5546 (a Triton GPU planner that hands each request a partition count proportional to its real context length) address exactly that shape. Measured on MiniMax-M3 MXFP4 + EAGLE3, TP4, same machine as the current published curve:

conc p90 interactivity throughput/chip
1 −0.0% −0.7%
10 +4.0% +3.6%
15 +11.8% +1.0%
20 +20.5% +2.7%

The gain grows with concurrency because that is when the batch has enough unevenness to rebalance; at conc 1 there is nothing to rebalance, and the curve confirms it. Past conc 20 it narrows again — the planner's task budget is 2 × CU, so each request's share falls as 512/batch and there is less left to redistribute.

Both paths are opt-in and gluon stays the default: the FlyDSL kernel is not fully tested on our shapes yet, so it is something you turn on, not something you inherit.

2. Usage

Argument Required Description
ATOM_PA_FLYDSL=1 No (default 0) Route the paged decode to FlyDSL where its domain covers the call. Off by default.
ATOM_PA_FLYDSL_PLAN=1 No (default 1) Use #5546's work planner on the dense decode. Requires ATOM_PA_FLYDSL=1; with FlyDSL off no plan is built at all.
ATOM_PA_FLYDSL=1 python3 -m atom.entrypoints.openai_server \
    --model amd/MiniMax-M3-MXFP4 --tensor-parallel-size 4 ...

Verify from the server log: pa_decode -> flydsl[N seqs] with no -> gluon lines, and flydsl work plan: num_seqs=... max_partitions=256 appearing once per cudagraph capture size per rank and never during the run.

3. Design

Aspect gluon (default) ATOM_PA_FLYDSL=1 + ATOM_PA_FLYDSL_PLAN=1
Split count dense_decode_splits(), capped at 32 same not used — plan.max_partitions instead
Partition allocation uniform across the batch uniform per request, by real context length, under a workgroup budget
Ceiling 32 (C++ PS reduce limit) 32 256, aiter's own default, never overridden
Where decided per pa_decode call per call once per forward, in the metadata builder
When allocated — — cudagraph capture only; runtime refreshes in place
Call sites all all in FlyDSL's domain whoever passes a work_plan — dense only

Two gates in series. The env is the opt-in; _flydsl_pa_decode_num_seqs() then mirrors the kernel's own validation so an unsupported shape falls back to gluon instead of raising from inside aiter. Every clause is a hard reject on aiter's side, not a preference: fp8 compute and fp8 page-16 cache, block_size ∈ {16, 64, 128} taken from the cache's dim −2, gfx942/gfx950, partition size and count bounds, head_dim ∈ {64} ∪ {128…1024 step 128} and agreeing with the cache's own num_hgroups × 16, q dtype, output dtype and shape matching q, a contiguous head_dim axis on both q and output, q heads divisible by kv heads, matching k/v cache dtypes, int32 and contiguous block_tables/context_lens, and a row count that divides evenly and stays inside the batch. The and short-circuits, so with the env off the capability check never runs.

The planner is kept off the sparse call sites. M3 issues 63 pa_decode calls per step, 57 of them sparse. Sparse reads a fixed top-k window, so its contexts are near-uniform — the one shape where the planner is a measured net loss (0.56× on an all-equal batch, and a flat curve across every ceiling, so it is not a tuning problem). There is no flag: work_plan defaults to None, and only the dense site in attention_mha.py hands one in, matching the boundary dense_decode_splits already draws. The vLLM and SGLang bridges run under someone else's forward context and pass none.

Plan lifetime vs cudagraph. The plan depends on context_lens alone, which every layer of one forward shares, so it is built once per forward rather than per call. Decode replays captured graphs, so it must exist at capture time or the static path is what gets recorded — silently, with a benchmark that still looks reasonable.

 build_for_cudagraph_capture(bs)        ← plan MINTED here, outside the graph
        │  one entry per (num_seqs, kv_heads, device), never replaced:
        │  a single slot would free tensors an earlier graph already baked in
        ▼
 warmup forward (eager)                 op reads plan → planned path recorded
        ▼
 torch.cuda.graph(...)  capture         only the refresh kernel is captured
        │                               (no allocation, no D2H → capturable)
        ▼
 replay, every step                     prepare_decode / prepare_mtp_decode
                                        refresh in place from live context_lens

Minting happens only on that first line. aiter's planner takes batch as a tl.constexpr, so an unseen value costs a kernel specialization — 65–72 ms cold — and a plan that can never be freed, because some captured graph may have baked its pointers in. A runtime batch with no plan is one no graph will replay, so the static path is the right answer for it rather than a stall. The cost of that rule is that the planner is inert under --enforce-eager.

Rows are running_bs long, not scheduled_bs: the op derives its own batch as q.shape[0] // max_seqlen_q, and aiter validates reduce_info.shape == (num_seqs, 2) against exactly that. Every draft pass republishes its plan through workinfos — each pass advances context_lens, so reusing the target's plan would point the kernel at the wrong KV ranges, which is a wrong answer rather than a slow one.

max_partitions is left unset, at aiter's own default. Setting it from the static split count clamps every request alike and removes the planner's whole mechanism; this tree did that once. Clamping it to 64 was considered and rejected — see §5.

Device visibility. numa_utils and the mooncake connector now read ROCR_VISIBLE_DEVICES before HIP_VISIBLE_DEVICES. ROCR filters and renumbers the device table before HIP indexes the result, so a recipe setting both cuts the visible count twice; dropping HIP_VISIBLE_DEVICES then left the NUMA binding reading nothing and put all four ranks on one node, which deadlocks at allocate_kv_cache.

Integration: needs aiter 94dca7bc6 or later — that commit (#4332, merged 2026-09-22) brings both the FlyDSL paged decode and #5546's planner into main. On an older aiter the import guard keeps the default path working; with ATOM_PA_FLYDSL=1 it would fail at startup.

4. Test Plan

  • tests/test_paged_attention_dispatch.py — dispatch, capability gate and plan wiring, CPU only.
  • gsm8k, full 1319 questions, splits 32 vs 128, order 32/128/128/32 — the accuracy precondition for splitting past the C++ reduce cap.
  • Planner drift against an fp32 reference, on the planned path, swept over the ceiling the planner actually reaches.
  • End-to-end agentic trace, TP4 + EAGLE3, MXFP4, 3600 s per point, against the same machine's gluon curve.

5. Test Result

Unit: 127 passed (74 → 127). Each new guard was verified to go red on its own reversion, by injecting the defect and re-running — including the two the previous round's tests did not catch: a create=True moved onto a runtime path, and the head_dim whitelist deleted outright.

Accuracy (gsm8k, 5-shot / 20-shot): 32 → 0.9674 / 0.9659, 128 → 0.9636 / 0.9644, all far above the 0.93 gate. Paired McNemar p=0.441 / 0.831; 98% of questions agree question-by-question and the 2% that disagree split evenly both ways.

Planner drift at the real ceiling. The plan's capacity is min(batch × max_partitions, max(batch, 2 × CU / kv_heads)) = 512 here, so each request's share is about 512/batch and at batch 1 the realized split count equals max_partitions exactly. The trace's single-request contexts are p50 120k / p90 334k / p99 764k tokens — 470 to 2986 tiles — so low-concurrency steps really do run at the 256 ceiling, four times past anything the old bf16 round-trip argument measured. Swept over nine cells (those three contexts × batch 1/8/24, one shared fp32 reference per cell):

max relative deviation ‖Δ‖/‖ref‖
across all 36 arms 2.84 – 4.76% 3.62 – 3.80%
paired 8 → 256, per cell ±0.5pp, both signs flat

No growth with the split count anywhere, so the ceiling stays at aiter's default. The sweep also shows the ceiling only ever binds at batch 1: at batch 8 and 24 the 64 and 256 arms receive identical per-request counts, so clamping to 64 would change nothing above batch 2 while capping the one case the planner has the most to rebalance.

End-to-end, same machine, paired against the pre-FlyDSL gluon curve (p90 interactivity = 1/itl_p90, SemiAnalysis convention):

conc gluon this PR Δ tok/s/chip Δ ITL p90
1 297.5 297.4 −0.0% −0.7% 3.36 → 3.36
2 274.9 291.6 +6.1% −2.6% 3.64 → 3.43
10 206.6 214.9 +4.0% +3.6% 4.84 → 4.65
15 172.8 193.3 +11.8% +1.0% 5.79 → 5.17
20 133.0 160.3 +20.5% +2.7% 7.52 → 6.24

Completed-request counts match within 1.3% at every point, so the deltas are rates rather than a different request mix. Routing was asserted per point: gluon=0, and the plan built only during capture.

High concurrency, measured on a second node, so stated against the published curve rather than added to the paired table above:

conc tok/s/chip vs published p90 interactivity vs published
24 42,661 +3.7% 138.9 +17.9%
28 46,608 +3.4% 118.2 +16.1%
32 47,782 +3.4% 85.6 +9.7%
40 58,624 +12.2% 87.7 +30.2%

conc 40 runs with LMCache CPU offload on, as does the published curve's own conc-40 point; the other three are GPU-resident on both sides.

6. Known Limitations

  1. No aiter version pin. The code assumes aiter.ops.flydsl.pa_decode exists whenever ATOM_PA_FLYDSL=1. It landed in 94dca7bc6 (2026-09-22); nothing in this repo enforces that floor.
  2. The planner is inert under --enforce-eager. Plans are minted only during cudagraph capture, which never happens there. This is the deliberate cost of not specializing a planner kernel mid-serving.
  3. The conc 2 point is noisy. +6.1% interactivity with −2.6% throughput does not match the shape of the other points; two independent runs at that concurrency agree with each other and disagree with this one. Treat it as run-to-run variance.
  4. The drift sweep is nine cells, not a tail. The bound it replaces was set by the worst of 370 shapes. Nine cells covering the production context and batch range show no growth, which is not the same as showing the tail is gone.
  5. Not attributed at kernel level. Per-kernel traces at bs=1 and bs=4 show the planner as a wash: the attention tile gets 11–38% faster, but PS reduce gets ~14% dearer because planned partials are packed per task. The end-to-end gain must come from larger, more uneven batches, which those two traces do not cover.
  6. Widens beyond MiniMax-M3. With ATOM_PA_FLYDSL=1, any fp8 page-16 MHA decode takes FlyDSL, including the vLLM bridge. The SGLang bridge passes compute_type=torch.bfloat16, which the gate rejects, so it stays on gluon.

Submission Checklist

yitingw1 and others added 8 commits September 21, 2026 10:32
`PA_DENSE_SPLIT_MAX = 32` is not a tuned value. It is the smaller of two
unrelated bounds that happen to agree near there: production aiter does not
build the C++ PS reduce past 64 and has no working fallback under it, and
`temporary_output` is bf16, so each extra split adds a round trip through the
PS combine.

aiter #4332 removes the first bound. Deciding whether the second one still
binds needs the cap measured against the shipping value on the same tree,
which a constant cannot express. `dense_decode_splits` computes
`min(PA_DENSE_SPLIT_MAX, cdiv(128, running_bs))`, so the cap only binds at
`running_bs <= 2` -- the difference is invisible without an A/B at that shape.

Default stays 32 and behaviour is unchanged. Raising it on production aiter
still fails at engine init rather than at runtime.

`int(os.getenv(...) or 32)` rather than a getenv default: an exported-but-empty
variable would otherwise raise `int('')` at import and take the engine down
before it starts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n env

ATOM_PA_FLYDSL=1 sends the paged decode to aiter's FlyDSL implementation
(aiter PR #4332) where FlyDSL's domain covers the call, and to gluon
otherwise. A measurement switch, not a shipping default: the two take the
same arguments and compute the same thing, and the env exists to A/B them
without maintaining two ATOM trees. Default 0, so nothing changes until an
operator opts in on an aiter that carries #4332.

_flydsl_pa_decode_num_seqs mirrors the kernel's own validation -- alibi,
sinks, externally quantized FP8 queries, ps=False, a positive sliding window,
a non-256 tile, a partition count outside [1, 256], an unsupported head dim,
and any cache that is not [nb, Hkv, D//16, page, 16] at 1 byte per element are
all hard rejects in aiter, not preferences. Enabling the env therefore cannot
turn a working configuration into an exception.

The returned count is the substance. ATOM pads two axes independently:
context_lens to running_bs for CUDA-graph identity, with
[scheduled_bs:running_bs] zeroed, and q to running_tokens for the MoE
all_gather. forward_context.Context stores both "because the ratio is not
always max_seqlen_q" and asserts no rectangle between them. gluon absorbs the
mismatch by deriving its batch as q.shape[0] // query_length and letting the
surplus zero-length rows do no work; FlyDSL demands the rectangle and raises

    ValueError: query.shape[0] (12) must equal
                context_lengths.shape[0] * query_length (4 * 4)

on an ordinary step. Recovering the count the way gluon does and slicing the
five per-sequence arguments to it hands FlyDSL the same rectangle, dropping
exactly the zeroed tail. All slices are views on dim 0.

The gate is deliberately not restricted to max_seqlen_q == 1. The dense path
is where FlyDSL's headroom over gluon lives, it runs max_seqlen_q ==
num_spec + 1, and FlyDSL carries a query_length == 4 MTP4 grid split tuned for
exactly that shape.

ATOM_PA_FLYDSL_PLAN=1 additionally opts into aiter #5546's GPU work planner.
The plan and its scratch are allocated once and only refreshed: allocation is
illegal inside graph capture and a captured graph bakes in the pointers, and a
planned call does not use the caller's static buffers -- planned partials are
packed as [kv_heads, capacity, query_rows(, D)] against the static API's
[num_seqs, kv_heads, partitions, query_rows(, D)]. The planner refuses batches
past 4096 while M3's sparse call site folds query tokens into num_seqs and
reaches 32768 on a prefill-as-decode step, so those fall back to the static
path rather than raising.

Measured negative on this workload and defaulted off: against the same tree
with the planner off it costs 3.86% at conc 1, 11.50% at conc 10 and 18.28% at
conc 20. The cause is not batch uniformity -- 31% of decode steps have max/min
KV length >= 4 -- but the split count, which plan_pa_decode and pa_decode
share: dense_decode_splits gives 16 at running_bs 5 and 8 at running_bs 24,
and at those counts a batch skewed 2.5x and one skewed 120x both measure
1.00x. #5546's own conclusion is that the data "support using dynamic plans
for uneven KV work, not enabling them universally".

Both routes are logged once per shape signature. A run where the env is set
but every call still lands on gluon is otherwise indistinguishable from one
where FlyDSL simply did not help.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…and keep it off the sparse call sites

`_flydsl_work_plan` handed `plan_pa_decode` the static split count from
`dense_decode_splits`, overwriting that argument's own default. The two are
different quantities: `get_recommended_splits` gives every request the same
count and documents itself as "not a variable-work scheduler", while the plan's
`max_partitions` is a per-request ceiling it then divides under a workgroup
budget. Clamping the ceiling to the static count leaves a long request with the
same share as a short one, which is the planner's entire mechanism. Read back
from `plan.reduce_info`: a 5-request batch at ceiling 16 splits [16,4,4,4,4]
with the long request pinned at the ceiling, and at the default 256 it splits
[256,20,21,20,21].

`ATOM_PA_FLYDSL_PLAN_MAX` now carries that ceiling, 0 meaning "omit the
argument and take aiter's default". Do not read the ceiling off aiter's unit
test: it is the only caller of `plan_pa_decode` in that tree and it binds the
two deliberately, building the plan from the same count it hands `pa_decode` so
the paths must agree numerically. That makes it a correctness test which cannot
see the conflation.

The planner was also a global switch inside `run_pa_decode_gluon`, so enabling
it reached the two sparse call sites as well -- 57 of the 63 `pa_decode` calls
per step. Their context is a fixed topk window, so there is no unevenness to
rebalance and only the refresh and the task packing remain. On uniform batches
the planner is a loss that the ceiling cannot rescue: 0.56x at B8/257 and 0.84x
at B200/200000, flat from ceiling 4 through 256. `allow_work_plan` moves that
choice to the caller, where the split policy already lives, and only the dense
site opts in -- the same boundary `dense_decode_splits` already draws.

A plan build now logs its shape once per cached key, which is the only evidence
that the planner went where it was meant to rather than being assumed to.

Measured at conc 20 on the agentic trace (one operating point), single-user
output throughput 157.19 -> 178.49 tok/s and ITL 7.97 -> 6.96 ms, against 139.20
for the same arm before these changes. Prefix hit 96.56 vs 96.63, input length
+1.6%, request count +1.0%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ur env switches into one

The plan is a function of `context_lens` alone, so every layer of one forward
wants the same one. Building it inside the attention op re-ran the planner
kernel for each of M3's three dense layers, and the module-level cache that
existed to paper over that had to key itself on shape to stay correct. It now
lives where the rest of the per-forward metadata is built: `prepare_decode`
attaches it, `prepare_mtp_decode` refreshes it for each draft pass (their
context lengths differ by a token, so reusing the target's plan would point the
kernel at the wrong KV ranges -- wrong output, not merely slower), and the op
reads it off `get_forward_context().attn_metadata`.

Measured before the move: 6 planner kernels per step, and `envs.__getattr__` on
all 63 `pa_decode` calls rather than the 6 that could use a plan. After: 4 and
1. The env read is the larger of the two -- one `__getattr__` costs ~0.97 us and
63 of them are ~0.5% of a 12.8 ms step.

Scratch allocation stays in the op. It is sized from the query tensor the op is
holding and happens once per shape, so only the per-step work moved out.

The env surface collapses to `ATOM_PA_FLYDSL_PLAN`, now on by default:

  * ATOM_PA_FLYDSL is gone. It was a measurement switch for A/B-ing FlyDSL
    against gluon without two trees; FlyDSL is the decode path. The capability
    check that falls back to gluon for shapes outside FlyDSL's domain stays --
    it mirrors the kernel's own validation and is not a switch.
  * ATOM_PA_FLYDSL_PLAN_MAX is gone, and the call now omits `max_partitions`
    entirely so `plan_pa_decode` takes its own default. The one time this tree
    chose the ceiling it chose the static split count, which clamps every
    request to the same share and removes the planner's mechanism.
  * ATOM_PA_DENSE_SPLIT_MAX is gone and PA_DENSE_SPLIT_MAX is a constant again.
    With the planner on, a planned call is told `plan.max_partitions`, so the
    cap no longer sets the production partition count; it survives for the
    gluon fallback, where both halves of its bound still hold. Raising it would
    only enlarge the static scratch a planned call never reads.
  * ATOM_PA_FLYDSL_PLAN defaults to 1. On the agentic trace, SA-convention
    interactivity (1/itl_p90) is +24.6% at conc 20 and +8.4% at conc 10, and
    the gain runs monotonically from the slow tail to the fast one (p10 +24.6%,
    p50 +19.3%, p90 +5.0%) -- it lifts the floor, which is the end the metric
    watches.

The routing log no longer hangs off the deleted env, and is unconditional: with
FlyDSL as the default path, which route a call took is worth always recording.

tests/test_paged_attention_dispatch.py: 74 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prose cut, facts kept. What stays is the four things a reader would otherwise
re-derive the hard way: the ceiling must not come from the static split count,
each draft pass needs its own refresh (reusing the target's plan points the
kernel at the wrong KV ranges -- wrong output, not merely slower), scratch is
allocated in the op because its shapes come from the query tensor, and the
FlyDSL/gluon split is a capability check rather than a switch. The measured
numbers move to the one place they inform a decision: why the planner defaults
on.

Also fixes a contradiction left by the previous commit. `run_pa_decode_gluon`
still described itself as a "MEASUREMENT SWITCH, not a shipping default" after
FlyDSL had become the path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review found the previous commit's premise broken: decode runs from captured
graphs (cudagraph_mode=FULL), and `build_for_cudagraph_capture` never attached
a plan, so the op saw None at capture time and the STATIC path is what got
recorded. Every later refresh then worked on a graph that does not read it --
no error, no slowdown, just a planner that quietly does nothing and an
end-to-end number that looks fine.

The capture flow (model_runner.py) is what makes the fix possible:

    3857  build_capture(bs)           outside the graph -- first plan allocated
    3913  self.model(...)             warmup, eager; the op sees the plan
    3969  with torch.cuda.graph(...)  capture; the plan is already there

So the plan is attached in `build_for_cudagraph_capture` too, and the builder
keeps one per (batch, kv_heads, device) instead of a single slot. The single
slot was the more dangerous half: `build_capture` runs once per capture-ladder
size, so the second rung replaced the first rung's plan while the first rung's
graph had already baked in its pointers.

Two more from the same review:

  * The plan was built from `context_lens[:scheduled_bs]` while the op derives
    its own batch as `q.shape[0] // max_seqlen_q`, which is `running_bs`.
    aiter validates `reduce_info.shape == (num_seqs, 2)` and raises, so any
    step whose batch was not exactly on the ladder would have killed the
    worker. `context_lens` is already `running_bs` long with the tail zeroed.
  * `prepare_mtp_decode` dropped the refresh result and relied on the returned
    object being the same one. It now publishes through `workinfos`, which the
    caller splats into the metadata, so a None result clears a stale plan
    rather than leaving it in place.

`flydsl_plan_matches` is extracted rather than inlined so the guard is
testable: a plan built for another batch now falls back to the static path
instead of reaching aiter's validate.

tests/test_paged_attention_dispatch.py: 74 -> 83. Eight of the nine are
falsifiable by reverting the behaviour they describe, each reddening exactly
one test; the ninth is a source check on the capture attach, labelled as such,
because driving a real capture needs a model runner and what it guards fails
silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…y default

aiter #4332 is unmerged and the kernel is not fully tested, so gluon goes
back to being the shipping decode path and FlyDSL becomes the opt-in:
ATOM_PA_FLYDSL=1, off by default. The two gates run in series -- the env,
then the capability check that mirrors the kernel's own validation -- and
the `and` short-circuits, so with the env off the check does not run.
Shapes outside FlyDSL's domain still fall back to gluon either way.

The work planner (aiter #5546) only feeds the FlyDSL kernel, so it now
requires ATOM_PA_FLYDSL=1 as well. Without that, a plan would be built and
refreshed every step for a kernel that never reads it, and the "flydsl work
plan" log line -- what an A/B arm is checked with -- would show up on a run
that is entirely gluon.

Tests 84 -> 86: the default must be off, the dispatch must consult the env
before the capability check, and the builder must not plan with FlyDSL off.
Each of the three goes red on its own reversion.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The parameter now matches the env that controls it, ATOM_PA_FLYDSL_PLAN.
"work plan" is aiter's own term for the object, but on its own it does not
say the plan belongs to the FlyDSL path -- which matters now that gluon is
the default and FlyDSL is the opt-in.

Comments trimmed throughout: the _flydsl_pa_decode_num_seqs docstring keeps
why it returns a count rather than a bool (ATOM pads its sequence and row
axes independently, so the two disagree by a padding slot on an ordinary
step; gluon absorbs that, FlyDSL raises) and drops the shape algebra and the
verbatim traceback.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2366 --add-label <label>

@yitingw1 yitingw1 changed the title # [MiniMax-M3] Route the paged decode to aiter's FlyDSL kernel with work planner [MiniMax-M3] Route the paged decode to aiter's FlyDSL kernel with work planner Sep 23, 2026
@valarLip

Copy link
Copy Markdown
Collaborator

Review — FlyDSL pa_decode routing

Reviewed 0e1374302 in a detached worktree. The new tests pass here (113 passed, 0 skipped).

The routing design is the right shape — a narrow gate in front of an opt-in kernel, with a static fallback behind it — but two things need to be resolved before this can land, and they are of different kinds.

The first is blunt:

pa_decode() is called with a ps= keyword that aiter's FlyDSL kernel does not accept and does not absorb via **kwargs. Every routed call raises TypeError. No test exercises the real call, which is why the suite is green.

The second is structural:

The gate's docstring promises that everything aiter hard-rejects is rejected here first, but about eight of aiter's rejects are missing; and per-forward state that every sibling artefact keeps as an AttentionMetaData field now lives in a process-global dict and a thread-local context lookup instead. Together these turn several would-be fallbacks into crashes, and several would-be crashes into silence.

Provenance: [verified] means the deciding code was read in the PR-head worktree or the command was run there. [reported] means the shape matches the code but was not traced end to end.


1. The routed path cannot run [verified]

base_attention.py:433 passes ps=ps. inspect.signature(aiter.ops.flydsl.pa_decode.pa_decode) ends ... alibi_slopes, sinks, sliding_window=0, work_plan=None — no ps, no **kwargs — and the same holds at 94dca7bc67, the aiter floor this PR names.

With ATOM_PA_FLYDSL=1 (which the recipe now sets), the first dense decode layer that passes the gate dies during cudagraph warmup with TypeError: pa_decode() got an unexpected keyword argument 'ps': worker dead, server still answering /metrics. That is precisely the failure mode the gate exists to prevent.

The argument is also redundant — the gate already rejects not ps before reaching this call.

The reason this shipped green is worth fixing alongside it: every wiring test monkeypatches plan_pa_decode with raising=False, which creates the attribute rather than asserting it exists, and nothing pins aiter's pa_decode signature. An inspect.signature assertion against both symbols would have caught this and would catch the next aiter rename.

2. MTP draft attends over a stale plan, and flydsl_plan_matches accepts it [verified]

GDNAttentionMetadataBuilder.prepare_mtp_decode (gdn_attn.py:1518) and Qwen4ExpMetadataBuilder.prepare_mtp_decode (qwen4_exp_attn.py:627) override without calling super() and return dicts with no flydsl_work_plan key. Both classes do inherit prepare_decode, which sets it — so the draft pass runs on the target's plan.

eagle_proposer.py:746 bumps attn_metadata.context_lens[:running_bs] += 1 in place before each draft step, and aiter's planner bakes per-task begin/end tiles plus seq_ctx into the plan (pa_decode_plan.py, tl.store(work + slot*4 + 3, seq_ctx, active)). flydsl_plan_matches compares only reduce_info.shape[0] and num_kv_heads — neither changes — so the stale plan is accepted, not refused.

On a GDN/Qwen4-Exp hybrid with MTP, fp8 page-16 KV, ATOM_PA_FLYDSL=1 and an eager or piecewise-eager draft step, the draft attends over the pre-bump context and never sees the KV of the token it just drafted. Silent acceptance-rate and accuracy loss, no exception.

The AST guard at tests/test_paged_attention_dispatch.py:629 parses aiter_attention only, so these two overrides are invisible to it.

3. Per-forward state left the metadata dataclass

flydsl_work_plan is the first per-forward attention artefact to skip AttentionMetaData entirely — work_meta_data, work_indptr and reduce_indptr are all declared fields, and gdn_attn.py:106 declares flydsl_prefill_metadata the same way. Two consequences, both invisible today and both load-bearing under TBO:

The plan is fetched from the thread-local forward context rather than from the fwd_ctx the caller already holds (base_attention.py:364) [verified]. attention_mha.paged_attention_triton takes fwd_ctx: ForwardContext, reads attn_metadata = fwd_ctx.attn_metadata at line 560 and threads context_lens, block_tables and max_seqlen_q off it explicitly; the new code calls get_forward_context() and re-reads. atom/utils/forward_context.py resolves thread-local first and falls back to the module global, so a TBO worker thread that never installed its own context reads whatever the other ubatch thread wrote last. Since flydsl_plan_matches compares only batch and kv-head count, two equal-sized ubatches pass and the kernel is handed the other ubatch's work_info.

_FLYDSL_PLAN_SCRATCH is one process-global triple shared by every layer, model, builder and stream (base_attention.py:245) [verified]. The static path it replaces allocates exp_sums / max_logits / temporary_output fresh per call (attention_mha.py:587-596), so two in-flight pa_decode calls could never alias. The new key (num_kv_heads, capacity, rows, head_dim, out_dtype, device.index) is identical across every rung of the capture ladder, because capacity = min(batch*max_parts, max(batch, ceil(2*CU/kv_heads))) is a constant 512 for every batch under the workgroup budget at kv_heads=1. Under capture_tbo_graph ("threads + multi-stream captured in graph") two equal-shaped microbatches write the same buffers concurrently — wrong logits, no error, vanishing under HIP_LAUNCH_BLOCKING.

This is only accidentally safe today, because ubatch metadata carries no plan and falls back to per-call scratch. Nothing states the single-stream assumption and no test pins it. Putting the plan on AttentionMetaData like its siblings resolves both facets at once.

4. The dummy-run short-circuit was dropped 30 lines from where it is applied [verified]

aiter_attention.py:892-908 passes running_bs == 0 or get_forward_context().context.is_dummy_run to _mtp_prepare_decode_metadata_kernel as a no-op flag, with the comment: "Dummy runs skip the draft attention, so keep this launch as a no-op: their synthetic context_lens can point past block_tables."

refresh_flydsl_plan(context_lens[:running_bs]) at line 932 then runs unconditionally over that same buffer. Every DP-padding or dummy draft step pays a planner launch it can never use, and because _flydsl_plans hands back the one plan object whose tensors the real target's captured graph baked pointers into, the dummy step rewrites the live plan's work_info from synthetic lengths. Correctness rests entirely on the next real step refreshing first; nothing states or tests that ordering.

5. The gate is narrower than its docstring claims [verified]

_flydsl_pa_decode_num_seqs (base_attention.py:188) says "Every clause below is a hard reject there." Missing versus aiter's pa_decode: num_hgroups != head_dim // 16 (pa_decode.py:337), block_tables.shape[0] != num_seqs (:313), output.shape != query.shape (:308), block_tables.dtype != int32 (:400), context_lengths.dtype != int32 (:403), query.dtype not in (bf16, f16) (:411), output.dtype != query.dtype (:414), num_q_heads % num_kv_heads != 0 (:379), query.stride(2) != 1 (:429), value_cache.dtype != expected_fp8 (:426).

The int32 one is worth calling out on its own: refresh_flydsl_plan does guard context_lens.dtype is not torch.int32. A model handing int64 lengths therefore gets work_plan=None and still routes into FlyDSL via the static path, where aiter raises TypeError: context_lengths must be int32 on the first decode step. The one dtype the author knew about is guarded on the path that survives it and unguarded on the path that cannot.

6. The measured bf16 split clamp is bypassed, and the comment retiring it cites no measurement [verified]

The original comment at base_attention.py:64 records two separate bounds behind PA_DENSE_SPLIT_MAX = 32: the C++ PS-reduce build limit (gluon-only, legitimately inapplicable to the planner) and a numerics bound — "temporary_output is bf16 ... the worst shape measured drifts 20pp further from an fp32 reference at 64 than at 8."

_flydsl_plan_scratch allocates temporary_output in output.dtype, i.e. bf16, so the round-trip is identical. plan_pa_decode defaults max_partitions to the device CU count (256 on MI355X) and the planner clamps per-sequence counts only at MAX_PARTS, so a long request in an uneven batch gets up to 256 splits — 8x past the measured-safe ceiling. Known Limitation #4 concedes the gsm8k arm tested 128, not 256.

Either re-measure the drift at the planner's actual ceiling, or pass max_partitions — which §7 wants anyway.

7. Taking aiter's max_partitions default disables its bounded and vectorized reduce, and a test pins that choice [verified]

int(work_plan.max_partitions) reaches aiter as max_context_partition_num → context_partition_num in launch_pa_decode_ps_reduce. There, bounded_plan_logits requires context_partition_num <= 64, and vectorize_plan_logits is gated on it (pa_decode.py:116-137). At the CU-count default of 256 both are permanently off, so every planned call compiles the unbounded, unvectorized reducer.

That directly explains this PR's own Known Limitation #5 ("PS reduce gets ~14% dearer"). And aiter's comment on the flag is a safety statement — "so out-of-bounds reads cannot wrap back into valid data" — not only a speed one.

test_ceiling_is_left_at_the_aiter_default asserts "max_partitions" not in seen, so capping at 64 goes red and reads as a regression. The test conflates "do not feed the static split count in" with "never set the ceiling"; those are different claims and only the first one is intended.

8. The plan cache is unbounded and the planner specializes on batch [verified, measured here]

_flydsl_plans (aiter_attention.py:821) is never evicted, and aiter's planner kernel takes batch as a tl.constexpr. Measured on this box: a first plan_pa_decode at a new batch value costs 65-72 ms cold (1.2-1.7 ms with a warm triton disk cache); steady-state refresh is 16 µs.

Under --enforce-eager, piecewise skips, above-ladder batches or non-unified DP steps, n tracks the real batch, so each new value mints a plan (never freed: GPU int32 capacity x 4 plus n x 2), one logger.info, and one kernel specialization, up to the 4096 cap. This is the per-shape-JIT-leaking-into-serving failure that base_attention.py:89-94 already guards against in this same file ("the PS reduce compiles one variant per distinct count"); the planner reintroduces it. Startup also pays it per capture rung — roughly 2.6 s over a 20-rung ladder x 2 q-buckets on a cold cache.

9. The feature-off path is no longer free [verified, measured here]

base_attention.py:341 sits outside if flydsl_seqs:. envs.<bool> costs 0.69 µs per access because atom/utils/envs.py:824 re-runs the lambda — and os.getenv — on every attribute access with no cache; the route_sig tuple plus set lookup costs another 0.12 µs.

At 63 pa_decode calls per step that is ~51 µs of CPU per decode step on every deployment that never enables this feature — about 1% of a 5 ms interactivity-bound step, and not elided by graph replay since attention is a piecewise split op. With the env on it is ~92 µs/step. Hoisting the env read to a module constant and moving the route log inside the guard fixes both.

Separately, paged_attention_triton still allocates its three torch.empty scratch tensors unconditionally and the planned branch then discards them — ~3 MB/step of pure allocator churn.

10. Three builders bake the static path into their graphs [reported]

build_for_cudagraph_capture in gdn_attn.py:1551, qwen4_exp_attn.py:673 and _build_ubatch_decode_metadata never attach a plan. The route decision is Python executed at capture time, so with flydsl_work_plan absent, getattr(md, "flydsl_work_plan", None) returns None and the static branch is what gets recorded. Every later refresh_flydsl_plan is then a 16 µs GPU kernel feeding a graph that never reads it.

The "plan refused" warning cannot fire here either, because work_plan is None skips it. So on GDN/Qwen4-Exp/TBO an A/B of ATOM_PA_FLYDSL_PLAN=0/1 measures pure overhead and reads as "the planner does not help" — exactly the ambiguity the routing log was added to remove.

11. Deleting HIP_VISIBLE_DEVICES from the recipe is correct, and it silently breaks NUMA binding [verified, measured here]

The deletion is a real fix: ROCR_VISIBLE_DEVICES=0,1,4,5 HIP_VISIBLE_DEVICES=0,1,4,5 yields torch.cuda.device_count() == 2 on this box (ROCR filters first, then HIP indexes into the filtered set and 4/5 are out of range), versus 4 with ROCR alone. But it arrives unexplained and unrelated to FlyDSL, and it has a cost.

numa_utils.py:54 reads os.environ.get("HIP_VISIBLE_DEVICES") or os.environ.get("CUDA_VISIBLE_DEVICES") and never consults ROCR_VISIBLE_DEVICES. With HIP removed, visible is empty, _physical_index falls back to identity, and logical ranks 2/3 map to physical GPUs 2/3 instead of 4/5. Physical cards sorted by BDF split [0,0,0,0,1,1,1,1] across nodes, ATOM_AUTO_NUMA_BIND defaults to 1, and the recipe sets no ATOM_NUMA_NODE — so all four ranks land on node 0, the condition this same recipe documents as fatal ("timed out on the allocate_kv_cache barrier after 600 s").

Fix-then-sweep: mooncake_connector.py:641 has the identical HIP-or-CUDA chain.

12. A test asserts the gate accepts a call aiter rejects [verified]

test_head_dim_stays_a_subset_of_aiters[256-True] (tests/test_paged_attention_dispatch.py:845) asserts self._call(q=torch.empty(16, 16, 256, ...)) == 4 while _call's baseline k_cache stays (4, 1, 8, 128, 16) — num_hgroups=8, which encodes head_dim 128. aiter rejects exactly that at pa_decode.py:337.

The docstring presents this acceptance as the contract ("Narrower than aiter on purpose; widening is what breaks"), so the missing k_cache.shape[2] * 16 == q.shape[-1] clause will not be added, and any deployment where those disagree routes into FlyDSL and dies on a shape error inside aiter.

13. Conventions

Smaller, but several are repo rules rather than preferences.

run_pa_decode_gluon now dispatches two kernels under a gluon-specific name. ATOM/CLAUDE.md Name-matches-function is mandatory — "When behavior changes, rename immediately". The PR edited the docstring from "Run the AITER paged-attention Gluon decode kernel" to drop the word rather than renaming, leaving four call sites — including the vLLM and SGLang bridges — importing a symbol named gluon with no signal that ATOM_PA_FLYDSL reroutes them.

Neither new env var is documented. grep FLYDSL docs/ finds neither ATOM_PA_FLYDSL nor ATOM_PA_FLYDSL_PLAN among the 96 documented ATOM_* vars, and docs/model_ops_guide.md:152 still states the decode kernel is torch.ops.aiter.pa_decode_gluon.

The comment at line 160 says the mirrored limits are "pinned against its source by a test", and they are not. TestFlyDSLConstantsMatchAiter pins only the block-size whitelist and the arch tuple; _FLYDSL_PA_MAX_PARTITIONS, _FLYDSL_PA_TILE and _FLYDSL_PLAN_MAX_BATCH are unpinned.

Five new assertions test source text rather than behaviour (inspect.getsource / ast.parse at tests lines 494, 508, 600, 686, 611-668). A test that greps the code it tests passes for reasons unrelated to what the code does. test_scratch_is_keyed_by_capacity also leaves ~25 MB of live CUDA buffers in the never-evicted module global with no teardown.

envs.py:525 says "+18.6% interactivity at conc 20"; the recipe says "+20.5%" for the same measurement.


Claims checked and not upheld

Recorded so they are not re-raised:

  • The planner is not inert. running_tokens == running_bs * max_seqlen_q holds by construction (forward_context.py:370-378), so flydsl_plan_matches passes on every decode step in the shipped TP4/no-DP recipe. The gate docstring's stated reason for the num_seqs design is factually wrong, but the design works.
  • monkeypatch.setattr(envs, ...) does not leak — the pre-existing autouse keep_envs_lazy fixture cleans it.
  • Zero-length padded rows come back as 0.0, not uninitialized memory.

…binding

Two gates in series guard the FlyDSL paged decode: ATOM_PA_FLYDSL is the
opt-in, and _flydsl_pa_decode_num_seqs mirrors aiter's own rejects so an
unsupported call falls back to gluon instead of raising inside the kernel
and killing the worker. The gate was missing several of those rejects --
q/output dtype and shape agreement, both head_dim stride checks, the
q-heads-over-kv-heads divisor, the v_cache dtype, and contiguity of the
two caches and the two per-sequence arrays.

Plans are now only ever minted during cudagraph capture. aiter's planner
takes batch as a tl.constexpr, so a runtime batch that no graph replays
would cost a cold kernel specialization and a plan nothing can free; the
static path is the right answer for it. Runtime only refreshes in place.
The one-shot "no plan for this batch" notice waits until a capture has
happened, since before that every call legitimately lands there and the
notice would be spent on a step that says nothing.

max_partitions stays unset, at plan_pa_decode's own default: every number
measured on this branch was measured there, and the review's suggestion to
clamp it to 64 is a perf claim nothing here backs. Measuring instead, on
the planned path, across nine cells spanning the trace's p50/p90/p99
contexts at batch 1/8/24: drift from an fp32 reference does not grow with
the split count anywhere, and the ceiling only ever binds at batch 1. The
plan's capacity is min(batch * max_partitions, max(batch, 2 * CU /
kv_heads)) = 512 here, so each sequence's share is about 512/batch and a
64 ceiling would have changed nothing above batch 2 while capping the one
place the planner has the most to rebalance.

numa_utils and the mooncake connector now read ROCR_VISIBLE_DEVICES first.
ROCR filters and renumbers before HIP indexes the result, so setting both
cut the visible device count twice; dropping HIP from the recipe left the
NUMA binding reading nothing and put all four ranks on one node.

Tests go from 86 to 127. The call-site contract is now checked by walking
the AST of all three builders and comparing the value of `create`, not its
presence in one file -- a sed that moved `create=True` onto a runtime path
had left every test green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@valarLip
valarLip merged commit 94cde4b into main Sep 23, 2026
74 of 80 checks passed
@valarLip
valarLip deleted the flydsl-pa-decode-routing branch September 23, 2026 12:01
yhl-amd added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Sep 26, 2026
Main migrated single-node AgentX to native srt-slurm (#3428) and deleted
benchmarks/single_node/agentic (#3461). Accept the script deletion and move
this PR's change into the declarative recipe: bump the engine image to
rocm/atom-dev:nightly_202609231248 (ROCm/ATOM#2366) and set ATOM_PA_FLYDSL=1
and ATOM_PA_FLYDSL_PLAN=1. The bash-only GPU-mask fix no longer applies.
The changelog entry follows the default eval policy.

中文:main 已将单节点 AgentX 迁移到 srt-slurm 并删除旧 bash 脚本。本 PR 改为在
YAML 配方中更新镜像至 nightly_202609231248 并启用 ATOM_PA_FLYDSL /
ATOM_PA_FLYDSL_PLAN;GPU mask 修复随旧脚本一起移除,changelog 采用默认 eval 策略。

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
yhl-amd added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Sep 28, 2026
Move the MiniMax-M3 MI355X ATOM AgentX srt-slurm recipe to
rocm/atom-dev:nightly_202609231248, which contains ROCm/ATOM#2366, and set
ATOM_PA_FLYDSL=1 and ATOM_PA_FLYDSL_PLAN=1. Drop TP4 C32 to bound the sweep.

中文:将 MiniMax-M3 MI355X ATOM AgentX srt-slurm 配方镜像更新为包含
ROCm/ATOM#2366 的 rocm/atom-dev:nightly_202609231248,并设置
ATOM_PA_FLYDSL=1 与 ATOM_PA_FLYDSL_PLAN=1;移除 TP4 C32 以控制 sweep 规模。

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
yitingw1 added a commit that referenced this pull request Oct 8, 2026
…plans

aiter #5809 fixes a plan's capacity from workgroup_budget when it is built,
and target and draft tune to different budgets (B16: 512 vs 256). Since
#2366 both share one capture plan per (batch, kv_heads), refreshed in turn,
so one of them could never get its budget; and a budget hit with a
different capacity made the op mint a second plan inside the captured graph.

- Key capture plans by (batch, kv_heads, budget, device). Equal budgets keep
  the single shared object of #2366.
- Builders tag attn_metadata with a role: target from prepare_decode and the
  capture builder, draft from prepare_mtp_decode.
- During the eager warmup before capture, the op resolves the budget once
  per (role, batch) from the real tensors; on a capacity mismatch it asks the
  builder to adopt the tuned plan and writes it back to the metadata, so the
  capture records it. During capture and once decided, the op uses the plan
  as given and never looks up a budget or builds another plan.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
yitingw1 added a commit that referenced this pull request Oct 9, 2026
…get plans

The latest aiter #5809 removed pa_decode_tuning (get_cached_budget,
make_shape, storage_key) and ships no tuned budgets, so importing it fails at
the first FlyDSL decode.

- Build every explicit plan with aiter's default workgroup budget (2*CU).
  Measured on the agentic context distribution, the default was the fastest
  budget at every dense and draft batch checked.
- Drop the per-role budget plans (role tags, adopt_flydsl_budget, plan_owner):
  target and EAGLE draft share one capture plan again, as in #2366.
- Keep the once-per-step sparse refresh, the 4096-row gluon fallback and the
  FlyDSL-only context bound.
- Tests: an explicit plan needs no tuning module and passes no budget (red if
  the old import comes back); the sparse refresh tests move into
  TestExplicitPlan.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
yitingw1 added a commit that referenced this pull request Oct 10, 2026
…plans

aiter #5809 fixes a plan's capacity from workgroup_budget when it is built,
and target and draft tune to different budgets (B16: 512 vs 256). Since
#2366 both share one capture plan per (batch, kv_heads), refreshed in turn,
so one of them could never get its budget; and a budget hit with a
different capacity made the op mint a second plan inside the captured graph.

- Key capture plans by (batch, kv_heads, budget, device). Equal budgets keep
  the single shared object of #2366.
- Builders tag attn_metadata with a role: target from prepare_decode and the
  capture builder, draft from prepare_mtp_decode.
- During the eager warmup before capture, the op resolves the budget once
  per (role, batch) from the real tensors; on a capacity mismatch it asks the
  builder to adopt the tuned plan and writes it back to the metadata, so the
  capture records it. During capture and once decided, the op uses the plan
  as given and never looks up a budget or builds another plan.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
yitingw1 added a commit that referenced this pull request Oct 10, 2026
…get plans

The latest aiter #5809 removed pa_decode_tuning (get_cached_budget,
make_shape, storage_key) and ships no tuned budgets, so importing it fails at
the first FlyDSL decode.

- Build every explicit plan with aiter's default workgroup budget (2*CU).
  Measured on the agentic context distribution, the default was the fastest
  budget at every dense and draft batch checked.
- Drop the per-role budget plans (role tags, adopt_flydsl_budget, plan_owner):
  target and EAGLE draft share one capture plan again, as in #2366.
- Keep the once-per-step sparse refresh, the 4096-row gluon fallback and the
  FlyDSL-only context bound.
- Tests: an explicit plan needs no tuning module and passes no budget (red if
  the old import comes back); the sparse refresh tests move into
  TestExplicitPlan.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
valarLip pushed a commit that referenced this pull request Oct 12, 2026
…t work plans) (#2514)

* fix(attention): adapt FlyDSL PA decode to aiter 5809

* fix(attention): give target and draft their own tuned FlyDSL capture plans

aiter #5809 fixes a plan's capacity from workgroup_budget when it is built,
and target and draft tune to different budgets (B16: 512 vs 256). Since
#2366 both share one capture plan per (batch, kv_heads), refreshed in turn,
so one of them could never get its budget; and a budget hit with a
different capacity made the op mint a second plan inside the captured graph.

- Key capture plans by (batch, kv_heads, budget, device). Equal budgets keep
  the single shared object of #2366.
- Builders tag attn_metadata with a role: target from prepare_decode and the
  capture builder, draft from prepare_mtp_decode.
- During the eager warmup before capture, the op resolves the budget once
  per (role, batch) from the real tensors; on a capacity mismatch it asks the
  builder to adopt the tuned plan and writes it back to the metadata, so the
  capture records it. During capture and once decided, the op uses the plan
  as given and never looks up a budget or builds another plan.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(attention): refresh the sparse FlyDSL plan once per step; review fixes

- MiniMax-M3 sparse decode: every sparse layer of a step selects the same
  per-row lengths (0 differences over 9,000 C15 steps x 56 layer pairs), so
  the shared runtime plan is refreshed by the first sparse layer of the
  forward only -- one planner launch per step instead of 57, in eager and in
  the captured graph. Opt-in per call site (refresh_once_per_forward); dense
  eager and plugin callers still refresh every call.
- Route calls with more rows than aiter's planner accepts (4096) to gluon:
  every FlyDSL call needs a plan now, and a long sparse prefill folded into
  rows used to raise inside the worker.
- Resolve the FlyDSL context bound only on the FlyDSL path, so a deployment
  with FlyDSL off pays nothing and gluon gains no new way to raise.
- GDN capture builder and Qwen4 draft metadata now carry the plan role and
  builder, so capture never looks up a budget there either.
- Skip the budget tests when aiter lacks pa_decode_tuning instead of erroring.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(flydsl): state the aiter #5809 requirement and the new PLAN=0 meaning

This branch needs aiter with #5809 and is merged after it; nothing probes
for it at runtime, so the docs now say ATOM and aiter must be upgraded
together. Also correct what ATOM_PA_FLYDSL_PLAN does since #5809: there is
no static FlyDSL path, PLAN=0 routes to gluon, dense plans take their budget
from the offline tuning cache, and the sparse sites use runtime plans.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(attention): use aiter's default FlyDSL budget; drop the tuned-budget plans

The latest aiter #5809 removed pa_decode_tuning (get_cached_budget,
make_shape, storage_key) and ships no tuned budgets, so importing it fails at
the first FlyDSL decode.

- Build every explicit plan with aiter's default workgroup budget (2*CU).
  Measured on the agentic context distribution, the default was the fastest
  budget at every dense and draft batch checked.
- Drop the per-role budget plans (role tags, adopt_flydsl_budget, plan_owner):
  target and EAGLE draft share one capture plan again, as in #2366.
- Keep the once-per-step sparse refresh, the 4096-row gluon fallback and the
  FlyDSL-only context bound.
- Tests: an explicit plan needs no tuning module and passes no budget (red if
  the old import comes back); the sparse refresh tests move into
  TestExplicitPlan.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(attention): refresh the sparse FlyDSL plan per step, inside capture too

Review fixes on top of the default-budget adaptation:

- Sparse decode refreshed its plan once per forward, marked on ATOM's
  ForwardContext. Capture runs the warmup and the capture forward on one
  context and stream, so the capture skipped the refresh and every graph
  replay ran on the warmup's dummy lengths. The vLLM plugin never sets that
  context for MiniMax-M3, so there the mark outlived the step and eager steps
  stopped refreshing after the first. The mark now lives on the step's decode
  metadata (plan_step_owner, passed by the native and vLLM sparse decode),
  keyed by capture state; without an owner every call refreshes. Step-scoped
  and per-call plans no longer share a cache entry.
- Dense runtime plans take aiter's default partition ceiling, like the
  capture-owned plans. Sparse and the vLLM bridge keep their static split
  count (plan_partition_cap).
- A context bound below the query window routes to gluon instead of raising.
- Pass the clamped bound to aiter; drop the dead no-plan scratch branch; fix
  stale comments and the PLAN env docs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* refactor(attention): drop ATOM_PA_FLYDSL_PLAN

aiter #5809 left FlyDSL no planner-less path, so ATOM_PA_FLYDSL=1
ATOM_PA_FLYDSL_PLAN=0 could only route to gluon -- exactly what
ATOM_PA_FLYDSL=0 does. Gate the builder and the op on ATOM_PA_FLYDSL alone,
remove the env and its docs row (folding the plan behaviour into the
ATOM_PA_FLYDSL row), and warn once at startup when the old variable is still
set, so a leftover PLAN=0 does not silently change meaning.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* style: black and ruff fixes for the FlyDSL plan changes

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs: trim FlyDSL plan comments and the ATOM_PA_FLYDSL docs row

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs: final review nits for the FlyDSL plan changes

Drop the stale "static path" note in the GDN capture builder, scope the
builder-minting test docstring to the builder, inline the leftover
flydsl_enabled alias, and say in the ATOM_PA_FLYDSL row which callers use
capture-owned vs runtime plans and which plan ceiling each takes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(attention): address review on the FlyDSL runtime plans

- Key runtime plans by TBO ubatch id too: ubatches share one compute
  stream, so two equal-sized ubatches could refresh one step-scoped plan
  with each other's lengths.
- Refresh the sparse plan once per step only when sparse_local_block >= 1
  (tail block forced, so every sparse layer has the same lengths);
  otherwise refresh on every call. Native and vLLM sites.
- Probe once for aiter #5809's FlyDSL API (required keyword-only work_plan
  and max_context_length); an older aiter now falls back to gluon with a
  warning instead of raising on the first decode.
- Stop passing max_seqlen_k as the dense scheduling bound: it differs
  between graph and eager and selects the kernel specialization. Both now
  use the block-table bound.
- Drop the dead sliding-window handling from runtime plans (the gate
  rejects windows), say "sequences" for the 4096 planner limit, fix stale
  comments, and note in the recipe which configuration its FlyDSL-vs-gluon
  numbers were measured on.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
gyohuangxin added a commit that referenced this pull request Oct 20, 2026
* feat: integrate gfx1250 FP4 MQA indexer (#2334)

* feat: integrate gfx1250 fp4 MQA indexer

* perf: use qlen1 FP4 MQA for decode

* style: format gfx1250 FP4 pool layout

* test: skip OPUS metadata tests without AITER

* Add Kimi-K3 (hybrid) LMCache KV offload to the vLLM plugin path (#2268)

* Add Kimi-K3 (hybrid) LMCache KV offload to the vLLM plugin path

AtomLMCacheOffloadConnector could not serve Kimi-K3. Two guards rejected it at
startup, and behind them sat a correctness problem: K3 is hybrid, so vLLM builds
two KV cache groups (MLA full attention plus KDA recurrent state), and a
restored MLA prefix is only valid if the KDA state at the same token boundary is
restored with it. Half a restore is not a crash and not a log line -- it is
wrong output.

The two groups are moved by two different mechanisms, and the asymmetry is
forced, not chosen. The MLA group keeps the existing DenseKVByteCodec path
untouched. The KDA group cannot be read positionally at all: in
--mamba-cache-mode align a mamba block table is not append-only (a superseded
state block is freed and nulled, speculative blocks relocate in place), so
indexing it by token // block_size can persist a null, freed, or live
speculative block under a valid prefix hash. The only safe source is vLLM's
explicit hand-off, which names the exact block holding a committed boundary
state. The pinned vLLM 0.28 ships it flat on the scheduler output as
SchedulerOutput.partial_tail_offloads, fed by
KVCacheManager.take_partial_tail_offloads; 0.29 moves the same
{req_id: [(group_id, block_id, boundary_tokens)]} payload under
kv_connector_block_state.boundary_state_offloads. step_boundary_offloads reads
whichever spelling the running vLLM provides, so the connector keeps working
across that upgrade rather than going silently hit-less on one side of it. The
hand-off is taken off the KV cache manager once, while the step is built, so it
is consumed in the step it arrives. Each accepted boundary is pinned via the
bound GPU block pool and released on rank quorum.

Correctness of the pair is enforced on lookup, not on save: the reported
external hit is capped at the largest chunk boundary the state index still
claims, so a state that was never stored or was evicted shortens a prefix
instead of corrupting it. That removes the need for any cross-group save
parking. A KDA miss additionally invalidates the attention blocks the dense leg
filled, so vLLM cannot cache an MLA prefix whose state never arrived.

Hybrid models also require vLLM's hybrid memory allocator, which vLLM
auto-disables for a connector that does not declare SupportsHMA -- without that
declaration K3 plus --kv-transfer-config does not mis-save, it does not boot.
The connector now declares it and implements request_finished_all_groups, which
retires request_finished for every model. SupportsHMA, bind_gpu_block_pool and
request_finished_all_groups are all present in the pinned 0.28
(atom/plugin/vllm/vllm-version.env, 2cf0a6915 = v0.28.0), so nothing here waits
on the 0.29 upgrade.

supports_divergent_local_hybrid_hits stays False: the reconciled cross-group
boundary vLLM then reports is the conservative one, and this connector caps the
external hit on its own.

Stays on the pinned LMCache version. The 0.5.5 upgrade was permitted, not
required, and nothing here needs it; measured on the GLM-5.2 arc, 0.5.5rc4 lost
to 0.4.5 by 5.72%, with ~97% of chunk cost host-side.

Verification: the CI unit scope (--ignore=tests/plugin) is 3 failed / 5519
passed with the 38 new tests included, and those three failures reproduce
unchanged on origin/main in a clean worktree, so none of them is this change.
Seven further modules (tests/diffusion, tests/test_attn_family.py) cannot be
collected on this host at all -- Python 3.10 has no typing.Self or enum.StrEnum
-- which is the interpreter, not the change; CI runs a newer one. The tests were
mutation-checked: ten real degradations of the earlier work (uncapped hit, each
collect_stores filter removed, quorum of one, indexing despite a failed rank,
boundary_hash off-by-one, dropping an unresolvable load, silently dropping an
unmapped layer) each turn them red, and so does dropping either spelling from
step_boundary_offloads, while reordering the two spellings -- an equivalent
mutation -- leaves them green. black and ruff pass on every touched file.
Nothing has been run on hardware yet: boot, two-pass accuracy against the
0.9507 / 0.9500 baseline, and hit rate are still outstanding, and the recipe
says so rather than implying measured numbers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Fix three K3 offload defects found on the first hardware boot

The connector had never been run against a real engine. Three defects showed
up, two of them loud and one that would not have shown up at all.

1. Multi-group recurrent state. vLLM does not build one KV cache group per
   layer family; it builds equal-sized groups. K3's attention bucket is 29
   layers and its KDA bucket is 69, so the group size is 29 and the recurrent
   state lands in THREE groups (23 layers each), not one. The recurrent leg now
   reads the grouping off the engine instead of assuming a single group, and
   treats all mamba groups as one logical state: one token boundary, one block
   id per group, stored as those groups' bytes concatenated in group order.
   None of the groups is restorable alone.

2. `split_kv_caches_by_group` returns a list indexed by group id, and
   `_init_kda_tier` called `.get()` on it, so every worker died with
   `'list' object has no attribute 'get'`. Extracted `gather_group_tensors`,
   which is what the two functions were missing: nothing in the suite fed one
   to the other, so the seam had no coverage at all. It has four tests now,
   including the ordering contract -- vLLM's own `layer_names` order, never
   `sorted()` -- because a page image is readable only by a run that rebuilds
   the same order.

3. The dense leg strided every segment by a token, not by a block. This one is
   silent. `DenseKVByteCodec` derives each segment's per-block byte stride as
   `numel // num_blocks`, and `register_kv_caches` took `num_blocks` from the
   registered tensor's leading dimension. That is the block count only when the
   attention backend's kernel block size equals the group's block size. ATOM's
   MLA backend declares `get_supported_kernel_block_sizes() == [1]`, so vLLM
   allocates one row per token and expands manager block id `b` into kernel ids
   `b*n .. b*n+n-1`, while the ids a KV connector receives stay manager ids. On
   K3 -- 1536-token blocks -- that made the stride 1536x too fine: every block
   id still in range, every transfer reported success, bytes restored from the
   wrong rows, nothing logged. `DenseKVByteCodec`'s own docstring says the
   leading dimension is not the block count on token-major MLA.

   `resolve_block_count` now takes the count from `KVCacheConfig.num_blocks`
   and reconciles it against the tensor: the leading dimension must be a whole
   number of blocks, and the implied kernel blocks per block must divide the
   block size. It is an identity wherever the two already agree, which is every
   model shipped on this path so far (GLM-5.2 measured `num_blocks=8192`
   against `tail_shape=(64, 576)`), so the mismatch is K3's alone.

   No reshape is needed: a manager block's kernel blocks are contiguous and
   consecutive, so its bytes are one unbroken run either way.

The recurrent leg bounds-checks a block id against its own tensors and so would
have failed loudly on the same mistake. The dense leg has no such check, which
is why (3) needed a test rather than a boot.

Boot verified on 8xMI355 TP8: three mamba groups on 0,1,2 and attention on 3,
hybrid memory allocator left on, deterministic smoke answer correct.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Record the validated K3 boot, and make its log readable

The boot passed on 8xMI355 TP8, so the recipe's Status section is now a
measurement rather than a requirement. Three things it said were wrong, and
each one was wrong in the direction that reads as "fine":

- "vLLM builds two KV cache groups". It builds equal-sized groups, so K3's 29
  MLA and 69 KDA layers come out as four: one attention group of 29 and three
  mamba groups of 23. The recipe's own verification greps were written against
  the two-group picture.
- "LMCACHE_CHUNK_SIZE=128 # must equal --block-size". The block size is not the
  one you pass. vLLM raises it to 1536 so the attention page is at least the
  mamba page, and `--block-size 128` does not prevent that -- it fixes only the
  requested size, and the hybrid alignment step overrides it afterwards. A
  chunk of 128 is not a multiple of 1536 and the connector refuses to start.
- The "did the recurrent leg come up" grep named one line and said it appears
  once per worker and once in the EngineCore. There are two different lines
  from two different processes: workers print `recurrent state tier up`, the
  EngineCore prints `recurrent state leg on group(s)`. The documented grep
  returns nothing on a healthy boot, which reads as "the leg is off".

Status now carries the measured geometry to diff a future boot against, and
calls out the two quantities that fail silently -- `bytes_per_block`, which
must be `block_size x bytes_per_token`, and `leading dim`, which is 1536x
`num_blocks` on this backend -- plus the tier arithmetic, since at the
documented defaults the tier is 5x smaller than the HBM pool and can only
measure a null result.

`summarize_layout_id` run-length encodes the per-layer tail of the layout id
for the log: 69 identical entries, 1,877 characters on one line per worker and
36 KiB per boot, which left the grep the recipe prescribes scrolling past its
own answer. 1,877 -> 112 characters. The encoding is lossless and order
preserving, and it is deliberately NOT what `build_layout_id` folds into the
storage key -- shortening what a human reads must not shorten what separates
one layout's bytes from another's, so a test pins the two apart.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Size the K3 offload measurement from the trace instead of from a guess

The tier-sizing paragraph derived the HBM pool from `GPU KV cache size`, which
is `max_concurrency x max_model_len` and not a capacity at all -- the same
quantity this file already warns is not a pool. Replace it with the pool
arithmetic that the boot line `num_blocks / Maximum concurrency` confirms
(55 blocks per request = 43 attention + 3 x 4 mamba, the mamba term independent
of max_model_len in align mode), and with the reuse-distance band measured
offline from two independent K3 agentic runs.

That measurement changes the recommendation. The band `[pool, tier)` holds only
2-3% of turns at the default 1584 blocks, so that arm could only ever return a
null result. At 1100 blocks it holds 44-45% -- and the default 20 GiB/rank tier
is already the larger end, so the experiment needs no extra host memory, where
growing the tier past the default pool would have wanted ~853 GiB pinned.

Also records why max_model_len must not be lowered to shrink the pool: this
trace's median input sequence already runs past the 65,536 window, so a shorter
one truncates the workload rather than the pool.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Measure the lane scaling within K3 instead of across models

The sizing section claimed reuse distance grows sub-linearly in lanes, on the
strength of an 8x-lanes-to-2.3x-distance comparison. That pair was a K3 run
against a GLM-5.2 run, so it varied block size, bytes per token and the
prompt-length distribution at the same time as the lane count, and credited the
product to lanes.

The same sweep that supplied the 4-lane distribution also has an 8-lane run,
which gives a within-model axis at fixed workload: distance moves x1.90 at p75
and x1.97 at p50, i.e. close to linear, against ISL distributions 7% apart. The
1-lane point in that sweep is unusable for the same reason the cross-model pair
was -- its median input sequence is 1.8x the others.

Extrapolating one doubling from the 8-lane run puts the band at 16 lanes, which
moves the default row from 2-3% to 8-16% and leaves 1100 blocks at 36-46%. The
recommendation is unchanged; the reason it is safe is now measured rather than
assumed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Store KDA boundaries vLLM's hand-off can never name

The recurrent leg reported exactly zero external hits on Kimi-K3 --
3,956,629 queries, 0 hits, over 900 seconds -- and nothing said why.

Two acceptance domains that cannot intersect. vLLM's boundary hand-off
is a *partial tail* by definition: `_cache_partial_tail_block` returns
early when `num_tokens % block_size == 0`, and earlier still when
`block_size == hash_block_size`. This planner accepts only boundaries
that are whole mamba blocks *and* chunk-aligned, because `cap_hit`
descends in chunk steps and can select nothing else. On K3 both block
sizes are 1536 (vLLM raises the attention block size to satisfy
"attention page >= mamba page", and the hash block size is their gcd),
so the hand-off is empty -- and every boundary it could ever carry is a
non-multiple of the chunk size, so widening the filter would not have
helped either.

Take the boundaries from the block pool's own content-addressed cache
instead, the map vLLM uses to serve a local mamba hit. Resolving by
hash rather than by row is what makes it safe where a positional read
is not: an align-mode mamba block that was superseded, freed, nulled or
relocated is no longer registered under that hash, so the failure the
module docstring warns about -- wrong bytes under a valid prefix hash --
cannot occur. `get_cached_block` spans every mamba group in one call and
returns nothing unless all of them are present, which is the
all-or-nothing this state needs anyway. The hand-off is still read, for
models where it does carry something.

The 900 silent seconds were the second defect. Every explanation on this
leg was `logger.debug`, `stats()` had no call site, and the plugin path
exports no `lmcache_*` metrics, so a by-construction null looked exactly
like a tier with nothing to do -- which is the trap `StateOffloadIndex`
documents in its own comment and added `stores_refused` for. Counters
now name each reason a boundary was not stored, and one INFO line a
minute prints them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* [Doc] Report the measured K3 LMCache offload pair in the recipe

The LMCache section said "accuracy and hit rate: not yet run" and left the
sizing table's 1100-block recommendation as the last word. Both are now
replaced by the pair that was actually run.

Measured, 1800 s per arm, TP8, conc 16, seed 1234, pool pinned to 320 blocks
(381,300 tokens) and tier 40 GiB/rank: +10.81% req/s, +13.24% output token
throughput, TTFT p50 -59.1%, preemptions 35 -> 2. Prefill recompute falls from
64.01% to 16.27% of prompt tokens, which is the whole mechanism.

Three things are written down because they are easy to get wrong later:

- the promised / delivered / counted rulers disagree, and the promised-to-
  delivered gap must be summed as sum(required - retrieved) over every rank
  line. The per-event shortcut misses partial retrievals and overstated it 39%.
- GLM-5.3's start-cut sweep is not licensed here: the OFF arm drifts -17.1%
  across cuts, so no steady-state row is quoted. The ratio's stability across
  eight cuts is what says the full-window figure is not a window artefact.
- the sign of the tier's benefit follows tier/pool, with the 792-block pair
  (-3.59%) as the counter-example.

Limits are stated rather than hidden: n=1 per arm, ISL/OSL not held fixed
across arms, host MemFree differed at launch, and the tier is deliberately
under-sized (8,847 allocation failures on the ON arm).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* [Doc] Move K3 LMCache offload into its own recipe

Kimi-K3.md had grown a 340-line LMCache section that was longer than the rest
of the recipe put together, and unlike M3 and GLM-5.2 it had no file of its own
to point people at. Split it out as Kimi-K3-LMCache-Byte-Offload.md, matching
the naming the other offload recipes already use, and leave a short pointer in
Kimi-K3.md carrying the one number a reader needs to decide whether to open it.

Content is moved, not rewritten. The cross-references that said "above" and
pointed at sections no longer in the same file now name Kimi-K3.md and its
anchor. Two things are corrected in passing: the launch block declared a 20
GiB/rank tier while the measured pair ran 40, and the GLM-5.3 offload recipe is
referenced by name rather than linked, because it is not on main yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* [Doc] Align the K3 offload recipe's structure with GLM-5.3's

The split file kept the section order it had inside Kimi-K3.md, which is not
the shape the other offload recipes use. Reorganise it onto GLM-5.3's skeleton
so the two can be read side by side: Server / Sizing the tier / Client /
Measured / Related, with the "why this model needs more than the M3 path"
material moved into the intro, where GLM-5.3 puts its mirror-image claim that
no model-specific code exists.

Content is moved, not rewritten; the only lines that disappear are a duplicated
intro paragraph left behind by the split and the five retitled headings.

Two sections are new rather than moved:

- Client is now its own top-level section instead of a code block buried in
  Measured, with the four load-bearing flags called out the way GLM-5.3 calls
  out its own. It also records that the base launch does not pass
  --enable-prompt-tokens-details, which is why aiperf's prompt-cache columns
  are empty and every hit rate in Measured is read from /metrics instead.
- "Verify it booted right" keeps K3's boot geometry as a sub-section of Server,
  since it has no GLM-5.3 counterpart and is a server-side check.

The "accuracy not yet run" note moves to the end of Measured, next to the
limits of what was measured, rather than sitting in the boot-check section.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* recipes: report the GLM-aligned K3 offload pair as the null it is

Replaces Measured with the pair that actually matches the launch in Server
(no --num-gpu-blocks-override, 64 GiB/rank, ATOM_SLRU read back from the
rank-0 config dump, --enable-prompt-tokens-details on).

The result is that the tier supplies 0.30% of prompt tokens and the
throughput difference is not measurable: aiperf's req/s says +0.21% and a
recomputation over each arm's record span says -3.73%, with n=1 per arm and
no noise floor, so the sign is undetermined. It is written up that way
rather than quoting the +3.08% out-tok/s.

The cause is structural, not a sizing error. With the pool unpinned vLLM
answers 85.05% of prompt tokens itself, and on the plugin path the connector
is only asked about what the pool missed, so the tier bids for the leftover
15% and wins 2.01% of it. The band model in Sizing promised 18.7%; the 62x
gap between that bound and the delivered 0.30% is now recorded next to it.

Sizing also gains GLM-5.3's cheaper cross-check (tier ~1.5x the reusable
working set): 20 distinct conversations x their largest ISL x 56,448 B =
30.5 GiB/rank, so 64 is 2.10x the set, above the rule rather than below it.
GLM-5.3's own 256 GiB/rank is 1.57x its 163 GiB set at TP4 and cannot be
copied to K3: at TP8 it would pin 2048 GiB against a ~90 GiB/rank ceiling
on this host.

Kimi-K3.md's pointer said +10.81% without saying that number needs a pinned
pool; it now leads with the default-pool null.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Keep a failed tier load from killing the engine on a hybrid model

vLLM's KV-load-failure recovery assumes a single KV cache group:

    (req_block_ids,) = self.kv_cache_manager.get_block_ids(req_id)

under a `# TODO (davidb): add support for hybrid memory allocator` in
`Scheduler._update_requests_with_invalid_blocks`. Kimi-K3 is hybrid (MLA
full-attention plus KDA recurrent layers), so `get_block_ids` returns one
list per group and the unpack raises inside EngineCore. That is not a
degraded request: it is `EngineDeadError`, every in-flight request 500s,
and the server shuts down. The path is reachable exactly when this
connector is in use with `"kv_load_failure_policy":"recompute"`, which
the K3 recipe requires -- the default `fail` turns a tier miss into a
user-visible 500 instead.

Rather than patch vLLM in place, select a subclass through the public
`scheduler_cls` config knob from ATOM's platform hook. The blast radius
is then one engine's scheduler instead of every `Scheduler` in the
process, and a mismatch with a future vLLM fails loudly at boot instead
of silently reverting behaviour. Be honest about what it does not buy:
the overridden method is private either way, so the version coupling is
the same as a monkeypatch's.

Two subclasses, because `get_scheduler_cls` consults `async_scheduling`
only while `scheduler_cls` is None; once set, the named class is
returned as-is, so returning a plain `Scheduler` would silently disable
async scheduling. `select_scheduler_cls` reads the already-resolved
`async_scheduling` bool -- the platform hook runs after vLLM decides it.

The override is skipped when vLLM's own method no longer contains the
single-group unpack, so this retires itself when upstream fixes it. It
is gated on a KV connector being configured (without one no load can
fail), not on the model being hybrid: group counts are not known at
config time, and the override handles a single group exactly as vLLM
does. Selection failure is logged, never fatal.

Not unit-tested: the CI unit job has no vLLM and both the subclasses and
the platform hook need it to import. Checked instead by the sweep's
preflight, which asserts the bases, the upstream-source probe, that the
hook is actually called from `check_and_update_config`, and that both
`async_scheduling` values resolve to the right qualified name.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Record the controlled-prefix working point in the K3 offload recipe

The recipe's Measured pair reports the tier supplying 0.30% of prompt tokens
and no measurable throughput difference. That is a property of the client used
there, not of an unpinned HBM pool: its reuse falls inside what HBM keeps, so
the connector only ever sees a 15% miss tail.

Re-measured at the same unpinned pool (both arms report the pool within 0.06%,
1,887,436 vs 1,886,245 tokens) with a controlled-prefix synthetic client whose
reuse distance is chosen to land in the band Sizing the tier defines, the tier
supplies ~82% and throughput gains +20.5 / +22.4 / +27.8% at conc 16 / 20 / 24
over 1800 s windows. Deltas are taken from request counts, not from aiperf's
two-decimal req/s. Every arm is n=1; there is no noise floor and the section
says so.

Three knobs differ between the two working points -- client, tier 64 -> 90
GiB/rank, and atom_head -- so nothing is isolated to one of them; the section
enumerates them rather than attributing the result to the tier.

The band arithmetic already in Sizing the tier predicts both the result and its
trend: d = 483,840 tokens sits above the reusable prefix (270,336 at conc 16,
falling to 101,376 at conc 24) and below the tier (1,711,961 tokens at 90
GiB/rank), and the lower edge drops as lanes are added, which is the direction
the gain moves.

Also qualified in place: the "factor of 62" paragraph, which read as a property
of unpinned pools; the 64 GiB/rank knee, which is that workload's; and the
pointer in Kimi-K3.md, which stated the null without its precondition.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Keep a KDA offload hit from serving another prefix's state

Copy the boundary page only after the forward and mamba postprocess
have written the accepted state, retry a boundary the pool does not
hold yet, and fail a load that does not end on a mamba block or cannot
name the attention blocks to invalidate.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Record the conc-32 ON arm and close the sweep's open prediction

The 2026-09-22 sweep left conc 32 with an OFF arm only: wait_for_gpus.sh gates
on 756 GiB of raw MemFree and the host sat at 553.  The GPUs were idle; the
memory was 1,068 GiB of K3 weights that each arm's NFS re-read had left in the
page cache, measured per-file with fincore and released with posix_fadvise
DONTNEED (clean pages, no drop_caches).  MemFree 467 -> 1539 and the arm ran.

It confirms the prediction the section already carried.  conc 32 is where the
reusable-prefix formula goes negative, so no prefix survives between turns and
every reuse must come from the tier: tier 87.73% of prompt tokens against
81.6-82.7% at the other three points, HBM local hit 5.70% against 10.4-11.1%,
and the largest gain of the set at +32.6% (3028 vs 2283 requests over identical
1800 s windows).  Recompute is what moves -- 6.57% ON against 19.96% OFF.

Also noted, so a reader does not have to re-derive it from server.log: that
arm's EngineDeadError is the driver's own SIGTERM racing AsyncLLM's output
handler at teardown, five seconds after a healthy engine log and after aiperf
had finished; was_cancelled=False and error_summary is empty.

The gain still carries no error bar (every arm is n=1) and the rising trend runs
with the fixed-pool bias run_sweep.sh documents, not against it.  Both limits
are stated in the section rather than left to the reader.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: perzhang <perzhang@amd.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* perf(decode): fold the graph's padded tail into the deferred-id kernel (#2368)

A decode step wrote `input_ids` in two launches: one Triton kernel filling
each carried-over request's span, then a `zero_()` over the rows a replayed
graph reads but the batch never scheduled. The second is a 5.6us dispatch
floor for ~1.5KB, and it fires on every step -- `bs` sits between two capture
buckets on all but the exact-fit ones.

The two regions are disjoint by construction: the tail starts where the last
request ends, so nothing orders them and one launch can do both. Programs past
`num_requests` zero `[cu[bs], width)` instead of filling a span.

`cu` stays the only statement of where requests end -- the padding programs
read `cu[bs]` themselves rather than taking a second copy of it from the host,
so the two cannot disagree. `width` is a required argument, because a caller
that forgets it leaves the padded rows carrying the previous step's ids, and
the MoE path does consume them.

`BLOCK_PAD` is one value so the tail costs one JIT specialization rather than
one per capture bucket, and the three new runtime ints are `do_not_specialize`
for the same reason.

Verified: GSM8K 1319 @ 3-shot on DeepSeek-V4.1-Flash DSpark-5, TP4, bf16 KV,
index fp8 -- flexible 0.9174 / strict 0.9166 against a 0.9224 baseline mean
(sigma ~0.005). Unit suite 8512 passed, unchanged from before the change.

* (recipe) add Kimi-K3 AgentX recipe (#2131)

* (recipe) add Kimi-K3 AgentX recipe

Document the per-concurrency ATOM serving bands for the Weka AgentX workload.

* (recipe) tune Kimi-K3 AgentX ReplaySSM and AITER comm reuse

Turn ReplaySSM off at C1/C2/C4 and on at C16, and keep AITER comm-group reuse only on C56/C64.

* (recipe) document container prerequisites and add the C14 point

Two container-level facts decide whether this recipe reproduces its own
numbers, and both of them fail quietly, so add a prerequisites section
covering each:

- triton must be 3.7.x. The kimi_k3_agentic_0903 image shipped 3.8.0, which
  costs ~2.1x on prefill TTFT here (C1 p50 0.79s -> 1.5s) while leaving decode
  untouched. Measured across eleven runs: the gap survives swapping aiter's
  .so between revisions, swapping the ATOM checkout, and switching between a
  full CI prebuild and a lean JIT build, so triton is the only variable that
  moves it.
- DRAFT_MODEL_PATH must name a local directory. The default is a Hugging Face
  repo id the image has no cache entry for, so leaving it unset makes every
  rank fetch the same 7 GB checkpoint with no error printed, the log stopped
  after "Loading drafter model...", and every GPU at 0%. Document the symptom
  next to the fix, because the hang is otherwise indistinguishable from a
  wedged server.

C14 was in the benchmark scripts but never in the recipe. It reuses the C16
server recipe verbatim, including a CUDA-graph width pinned to 32; deriving it
from 2*CONC gives 28 and the server fails during CUDA-graph warmup. Routing
GRAPH_MAX through CUDAGRAPH_MAX_NUM_SEQS lets the C14 arm pin it while every
other concurrency keeps the 2*CONC default unchanged.

Verified against run_kimik3_atom_dspark_server.sh: all eleven concurrencies now
agree item by item across DCP, max-num-seqs, batched tokens, GPU utilization,
LMCache, ReplaySSM, spec tokens, acceptance length, AITER comm reuse and the
recomputed graph_max.

* (recipe) set ATOM_GDN_SSM_DTYPE=fp16 for Kimi-K3 AgentX

Apply the KDA temporal state dtype optimization from ATOM #2130.

Co-authored-by: Cursor <cursoragent@cursor.com>

* (recipe) Kimi-K3 AgentX: per-concurrency params from the 12-point MI355X sweep (#2208)

* (recipe) Kimi-K3 AgentX: band params from the 12-point MI355X sweep

Replaces the per-concurrency values with the ones a full 12-point AgentX
sweep actually ran on MI355X (8x, ATOM 97c15a73 + aiter 6b3f14b5f,
aiperf 0.12.0, 3600 s profiling window per point).

  * GPU util 0.88/0.86 -> 0.90 at every band. Measured on this box as
    available_for_kv 39.42 -> 51.11 GB, a +40.6% KV sub-pool.
  * max-num-batched-tokens 4096 -> 8192 at C8 and C12, so it is now 8192
    everywhere. Cross-round TTFT p50 deltas move monotonically through
    zero across C8..C56, consistent with the larger prefill chunk.
  * AITER_REUSE_IDENTICAL_COMM_GROUPS is now a DCP rule rather than two
    hardcoded branches: 1 at every DCP=8 band, 0 at DCP=1. C32/C40/C48
    measured faster with it on; C1 measured 3.3-3.9% WORSE ITL with it
    on, which is why the DCP=1 bands keep it off. C8-C16 are still under
    A/B and may move in a follow-up.
  * Adds the missing C48 band (max-num-seqs 96, graph_max 96). It was in
    neither the table nor the case, so CONC=48 hit the unsupported exit.
  * Pins ATOM_USE_FLYDSL_GATHER_KV_B_PROJ=1. It already defaults to 1
    (atom/utils/envs.py), so this changes nothing today -- it is set
    explicitly so the recipe keeps reproducing the measured numbers if
    the default flips.

DCP, spec tokens, acceptance length, LMCache sizes, ReplaySSM,
max-num-seqs and graph_max are unchanged; every row was re-verified
against the sweep's own launcher.

* (recipe) Kimi-K3 AgentX: hoist band-invariant values out of the case

The per-concurrency case repeated nine values that are now identical across
every band, so a single-band change had twelve places to go wrong. Hoist the
shared defaults above the case and leave only the genuine per-band deltas
inside it: 119 lines -> 51.

Behaviour is unchanged. All twelve concurrencies were evaluated before and
after and every field -- DCP, max-num-seqs, batched tokens, GPU util,
ENABLE_LMCACHE, LMCache size, ReplaySSM, reuse, spec tokens, acceptance
length, CUDAGRAPH_MAX_NUM_SEQS and the derived graph_max -- is byte-identical.

max-num-seqs is now derived as 2*CONC on the throughput bands rather than
written out five times; it was already exactly 2*CONC there. The two bands
that genuinely deviate keep explicit overrides with the reason attached: C12
at 24, and C14's pinned CUDAGRAPH_MAX_NUM_SEQS=32.

LMCACHE_MAX_LOCAL_CPU_SIZE now carries its 128 default on the DCP=1 bands
too. That is inert: it is only exported inside the ENABLE_LMCACHE==1 branch,
which those bands do not take.

* (recipe) trim the ATOM_USE_FLYDSL_GATHER_KV_B_PROJ comment

* (recipe) Kimi-K3 AgentX: comm-group reuse off below C56

A seven-point A/B on MI355X (C8/12/14/16/32/40/48, one reuse=1 leg and one
reuse=0 leg each, identical builds) measured reuse=0 faster at every point:
ITL p50 -7.3% at C8 narrowing to -1.9% at C48, total tok/chip +3.8% to +1.2%.
Split into 10-minute buckets, 39 of 42 buckets favour reuse=0 (one-sided
p ~ 3e-9), which is well clear of the ~7% run-to-run noise floor that makes
any single full-window comparison on this workload unreliable.

The advantage shrinks monotonically as the KV pool tightens -- reuse=0 gives
up roughly 13k blocks, and its peak KV usage climbs from 77.6% at C8 to 97.6%
at C48, where it is the first point to spend any time above 95% and where
maxRunning starts being pushed back. C56/C64 were not measured and use a
larger LMCache tier and max-num-seqs, so they keep the previous setting.

This also removes the claim that the 12-point sweep measured enabling reuse at
C32/C40/C48 as a gain. That reading came from single-run full-window
aggregates and does not survive the paired A/B.

* (recipe) Kimi-K3 AgentX: C72 + C80 bands, drop the C14 CUDA-graph pin (#2213)

* (recipe) Kimi-K3 AgentX: add the C72 throughput band

Same shape as C64 -- DCP=8, no spec, 192 GiB LMCache, ReplaySSM off -- with
max-num-seqs following the 2*CONC rule every band from C32 up obeys, so 144,
and graph_max 144 with spec disabled.

Measured on MI355X against the C64 band it extends: total tok/chip 12794 vs
12221, so C64 is not the throughput plateau. ITL p50 moves 77.11 -> 88.60 and
P90 interactivity 9.9 -> 8.4, which is the trade the band exists to expose.
maxRunning reached 119, so the wider capture range is genuinely reached rather
than captured and left unused.

Reuse stays enabled here: the CONC >= 56 gate already covers C72, and a paired
A/B measured reuse=1 ahead at C72 by a wider margin than at C64 (ITL bucket
mean +4.5% and total tok/chip +3.1% in its favour, 6 of 6 ten-minute buckets
agreeing), consistent with C72 sitting further past the C56 crossover.

* (recipe) Kimi-K3 AgentX: drop the C14 CUDA-graph pin

C14 pinned CUDAGRAPH_MAX_NUM_SEQS=32 with the justification that deriving
graph_max from 2*CONC "would give 28 and the server fails during CUDA-graph
warmup". That is false on ATOM 97c15a73, and the pin has no effect.

Capture sizes are SEQUENCES, not tokens. ModelRunner truncates the declared
ladder to max_schedulable_decode_bs() = min(max_num_seqs, max_num_batched_tokens
// full_q_len) (atom/model_engine/model_runner.py:142) -- it divides the
speculative width out, while the launcher multiplies it in. For C14 that is
min(32, 8192//4) = 32, independent of CUDAGRAPH_MAX_NUM_SEQS. Sizes above the
bound are dropped with a warning, not asserted on; a batch past the ladder just
falls back to eager.

So both settings capture exactly bs=2..32:

  pinned   CGMS=32 -> declared [2..128] -> "= 2048) = 32; dropping" x8 ranks
  default  CGMS=28 -> declared [2..112] -> "= 2048) = 32; dropping" x8 ranks

Measured on 8xMI355X, AgentX 3600 s, C14, AITER_REUSE_IDENTICAL_COMM_GROUPS=0
both legs, ATOM 97c15a73 / aiter 6b3f14b5f / triton 3.7.0:

               TTFT p50  TTFT p90  ITL p50  ITL p90  Inter p90  tot tok/s
  CGMS=32         804.3    2140.0    12.42    19.18       52.1      57148
  CGMS=28         775.8    2028.4    12.36    19.18       52.1      57060

ITL p90 and Inter p90 are equal to the digit; throughput differs by -0.15%.
Ten-minute buckets have identical n per bucket and ITL deltas of
-5.4/+1.2/+0.2/-0.9/-1.0/+3.6 %, three each way, all inside the ~7% run-to-run
noise floor. The server reached READY in ~330 s unpinned.

The table's graph_max for C14 becomes 2*14*(1+3) = 112. With the pin gone, every
row now satisfies the stated rule graph_max = 2 * CONC * (1 + spec); C14 was the
only exception.

Co-Authored-By: Claude <noreply@anthropic.com>

* (recipe) Kimi-K3 AgentX: add the C80 throughput band

C80 is the same shape as C64/C72 -- DCP=8, no spec, 192 GiB LMCache,
max-num-seqs following the 2*CONC rule (160). It needs no new branch: the
existing `CONC -ge 56` guards already pick up the larger CPU tier and
AITER_REUSE_IDENTICAL_COMM_GROUPS=1.

Measured on 8xMI355X, AgentX 3600 s, ATOM 97c15a73 / aiter 6b3f14b5f /
triton 3.7.0, reuse=1:

  CONC  TTFT p50  TTFT p90  ITL p50  ITL p90  Inter p90  tot tok/chip  out tok/chip
    64    1356.1    3802.3    77.11   100.66        9.9         12221         88.56
    72    1489.8    3963.2    88.60   119.50        8.4         12794         89.42
    80    1549.8    4405.3   103.07   130.43        7.7         12839         95.75

**Total throughput plateaus at C72.** C64 -> C72 is +4.7% on tokens/chip;
C72 -> C80 is +0.35%, well inside the ~7% run-to-run noise floor, while ITL p90
rises 9% and Inter p90 falls from 8.4 to 7.7.

Read `tot` with care at the top of the ladder. C80's ISL drops to 107777 (vs
C72's 116894) because the trace serves more, shorter requests (3443 vs 3165), so
output throughput still climbs +7.1% (715 -> 766 tok/s) while total flattens.
The two agree at every band from C8 to C72 and diverge for the first time here.

Saturation is unambiguous across three independent signals:

              KV usage samples >=95%   KV max   maxRunning
  C64                      2/528        96.5%           84
  C72                     17/545        96.2%          119
  C80                     48/556        99.7%          114

maxRunning stops rising at C80 while KV hits 99.7% -- the scheduler is bounded
by the KV pool, not by the concurrency parameter. This is the same ordering
variable that decides the AITER_REUSE_IDENTICAL_COMM_GROUPS crossover at C56.

C80 is included for completeness of the band table. C72 is the knee.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: xytpai <xytpai@foxmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: gbyu-amd <Guanbao.Yu@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>

* feat(v41): enable TP2 DSpark and add AgentX recipe (#2373)

* feat(v41): enable TP2 DSpark and add AgentX recipe

* fix(v41): admit fixed acceptance schedules without an opt-in gate

* docs(v41): simplify AgentX recipe and remove auxiliary CSV

* docs(v41): present TP2 and TP4 performance in separate tables

* docs(v41): use direct server and client recipe commands

* style(v41): format DSpark runtime admission test

---------

Co-authored-by: ZhangLirong <219599949+ZhangLirong-amd@users.noreply.github.com>

* [MiniMax-M3] Route the paged decode to aiter's FlyDSL kernel with work planner (#2366)

* [MiniMax-M3] Make the dense paged-decode split cap overridable

`PA_DENSE_SPLIT_MAX = 32` is not a tuned value. It is the smaller of two
unrelated bounds that happen to agree near there: production aiter does not
build the C++ PS reduce past 64 and has no working fallback under it, and
`temporary_output` is bf16, so each extra split adds a round trip through the
PS combine.

aiter #4332 removes the first bound. Deciding whether the second one still
binds needs the cap measured against the shipping value on the same tree,
which a constant cannot express. `dense_decode_splits` computes
`min(PA_DENSE_SPLIT_MAX, cdiv(128, running_bs))`, so the cap only binds at
`running_bs <= 2` -- the difference is invisible without an A/B at that shape.

Default stays 32 and behaviour is unchanged. Raising it on production aiter
still fails at engine init rather than at runtime.

`int(os.getenv(...) or 32)` rather than a getenv default: an exported-but-empty
variable would otherwise raise `int('')` at import and take the engine down
before it starts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [MiniMax-M3] Route the paged decode to aiter's FlyDSL kernel behind an env

ATOM_PA_FLYDSL=1 sends the paged decode to aiter's FlyDSL implementation
(aiter PR #4332) where FlyDSL's domain covers the call, and to gluon
otherwise. A measurement switch, not a shipping default: the two take the
same arguments and compute the same thing, and the env exists to A/B them
without maintaining two ATOM trees. Default 0, so nothing changes until an
operator opts in on an aiter that carries #4332.

_flydsl_pa_decode_num_seqs mirrors the kernel's own validation -- alibi,
sinks, externally quantized FP8 queries, ps=False, a positive sliding window,
a non-256 tile, a partition count outside [1, 256], an unsupported head dim,
and any cache that is not [nb, Hkv, D//16, page, 16] at 1 byte per element are
all hard rejects in aiter, not preferences. Enabling the env therefore cannot
turn a working configuration into an exception.

The returned count is the substance. ATOM pads two axes independently:
context_lens to running_bs for CUDA-graph identity, with
[scheduled_bs:running_bs] zeroed, and q to running_tokens for the MoE
all_gather. forward_context.Context stores both "because the ratio is not
always max_seqlen_q" and asserts no rectangle between them. gluon absorbs the
mismatch by deriving its batch as q.shape[0] // query_length and letting the
surplus zero-length rows do no work; FlyDSL demands the rectangle and raises

    ValueError: query.shape[0] (12) must equal
                context_lengths.shape[0] * query_length (4 * 4)

on an ordinary step. Recovering the count the way gluon does and slicing the
five per-sequence arguments to it hands FlyDSL the same rectangle, dropping
exactly the zeroed tail. All slices are views on dim 0.

The gate is deliberately not restricted to max_seqlen_q == 1. The dense path
is where FlyDSL's headroom over gluon lives, it runs max_seqlen_q ==
num_spec + 1, and FlyDSL carries a query_length == 4 MTP4 grid split tuned for
exactly that shape.

ATOM_PA_FLYDSL_PLAN=1 additionally opts into aiter #5546's GPU work planner.
The plan and its scratch are allocated once and only refreshed: allocation is
illegal inside graph capture and a captured graph bakes in the pointers, and a
planned call does not use the caller's static buffers -- planned partials are
packed as [kv_heads, capacity, query_rows(, D)] against the static API's
[num_seqs, kv_heads, partitions, query_rows(, D)]. The planner refuses batches
past 4096 while M3's sparse call site folds query tokens into num_seqs and
reaches 32768 on a prefill-as-decode step, so those fall back to the static
path rather than raising.

Measured negative on this workload and defaulted off: against the same tree
with the planner off it costs 3.86% at conc 1, 11.50% at conc 10 and 18.28% at
conc 20. The cause is not batch uniformity -- 31% of decode steps have max/min
KV length >= 4 -- but the split count, which plan_pa_decode and pa_decode
share: dense_decode_splits gives 16 at running_bs 5 and 8 at running_bs 24,
and at those counts a batch skewed 2.5x and one skewed 120x both measure
1.00x. #5546's own conclusion is that the data "support using dynamic plans
for uneven KV work, not enabling them universally".

Both routes are logged once per shape signature. A run where the env is set
but every call still lands on gluon is otherwise indistinguishable from one
where FlyDSL simply did not help.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [MiniMax-M3] Give the FlyDSL work planner its own partition ceiling, and keep it off the sparse call sites

`_flydsl_work_plan` handed `plan_pa_decode` the static split count from
`dense_decode_splits`, overwriting that argument's own default. The two are
different quantities: `get_recommended_splits` gives every request the same
count and documents itself as "not a variable-work scheduler", while the plan's
`max_partitions` is a per-request ceiling it then divides under a workgroup
budget. Clamping the ceiling to the static count leaves a long request with the
same share as a short one, which is the planner's entire mechanism. Read back
from `plan.reduce_info`: a 5-request batch at ceiling 16 splits [16,4,4,4,4]
with the long request pinned at the ceiling, and at the default 256 it splits
[256,20,21,20,21].

`ATOM_PA_FLYDSL_PLAN_MAX` now carries that ceiling, 0 meaning "omit the
argument and take aiter's default". Do not read the ceiling off aiter's unit
test: it is the only caller of `plan_pa_decode` in that tree and it binds the
two deliberately, building the plan from the same count it hands `pa_decode` so
the paths must agree numerically. That makes it a correctness test which cannot
see the conflation.

The planner was also a global switch inside `run_pa_decode_gluon`, so enabling
it reached the two sparse call sites as well -- 57 of the 63 `pa_decode` calls
per step. Their context is a fixed topk window, so there is no unevenness to
rebalance and only the refresh and the task packing remain. On uniform batches
the planner is a loss that the ceiling cannot rescue: 0.56x at B8/257 and 0.84x
at B200/200000, flat from ceiling 4 through 256. `allow_work_plan` moves that
choice to the caller, where the split policy already lives, and only the dense
site opts in -- the same boundary `dense_decode_splits` already draws.

A plan build now logs its shape once per cached key, which is the only evidence
that the planner went where it was meant to rather than being assumed to.

Measured at conc 20 on the agentic trace (one operating point), single-user
output throughput 157.19 -> 178.49 tok/s and ITL 7.97 -> 6.96 ms, against 139.20
for the same arm before these changes. Prefix hit 96.56 vs 96.63, input length
+1.6%, request count +1.0%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [MiniMax-M3] Build the FlyDSL work plan once per forward, and fold four env switches into one

The plan is a function of `context_lens` alone, so every layer of one forward
wants the same one. Building it inside the attention op re-ran the planner
kernel for each of M3's three dense layers, and the module-level cache that
existed to paper over that had to key itself on shape to stay correct. It now
lives where the rest of the per-forward metadata is built: `prepare_decode`
attaches it, `prepare_mtp_decode` refreshes it for each draft pass (their
context lengths differ by a token, so reusing the target's plan would point the
kernel at the wrong KV ranges -- wrong output, not merely slower), and the op
reads it off `get_forward_context().attn_metadata`.

Measured before the move: 6 planner kernels per step, and `envs.__getattr__` on
all 63 `pa_decode` calls rather than the 6 that could use a plan. After: 4 and
1. The env read is the larger of the two -- one `__getattr__` costs ~0.97 us and
63 of them are ~0.5% of a 12.8 ms step.

Scratch allocation stays in the op. It is sized from the query tensor the op is
holding and happens once per shape, so only the per-step work moved out.

The env surface collapses to `ATOM_PA_FLYDSL_PLAN`, now on by default:

  * ATOM_PA_FLYDSL is gone. It was a measurement switch for A/B-ing FlyDSL
    against gluon without two trees; FlyDSL is the decode path. The capability
    check that falls back to gluon for shapes outside FlyDSL's domain stays --
    it mirrors the kernel's own validation and is not a switch.
  * ATOM_PA_FLYDSL_PLAN_MAX is gone, and the call now omits `max_partitions`
    entirely so `plan_pa_decode` takes its own default. The one time this tree
    chose the ceiling it chose the static split count, which clamps every
    request to the same share and removes the planner's mechanism.
  * ATOM_PA_DENSE_SPLIT_MAX is gone and PA_DENSE_SPLIT_MAX is a constant again.
    With the planner on, a planned call is told `plan.max_partitions`, so the
    cap no longer sets the production partition count; it survives for the
    gluon fallback, where both halves of its bound still hold. Raising it would
    only enlarge the static scratch a planned call never reads.
  * ATOM_PA_FLYDSL_PLAN defaults to 1. On the agentic trace, SA-convention
    interactivity (1/itl_p90) is +24.6% at conc 20 and +8.4% at conc 10, and
    the gain runs monotonically from the slow tail to the fast one (p10 +24.6%,
    p50 +19.3%, p90 +5.0%) -- it lifts the floor, which is the end the metric
    watches.

The routing log no longer hangs off the deleted env, and is unconditional: with
FlyDSL as the default path, which route a call took is worth always recording.

tests/test_paged_attention_dispatch.py: 74 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [MiniMax-M3] Trim the FlyDSL work-plan comments

Prose cut, facts kept. What stays is the four things a reader would otherwise
re-derive the hard way: the ceiling must not come from the static split count,
each draft pass needs its own refresh (reusing the target's plan points the
kernel at the wrong KV ranges -- wrong output, not merely slower), scratch is
allocated in the op because its shapes come from the query tensor, and the
FlyDSL/gluon split is a capability check rather than a switch. The measured
numbers move to the one place they inform a decision: why the planner defaults
on.

Also fixes a contradiction left by the previous commit. `run_pa_decode_gluon`
still described itself as a "MEASUREMENT SWITCH, not a shipping default" after
FlyDSL had become the path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [MiniMax-M3] Make the per-forward work plan survive cudagraph capture

Review found the previous commit's premise broken: decode runs from captured
graphs (cudagraph_mode=FULL), and `build_for_cudagraph_capture` never attached
a plan, so the op saw None at capture time and the STATIC path is what got
recorded. Every later refresh then worked on a graph that does not read it --
no error, no slowdown, just a planner that quietly does nothing and an
end-to-end number that looks fine.

The capture flow (model_runner.py) is what makes the fix possible:

    3857  build_capture(bs)           outside the graph -- first plan allocated
    3913  self.model(...)             warmup, eager; the op sees the plan
    3969  with torch.cuda.graph(...)  capture; the plan is already there

So the plan is attached in `build_for_cudagraph_capture` too, and the builder
keeps one per (batch, kv_heads, device) instead of a single slot. The single
slot was the more dangerous half: `build_capture` runs once per capture-ladder
size, so the second rung replaced the first rung's plan while the first rung's
graph had already baked in its pointers.

Two more from the same review:

  * The plan was built from `context_lens[:scheduled_bs]` while the op derives
    its own batch as `q.shape[0] // max_seqlen_q`, which is `running_bs`.
    aiter validates `reduce_info.shape == (num_seqs, 2)` and raises, so any
    step whose batch was not exactly on the ladder would have killed the
    worker. `context_lens` is already `running_bs` long with the tail zeroed.
  * `prepare_mtp_decode` dropped the refresh result and relied on the returned
    object being the same one. It now publishes through `workinfos`, which the
    caller splats into the metadata, so a None result clears a stale plan
    rather than leaving it in place.

`flydsl_plan_matches` is extracted rather than inlined so the guard is
testable: a plan built for another batch now falls back to the static path
instead of reaching aiter's validate.

tests/test_paged_attention_dispatch.py: 74 -> 83. Eight of the nine are
falsifiable by reverting the behaviour they describe, each reddening exactly
one test; the ninth is a source check on the capture attach, labelled as such,
because driving a real capture needs a model runner and what it guards fails
silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [MiniMax-M3] Put the FlyDSL decode kernel back behind an env, gluon by default

aiter #4332 is unmerged and the kernel is not fully tested, so gluon goes
back to being the shipping decode path and FlyDSL becomes the opt-in:
ATOM_PA_FLYDSL=1, off by default. The two gates run in series -- the env,
then the capability check that mirrors the kernel's own validation -- and
the `and` short-circuits, so with the env off the check does not run.
Shapes outside FlyDSL's domain still fall back to gluon either way.

The work planner (aiter #5546) only feeds the FlyDSL kernel, so it now
requires ATOM_PA_FLYDSL=1 as well. Without that, a plan would be built and
refreshed every step for a kernel that never reads it, and the "flydsl work
plan" log line -- what an A/B arm is checked with -- would show up on a run
that is entirely gluon.

Tests 84 -> 86: the default must be off, the dispatch must consult the env
before the capability check, and the builder must not plan with FlyDSL off.
Each of the three goes red on its own reversion.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [MiniMax-M3] Rename allow_work_plan to allow_flydsl_plan, trim comments

The parameter now matches the env that controls it, ATOM_PA_FLYDSL_PLAN.
"work plan" is aiter's own term for the object, but on its own it does not
say the plan belongs to the FlyDSL path -- which matters now that gluon is
the default and FlyDSL is the opt-in.

Comments trimmed throughout: the _flydsl_pa_decode_num_seqs docstring keeps
why it returns a count rather than a bool (ATOM pads its sequence and row
axes independently, so the two disagree by a padding slot on an ordinary
step; gluon absorbs that, FlyDSL raises) and drops the shape algebra and the
verbatim traceback.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fix format

* Update docs

* Fix review comments

* Address PR #2366 review: capability gate, plan lifetime, ROCR device binding

Two gates in series guard the FlyDSL paged decode: ATOM_PA_FLYDSL is the
opt-in, and _flydsl_pa_decode_num_seqs mirrors aiter's own rejects so an
unsupported call falls back to gluon instead of raising inside the kernel
and killing the worker. The gate was missing several of those rejects --
q/output dtype and shape agreement, both head_dim stride checks, the
q-heads-over-kv-heads divisor, the v_cache dtype, and contiguity of the
two caches and the two per-sequence arrays.

Plans are now only ever minted during cudagraph capture. aiter's planner
takes batch as a tl.constexpr, so a runtime batch that no graph replays
would cost a cold kernel specialization and a plan nothing can free; the
static path is the right answer for it. Runtime only refreshes in place.
The one-shot "no plan for this batch" notice waits until a capture has
happened, since before that every call legitimately lands there and the
notice would be spent on a step that says nothing.

max_partitions stays unset, at plan_pa_decode's own default: every number
measured on this branch was measured there, and the review's suggestion to
clamp it to 64 is a perf claim nothing here backs. Measuring instead, on
the planned path, across nine cells spanning the trace's p50/p90/p99
contexts at batch 1/8/24: drift from an fp32 reference does not grow with
the split count anywhere, and the ceiling only ever binds at batch 1. The
plan's capacity is min(batch * max_partitions, max(batch, 2 * CU /
kv_heads)) = 512 here, so each sequence's share is about 512/batch and a
64 ceiling would have changed nothing above batch 2 while capping the one
place the planner has the most to rebalance.

numa_utils and the mooncake connector now read ROCR_VISIBLE_DEVICES first.
ROCR filters and renumbers before HIP indexes the result, so setting both
cut the visible device count twice; dropping HIP from the recipe left the
NUMA binding reading nothing and put all four ranks on one node.

Tests go from 86 to 127. The call-site contract is now checked by walking
the AST of all three builders and comparing the value of `create`, not its
presence in one file -- a sed that moved `create=True` onto a runtime path
had left every test green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [ATOM][PD]: Transfer prefill prompt token ids to decode (#2331)

* feat(pd): relay prefill prompt token ids to decode

Decode was re-rendering and tokenizing the same prompt prefill already
tokenized. Prefill echoes the ids, atomesh carries them in
kv_transfer_params, and decode skips tokenize.

Co-authored-by: Cursor <cursoragent@cursor.com>

* perf(pd): avoid redundant prompt token ID serialization

Serialize completed OpenAI responses through JSONResponse to avoid
FastAPI recursively visiting every prompt token ID. After validating the
prefill IDs on decode, remove the duplicate prompt from KV metadata before
engine IPC while preserving the original request and other transfer fields.

Cover HTTP response compatibility, prompt-ID round trips, and metadata
forwarding for chat/completions, streaming, and fanout requests.

Validation: 103 passed in the prompt-token-ID and protocol tests.

* docs(pd): simplify token ID comments and drop recipe changes

Shorten token ID comments and docstrings across the API, router, and tests.
Restore the PD disaggregation recipe to origin/main without changing runtime
behavior.

Validation: 103 related tests passed; Black passed; no new Ruff findings.

* simplify the test cases

Signed-off-by: Mengqing Cao <cmq0113@gmail.com>

---------

Signed-off-by: Mengqing Cao <cmq0113@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(moe): prefill-only mega combine wire (#2347)

* feat(moe): prefill-only mega combine wire

* fix(moe): DP-agree the mega combine wire

* refactor(moe): move combine wire env to envs.py

---------

Co-authored-by: honglie <hyi@amd.com>

* fix(offload): one tier lookup per waiting request, not one per step (#2305)

`get_num_new_matched_tokens` runs before allocation and is gated only on
`num_computed_tokens == 0`, so a request that fails to allocate goes back
to the head of the waiting queue and is asked the identical question next
step. On a full KV cache that turns one lookup into one lookup per
scheduler step: a prompt copy, ~5k chunk hashes and a blocking round trip
to the tier worker, all on the scheduler thread.

Memoize the one expensive input -- the tier `hit` -- and re-derive the
answer from it on every call. The frontier is not part of the question:
how much of the prompt the tier holds is a property of the prompt, and the
frontier only decides how much of that hit is still worth transferring.
Nothing tells the connector when the tier's answer changes, so the memo is
a bounded optimisation rather than a cache: a hit is replayed at most
`OFFLOAD_LOOKUP_MEMO_STEPS` times and a non-answer backs off for
`OFFLOAD_LOOKUP_RETRY_STEPS` (both default 32).

A memo-served answer carries no pin, and an answer needs none -- the
request only waits. A transfer does. `build_connector_meta`'s load-emit
loop is the single choke point where a retrieve reaches the worker on both
the native and the plugin path, so the pin is re-taken there, confirm-only:
the tier is asked again and the load is dropped as `tier_lost_prefix` if it
no longer holds what the spec promised. That is one lookup per load
actually dispatched.

The MP client's `lookup` now returns `None` on deadline expiry instead of
0: "the worker did not answer in time" is not a hit of zero and must not be
remembered as one.

Applied to both `ChunkedOffloadSchedulerBase` and `DSV4OffloadScheduler`.
The latter releases its pins each step, so it always answers
`_ensure_lookup_pin` with a fresh lookup.

Co-authored-by: perzhang <perzhang@amd.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* ci: check container model mount in atom test (#2372)

* perf(v41): page the index plane at the candidate block length (#2377)

* perf(attention): reduce V4 fused decode normalization overhead

Normalize once per head and improve occupancy for nonquantized D512 32-head tiles. Cover ragged CSR, sinks, quantized KV, and changing graph inputs with an independent reference.

* perf(attention): specialize gfx950 fused decode with shared KV panels

Use explicit MFMA layouts for BF16 D512 head tiles and reuse KV panels through LDS. Preserve the existing split and fallback paths; cover graph inputs, tile boundaries, and large KV addresses.

* Generalize V4 fused decode across CDNA3 and CDNA4

* perf(v41): page the index plane at the candidate block length

A Reindex layer scored every column of its context and then drove the ones
outside the candidate source's blocks to -inf. The blocks it keeps are exactly
a block table -- if the plane is paged at the length they are picked in -- so
the same selection is reachable by scoring `candidate_topk_blocks` blocks and
never addressing the rest.

`V41PoolGeometry.index_block_rows` declares that length, and the backend sets
it to `candidate_block_size`. The plane's bytes do not change: data and scales
are packed with nothing between them, so a row's share is 132 B at every block
length, and only the granularity at which the two regions interleave moves.
aiter supports the short page from ROCm/aiter#5749; a GPU whose scorer cannot
page that short refuses when asked, which is not a silent wrong answer.

`restrict_to_candidates` has no caller left. It is kept, renamed, as the judge
of the compaction: restating it in the test would be the same addressing
written twice.

Two conflations came out with it:

* `index_write` spelled the MFMA tile width, the shuffle group and the block
  length with one name. All three were 16 while 16 was the only page the
  scorer took. A block of 8 shuffles in groups of 8 by 16 and tells them apart;
  at 16 the new form is byte-for-byte the old one.
* a hub id reached `CheckpointReader` as a path, so V4.1's Engram loader built
  `deepseek-ai/DeepSeek-V4.1-Flash/model.safetensors.index.json` and reported a
  missing checkpoint for a name nobody had resolved. `local_model_dir` is now
  the one place that turns either spelling into a directory; the three probes
  that answered "no such tensor" for every remote checkpoint go through it too,
  and the reader refuses a non-directory by name.

Gates: unit suite 8609 passed / 3 failed, the 3 reproducing on origin/main.
GSM8K 1319 @ 3-shot, DSpark-5, TP4, --level 0: flexible 0.9227 / strict 0.9234
against a 0.9224 +- 0.005 baseline. On a 127k-token context, 225 in-forward
comparisons of the two paths over 4 ranks disagreed on nothing -- that is the
gate for the regime GSM8K cannot reach, where the candidates are a strict
subset. Scoring is 0.80% of a decode step's GPU time in a 100k/64/conc-8 trace,
which is this optimization's ceiling there; the prefill share is 6.5%.

What the change buys is not measured. The trace carries no per-layer label, so
calls cannot be attributed to producer or consumer, and an earlier attempt to
split them by duration was wrong: a compacted call in prefill still scores
16384 columns over 16384 rows and is not short. gfx1250 is untested.

* [DCP] Allow query replication (QREP) with MTP (#2290)

* [DCP] Allow query replication (QREP) with speculative decode; fix stale sparse+MTP docs

qrep_unsupported_reason() unconditionally disabled QREP whenever
speculative_config was set, on the suspicion that QREP's full-group-head
q_out might conflict with the cprr round-robin MLA kernel MTP's qlen>1
verify path uses. Auditing the original commit and the two code paths shows
this was first-cut caution ("not yet wired"), not an observed conflict:
QREP only changes how q_out is produced (load-time replicated q_proj vs. a
runtime AllGather), and cprr's mask keys off KV-side round-robin position,
with no dependency on q_out's provenance.

Validated on GLM-5.2-FP8 (sparse DSA), TP8/DCP8, fp8 KV, MTP=3, gsm8k
nshot=20, same commit, only enable_query_replication differing:
  QREP off: flexible-extract 0.9492 +/- 0.0060, strict-match 0.9492 +/- 0.0060
  QREP on:  flexible-extract 0.9469 +/- 0.0062, strict-match 0.9477 +/- 0.0061
Delta is within combined stderr; both arms completed all 1319 requests
without error.

Also fixes docs/context_parallel_guide.md, which still claimed sparse DSA +
DCP + MTP was rejected by an assert -- that restriction was already lifted
(the sharded index cache's candidate exchange handles qlen>1 directly) but
the docs were never updated to match.

Dense MLA + MTP + QREP was not independently re-validated this round; the
mechanism argument applies equally but flagging the gap for reviewers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* [DCP] Gate QREP per-layer on actual q_proj width, not just the config flag

Removing the speculative_config gate (previous commit) made QREP reachable
for every model that runs under DCP, but only DeepSeek/GLM's decoder layer
and Kimi-K3's main model actually build q_proj with qrep_tp_override. Three
q_proj construction sites never opted in and would have silently misread a
plain per-rank-width q as the wide QREP layout, or crashed:
  - atom/models/eagle3_deepseek_mla.py (eagle3 draft, its own MLAModules)
  - atom/models/kimi_k3_dspark.py (DSpark draft, its own MLAModules)
  - atom/models/glm5_next.py (GLM-5.3-Flash; q_proj wrapped in _ZeroRopePad,
    which has no .weight at all)

All three used to be protected incidentally by the speculative_config gate
(eagle3/DSpark always set it; GLM-5.3 + MTP would too), so removing that
gate without this fix would have made them reachable and broken.

Fix: MLAAttention now…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants