Repository navigation
[MiniMax-M3] Route the paged decode to aiter's FlyDSL kernel with work planner - #2366
Conversation
`PA_DENSE_SPLIT_MAX = 32` is not a tuned value. It is the smaller of two
unrelated bounds that happen to agree near there: production aiter does not
build the C++ PS reduce past 64 and has no working fallback under it, and
`temporary_output` is bf16, so each extra split adds a round trip through the
PS combine.
aiter #4332 removes the first bound. Deciding whether the second one still
binds needs the cap measured against the shipping value on the same tree,
which a constant cannot express. `dense_decode_splits` computes
`min(PA_DENSE_SPLIT_MAX, cdiv(128, running_bs))`, so the cap only binds at
`running_bs <= 2` -- the difference is invisible without an A/B at that shape.
Default stays 32 and behaviour is unchanged. Raising it on production aiter
still fails at engine init rather than at runtime.
`int(os.getenv(...) or 32)` rather than a getenv default: an exported-but-empty
variable would otherwise raise `int('')` at import and take the engine down
before it starts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n env
ATOM_PA_FLYDSL=1 sends the paged decode to aiter's FlyDSL implementation
(aiter PR #4332) where FlyDSL's domain covers the call, and to gluon
otherwise. A measurement switch, not a shipping default: the two take the
same arguments and compute the same thing, and the env exists to A/B them
without maintaining two ATOM trees. Default 0, so nothing changes until an
operator opts in on an aiter that carries #4332.
_flydsl_pa_decode_num_seqs mirrors the kernel's own validation -- alibi,
sinks, externally quantized FP8 queries, ps=False, a positive sliding window,
a non-256 tile, a partition count outside [1, 256], an unsupported head dim,
and any cache that is not [nb, Hkv, D//16, page, 16] at 1 byte per element are
all hard rejects in aiter, not preferences. Enabling the env therefore cannot
turn a working configuration into an exception.
The returned count is the substance. ATOM pads two axes independently:
context_lens to running_bs for CUDA-graph identity, with
[scheduled_bs:running_bs] zeroed, and q to running_tokens for the MoE
all_gather. forward_context.Context stores both "because the ratio is not
always max_seqlen_q" and asserts no rectangle between them. gluon absorbs the
mismatch by deriving its batch as q.shape[0] // query_length and letting the
surplus zero-length rows do no work; FlyDSL demands the rectangle and raises
ValueError: query.shape[0] (12) must equal
context_lengths.shape[0] * query_length (4 * 4)
on an ordinary step. Recovering the count the way gluon does and slicing the
five per-sequence arguments to it hands FlyDSL the same rectangle, dropping
exactly the zeroed tail. All slices are views on dim 0.
The gate is deliberately not restricted to max_seqlen_q == 1. The dense path
is where FlyDSL's headroom over gluon lives, it runs max_seqlen_q ==
num_spec + 1, and FlyDSL carries a query_length == 4 MTP4 grid split tuned for
exactly that shape.
ATOM_PA_FLYDSL_PLAN=1 additionally opts into aiter #5546's GPU work planner.
The plan and its scratch are allocated once and only refreshed: allocation is
illegal inside graph capture and a captured graph bakes in the pointers, and a
planned call does not use the caller's static buffers -- planned partials are
packed as [kv_heads, capacity, query_rows(, D)] against the static API's
[num_seqs, kv_heads, partitions, query_rows(, D)]. The planner refuses batches
past 4096 while M3's sparse call site folds query tokens into num_seqs and
reaches 32768 on a prefill-as-decode step, so those fall back to the static
path rather than raising.
Measured negative on this workload and defaulted off: against the same tree
with the planner off it costs 3.86% at conc 1, 11.50% at conc 10 and 18.28% at
conc 20. The cause is not batch uniformity -- 31% of decode steps have max/min
KV length >= 4 -- but the split count, which plan_pa_decode and pa_decode
share: dense_decode_splits gives 16 at running_bs 5 and 8 at running_bs 24,
and at those counts a batch skewed 2.5x and one skewed 120x both measure
1.00x. #5546's own conclusion is that the data "support using dynamic plans
for uneven KV work, not enabling them universally".
Both routes are logged once per shape signature. A run where the env is set
but every call still lands on gluon is otherwise indistinguishable from one
where FlyDSL simply did not help.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…and keep it off the sparse call sites `_flydsl_work_plan` handed `plan_pa_decode` the static split count from `dense_decode_splits`, overwriting that argument's own default. The two are different quantities: `get_recommended_splits` gives every request the same count and documents itself as "not a variable-work scheduler", while the plan's `max_partitions` is a per-request ceiling it then divides under a workgroup budget. Clamping the ceiling to the static count leaves a long request with the same share as a short one, which is the planner's entire mechanism. Read back from `plan.reduce_info`: a 5-request batch at ceiling 16 splits [16,4,4,4,4] with the long request pinned at the ceiling, and at the default 256 it splits [256,20,21,20,21]. `ATOM_PA_FLYDSL_PLAN_MAX` now carries that ceiling, 0 meaning "omit the argument and take aiter's default". Do not read the ceiling off aiter's unit test: it is the only caller of `plan_pa_decode` in that tree and it binds the two deliberately, building the plan from the same count it hands `pa_decode` so the paths must agree numerically. That makes it a correctness test which cannot see the conflation. The planner was also a global switch inside `run_pa_decode_gluon`, so enabling it reached the two sparse call sites as well -- 57 of the 63 `pa_decode` calls per step. Their context is a fixed topk window, so there is no unevenness to rebalance and only the refresh and the task packing remain. On uniform batches the planner is a loss that the ceiling cannot rescue: 0.56x at B8/257 and 0.84x at B200/200000, flat from ceiling 4 through 256. `allow_work_plan` moves that choice to the caller, where the split policy already lives, and only the dense site opts in -- the same boundary `dense_decode_splits` already draws. A plan build now logs its shape once per cached key, which is the only evidence that the planner went where it was meant to rather than being assumed to. Measured at conc 20 on the agentic trace (one operating point), single-user output throughput 157.19 -> 178.49 tok/s and ITL 7.97 -> 6.96 ms, against 139.20 for the same arm before these changes. Prefix hit 96.56 vs 96.63, input length +1.6%, request count +1.0%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ur env switches into one
The plan is a function of `context_lens` alone, so every layer of one forward
wants the same one. Building it inside the attention op re-ran the planner
kernel for each of M3's three dense layers, and the module-level cache that
existed to paper over that had to key itself on shape to stay correct. It now
lives where the rest of the per-forward metadata is built: `prepare_decode`
attaches it, `prepare_mtp_decode` refreshes it for each draft pass (their
context lengths differ by a token, so reusing the target's plan would point the
kernel at the wrong KV ranges -- wrong output, not merely slower), and the op
reads it off `get_forward_context().attn_metadata`.
Measured before the move: 6 planner kernels per step, and `envs.__getattr__` on
all 63 `pa_decode` calls rather than the 6 that could use a plan. After: 4 and
1. The env read is the larger of the two -- one `__getattr__` costs ~0.97 us and
63 of them are ~0.5% of a 12.8 ms step.
Scratch allocation stays in the op. It is sized from the query tensor the op is
holding and happens once per shape, so only the per-step work moved out.
The env surface collapses to `ATOM_PA_FLYDSL_PLAN`, now on by default:
* ATOM_PA_FLYDSL is gone. It was a measurement switch for A/B-ing FlyDSL
against gluon without two trees; FlyDSL is the decode path. The capability
check that falls back to gluon for shapes outside FlyDSL's domain stays --
it mirrors the kernel's own validation and is not a switch.
* ATOM_PA_FLYDSL_PLAN_MAX is gone, and the call now omits `max_partitions`
entirely so `plan_pa_decode` takes its own default. The one time this tree
chose the ceiling it chose the static split count, which clamps every
request to the same share and removes the planner's mechanism.
* ATOM_PA_DENSE_SPLIT_MAX is gone and PA_DENSE_SPLIT_MAX is a constant again.
With the planner on, a planned call is told `plan.max_partitions`, so the
cap no longer sets the production partition count; it survives for the
gluon fallback, where both halves of its bound still hold. Raising it would
only enlarge the static scratch a planned call never reads.
* ATOM_PA_FLYDSL_PLAN defaults to 1. On the agentic trace, SA-convention
interactivity (1/itl_p90) is +24.6% at conc 20 and +8.4% at conc 10, and
the gain runs monotonically from the slow tail to the fast one (p10 +24.6%,
p50 +19.3%, p90 +5.0%) -- it lifts the floor, which is the end the metric
watches.
The routing log no longer hangs off the deleted env, and is unconditional: with
FlyDSL as the default path, which route a call took is worth always recording.
tests/test_paged_attention_dispatch.py: 74 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prose cut, facts kept. What stays is the four things a reader would otherwise re-derive the hard way: the ceiling must not come from the static split count, each draft pass needs its own refresh (reusing the target's plan points the kernel at the wrong KV ranges -- wrong output, not merely slower), scratch is allocated in the op because its shapes come from the query tensor, and the FlyDSL/gluon split is a capability check rather than a switch. The measured numbers move to the one place they inform a decision: why the planner defaults on. Also fixes a contradiction left by the previous commit. `run_pa_decode_gluon` still described itself as a "MEASUREMENT SWITCH, not a shipping default" after FlyDSL had become the path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review found the previous commit's premise broken: decode runs from captured
graphs (cudagraph_mode=FULL), and `build_for_cudagraph_capture` never attached
a plan, so the op saw None at capture time and the STATIC path is what got
recorded. Every later refresh then worked on a graph that does not read it --
no error, no slowdown, just a planner that quietly does nothing and an
end-to-end number that looks fine.
The capture flow (model_runner.py) is what makes the fix possible:
3857 build_capture(bs) outside the graph -- first plan allocated
3913 self.model(...) warmup, eager; the op sees the plan
3969 with torch.cuda.graph(...) capture; the plan is already there
So the plan is attached in `build_for_cudagraph_capture` too, and the builder
keeps one per (batch, kv_heads, device) instead of a single slot. The single
slot was the more dangerous half: `build_capture` runs once per capture-ladder
size, so the second rung replaced the first rung's plan while the first rung's
graph had already baked in its pointers.
Two more from the same review:
* The plan was built from `context_lens[:scheduled_bs]` while the op derives
its own batch as `q.shape[0] // max_seqlen_q`, which is `running_bs`.
aiter validates `reduce_info.shape == (num_seqs, 2)` and raises, so any
step whose batch was not exactly on the ladder would have killed the
worker. `context_lens` is already `running_bs` long with the tail zeroed.
* `prepare_mtp_decode` dropped the refresh result and relied on the returned
object being the same one. It now publishes through `workinfos`, which the
caller splats into the metadata, so a None result clears a stale plan
rather than leaving it in place.
`flydsl_plan_matches` is extracted rather than inlined so the guard is
testable: a plan built for another batch now falls back to the static path
instead of reaching aiter's validate.
tests/test_paged_attention_dispatch.py: 74 -> 83. Eight of the nine are
falsifiable by reverting the behaviour they describe, each reddening exactly
one test; the ninth is a source check on the capture attach, labelled as such,
because driving a real capture needs a model runner and what it guards fails
silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…y default aiter #4332 is unmerged and the kernel is not fully tested, so gluon goes back to being the shipping decode path and FlyDSL becomes the opt-in: ATOM_PA_FLYDSL=1, off by default. The two gates run in series -- the env, then the capability check that mirrors the kernel's own validation -- and the `and` short-circuits, so with the env off the check does not run. Shapes outside FlyDSL's domain still fall back to gluon either way. The work planner (aiter #5546) only feeds the FlyDSL kernel, so it now requires ATOM_PA_FLYDSL=1 as well. Without that, a plan would be built and refreshed every step for a kernel that never reads it, and the "flydsl work plan" log line -- what an A/B arm is checked with -- would show up on a run that is entirely gluon. Tests 84 -> 86: the default must be off, the dispatch must consult the env before the capability check, and the builder must not plan with FlyDSL off. Each of the three goes red on its own reversion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The parameter now matches the env that controls it, ATOM_PA_FLYDSL_PLAN. "work plan" is aiter's own term for the object, but on its own it does not say the plan belongs to the FlyDSL path -- which matters now that gluon is the default and FlyDSL is the opt-in. Comments trimmed throughout: the _flydsl_pa_decode_num_seqs docstring keeps why it returns a count rather than a bool (ATOM pads its sequence and row axes independently, so the two disagree by a padding slot on an ordinary step; gluon absorbs that, FlyDSL raises) and drops the shape algebra and the verbatim traceback. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
Review — FlyDSL pa_decode routingReviewed The routing design is the right shape — a narrow gate in front of an opt-in kernel, with a static fallback behind it — but two things need to be resolved before this can land, and they are of different kinds. The first is blunt:
The second is structural:
Provenance: [verified] means the deciding code was read in the PR-head worktree or the command was run there. [reported] means the shape matches the code but was not traced end to end. 1. The routed path cannot run [verified]
With The argument is also redundant — the gate already rejects The reason this shipped green is worth fixing alongside it: every wiring test monkeypatches 2. MTP draft attends over a stale plan, and
|
…binding Two gates in series guard the FlyDSL paged decode: ATOM_PA_FLYDSL is the opt-in, and _flydsl_pa_decode_num_seqs mirrors aiter's own rejects so an unsupported call falls back to gluon instead of raising inside the kernel and killing the worker. The gate was missing several of those rejects -- q/output dtype and shape agreement, both head_dim stride checks, the q-heads-over-kv-heads divisor, the v_cache dtype, and contiguity of the two caches and the two per-sequence arrays. Plans are now only ever minted during cudagraph capture. aiter's planner takes batch as a tl.constexpr, so a runtime batch that no graph replays would cost a cold kernel specialization and a plan nothing can free; the static path is the right answer for it. Runtime only refreshes in place. The one-shot "no plan for this batch" notice waits until a capture has happened, since before that every call legitimately lands there and the notice would be spent on a step that says nothing. max_partitions stays unset, at plan_pa_decode's own default: every number measured on this branch was measured there, and the review's suggestion to clamp it to 64 is a perf claim nothing here backs. Measuring instead, on the planned path, across nine cells spanning the trace's p50/p90/p99 contexts at batch 1/8/24: drift from an fp32 reference does not grow with the split count anywhere, and the ceiling only ever binds at batch 1. The plan's capacity is min(batch * max_partitions, max(batch, 2 * CU / kv_heads)) = 512 here, so each sequence's share is about 512/batch and a 64 ceiling would have changed nothing above batch 2 while capping the one place the planner has the most to rebalance. numa_utils and the mooncake connector now read ROCR_VISIBLE_DEVICES first. ROCR filters and renumbers before HIP indexes the result, so setting both cut the visible device count twice; dropping HIP from the recipe left the NUMA binding reading nothing and put all four ranks on one node. Tests go from 86 to 127. The call-site contract is now checked by walking the AST of all three builders and comparing the value of `create`, not its presence in one file -- a sed that moved `create=True` onto a runtime path had left every test green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Main migrated single-node AgentX to native srt-slurm (#3428) and deleted benchmarks/single_node/agentic (#3461). Accept the script deletion and move this PR's change into the declarative recipe: bump the engine image to rocm/atom-dev:nightly_202609231248 (ROCm/ATOM#2366) and set ATOM_PA_FLYDSL=1 and ATOM_PA_FLYDSL_PLAN=1. The bash-only GPU-mask fix no longer applies. The changelog entry follows the default eval policy. 中文:main 已将单节点 AgentX 迁移到 srt-slurm 并删除旧 bash 脚本。本 PR 改为在 YAML 配方中更新镜像至 nightly_202609231248 并启用 ATOM_PA_FLYDSL / ATOM_PA_FLYDSL_PLAN;GPU mask 修复随旧脚本一起移除,changelog 采用默认 eval 策略。 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Move the MiniMax-M3 MI355X ATOM AgentX srt-slurm recipe to rocm/atom-dev:nightly_202609231248, which contains ROCm/ATOM#2366, and set ATOM_PA_FLYDSL=1 and ATOM_PA_FLYDSL_PLAN=1. Drop TP4 C32 to bound the sweep. 中文:将 MiniMax-M3 MI355X ATOM AgentX srt-slurm 配方镜像更新为包含 ROCm/ATOM#2366 的 rocm/atom-dev:nightly_202609231248,并设置 ATOM_PA_FLYDSL=1 与 ATOM_PA_FLYDSL_PLAN=1;移除 TP4 C32 以控制 sweep 规模。 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…plans aiter #5809 fixes a plan's capacity from workgroup_budget when it is built, and target and draft tune to different budgets (B16: 512 vs 256). Since #2366 both share one capture plan per (batch, kv_heads), refreshed in turn, so one of them could never get its budget; and a budget hit with a different capacity made the op mint a second plan inside the captured graph. - Key capture plans by (batch, kv_heads, budget, device). Equal budgets keep the single shared object of #2366. - Builders tag attn_metadata with a role: target from prepare_decode and the capture builder, draft from prepare_mtp_decode. - During the eager warmup before capture, the op resolves the budget once per (role, batch) from the real tensors; on a capacity mismatch it asks the builder to adopt the tuned plan and writes it back to the metadata, so the capture records it. During capture and once decided, the op uses the plan as given and never looks up a budget or builds another plan. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…get plans The latest aiter #5809 removed pa_decode_tuning (get_cached_budget, make_shape, storage_key) and ships no tuned budgets, so importing it fails at the first FlyDSL decode. - Build every explicit plan with aiter's default workgroup budget (2*CU). Measured on the agentic context distribution, the default was the fastest budget at every dense and draft batch checked. - Drop the per-role budget plans (role tags, adopt_flydsl_budget, plan_owner): target and EAGLE draft share one capture plan again, as in #2366. - Keep the once-per-step sparse refresh, the 4096-row gluon fallback and the FlyDSL-only context bound. - Tests: an explicit plan needs no tuning module and passes no budget (red if the old import comes back); the sparse refresh tests move into TestExplicitPlan. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…plans aiter #5809 fixes a plan's capacity from workgroup_budget when it is built, and target and draft tune to different budgets (B16: 512 vs 256). Since #2366 both share one capture plan per (batch, kv_heads), refreshed in turn, so one of them could never get its budget; and a budget hit with a different capacity made the op mint a second plan inside the captured graph. - Key capture plans by (batch, kv_heads, budget, device). Equal budgets keep the single shared object of #2366. - Builders tag attn_metadata with a role: target from prepare_decode and the capture builder, draft from prepare_mtp_decode. - During the eager warmup before capture, the op resolves the budget once per (role, batch) from the real tensors; on a capacity mismatch it asks the builder to adopt the tuned plan and writes it back to the metadata, so the capture records it. During capture and once decided, the op uses the plan as given and never looks up a budget or builds another plan. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…get plans The latest aiter #5809 removed pa_decode_tuning (get_cached_budget, make_shape, storage_key) and ships no tuned budgets, so importing it fails at the first FlyDSL decode. - Build every explicit plan with aiter's default workgroup budget (2*CU). Measured on the agentic context distribution, the default was the fastest budget at every dense and draft batch checked. - Drop the per-role budget plans (role tags, adopt_flydsl_budget, plan_owner): target and EAGLE draft share one capture plan again, as in #2366. - Keep the once-per-step sparse refresh, the 4096-row gluon fallback and the FlyDSL-only context bound. - Tests: an explicit plan needs no tuning module and passes no budget (red if the old import comes back); the sparse refresh tests move into TestExplicitPlan. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…t work plans) (#2514) * fix(attention): adapt FlyDSL PA decode to aiter 5809 * fix(attention): give target and draft their own tuned FlyDSL capture plans aiter #5809 fixes a plan's capacity from workgroup_budget when it is built, and target and draft tune to different budgets (B16: 512 vs 256). Since #2366 both share one capture plan per (batch, kv_heads), refreshed in turn, so one of them could never get its budget; and a budget hit with a different capacity made the op mint a second plan inside the captured graph. - Key capture plans by (batch, kv_heads, budget, device). Equal budgets keep the single shared object of #2366. - Builders tag attn_metadata with a role: target from prepare_decode and the capture builder, draft from prepare_mtp_decode. - During the eager warmup before capture, the op resolves the budget once per (role, batch) from the real tensors; on a capacity mismatch it asks the builder to adopt the tuned plan and writes it back to the metadata, so the capture records it. During capture and once decided, the op uses the plan as given and never looks up a budget or builds another plan. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(attention): refresh the sparse FlyDSL plan once per step; review fixes - MiniMax-M3 sparse decode: every sparse layer of a step selects the same per-row lengths (0 differences over 9,000 C15 steps x 56 layer pairs), so the shared runtime plan is refreshed by the first sparse layer of the forward only -- one planner launch per step instead of 57, in eager and in the captured graph. Opt-in per call site (refresh_once_per_forward); dense eager and plugin callers still refresh every call. - Route calls with more rows than aiter's planner accepts (4096) to gluon: every FlyDSL call needs a plan now, and a long sparse prefill folded into rows used to raise inside the worker. - Resolve the FlyDSL context bound only on the FlyDSL path, so a deployment with FlyDSL off pays nothing and gluon gains no new way to raise. - GDN capture builder and Qwen4 draft metadata now carry the plan role and builder, so capture never looks up a budget there either. - Skip the budget tests when aiter lacks pa_decode_tuning instead of erroring. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(flydsl): state the aiter #5809 requirement and the new PLAN=0 meaning This branch needs aiter with #5809 and is merged after it; nothing probes for it at runtime, so the docs now say ATOM and aiter must be upgraded together. Also correct what ATOM_PA_FLYDSL_PLAN does since #5809: there is no static FlyDSL path, PLAN=0 routes to gluon, dense plans take their budget from the offline tuning cache, and the sparse sites use runtime plans. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(attention): use aiter's default FlyDSL budget; drop the tuned-budget plans The latest aiter #5809 removed pa_decode_tuning (get_cached_budget, make_shape, storage_key) and ships no tuned budgets, so importing it fails at the first FlyDSL decode. - Build every explicit plan with aiter's default workgroup budget (2*CU). Measured on the agentic context distribution, the default was the fastest budget at every dense and draft batch checked. - Drop the per-role budget plans (role tags, adopt_flydsl_budget, plan_owner): target and EAGLE draft share one capture plan again, as in #2366. - Keep the once-per-step sparse refresh, the 4096-row gluon fallback and the FlyDSL-only context bound. - Tests: an explicit plan needs no tuning module and passes no budget (red if the old import comes back); the sparse refresh tests move into TestExplicitPlan. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(attention): refresh the sparse FlyDSL plan per step, inside capture too Review fixes on top of the default-budget adaptation: - Sparse decode refreshed its plan once per forward, marked on ATOM's ForwardContext. Capture runs the warmup and the capture forward on one context and stream, so the capture skipped the refresh and every graph replay ran on the warmup's dummy lengths. The vLLM plugin never sets that context for MiniMax-M3, so there the mark outlived the step and eager steps stopped refreshing after the first. The mark now lives on the step's decode metadata (plan_step_owner, passed by the native and vLLM sparse decode), keyed by capture state; without an owner every call refreshes. Step-scoped and per-call plans no longer share a cache entry. - Dense runtime plans take aiter's default partition ceiling, like the capture-owned plans. Sparse and the vLLM bridge keep their static split count (plan_partition_cap). - A context bound below the query window routes to gluon instead of raising. - Pass the clamped bound to aiter; drop the dead no-plan scratch branch; fix stale comments and the PLAN env docs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * refactor(attention): drop ATOM_PA_FLYDSL_PLAN aiter #5809 left FlyDSL no planner-less path, so ATOM_PA_FLYDSL=1 ATOM_PA_FLYDSL_PLAN=0 could only route to gluon -- exactly what ATOM_PA_FLYDSL=0 does. Gate the builder and the op on ATOM_PA_FLYDSL alone, remove the env and its docs row (folding the plan behaviour into the ATOM_PA_FLYDSL row), and warn once at startup when the old variable is still set, so a leftover PLAN=0 does not silently change meaning. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * style: black and ruff fixes for the FlyDSL plan changes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs: trim FlyDSL plan comments and the ATOM_PA_FLYDSL docs row Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs: final review nits for the FlyDSL plan changes Drop the stale "static path" note in the GDN capture builder, scope the builder-minting test docstring to the builder, inline the leftover flydsl_enabled alias, and say in the ATOM_PA_FLYDSL row which callers use capture-owned vs runtime plans and which plan ceiling each takes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(attention): address review on the FlyDSL runtime plans - Key runtime plans by TBO ubatch id too: ubatches share one compute stream, so two equal-sized ubatches could refresh one step-scoped plan with each other's lengths. - Refresh the sparse plan once per step only when sparse_local_block >= 1 (tail block forced, so every sparse layer has the same lengths); otherwise refresh on every call. Native and vLLM sites. - Probe once for aiter #5809's FlyDSL API (required keyword-only work_plan and max_context_length); an older aiter now falls back to gluon with a warning instead of raising on the first decode. - Stop passing max_seqlen_k as the dense scheduling bound: it differs between graph and eager and selects the kernel specialization. Both now use the block-table bound. - Drop the dead sliding-window handling from runtime plans (the gate rejects windows), say "sequences" for the 4096 planner limit, fix stale comments, and note in the recipe which configuration its FlyDSL-vs-gluon numbers were measured on. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat: integrate gfx1250 FP4 MQA indexer (#2334)
* feat: integrate gfx1250 fp4 MQA indexer
* perf: use qlen1 FP4 MQA for decode
* style: format gfx1250 FP4 pool layout
* test: skip OPUS metadata tests without AITER
* Add Kimi-K3 (hybrid) LMCache KV offload to the vLLM plugin path (#2268)
* Add Kimi-K3 (hybrid) LMCache KV offload to the vLLM plugin path
AtomLMCacheOffloadConnector could not serve Kimi-K3. Two guards rejected it at
startup, and behind them sat a correctness problem: K3 is hybrid, so vLLM builds
two KV cache groups (MLA full attention plus KDA recurrent state), and a
restored MLA prefix is only valid if the KDA state at the same token boundary is
restored with it. Half a restore is not a crash and not a log line -- it is
wrong output.
The two groups are moved by two different mechanisms, and the asymmetry is
forced, not chosen. The MLA group keeps the existing DenseKVByteCodec path
untouched. The KDA group cannot be read positionally at all: in
--mamba-cache-mode align a mamba block table is not append-only (a superseded
state block is freed and nulled, speculative blocks relocate in place), so
indexing it by token // block_size can persist a null, freed, or live
speculative block under a valid prefix hash. The only safe source is vLLM's
explicit hand-off, which names the exact block holding a committed boundary
state. The pinned vLLM 0.28 ships it flat on the scheduler output as
SchedulerOutput.partial_tail_offloads, fed by
KVCacheManager.take_partial_tail_offloads; 0.29 moves the same
{req_id: [(group_id, block_id, boundary_tokens)]} payload under
kv_connector_block_state.boundary_state_offloads. step_boundary_offloads reads
whichever spelling the running vLLM provides, so the connector keeps working
across that upgrade rather than going silently hit-less on one side of it. The
hand-off is taken off the KV cache manager once, while the step is built, so it
is consumed in the step it arrives. Each accepted boundary is pinned via the
bound GPU block pool and released on rank quorum.
Correctness of the pair is enforced on lookup, not on save: the reported
external hit is capped at the largest chunk boundary the state index still
claims, so a state that was never stored or was evicted shortens a prefix
instead of corrupting it. That removes the need for any cross-group save
parking. A KDA miss additionally invalidates the attention blocks the dense leg
filled, so vLLM cannot cache an MLA prefix whose state never arrived.
Hybrid models also require vLLM's hybrid memory allocator, which vLLM
auto-disables for a connector that does not declare SupportsHMA -- without that
declaration K3 plus --kv-transfer-config does not mis-save, it does not boot.
The connector now declares it and implements request_finished_all_groups, which
retires request_finished for every model. SupportsHMA, bind_gpu_block_pool and
request_finished_all_groups are all present in the pinned 0.28
(atom/plugin/vllm/vllm-version.env, 2cf0a6915 = v0.28.0), so nothing here waits
on the 0.29 upgrade.
supports_divergent_local_hybrid_hits stays False: the reconciled cross-group
boundary vLLM then reports is the conservative one, and this connector caps the
external hit on its own.
Stays on the pinned LMCache version. The 0.5.5 upgrade was permitted, not
required, and nothing here needs it; measured on the GLM-5.2 arc, 0.5.5rc4 lost
to 0.4.5 by 5.72%, with ~97% of chunk cost host-side.
Verification: the CI unit scope (--ignore=tests/plugin) is 3 failed / 5519
passed with the 38 new tests included, and those three failures reproduce
unchanged on origin/main in a clean worktree, so none of them is this change.
Seven further modules (tests/diffusion, tests/test_attn_family.py) cannot be
collected on this host at all -- Python 3.10 has no typing.Self or enum.StrEnum
-- which is the interpreter, not the change; CI runs a newer one. The tests were
mutation-checked: ten real degradations of the earlier work (uncapped hit, each
collect_stores filter removed, quorum of one, indexing despite a failed rank,
boundary_hash off-by-one, dropping an unresolvable load, silently dropping an
unmapped layer) each turn them red, and so does dropping either spelling from
step_boundary_offloads, while reordering the two spellings -- an equivalent
mutation -- leaves them green. black and ruff pass on every touched file.
Nothing has been run on hardware yet: boot, two-pass accuracy against the
0.9507 / 0.9500 baseline, and hit rate are still outstanding, and the recipe
says so rather than implying measured numbers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Fix three K3 offload defects found on the first hardware boot
The connector had never been run against a real engine. Three defects showed
up, two of them loud and one that would not have shown up at all.
1. Multi-group recurrent state. vLLM does not build one KV cache group per
layer family; it builds equal-sized groups. K3's attention bucket is 29
layers and its KDA bucket is 69, so the group size is 29 and the recurrent
state lands in THREE groups (23 layers each), not one. The recurrent leg now
reads the grouping off the engine instead of assuming a single group, and
treats all mamba groups as one logical state: one token boundary, one block
id per group, stored as those groups' bytes concatenated in group order.
None of the groups is restorable alone.
2. `split_kv_caches_by_group` returns a list indexed by group id, and
`_init_kda_tier` called `.get()` on it, so every worker died with
`'list' object has no attribute 'get'`. Extracted `gather_group_tensors`,
which is what the two functions were missing: nothing in the suite fed one
to the other, so the seam had no coverage at all. It has four tests now,
including the ordering contract -- vLLM's own `layer_names` order, never
`sorted()` -- because a page image is readable only by a run that rebuilds
the same order.
3. The dense leg strided every segment by a token, not by a block. This one is
silent. `DenseKVByteCodec` derives each segment's per-block byte stride as
`numel // num_blocks`, and `register_kv_caches` took `num_blocks` from the
registered tensor's leading dimension. That is the block count only when the
attention backend's kernel block size equals the group's block size. ATOM's
MLA backend declares `get_supported_kernel_block_sizes() == [1]`, so vLLM
allocates one row per token and expands manager block id `b` into kernel ids
`b*n .. b*n+n-1`, while the ids a KV connector receives stay manager ids. On
K3 -- 1536-token blocks -- that made the stride 1536x too fine: every block
id still in range, every transfer reported success, bytes restored from the
wrong rows, nothing logged. `DenseKVByteCodec`'s own docstring says the
leading dimension is not the block count on token-major MLA.
`resolve_block_count` now takes the count from `KVCacheConfig.num_blocks`
and reconciles it against the tensor: the leading dimension must be a whole
number of blocks, and the implied kernel blocks per block must divide the
block size. It is an identity wherever the two already agree, which is every
model shipped on this path so far (GLM-5.2 measured `num_blocks=8192`
against `tail_shape=(64, 576)`), so the mismatch is K3's alone.
No reshape is needed: a manager block's kernel blocks are contiguous and
consecutive, so its bytes are one unbroken run either way.
The recurrent leg bounds-checks a block id against its own tensors and so would
have failed loudly on the same mistake. The dense leg has no such check, which
is why (3) needed a test rather than a boot.
Boot verified on 8xMI355 TP8: three mamba groups on 0,1,2 and attention on 3,
hybrid memory allocator left on, deterministic smoke answer correct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Record the validated K3 boot, and make its log readable
The boot passed on 8xMI355 TP8, so the recipe's Status section is now a
measurement rather than a requirement. Three things it said were wrong, and
each one was wrong in the direction that reads as "fine":
- "vLLM builds two KV cache groups". It builds equal-sized groups, so K3's 29
MLA and 69 KDA layers come out as four: one attention group of 29 and three
mamba groups of 23. The recipe's own verification greps were written against
the two-group picture.
- "LMCACHE_CHUNK_SIZE=128 # must equal --block-size". The block size is not the
one you pass. vLLM raises it to 1536 so the attention page is at least the
mamba page, and `--block-size 128` does not prevent that -- it fixes only the
requested size, and the hybrid alignment step overrides it afterwards. A
chunk of 128 is not a multiple of 1536 and the connector refuses to start.
- The "did the recurrent leg come up" grep named one line and said it appears
once per worker and once in the EngineCore. There are two different lines
from two different processes: workers print `recurrent state tier up`, the
EngineCore prints `recurrent state leg on group(s)`. The documented grep
returns nothing on a healthy boot, which reads as "the leg is off".
Status now carries the measured geometry to diff a future boot against, and
calls out the two quantities that fail silently -- `bytes_per_block`, which
must be `block_size x bytes_per_token`, and `leading dim`, which is 1536x
`num_blocks` on this backend -- plus the tier arithmetic, since at the
documented defaults the tier is 5x smaller than the HBM pool and can only
measure a null result.
`summarize_layout_id` run-length encodes the per-layer tail of the layout id
for the log: 69 identical entries, 1,877 characters on one line per worker and
36 KiB per boot, which left the grep the recipe prescribes scrolling past its
own answer. 1,877 -> 112 characters. The encoding is lossless and order
preserving, and it is deliberately NOT what `build_layout_id` folds into the
storage key -- shortening what a human reads must not shorten what separates
one layout's bytes from another's, so a test pins the two apart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Size the K3 offload measurement from the trace instead of from a guess
The tier-sizing paragraph derived the HBM pool from `GPU KV cache size`, which
is `max_concurrency x max_model_len` and not a capacity at all -- the same
quantity this file already warns is not a pool. Replace it with the pool
arithmetic that the boot line `num_blocks / Maximum concurrency` confirms
(55 blocks per request = 43 attention + 3 x 4 mamba, the mamba term independent
of max_model_len in align mode), and with the reuse-distance band measured
offline from two independent K3 agentic runs.
That measurement changes the recommendation. The band `[pool, tier)` holds only
2-3% of turns at the default 1584 blocks, so that arm could only ever return a
null result. At 1100 blocks it holds 44-45% -- and the default 20 GiB/rank tier
is already the larger end, so the experiment needs no extra host memory, where
growing the tier past the default pool would have wanted ~853 GiB pinned.
Also records why max_model_len must not be lowered to shrink the pool: this
trace's median input sequence already runs past the 65,536 window, so a shorter
one truncates the workload rather than the pool.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Measure the lane scaling within K3 instead of across models
The sizing section claimed reuse distance grows sub-linearly in lanes, on the
strength of an 8x-lanes-to-2.3x-distance comparison. That pair was a K3 run
against a GLM-5.2 run, so it varied block size, bytes per token and the
prompt-length distribution at the same time as the lane count, and credited the
product to lanes.
The same sweep that supplied the 4-lane distribution also has an 8-lane run,
which gives a within-model axis at fixed workload: distance moves x1.90 at p75
and x1.97 at p50, i.e. close to linear, against ISL distributions 7% apart. The
1-lane point in that sweep is unusable for the same reason the cross-model pair
was -- its median input sequence is 1.8x the others.
Extrapolating one doubling from the 8-lane run puts the band at 16 lanes, which
moves the default row from 2-3% to 8-16% and leaves 1100 blocks at 36-46%. The
recommendation is unchanged; the reason it is safe is now measured rather than
assumed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Store KDA boundaries vLLM's hand-off can never name
The recurrent leg reported exactly zero external hits on Kimi-K3 --
3,956,629 queries, 0 hits, over 900 seconds -- and nothing said why.
Two acceptance domains that cannot intersect. vLLM's boundary hand-off
is a *partial tail* by definition: `_cache_partial_tail_block` returns
early when `num_tokens % block_size == 0`, and earlier still when
`block_size == hash_block_size`. This planner accepts only boundaries
that are whole mamba blocks *and* chunk-aligned, because `cap_hit`
descends in chunk steps and can select nothing else. On K3 both block
sizes are 1536 (vLLM raises the attention block size to satisfy
"attention page >= mamba page", and the hash block size is their gcd),
so the hand-off is empty -- and every boundary it could ever carry is a
non-multiple of the chunk size, so widening the filter would not have
helped either.
Take the boundaries from the block pool's own content-addressed cache
instead, the map vLLM uses to serve a local mamba hit. Resolving by
hash rather than by row is what makes it safe where a positional read
is not: an align-mode mamba block that was superseded, freed, nulled or
relocated is no longer registered under that hash, so the failure the
module docstring warns about -- wrong bytes under a valid prefix hash --
cannot occur. `get_cached_block` spans every mamba group in one call and
returns nothing unless all of them are present, which is the
all-or-nothing this state needs anyway. The hand-off is still read, for
models where it does carry something.
The 900 silent seconds were the second defect. Every explanation on this
leg was `logger.debug`, `stats()` had no call site, and the plugin path
exports no `lmcache_*` metrics, so a by-construction null looked exactly
like a tier with nothing to do -- which is the trap `StateOffloadIndex`
documents in its own comment and added `stores_refused` for. Counters
now name each reason a boundary was not stored, and one INFO line a
minute prints them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [Doc] Report the measured K3 LMCache offload pair in the recipe
The LMCache section said "accuracy and hit rate: not yet run" and left the
sizing table's 1100-block recommendation as the last word. Both are now
replaced by the pair that was actually run.
Measured, 1800 s per arm, TP8, conc 16, seed 1234, pool pinned to 320 blocks
(381,300 tokens) and tier 40 GiB/rank: +10.81% req/s, +13.24% output token
throughput, TTFT p50 -59.1%, preemptions 35 -> 2. Prefill recompute falls from
64.01% to 16.27% of prompt tokens, which is the whole mechanism.
Three things are written down because they are easy to get wrong later:
- the promised / delivered / counted rulers disagree, and the promised-to-
delivered gap must be summed as sum(required - retrieved) over every rank
line. The per-event shortcut misses partial retrievals and overstated it 39%.
- GLM-5.3's start-cut sweep is not licensed here: the OFF arm drifts -17.1%
across cuts, so no steady-state row is quoted. The ratio's stability across
eight cuts is what says the full-window figure is not a window artefact.
- the sign of the tier's benefit follows tier/pool, with the 792-block pair
(-3.59%) as the counter-example.
Limits are stated rather than hidden: n=1 per arm, ISL/OSL not held fixed
across arms, host MemFree differed at launch, and the tier is deliberately
under-sized (8,847 allocation failures on the ON arm).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [Doc] Move K3 LMCache offload into its own recipe
Kimi-K3.md had grown a 340-line LMCache section that was longer than the rest
of the recipe put together, and unlike M3 and GLM-5.2 it had no file of its own
to point people at. Split it out as Kimi-K3-LMCache-Byte-Offload.md, matching
the naming the other offload recipes already use, and leave a short pointer in
Kimi-K3.md carrying the one number a reader needs to decide whether to open it.
Content is moved, not rewritten. The cross-references that said "above" and
pointed at sections no longer in the same file now name Kimi-K3.md and its
anchor. Two things are corrected in passing: the launch block declared a 20
GiB/rank tier while the measured pair ran 40, and the GLM-5.3 offload recipe is
referenced by name rather than linked, because it is not on main yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* [Doc] Align the K3 offload recipe's structure with GLM-5.3's
The split file kept the section order it had inside Kimi-K3.md, which is not
the shape the other offload recipes use. Reorganise it onto GLM-5.3's skeleton
so the two can be read side by side: Server / Sizing the tier / Client /
Measured / Related, with the "why this model needs more than the M3 path"
material moved into the intro, where GLM-5.3 puts its mirror-image claim that
no model-specific code exists.
Content is moved, not rewritten; the only lines that disappear are a duplicated
intro paragraph left behind by the split and the five retitled headings.
Two sections are new rather than moved:
- Client is now its own top-level section instead of a code block buried in
Measured, with the four load-bearing flags called out the way GLM-5.3 calls
out its own. It also records that the base launch does not pass
--enable-prompt-tokens-details, which is why aiperf's prompt-cache columns
are empty and every hit rate in Measured is read from /metrics instead.
- "Verify it booted right" keeps K3's boot geometry as a sub-section of Server,
since it has no GLM-5.3 counterpart and is a server-side check.
The "accuracy not yet run" note moves to the end of Measured, next to the
limits of what was measured, rather than sitting in the boot-check section.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* recipes: report the GLM-aligned K3 offload pair as the null it is
Replaces Measured with the pair that actually matches the launch in Server
(no --num-gpu-blocks-override, 64 GiB/rank, ATOM_SLRU read back from the
rank-0 config dump, --enable-prompt-tokens-details on).
The result is that the tier supplies 0.30% of prompt tokens and the
throughput difference is not measurable: aiperf's req/s says +0.21% and a
recomputation over each arm's record span says -3.73%, with n=1 per arm and
no noise floor, so the sign is undetermined. It is written up that way
rather than quoting the +3.08% out-tok/s.
The cause is structural, not a sizing error. With the pool unpinned vLLM
answers 85.05% of prompt tokens itself, and on the plugin path the connector
is only asked about what the pool missed, so the tier bids for the leftover
15% and wins 2.01% of it. The band model in Sizing promised 18.7%; the 62x
gap between that bound and the delivered 0.30% is now recorded next to it.
Sizing also gains GLM-5.3's cheaper cross-check (tier ~1.5x the reusable
working set): 20 distinct conversations x their largest ISL x 56,448 B =
30.5 GiB/rank, so 64 is 2.10x the set, above the rule rather than below it.
GLM-5.3's own 256 GiB/rank is 1.57x its 163 GiB set at TP4 and cannot be
copied to K3: at TP8 it would pin 2048 GiB against a ~90 GiB/rank ceiling
on this host.
Kimi-K3.md's pointer said +10.81% without saying that number needs a pinned
pool; it now leads with the default-pool null.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Keep a failed tier load from killing the engine on a hybrid model
vLLM's KV-load-failure recovery assumes a single KV cache group:
(req_block_ids,) = self.kv_cache_manager.get_block_ids(req_id)
under a `# TODO (davidb): add support for hybrid memory allocator` in
`Scheduler._update_requests_with_invalid_blocks`. Kimi-K3 is hybrid (MLA
full-attention plus KDA recurrent layers), so `get_block_ids` returns one
list per group and the unpack raises inside EngineCore. That is not a
degraded request: it is `EngineDeadError`, every in-flight request 500s,
and the server shuts down. The path is reachable exactly when this
connector is in use with `"kv_load_failure_policy":"recompute"`, which
the K3 recipe requires -- the default `fail` turns a tier miss into a
user-visible 500 instead.
Rather than patch vLLM in place, select a subclass through the public
`scheduler_cls` config knob from ATOM's platform hook. The blast radius
is then one engine's scheduler instead of every `Scheduler` in the
process, and a mismatch with a future vLLM fails loudly at boot instead
of silently reverting behaviour. Be honest about what it does not buy:
the overridden method is private either way, so the version coupling is
the same as a monkeypatch's.
Two subclasses, because `get_scheduler_cls` consults `async_scheduling`
only while `scheduler_cls` is None; once set, the named class is
returned as-is, so returning a plain `Scheduler` would silently disable
async scheduling. `select_scheduler_cls` reads the already-resolved
`async_scheduling` bool -- the platform hook runs after vLLM decides it.
The override is skipped when vLLM's own method no longer contains the
single-group unpack, so this retires itself when upstream fixes it. It
is gated on a KV connector being configured (without one no load can
fail), not on the model being hybrid: group counts are not known at
config time, and the override handles a single group exactly as vLLM
does. Selection failure is logged, never fatal.
Not unit-tested: the CI unit job has no vLLM and both the subclasses and
the platform hook need it to import. Checked instead by the sweep's
preflight, which asserts the bases, the upstream-source probe, that the
hook is actually called from `check_and_update_config`, and that both
`async_scheduling` values resolve to the right qualified name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Record the controlled-prefix working point in the K3 offload recipe
The recipe's Measured pair reports the tier supplying 0.30% of prompt tokens
and no measurable throughput difference. That is a property of the client used
there, not of an unpinned HBM pool: its reuse falls inside what HBM keeps, so
the connector only ever sees a 15% miss tail.
Re-measured at the same unpinned pool (both arms report the pool within 0.06%,
1,887,436 vs 1,886,245 tokens) with a controlled-prefix synthetic client whose
reuse distance is chosen to land in the band Sizing the tier defines, the tier
supplies ~82% and throughput gains +20.5 / +22.4 / +27.8% at conc 16 / 20 / 24
over 1800 s windows. Deltas are taken from request counts, not from aiperf's
two-decimal req/s. Every arm is n=1; there is no noise floor and the section
says so.
Three knobs differ between the two working points -- client, tier 64 -> 90
GiB/rank, and atom_head -- so nothing is isolated to one of them; the section
enumerates them rather than attributing the result to the tier.
The band arithmetic already in Sizing the tier predicts both the result and its
trend: d = 483,840 tokens sits above the reusable prefix (270,336 at conc 16,
falling to 101,376 at conc 24) and below the tier (1,711,961 tokens at 90
GiB/rank), and the lower edge drops as lanes are added, which is the direction
the gain moves.
Also qualified in place: the "factor of 62" paragraph, which read as a property
of unpinned pools; the 64 GiB/rank knee, which is that workload's; and the
pointer in Kimi-K3.md, which stated the null without its precondition.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Keep a KDA offload hit from serving another prefix's state
Copy the boundary page only after the forward and mamba postprocess
have written the accepted state, retry a boundary the pool does not
hold yet, and fail a load that does not end on a mamba block or cannot
name the attention blocks to invalidate.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Record the conc-32 ON arm and close the sweep's open prediction
The 2026-09-22 sweep left conc 32 with an OFF arm only: wait_for_gpus.sh gates
on 756 GiB of raw MemFree and the host sat at 553. The GPUs were idle; the
memory was 1,068 GiB of K3 weights that each arm's NFS re-read had left in the
page cache, measured per-file with fincore and released with posix_fadvise
DONTNEED (clean pages, no drop_caches). MemFree 467 -> 1539 and the arm ran.
It confirms the prediction the section already carried. conc 32 is where the
reusable-prefix formula goes negative, so no prefix survives between turns and
every reuse must come from the tier: tier 87.73% of prompt tokens against
81.6-82.7% at the other three points, HBM local hit 5.70% against 10.4-11.1%,
and the largest gain of the set at +32.6% (3028 vs 2283 requests over identical
1800 s windows). Recompute is what moves -- 6.57% ON against 19.96% OFF.
Also noted, so a reader does not have to re-derive it from server.log: that
arm's EngineDeadError is the driver's own SIGTERM racing AsyncLLM's output
handler at teardown, five seconds after a healthy engine log and after aiperf
had finished; was_cancelled=False and error_summary is empty.
The gain still carries no error bar (every arm is n=1) and the rising trend runs
with the fixed-pool bias run_sweep.sh documents, not against it. Both limits
are stated in the section rather than left to the reader.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: perzhang <perzhang@amd.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(decode): fold the graph's padded tail into the deferred-id kernel (#2368)
A decode step wrote `input_ids` in two launches: one Triton kernel filling
each carried-over request's span, then a `zero_()` over the rows a replayed
graph reads but the batch never scheduled. The second is a 5.6us dispatch
floor for ~1.5KB, and it fires on every step -- `bs` sits between two capture
buckets on all but the exact-fit ones.
The two regions are disjoint by construction: the tail starts where the last
request ends, so nothing orders them and one launch can do both. Programs past
`num_requests` zero `[cu[bs], width)` instead of filling a span.
`cu` stays the only statement of where requests end -- the padding programs
read `cu[bs]` themselves rather than taking a second copy of it from the host,
so the two cannot disagree. `width` is a required argument, because a caller
that forgets it leaves the padded rows carrying the previous step's ids, and
the MoE path does consume them.
`BLOCK_PAD` is one value so the tail costs one JIT specialization rather than
one per capture bucket, and the three new runtime ints are `do_not_specialize`
for the same reason.
Verified: GSM8K 1319 @ 3-shot on DeepSeek-V4.1-Flash DSpark-5, TP4, bf16 KV,
index fp8 -- flexible 0.9174 / strict 0.9166 against a 0.9224 baseline mean
(sigma ~0.005). Unit suite 8512 passed, unchanged from before the change.
* (recipe) add Kimi-K3 AgentX recipe (#2131)
* (recipe) add Kimi-K3 AgentX recipe
Document the per-concurrency ATOM serving bands for the Weka AgentX workload.
* (recipe) tune Kimi-K3 AgentX ReplaySSM and AITER comm reuse
Turn ReplaySSM off at C1/C2/C4 and on at C16, and keep AITER comm-group reuse only on C56/C64.
* (recipe) document container prerequisites and add the C14 point
Two container-level facts decide whether this recipe reproduces its own
numbers, and both of them fail quietly, so add a prerequisites section
covering each:
- triton must be 3.7.x. The kimi_k3_agentic_0903 image shipped 3.8.0, which
costs ~2.1x on prefill TTFT here (C1 p50 0.79s -> 1.5s) while leaving decode
untouched. Measured across eleven runs: the gap survives swapping aiter's
.so between revisions, swapping the ATOM checkout, and switching between a
full CI prebuild and a lean JIT build, so triton is the only variable that
moves it.
- DRAFT_MODEL_PATH must name a local directory. The default is a Hugging Face
repo id the image has no cache entry for, so leaving it unset makes every
rank fetch the same 7 GB checkpoint with no error printed, the log stopped
after "Loading drafter model...", and every GPU at 0%. Document the symptom
next to the fix, because the hang is otherwise indistinguishable from a
wedged server.
C14 was in the benchmark scripts but never in the recipe. It reuses the C16
server recipe verbatim, including a CUDA-graph width pinned to 32; deriving it
from 2*CONC gives 28 and the server fails during CUDA-graph warmup. Routing
GRAPH_MAX through CUDAGRAPH_MAX_NUM_SEQS lets the C14 arm pin it while every
other concurrency keeps the 2*CONC default unchanged.
Verified against run_kimik3_atom_dspark_server.sh: all eleven concurrencies now
agree item by item across DCP, max-num-seqs, batched tokens, GPU utilization,
LMCache, ReplaySSM, spec tokens, acceptance length, AITER comm reuse and the
recomputed graph_max.
* (recipe) set ATOM_GDN_SSM_DTYPE=fp16 for Kimi-K3 AgentX
Apply the KDA temporal state dtype optimization from ATOM #2130.
Co-authored-by: Cursor <cursoragent@cursor.com>
* (recipe) Kimi-K3 AgentX: per-concurrency params from the 12-point MI355X sweep (#2208)
* (recipe) Kimi-K3 AgentX: band params from the 12-point MI355X sweep
Replaces the per-concurrency values with the ones a full 12-point AgentX
sweep actually ran on MI355X (8x, ATOM 97c15a73 + aiter 6b3f14b5f,
aiperf 0.12.0, 3600 s profiling window per point).
* GPU util 0.88/0.86 -> 0.90 at every band. Measured on this box as
available_for_kv 39.42 -> 51.11 GB, a +40.6% KV sub-pool.
* max-num-batched-tokens 4096 -> 8192 at C8 and C12, so it is now 8192
everywhere. Cross-round TTFT p50 deltas move monotonically through
zero across C8..C56, consistent with the larger prefill chunk.
* AITER_REUSE_IDENTICAL_COMM_GROUPS is now a DCP rule rather than two
hardcoded branches: 1 at every DCP=8 band, 0 at DCP=1. C32/C40/C48
measured faster with it on; C1 measured 3.3-3.9% WORSE ITL with it
on, which is why the DCP=1 bands keep it off. C8-C16 are still under
A/B and may move in a follow-up.
* Adds the missing C48 band (max-num-seqs 96, graph_max 96). It was in
neither the table nor the case, so CONC=48 hit the unsupported exit.
* Pins ATOM_USE_FLYDSL_GATHER_KV_B_PROJ=1. It already defaults to 1
(atom/utils/envs.py), so this changes nothing today -- it is set
explicitly so the recipe keeps reproducing the measured numbers if
the default flips.
DCP, spec tokens, acceptance length, LMCache sizes, ReplaySSM,
max-num-seqs and graph_max are unchanged; every row was re-verified
against the sweep's own launcher.
* (recipe) Kimi-K3 AgentX: hoist band-invariant values out of the case
The per-concurrency case repeated nine values that are now identical across
every band, so a single-band change had twelve places to go wrong. Hoist the
shared defaults above the case and leave only the genuine per-band deltas
inside it: 119 lines -> 51.
Behaviour is unchanged. All twelve concurrencies were evaluated before and
after and every field -- DCP, max-num-seqs, batched tokens, GPU util,
ENABLE_LMCACHE, LMCache size, ReplaySSM, reuse, spec tokens, acceptance
length, CUDAGRAPH_MAX_NUM_SEQS and the derived graph_max -- is byte-identical.
max-num-seqs is now derived as 2*CONC on the throughput bands rather than
written out five times; it was already exactly 2*CONC there. The two bands
that genuinely deviate keep explicit overrides with the reason attached: C12
at 24, and C14's pinned CUDAGRAPH_MAX_NUM_SEQS=32.
LMCACHE_MAX_LOCAL_CPU_SIZE now carries its 128 default on the DCP=1 bands
too. That is inert: it is only exported inside the ENABLE_LMCACHE==1 branch,
which those bands do not take.
* (recipe) trim the ATOM_USE_FLYDSL_GATHER_KV_B_PROJ comment
* (recipe) Kimi-K3 AgentX: comm-group reuse off below C56
A seven-point A/B on MI355X (C8/12/14/16/32/40/48, one reuse=1 leg and one
reuse=0 leg each, identical builds) measured reuse=0 faster at every point:
ITL p50 -7.3% at C8 narrowing to -1.9% at C48, total tok/chip +3.8% to +1.2%.
Split into 10-minute buckets, 39 of 42 buckets favour reuse=0 (one-sided
p ~ 3e-9), which is well clear of the ~7% run-to-run noise floor that makes
any single full-window comparison on this workload unreliable.
The advantage shrinks monotonically as the KV pool tightens -- reuse=0 gives
up roughly 13k blocks, and its peak KV usage climbs from 77.6% at C8 to 97.6%
at C48, where it is the first point to spend any time above 95% and where
maxRunning starts being pushed back. C56/C64 were not measured and use a
larger LMCache tier and max-num-seqs, so they keep the previous setting.
This also removes the claim that the 12-point sweep measured enabling reuse at
C32/C40/C48 as a gain. That reading came from single-run full-window
aggregates and does not survive the paired A/B.
* (recipe) Kimi-K3 AgentX: C72 + C80 bands, drop the C14 CUDA-graph pin (#2213)
* (recipe) Kimi-K3 AgentX: add the C72 throughput band
Same shape as C64 -- DCP=8, no spec, 192 GiB LMCache, ReplaySSM off -- with
max-num-seqs following the 2*CONC rule every band from C32 up obeys, so 144,
and graph_max 144 with spec disabled.
Measured on MI355X against the C64 band it extends: total tok/chip 12794 vs
12221, so C64 is not the throughput plateau. ITL p50 moves 77.11 -> 88.60 and
P90 interactivity 9.9 -> 8.4, which is the trade the band exists to expose.
maxRunning reached 119, so the wider capture range is genuinely reached rather
than captured and left unused.
Reuse stays enabled here: the CONC >= 56 gate already covers C72, and a paired
A/B measured reuse=1 ahead at C72 by a wider margin than at C64 (ITL bucket
mean +4.5% and total tok/chip +3.1% in its favour, 6 of 6 ten-minute buckets
agreeing), consistent with C72 sitting further past the C56 crossover.
* (recipe) Kimi-K3 AgentX: drop the C14 CUDA-graph pin
C14 pinned CUDAGRAPH_MAX_NUM_SEQS=32 with the justification that deriving
graph_max from 2*CONC "would give 28 and the server fails during CUDA-graph
warmup". That is false on ATOM 97c15a73, and the pin has no effect.
Capture sizes are SEQUENCES, not tokens. ModelRunner truncates the declared
ladder to max_schedulable_decode_bs() = min(max_num_seqs, max_num_batched_tokens
// full_q_len) (atom/model_engine/model_runner.py:142) -- it divides the
speculative width out, while the launcher multiplies it in. For C14 that is
min(32, 8192//4) = 32, independent of CUDAGRAPH_MAX_NUM_SEQS. Sizes above the
bound are dropped with a warning, not asserted on; a batch past the ladder just
falls back to eager.
So both settings capture exactly bs=2..32:
pinned CGMS=32 -> declared [2..128] -> "= 2048) = 32; dropping" x8 ranks
default CGMS=28 -> declared [2..112] -> "= 2048) = 32; dropping" x8 ranks
Measured on 8xMI355X, AgentX 3600 s, C14, AITER_REUSE_IDENTICAL_COMM_GROUPS=0
both legs, ATOM 97c15a73 / aiter 6b3f14b5f / triton 3.7.0:
TTFT p50 TTFT p90 ITL p50 ITL p90 Inter p90 tot tok/s
CGMS=32 804.3 2140.0 12.42 19.18 52.1 57148
CGMS=28 775.8 2028.4 12.36 19.18 52.1 57060
ITL p90 and Inter p90 are equal to the digit; throughput differs by -0.15%.
Ten-minute buckets have identical n per bucket and ITL deltas of
-5.4/+1.2/+0.2/-0.9/-1.0/+3.6 %, three each way, all inside the ~7% run-to-run
noise floor. The server reached READY in ~330 s unpinned.
The table's graph_max for C14 becomes 2*14*(1+3) = 112. With the pin gone, every
row now satisfies the stated rule graph_max = 2 * CONC * (1 + spec); C14 was the
only exception.
Co-Authored-By: Claude <noreply@anthropic.com>
* (recipe) Kimi-K3 AgentX: add the C80 throughput band
C80 is the same shape as C64/C72 -- DCP=8, no spec, 192 GiB LMCache,
max-num-seqs following the 2*CONC rule (160). It needs no new branch: the
existing `CONC -ge 56` guards already pick up the larger CPU tier and
AITER_REUSE_IDENTICAL_COMM_GROUPS=1.
Measured on 8xMI355X, AgentX 3600 s, ATOM 97c15a73 / aiter 6b3f14b5f /
triton 3.7.0, reuse=1:
CONC TTFT p50 TTFT p90 ITL p50 ITL p90 Inter p90 tot tok/chip out tok/chip
64 1356.1 3802.3 77.11 100.66 9.9 12221 88.56
72 1489.8 3963.2 88.60 119.50 8.4 12794 89.42
80 1549.8 4405.3 103.07 130.43 7.7 12839 95.75
**Total throughput plateaus at C72.** C64 -> C72 is +4.7% on tokens/chip;
C72 -> C80 is +0.35%, well inside the ~7% run-to-run noise floor, while ITL p90
rises 9% and Inter p90 falls from 8.4 to 7.7.
Read `tot` with care at the top of the ladder. C80's ISL drops to 107777 (vs
C72's 116894) because the trace serves more, shorter requests (3443 vs 3165), so
output throughput still climbs +7.1% (715 -> 766 tok/s) while total flattens.
The two agree at every band from C8 to C72 and diverge for the first time here.
Saturation is unambiguous across three independent signals:
KV usage samples >=95% KV max maxRunning
C64 2/528 96.5% 84
C72 17/545 96.2% 119
C80 48/556 99.7% 114
maxRunning stops rising at C80 while KV hits 99.7% -- the scheduler is bounded
by the KV pool, not by the concurrency parameter. This is the same ordering
variable that decides the AITER_REUSE_IDENTICAL_COMM_GROUPS crossover at C56.
C80 is included for completeness of the band table. C72 is the knee.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
---------
Co-authored-by: xytpai <xytpai@foxmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: gbyu-amd <Guanbao.Yu@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
* feat(v41): enable TP2 DSpark and add AgentX recipe (#2373)
* feat(v41): enable TP2 DSpark and add AgentX recipe
* fix(v41): admit fixed acceptance schedules without an opt-in gate
* docs(v41): simplify AgentX recipe and remove auxiliary CSV
* docs(v41): present TP2 and TP4 performance in separate tables
* docs(v41): use direct server and client recipe commands
* style(v41): format DSpark runtime admission test
---------
Co-authored-by: ZhangLirong <219599949+ZhangLirong-amd@users.noreply.github.com>
* [MiniMax-M3] Route the paged decode to aiter's FlyDSL kernel with work planner (#2366)
* [MiniMax-M3] Make the dense paged-decode split cap overridable
`PA_DENSE_SPLIT_MAX = 32` is not a tuned value. It is the smaller of two
unrelated bounds that happen to agree near there: production aiter does not
build the C++ PS reduce past 64 and has no working fallback under it, and
`temporary_output` is bf16, so each extra split adds a round trip through the
PS combine.
aiter #4332 removes the first bound. Deciding whether the second one still
binds needs the cap measured against the shipping value on the same tree,
which a constant cannot express. `dense_decode_splits` computes
`min(PA_DENSE_SPLIT_MAX, cdiv(128, running_bs))`, so the cap only binds at
`running_bs <= 2` -- the difference is invisible without an A/B at that shape.
Default stays 32 and behaviour is unchanged. Raising it on production aiter
still fails at engine init rather than at runtime.
`int(os.getenv(...) or 32)` rather than a getenv default: an exported-but-empty
variable would otherwise raise `int('')` at import and take the engine down
before it starts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [MiniMax-M3] Route the paged decode to aiter's FlyDSL kernel behind an env
ATOM_PA_FLYDSL=1 sends the paged decode to aiter's FlyDSL implementation
(aiter PR #4332) where FlyDSL's domain covers the call, and to gluon
otherwise. A measurement switch, not a shipping default: the two take the
same arguments and compute the same thing, and the env exists to A/B them
without maintaining two ATOM trees. Default 0, so nothing changes until an
operator opts in on an aiter that carries #4332.
_flydsl_pa_decode_num_seqs mirrors the kernel's own validation -- alibi,
sinks, externally quantized FP8 queries, ps=False, a positive sliding window,
a non-256 tile, a partition count outside [1, 256], an unsupported head dim,
and any cache that is not [nb, Hkv, D//16, page, 16] at 1 byte per element are
all hard rejects in aiter, not preferences. Enabling the env therefore cannot
turn a working configuration into an exception.
The returned count is the substance. ATOM pads two axes independently:
context_lens to running_bs for CUDA-graph identity, with
[scheduled_bs:running_bs] zeroed, and q to running_tokens for the MoE
all_gather. forward_context.Context stores both "because the ratio is not
always max_seqlen_q" and asserts no rectangle between them. gluon absorbs the
mismatch by deriving its batch as q.shape[0] // query_length and letting the
surplus zero-length rows do no work; FlyDSL demands the rectangle and raises
ValueError: query.shape[0] (12) must equal
context_lengths.shape[0] * query_length (4 * 4)
on an ordinary step. Recovering the count the way gluon does and slicing the
five per-sequence arguments to it hands FlyDSL the same rectangle, dropping
exactly the zeroed tail. All slices are views on dim 0.
The gate is deliberately not restricted to max_seqlen_q == 1. The dense path
is where FlyDSL's headroom over gluon lives, it runs max_seqlen_q ==
num_spec + 1, and FlyDSL carries a query_length == 4 MTP4 grid split tuned for
exactly that shape.
ATOM_PA_FLYDSL_PLAN=1 additionally opts into aiter #5546's GPU work planner.
The plan and its scratch are allocated once and only refreshed: allocation is
illegal inside graph capture and a captured graph bakes in the pointers, and a
planned call does not use the caller's static buffers -- planned partials are
packed as [kv_heads, capacity, query_rows(, D)] against the static API's
[num_seqs, kv_heads, partitions, query_rows(, D)]. The planner refuses batches
past 4096 while M3's sparse call site folds query tokens into num_seqs and
reaches 32768 on a prefill-as-decode step, so those fall back to the static
path rather than raising.
Measured negative on this workload and defaulted off: against the same tree
with the planner off it costs 3.86% at conc 1, 11.50% at conc 10 and 18.28% at
conc 20. The cause is not batch uniformity -- 31% of decode steps have max/min
KV length >= 4 -- but the split count, which plan_pa_decode and pa_decode
share: dense_decode_splits gives 16 at running_bs 5 and 8 at running_bs 24,
and at those counts a batch skewed 2.5x and one skewed 120x both measure
1.00x. #5546's own conclusion is that the data "support using dynamic plans
for uneven KV work, not enabling them universally".
Both routes are logged once per shape signature. A run where the env is set
but every call still lands on gluon is otherwise indistinguishable from one
where FlyDSL simply did not help.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [MiniMax-M3] Give the FlyDSL work planner its own partition ceiling, and keep it off the sparse call sites
`_flydsl_work_plan` handed `plan_pa_decode` the static split count from
`dense_decode_splits`, overwriting that argument's own default. The two are
different quantities: `get_recommended_splits` gives every request the same
count and documents itself as "not a variable-work scheduler", while the plan's
`max_partitions` is a per-request ceiling it then divides under a workgroup
budget. Clamping the ceiling to the static count leaves a long request with the
same share as a short one, which is the planner's entire mechanism. Read back
from `plan.reduce_info`: a 5-request batch at ceiling 16 splits [16,4,4,4,4]
with the long request pinned at the ceiling, and at the default 256 it splits
[256,20,21,20,21].
`ATOM_PA_FLYDSL_PLAN_MAX` now carries that ceiling, 0 meaning "omit the
argument and take aiter's default". Do not read the ceiling off aiter's unit
test: it is the only caller of `plan_pa_decode` in that tree and it binds the
two deliberately, building the plan from the same count it hands `pa_decode` so
the paths must agree numerically. That makes it a correctness test which cannot
see the conflation.
The planner was also a global switch inside `run_pa_decode_gluon`, so enabling
it reached the two sparse call sites as well -- 57 of the 63 `pa_decode` calls
per step. Their context is a fixed topk window, so there is no unevenness to
rebalance and only the refresh and the task packing remain. On uniform batches
the planner is a loss that the ceiling cannot rescue: 0.56x at B8/257 and 0.84x
at B200/200000, flat from ceiling 4 through 256. `allow_work_plan` moves that
choice to the caller, where the split policy already lives, and only the dense
site opts in -- the same boundary `dense_decode_splits` already draws.
A plan build now logs its shape once per cached key, which is the only evidence
that the planner went where it was meant to rather than being assumed to.
Measured at conc 20 on the agentic trace (one operating point), single-user
output throughput 157.19 -> 178.49 tok/s and ITL 7.97 -> 6.96 ms, against 139.20
for the same arm before these changes. Prefix hit 96.56 vs 96.63, input length
+1.6%, request count +1.0%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [MiniMax-M3] Build the FlyDSL work plan once per forward, and fold four env switches into one
The plan is a function of `context_lens` alone, so every layer of one forward
wants the same one. Building it inside the attention op re-ran the planner
kernel for each of M3's three dense layers, and the module-level cache that
existed to paper over that had to key itself on shape to stay correct. It now
lives where the rest of the per-forward metadata is built: `prepare_decode`
attaches it, `prepare_mtp_decode` refreshes it for each draft pass (their
context lengths differ by a token, so reusing the target's plan would point the
kernel at the wrong KV ranges -- wrong output, not merely slower), and the op
reads it off `get_forward_context().attn_metadata`.
Measured before the move: 6 planner kernels per step, and `envs.__getattr__` on
all 63 `pa_decode` calls rather than the 6 that could use a plan. After: 4 and
1. The env read is the larger of the two -- one `__getattr__` costs ~0.97 us and
63 of them are ~0.5% of a 12.8 ms step.
Scratch allocation stays in the op. It is sized from the query tensor the op is
holding and happens once per shape, so only the per-step work moved out.
The env surface collapses to `ATOM_PA_FLYDSL_PLAN`, now on by default:
* ATOM_PA_FLYDSL is gone. It was a measurement switch for A/B-ing FlyDSL
against gluon without two trees; FlyDSL is the decode path. The capability
check that falls back to gluon for shapes outside FlyDSL's domain stays --
it mirrors the kernel's own validation and is not a switch.
* ATOM_PA_FLYDSL_PLAN_MAX is gone, and the call now omits `max_partitions`
entirely so `plan_pa_decode` takes its own default. The one time this tree
chose the ceiling it chose the static split count, which clamps every
request to the same share and removes the planner's mechanism.
* ATOM_PA_DENSE_SPLIT_MAX is gone and PA_DENSE_SPLIT_MAX is a constant again.
With the planner on, a planned call is told `plan.max_partitions`, so the
cap no longer sets the production partition count; it survives for the
gluon fallback, where both halves of its bound still hold. Raising it would
only enlarge the static scratch a planned call never reads.
* ATOM_PA_FLYDSL_PLAN defaults to 1. On the agentic trace, SA-convention
interactivity (1/itl_p90) is +24.6% at conc 20 and +8.4% at conc 10, and
the gain runs monotonically from the slow tail to the fast one (p10 +24.6%,
p50 +19.3%, p90 +5.0%) -- it lifts the floor, which is the end the metric
watches.
The routing log no longer hangs off the deleted env, and is unconditional: with
FlyDSL as the default path, which route a call took is worth always recording.
tests/test_paged_attention_dispatch.py: 74 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [MiniMax-M3] Trim the FlyDSL work-plan comments
Prose cut, facts kept. What stays is the four things a reader would otherwise
re-derive the hard way: the ceiling must not come from the static split count,
each draft pass needs its own refresh (reusing the target's plan points the
kernel at the wrong KV ranges -- wrong output, not merely slower), scratch is
allocated in the op because its shapes come from the query tensor, and the
FlyDSL/gluon split is a capability check rather than a switch. The measured
numbers move to the one place they inform a decision: why the planner defaults
on.
Also fixes a contradiction left by the previous commit. `run_pa_decode_gluon`
still described itself as a "MEASUREMENT SWITCH, not a shipping default" after
FlyDSL had become the path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [MiniMax-M3] Make the per-forward work plan survive cudagraph capture
Review found the previous commit's premise broken: decode runs from captured
graphs (cudagraph_mode=FULL), and `build_for_cudagraph_capture` never attached
a plan, so the op saw None at capture time and the STATIC path is what got
recorded. Every later refresh then worked on a graph that does not read it --
no error, no slowdown, just a planner that quietly does nothing and an
end-to-end number that looks fine.
The capture flow (model_runner.py) is what makes the fix possible:
3857 build_capture(bs) outside the graph -- first plan allocated
3913 self.model(...) warmup, eager; the op sees the plan
3969 with torch.cuda.graph(...) capture; the plan is already there
So the plan is attached in `build_for_cudagraph_capture` too, and the builder
keeps one per (batch, kv_heads, device) instead of a single slot. The single
slot was the more dangerous half: `build_capture` runs once per capture-ladder
size, so the second rung replaced the first rung's plan while the first rung's
graph had already baked in its pointers.
Two more from the same review:
* The plan was built from `context_lens[:scheduled_bs]` while the op derives
its own batch as `q.shape[0] // max_seqlen_q`, which is `running_bs`.
aiter validates `reduce_info.shape == (num_seqs, 2)` and raises, so any
step whose batch was not exactly on the ladder would have killed the
worker. `context_lens` is already `running_bs` long with the tail zeroed.
* `prepare_mtp_decode` dropped the refresh result and relied on the returned
object being the same one. It now publishes through `workinfos`, which the
caller splats into the metadata, so a None result clears a stale plan
rather than leaving it in place.
`flydsl_plan_matches` is extracted rather than inlined so the guard is
testable: a plan built for another batch now falls back to the static path
instead of reaching aiter's validate.
tests/test_paged_attention_dispatch.py: 74 -> 83. Eight of the nine are
falsifiable by reverting the behaviour they describe, each reddening exactly
one test; the ninth is a source check on the capture attach, labelled as such,
because driving a real capture needs a model runner and what it guards fails
silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [MiniMax-M3] Put the FlyDSL decode kernel back behind an env, gluon by default
aiter #4332 is unmerged and the kernel is not fully tested, so gluon goes
back to being the shipping decode path and FlyDSL becomes the opt-in:
ATOM_PA_FLYDSL=1, off by default. The two gates run in series -- the env,
then the capability check that mirrors the kernel's own validation -- and
the `and` short-circuits, so with the env off the check does not run.
Shapes outside FlyDSL's domain still fall back to gluon either way.
The work planner (aiter #5546) only feeds the FlyDSL kernel, so it now
requires ATOM_PA_FLYDSL=1 as well. Without that, a plan would be built and
refreshed every step for a kernel that never reads it, and the "flydsl work
plan" log line -- what an A/B arm is checked with -- would show up on a run
that is entirely gluon.
Tests 84 -> 86: the default must be off, the dispatch must consult the env
before the capability check, and the builder must not plan with FlyDSL off.
Each of the three goes red on its own reversion.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [MiniMax-M3] Rename allow_work_plan to allow_flydsl_plan, trim comments
The parameter now matches the env that controls it, ATOM_PA_FLYDSL_PLAN.
"work plan" is aiter's own term for the object, but on its own it does not
say the plan belongs to the FlyDSL path -- which matters now that gluon is
the default and FlyDSL is the opt-in.
Comments trimmed throughout: the _flydsl_pa_decode_num_seqs docstring keeps
why it returns a count rather than a bool (ATOM pads its sequence and row
axes independently, so the two disagree by a padding slot on an ordinary
step; gluon absorbs that, FlyDSL raises) and drops the shape algebra and the
verbatim traceback.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Fix format
* Update docs
* Fix review comments
* Address PR #2366 review: capability gate, plan lifetime, ROCR device binding
Two gates in series guard the FlyDSL paged decode: ATOM_PA_FLYDSL is the
opt-in, and _flydsl_pa_decode_num_seqs mirrors aiter's own rejects so an
unsupported call falls back to gluon instead of raising inside the kernel
and killing the worker. The gate was missing several of those rejects --
q/output dtype and shape agreement, both head_dim stride checks, the
q-heads-over-kv-heads divisor, the v_cache dtype, and contiguity of the
two caches and the two per-sequence arrays.
Plans are now only ever minted during cudagraph capture. aiter's planner
takes batch as a tl.constexpr, so a runtime batch that no graph replays
would cost a cold kernel specialization and a plan nothing can free; the
static path is the right answer for it. Runtime only refreshes in place.
The one-shot "no plan for this batch" notice waits until a capture has
happened, since before that every call legitimately lands there and the
notice would be spent on a step that says nothing.
max_partitions stays unset, at plan_pa_decode's own default: every number
measured on this branch was measured there, and the review's suggestion to
clamp it to 64 is a perf claim nothing here backs. Measuring instead, on
the planned path, across nine cells spanning the trace's p50/p90/p99
contexts at batch 1/8/24: drift from an fp32 reference does not grow with
the split count anywhere, and the ceiling only ever binds at batch 1. The
plan's capacity is min(batch * max_partitions, max(batch, 2 * CU /
kv_heads)) = 512 here, so each sequence's share is about 512/batch and a
64 ceiling would have changed nothing above batch 2 while capping the one
place the planner has the most to rebalance.
numa_utils and the mooncake connector now read ROCR_VISIBLE_DEVICES first.
ROCR filters and renumbers before HIP indexes the result, so setting both
cut the visible device count twice; dropping HIP from the recipe left the
NUMA binding reading nothing and put all four ranks on one node.
Tests go from 86 to 127. The call-site contract is now checked by walking
the AST of all three builders and comparing the value of `create`, not its
presence in one file -- a sed that moved `create=True` onto a runtime path
had left every test green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* [ATOM][PD]: Transfer prefill prompt token ids to decode (#2331)
* feat(pd): relay prefill prompt token ids to decode
Decode was re-rendering and tokenizing the same prompt prefill already
tokenized. Prefill echoes the ids, atomesh carries them in
kv_transfer_params, and decode skips tokenize.
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(pd): avoid redundant prompt token ID serialization
Serialize completed OpenAI responses through JSONResponse to avoid
FastAPI recursively visiting every prompt token ID. After validating the
prefill IDs on decode, remove the duplicate prompt from KV metadata before
engine IPC while preserving the original request and other transfer fields.
Cover HTTP response compatibility, prompt-ID round trips, and metadata
forwarding for chat/completions, streaming, and fanout requests.
Validation: 103 passed in the prompt-token-ID and protocol tests.
* docs(pd): simplify token ID comments and drop recipe changes
Shorten token ID comments and docstrings across the API, router, and tests.
Restore the PD disaggregation recipe to origin/main without changing runtime
behavior.
Validation: 103 related tests passed; Black passed; no new Ruff findings.
* simplify the test cases
Signed-off-by: Mengqing Cao <cmq0113@gmail.com>
---------
Signed-off-by: Mengqing Cao <cmq0113@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(moe): prefill-only mega combine wire (#2347)
* feat(moe): prefill-only mega combine wire
* fix(moe): DP-agree the mega combine wire
* refactor(moe): move combine wire env to envs.py
---------
Co-authored-by: honglie <hyi@amd.com>
* fix(offload): one tier lookup per waiting request, not one per step (#2305)
`get_num_new_matched_tokens` runs before allocation and is gated only on
`num_computed_tokens == 0`, so a request that fails to allocate goes back
to the head of the waiting queue and is asked the identical question next
step. On a full KV cache that turns one lookup into one lookup per
scheduler step: a prompt copy, ~5k chunk hashes and a blocking round trip
to the tier worker, all on the scheduler thread.
Memoize the one expensive input -- the tier `hit` -- and re-derive the
answer from it on every call. The frontier is not part of the question:
how much of the prompt the tier holds is a property of the prompt, and the
frontier only decides how much of that hit is still worth transferring.
Nothing tells the connector when the tier's answer changes, so the memo is
a bounded optimisation rather than a cache: a hit is replayed at most
`OFFLOAD_LOOKUP_MEMO_STEPS` times and a non-answer backs off for
`OFFLOAD_LOOKUP_RETRY_STEPS` (both default 32).
A memo-served answer carries no pin, and an answer needs none -- the
request only waits. A transfer does. `build_connector_meta`'s load-emit
loop is the single choke point where a retrieve reaches the worker on both
the native and the plugin path, so the pin is re-taken there, confirm-only:
the tier is asked again and the load is dropped as `tier_lost_prefix` if it
no longer holds what the spec promised. That is one lookup per load
actually dispatched.
The MP client's `lookup` now returns `None` on deadline expiry instead of
0: "the worker did not answer in time" is not a hit of zero and must not be
remembered as one.
Applied to both `ChunkedOffloadSchedulerBase` and `DSV4OffloadScheduler`.
The latter releases its pins each step, so it always answers
`_ensure_lookup_pin` with a fresh lookup.
Co-authored-by: perzhang <perzhang@amd.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* ci: check container model mount in atom test (#2372)
* perf(v41): page the index plane at the candidate block length (#2377)
* perf(attention): reduce V4 fused decode normalization overhead
Normalize once per head and improve occupancy for nonquantized D512 32-head tiles. Cover ragged CSR, sinks, quantized KV, and changing graph inputs with an independent reference.
* perf(attention): specialize gfx950 fused decode with shared KV panels
Use explicit MFMA layouts for BF16 D512 head tiles and reuse KV panels through LDS. Preserve the existing split and fallback paths; cover graph inputs, tile boundaries, and large KV addresses.
* Generalize V4 fused decode across CDNA3 and CDNA4
* perf(v41): page the index plane at the candidate block length
A Reindex layer scored every column of its context and then drove the ones
outside the candidate source's blocks to -inf. The blocks it keeps are exactly
a block table -- if the plane is paged at the length they are picked in -- so
the same selection is reachable by scoring `candidate_topk_blocks` blocks and
never addressing the rest.
`V41PoolGeometry.index_block_rows` declares that length, and the backend sets
it to `candidate_block_size`. The plane's bytes do not change: data and scales
are packed with nothing between them, so a row's share is 132 B at every block
length, and only the granularity at which the two regions interleave moves.
aiter supports the short page from ROCm/aiter#5749; a GPU whose scorer cannot
page that short refuses when asked, which is not a silent wrong answer.
`restrict_to_candidates` has no caller left. It is kept, renamed, as the judge
of the compaction: restating it in the test would be the same addressing
written twice.
Two conflations came out with it:
* `index_write` spelled the MFMA tile width, the shuffle group and the block
length with one name. All three were 16 while 16 was the only page the
scorer took. A block of 8 shuffles in groups of 8 by 16 and tells them apart;
at 16 the new form is byte-for-byte the old one.
* a hub id reached `CheckpointReader` as a path, so V4.1's Engram loader built
`deepseek-ai/DeepSeek-V4.1-Flash/model.safetensors.index.json` and reported a
missing checkpoint for a name nobody had resolved. `local_model_dir` is now
the one place that turns either spelling into a directory; the three probes
that answered "no such tensor" for every remote checkpoint go through it too,
and the reader refuses a non-directory by name.
Gates: unit suite 8609 passed / 3 failed, the 3 reproducing on origin/main.
GSM8K 1319 @ 3-shot, DSpark-5, TP4, --level 0: flexible 0.9227 / strict 0.9234
against a 0.9224 +- 0.005 baseline. On a 127k-token context, 225 in-forward
comparisons of the two paths over 4 ranks disagreed on nothing -- that is the
gate for the regime GSM8K cannot reach, where the candidates are a strict
subset. Scoring is 0.80% of a decode step's GPU time in a 100k/64/conc-8 trace,
which is this optimization's ceiling there; the prefill share is 6.5%.
What the change buys is not measured. The trace carries no per-layer label, so
calls cannot be attributed to producer or consumer, and an earlier attempt to
split them by duration was wrong: a compacted call in prefill still scores
16384 columns over 16384 rows and is not short. gfx1250 is untested.
* [DCP] Allow query replication (QREP) with MTP (#2290)
* [DCP] Allow query replication (QREP) with speculative decode; fix stale sparse+MTP docs
qrep_unsupported_reason() unconditionally disabled QREP whenever
speculative_config was set, on the suspicion that QREP's full-group-head
q_out might conflict with the cprr round-robin MLA kernel MTP's qlen>1
verify path uses. Auditing the original commit and the two code paths shows
this was first-cut caution ("not yet wired"), not an observed conflict:
QREP only changes how q_out is produced (load-time replicated q_proj vs. a
runtime AllGather), and cprr's mask keys off KV-side round-robin position,
with no dependency on q_out's provenance.
Validated on GLM-5.2-FP8 (sparse DSA), TP8/DCP8, fp8 KV, MTP=3, gsm8k
nshot=20, same commit, only enable_query_replication differing:
QREP off: flexible-extract 0.9492 +/- 0.0060, strict-match 0.9492 +/- 0.0060
QREP on: flexible-extract 0.9469 +/- 0.0062, strict-match 0.9477 +/- 0.0061
Delta is within combined stderr; both arms completed all 1319 requests
without error.
Also fixes docs/context_parallel_guide.md, which still claimed sparse DSA +
DCP + MTP was rejected by an assert -- that restriction was already lifted
(the sharded index cache's candidate exchange handles qlen>1 directly) but
the docs were never updated to match.
Dense MLA + MTP + QREP was not independently re-validated this round; the
mechanism argument applies equally but flagging the gap for reviewers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* [DCP] Gate QREP per-layer on actual q_proj width, not just the config flag
Removing the speculative_config gate (previous commit) made QREP reachable
for every model that runs under DCP, but only DeepSeek/GLM's decoder layer
and Kimi-K3's main model actually build q_proj with qrep_tp_override. Three
q_proj construction sites never opted in and would have silently misread a
plain per-rank-width q as the wide QREP layout, or crashed:
- atom/models/eagle3_deepseek_mla.py (eagle3 draft, its own MLAModules)
- atom/models/kimi_k3_dspark.py (DSpark draft, its own MLAModules)
- atom/models/glm5_next.py (GLM-5.3-Flash; q_proj wrapped in _ZeroRopePad,
which has no .weight at all)
All three used to be protected incidentally by the speculative_config gate
(eagle3/DSpark always set it; GLM-5.3 + MTP would too), so removing that
gate without this fix would have made them reachable and broken.
Fix: MLAAttention now…
1. Motivation
gluon's paged decode gives every request in a batch the same number of KV splits. On an agentic trace a decode batch is wildly uneven — measured over 30,400 steps at conc 32, the step-internal
max/mincontext ratio is p50 2.49, p90 22.2, max 1610, and 31.2% of steps sit above 4× — so one long request owns the critical path while the short ones idle.aiter #4332 (FlyDSL paged attention) plus #5546 (a Triton GPU planner that hands each request a partition count proportional to its real context length) address exactly that shape. Measured on MiniMax-M3 MXFP4 + EAGLE3, TP4, same machine as the current published curve:
The gain grows with concurrency because that is when the batch has enough unevenness to rebalance; at conc 1 there is nothing to rebalance, and the curve confirms it. Past conc 20 it narrows again — the planner's task budget is
2 × CU, so each request's share falls as512/batchand there is less left to redistribute.Both paths are opt-in and gluon stays the default: the FlyDSL kernel is not fully tested on our shapes yet, so it is something you turn on, not something you inherit.
2. Usage
ATOM_PA_FLYDSL=10)ATOM_PA_FLYDSL_PLAN=11)ATOM_PA_FLYDSL=1; with FlyDSL off no plan is built at all.ATOM_PA_FLYDSL=1 python3 -m atom.entrypoints.openai_server \ --model amd/MiniMax-M3-MXFP4 --tensor-parallel-size 4 ...Verify from the server log:
pa_decode -> flydsl[N seqs]with no-> gluonlines, andflydsl work plan: num_seqs=... max_partitions=256appearing once per cudagraph capture size per rank and never during the run.3. Design
ATOM_PA_FLYDSL=1+ ATOM_PA_FLYDSL_PLAN=1dense_decode_splits(), capped at 32plan.max_partitionsinsteadpa_decodecallwork_plan— dense onlyTwo gates in series. The env is the opt-in;
_flydsl_pa_decode_num_seqs()then mirrors the kernel's own validation so an unsupported shape falls back to gluon instead of raising from inside aiter. Every clause is a hard reject on aiter's side, not a preference: fp8 compute and fp8 page-16 cache,block_size ∈ {16, 64, 128}taken from the cache's dim −2, gfx942/gfx950, partition size and count bounds,head_dim ∈ {64} ∪ {128…1024 step 128}and agreeing with the cache's ownnum_hgroups × 16, q dtype, output dtype and shape matching q, a contiguous head_dim axis on both q and output, q heads divisible by kv heads, matching k/v cache dtypes, int32 and contiguousblock_tables/context_lens, and a row count that divides evenly and stays inside the batch. Theandshort-circuits, so with the env off the capability check never runs.The planner is kept off the sparse call sites. M3 issues 63
pa_decodecalls per step, 57 of them sparse. Sparse reads a fixed top-k window, so its contexts are near-uniform — the one shape where the planner is a measured net loss (0.56× on an all-equal batch, and a flat curve across every ceiling, so it is not a tuning problem). There is no flag:work_plandefaults toNone, and only the dense site inattention_mha.pyhands one in, matching the boundarydense_decode_splitsalready draws. The vLLM and SGLang bridges run under someone else's forward context and pass none.Plan lifetime vs cudagraph. The plan depends on
context_lensalone, which every layer of one forward shares, so it is built once per forward rather than per call. Decode replays captured graphs, so it must exist at capture time or the static path is what gets recorded — silently, with a benchmark that still looks reasonable.Minting happens only on that first line. aiter's planner takes batch as a
tl.constexpr, so an unseen value costs a kernel specialization — 65–72 ms cold — and a plan that can never be freed, because some captured graph may have baked its pointers in. A runtime batch with no plan is one no graph will replay, so the static path is the right answer for it rather than a stall. The cost of that rule is that the planner is inert under--enforce-eager.Rows are
running_bslong, notscheduled_bs: the op derives its own batch asq.shape[0] // max_seqlen_q, and aiter validatesreduce_info.shape == (num_seqs, 2)against exactly that. Every draft pass republishes its plan throughworkinfos— each pass advancescontext_lens, so reusing the target's plan would point the kernel at the wrong KV ranges, which is a wrong answer rather than a slow one.max_partitionsis left unset, at aiter's own default. Setting it from the static split count clamps every request alike and removes the planner's whole mechanism; this tree did that once. Clamping it to 64 was considered and rejected — see §5.Device visibility.
numa_utilsand the mooncake connector now readROCR_VISIBLE_DEVICESbeforeHIP_VISIBLE_DEVICES. ROCR filters and renumbers the device table before HIP indexes the result, so a recipe setting both cuts the visible count twice; droppingHIP_VISIBLE_DEVICESthen left the NUMA binding reading nothing and put all four ranks on one node, which deadlocks atallocate_kv_cache.Integration: needs aiter
94dca7bc6or later — that commit (#4332, merged 2026-09-22) brings both the FlyDSL paged decode and #5546's planner intomain. On an older aiter the import guard keeps the default path working; withATOM_PA_FLYDSL=1it would fail at startup.4. Test Plan
tests/test_paged_attention_dispatch.py— dispatch, capability gate and plan wiring, CPU only.5. Test Result
Unit: 127 passed (74 → 127). Each new guard was verified to go red on its own reversion, by injecting the defect and re-running — including the two the previous round's tests did not catch: a
create=Truemoved onto a runtime path, and the head_dim whitelist deleted outright.Accuracy (gsm8k, 5-shot / 20-shot): 32 → 0.9674 / 0.9659, 128 → 0.9636 / 0.9644, all far above the 0.93 gate. Paired McNemar p=0.441 / 0.831; 98% of questions agree question-by-question and the 2% that disagree split evenly both ways.
Planner drift at the real ceiling. The plan's
capacityismin(batch × max_partitions, max(batch, 2 × CU / kv_heads))= 512 here, so each request's share is about512/batchand at batch 1 the realized split count equalsmax_partitionsexactly. The trace's single-request contexts are p50 120k / p90 334k / p99 764k tokens — 470 to 2986 tiles — so low-concurrency steps really do run at the 256 ceiling, four times past anything the old bf16 round-trip argument measured. Swept over nine cells (those three contexts × batch 1/8/24, one shared fp32 reference per cell):No growth with the split count anywhere, so the ceiling stays at aiter's default. The sweep also shows the ceiling only ever binds at batch 1: at batch 8 and 24 the 64 and 256 arms receive identical per-request counts, so clamping to 64 would change nothing above batch 2 while capping the one case the planner has the most to rebalance.
End-to-end, same machine, paired against the pre-FlyDSL gluon curve (p90 interactivity =
1/itl_p90, SemiAnalysis convention):Completed-request counts match within 1.3% at every point, so the deltas are rates rather than a different request mix. Routing was asserted per point:
gluon=0, and the plan built only during capture.High concurrency, measured on a second node, so stated against the published curve rather than added to the paired table above:
conc 40 runs with LMCache CPU offload on, as does the published curve's own conc-40 point; the other three are GPU-resident on both sides.
6. Known Limitations
aiter.ops.flydsl.pa_decodeexists wheneverATOM_PA_FLYDSL=1. It landed in94dca7bc6(2026-09-22); nothing in this repo enforces that floor.--enforce-eager. Plans are minted only during cudagraph capture, which never happens there. This is the deliberate cost of not specializing a planner kernel mid-serving.ATOM_PA_FLYDSL=1, any fp8 page-16 MHA decode takes FlyDSL, including the vLLM bridge. The SGLang bridge passescompute_type=torch.bfloat16, which the gate rejects, so it stays on gluon.Submission Checklist