You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[MiniMax-M3] Score the lightning indexer with aiter's FlyDSL kernels - #2493
[MiniMax-M3] Score the lightning indexer with aiter's FlyDSL kernels
Branch: ganyi/m3-flydsl-index-score-integration, merged with main (32981073c); head e55d2d3cdc49063febf0244859f1d7d2e88c6b34; 18 files, +1551/−449.
What this changes
MiniMax-M3's lightning indexer scores every 128-token block of index keys and keeps the top-k. Both phases did that in Triton. This routes them to aiter's FlyDSL MFMA kernels.
Selection is unchanged. The scorers agree on every block a row can pick, and aiter's alloc_score writes the layout the Triton kernel wrote below S * H = 16, and above it the decode op asks for that layout with out= (see "Two aiter contracts"), so everything downstream is blind to the choice. This is a pure performance change — no accuracy decision is being made here.
prefill_index_score keeps a backend= parameter that production never passes. It is the entry point for TestPrefillScorersAgree, which is what backs the claim above: the Triton _index_block_score_kernel is the reference the FlyDSL scorer is checked against, block by block. Remove the parameter and that claim loses its oracle.
Phase
Before
After
Decode
_decode_index_score_tiled_kernel (Triton)
minimax_m3_index_score (FlyDSL) — the only scorer
Decode, indexer CP
_context_score (Triton)
same FlyDSL kernel, via cp_world/cp_rank
Prefill
_index_block_score_kernel (Triton)
minimax_m3_index_score_prefill (FlyDSL). The Triton kernel stays as the tests' reference, reachable by naming it, not by configuration
Decode keeps no fallback. The two Triton decode kernels are deleted and the aiter import is hard, so an aiter that lacks the kernel is an ImportError at module load rather than a silent downgrade discovered 57 layers into a decode step.
Performance
End-to-end, re-measured at the merged head
ganyi/m3-flydsl-index-score-integration at d693b2f0e (a merge of origin/main32981073c) against origin/main32981073c. The commits after d693b2f0e up to e55d2d3cd change tests, recipes, the text of the index-cache refusal and comments / docstrings, and no scoring code (git diff d693b2f0e e55d2d3cd -- atom touches only the refusal messages and comments / docstrings). Setup: aiter 95d211284, flydsl 0.5.0. MiniMax-M3-MXFP4, TP4 on one quad of MI355X shared by every run, indexer CP on, --kv_cache_dtype fp8, --block-size 128, --max-model-len unset (the engine takes the model's 1,048,576; CI runs 32768, and a server at that bound was not measured), eagle3 (GQA draft), ATOM_PA_FLYDSL=1, aiperf inferencex-agentx-mvp, concurrency 25, 900 s per run, seed 42. Two servers per arm, order main, branch, main, branch.
One deviation: spec 3 at a forced acceptance length of 3, not spec 7 / 5. Current main refuses --num-speculative-tokens 7 at config time (num_speculative_tokens must be between 1 and 4), for both arms.
Throughput: +1.2% (arm means 1345.5 vs 1329.7; +0.95% and +1.42% in the two cycles from the unrounded values, +1.43% from the table's rounded ones; the arms repeat within 0.3%). ITL: no difference in cycle 1 (-0.4% whole run, -0.0% paired per request), -7.1% / -4.3% in cycle 2, which is the main arm's second server running 8% slower on ITL than its first; the median ITL is lower on the branch in both cycles (-3.2%, -8.9%). Paired on (conversation, turn) and pooled over the runs, ITL is -2.0% (95% CI -3.7..-0.4, request-level) and request latency -1.5% (CI -3.2..+0.3). TTFT is unchanged (+0.4%, CI -3.4..+4.4). This is smaller than the figures this description carried before (+4.3% throughput, -5.1% ITL).
Scorer kernels, re-measured (one MI355X)
Decode scorer, one index head (TP4), bound = 1M tokens, graph-replayed: FlyDSL is slower than the Triton scorer it replaces at 32K (14 -> 17 us at B=8, 17 -> 25 us at B=16), level at 128K, and 1-8% faster from 512K (e.g. B=32: 391 -> 360 us at 512K, 720 -> 699 us at 1M).
Prefill scorer, 32768-token chunk, one index head, fp8 cache: 302 -> 241 us (-20%) at 32768 tokens of prefix, 455 -> 350 us (-23%) at 69632, 633 -> 506 us (-20%) at 118784. This replaces the "-36% / -42% / -46%" table, which was taken on an older aiter against an older Triton scorer.
The earlier finding that FlyDSL is 3.6 to 4.8 times slower than Triton for a bf16 index cache (measured on an older aiter that still had the bf16 path, not re-run) is why the index cache is fp8 only. On M3 the index cache dtype follows --kv_cache_dtype unless --index_cache_dtype is given (atom/config.py:3071-3091), so in practice --kv_cache_dtype fp8 selects the supported path.
Accuracy
GSM8K, 5-shot chat, temperature 0, real eagle3 (MHA draft, 3 speculative tokens, acceptance not forced), fp8 KV and index, indexer CP off, same prompts for both arms, main 32981073c against d693b2f0e: 600 questions, two servers per arm: main 91.08%, branch 91.83%, branch - main +0.75 points (paired SE 0.66, 95% CI -0.55..+2.05). A single-server pair on the first 200 of those questions read 95.0% vs 93.0% (-2.0 points, paired SE 1.6, 7 questions only main right, 3 only the branch), which is inside the noise of 200 questions. No regression is detectable at this scale; one of about half a point is not excluded, and the prompts are short, so long-context selection is not covered (the scorer-against-the-definition GPU tests cover that at kernel level).
Earlier accuracy measurement (older head, kept for the scorer-profile and acceptance checks)
The throughput runs above force an acceptance length, which makes the generated text meaningless, so accuracy was measured separately, without forcing acceptance.
Setup. MiniMax-M3-MXFP4, TP4, real eagle3 (MiniMax-M3-EAGLE3, MHA draft) with 3 speculative tokens, --index-cache-dtype fp8, fp8 KV cache, ATOM_FORCE_ATTN_TRITON=1, indexer CP off. Image rocm/atom-dev:nightly_202610171506, GPUs 4–7. gsm8k 5-shot, chat template, temperature 0, 1319 docs, three reps per arm, one server launch per arm. The arms are the whole PR: head c0a9913b4 (FlyDSL scorers) against its merge-base 661ef7661 (Triton scorers). Each arm ran from its own source tree, and the import paths were checked.
rep1
rep2
rep3
mean
FlyDSL (this PR), flexible
0.9439
0.9348
0.9424
0.9404
Triton (base), flexible
0.9431
0.9386
0.9416
0.9411
FlyDSL, strict
0.9447
0.9348
0.9431
0.9409
Triton, strict
0.9447
0.9401
0.9424
0.9424
Per-rep stderr is about ±0.0065. FlyDSL minus Triton is −0.07 points flexible and −0.15 points strict; the paired per-doc difference of the 3-rep means (flexible) is −0.08 points, 95% CI [−0.68, +0.53]. The recipe's reference for eagle3 is 0.9469 flexible / 0.9477 strict, about 0.5 points above both arms.
At temperature 0, identical-config reps already differ in about 78% of response texts (eagle3 and batching), so text differences carry no signal. The answer differs on 3.6–4.7% of docs between two reps of the same arm and on 4.3% between the arms, with right→wrong and wrong→right balanced. The 118 docs whose answer is unstable within an arm are the same 118 across arms; no doc disagrees only across arms.
Which scorer ran was checked with the torch profiler on all four ranks, not assumed. The FlyDSL arm shows m3_index_score_prefill_* and m3_index_score_S4_H1_fp8_* kernels and no Triton scorer kernel; the Triton arm shows _index_block_score_kernel and _decode_index_score_tiled_kernel and no m3_index_score*.
What this does and does not show.
No regression is detectable at gsm8k scale. The 3-rep mean has a standard error of about 0.6 points, so a regression under about 1 point is not ruled out.
Indexer CP is off in these runs. Long-context behaviour is not covered: gsm8k prompts are short.
The two arms differ in more than the scorer: base imports the top-k helper through an aiter path that this image no longer has, so it may fall back to the Triton top-k selector. Both arms' profiles show the Triton selector and neither shows the aiter one, and the serve logs show no import error, so the selector looks the same in both, but this was not proven.
Performance was not measured here.
The work map, and where it is built
The FlyDSL decode kernel takes a packed dispatch map so a ragged batch does not pay for its holes. Building it is ~15 trivial device ops whose cost is almost entirely launch floor: ~120 µs a call, against a score kernel of 17–250 µs. Per sparse layer that costs more than the kernel it feeds.
It is therefore built once per decode step in the metadata builder (MiniMaxM3SparseMetadata.index_score_work_map) and threaded to both the plain and the indexer-CP decode paths. Two properties this forces:
A persistent buffer. A captured decode replays against the pointer baked in at capture, so the map is filled into a preallocated tensor, the same way n_valid_column_per_row already is.
A bound that does not move. The map's row count is the kernel's grid, and the grid is baked at capture while the map is refilled each step outside it. So the bound is ceil(max_model_len / 128) — never the batch's longest request, which would reshape the map under a fixed grid. Under indexer CP it is this rank's local bound, ceil(that / world), because the shard is what the kernel is dispatched against.
Omitting the map is still correct — decode_index_score builds its own — which is what the test and plugin paths do.
Two aiter contracts, now checked instead of assumed
alloc_score returns a permuted view. Past S * H >= 16 (S = query tokens per request, H = index heads on the rank; aiter's TRANSPOSE_MIN_F) aiter allocates [max_block, total_q, H] and hands back a feature-contiguous [H, total_q, max_block] permutation, which coalesces the scorer's store. That is a decode with 4 or more query tokens at CP world 4 (H = 4), 8 at TP2 (H = 2), or 16 at TP4 without CP (H = 1). local_candidate_keys rejects a non-contiguous score outright — exactly the check aiter's own docstring asks callers to make. The CP path now requests the layout it needs through a new out= on decode_index_score.
Both layouts carry bit-identical values (verified directly: max abs diff 0.0), so this costs the store's coalescing and nothing else, and it beats copying after the fact. _select_with_aiter declines on the same condition, since its flattening is a .view.
Block tables must cover max_block. A recent aiter rejects a table narrower than the bound rather than reading past the row. Production never produces that pair — the engine sizes block tables from max_model_len, the same quantity the bound comes from — so the tests were widened to match production rather than the check relaxed. Under CP the widening is by world * pad, since the bound is local and the table is global.
Tests
At 7d90b2426, with no GPU, on the touched M3 / attention / TBO test files (tests/test_minimax_m3_*.py, tests/model_ops/test_indexer_cp_parity.py, tests/model_ops/test_triton_dynamic_shapes.py, tests/test_indexer_cp_gate.py, tests/test_tbo_decode_ubatch_split.py, tests/test_h2d_attention_publication.py, tests/test_mtp_index_share.py, tests/test_mla_index_cache.py, tests/test_paged_attention_dispatch.py, tests/model_ops/test_sparse_indexer_chunk.py) the branch has 2 failed, 853 passed, 269 skipped, against main (32981073c) 2 failed, 922 passed, 206 skipped. Both failures are environment ones and appear on both sides: tests/test_paged_attention_dispatch.py::TestDenseDecodeSplits::test_batch_one_asks_for_the_ceiling (needs a GPU), and the subprocess test in tests/test_minimax_m3_flydsl_optional.py (named test_importing_needs_no_flydsl_... on the branch, test_bf16_and_fp8_need_no_flydsl_... on main), which on the branch fails with No module named 'aiter.ops.triton.fused_kv_cache' from the container image's stock aiter. Fewer tests pass on the branch because the PR deletes the Triton decode scorers and their tests. e55d2d3cd (comments and docstrings only) was checked on the two test files it touches (tests/test_minimax_m3_index_topk.py, tests/model_ops/test_indexer_cp_parity.py): 42 passed, 151 skipped, 0 failed at 7d90b2426 and at e55d2d3cd; the full selection above was not re-run at e55d2d3cd.
The GPU run (one MI355X) and the wider tracked-suite comparison were not re-run at this head; the counts quoted earlier in this description are gone.
A user-visible narrowing: the index cache is fp8 only
Before: index_cache_dtype defaults to kv_cache_dtype and then to "bf16", and the Triton decode scorer served it.
After: the FlyDSL scorers serve fp8 only (aiter#5626 cut the bf16 cache path; it is on aiter main now, as 1305f0c8b), and this PR deletes the Triton decode scorers, so a bf16 index cache is refused at startup, before any request, with a message naming the fp8 flag of the front-end that refuses (below). On vLLM the index cache follows --kv-cache-dtype, so fp8 there is also the main KV cache's dtype.
The shipped recipes and CI entries pass fp8 now: this PR changed recipes/atom_vllm/MiniMax-M3.md (--kv-cache-dtype auto to fp8), recipes/mesh/Agentic-MiniMax-M3.md (no KV dtype to --kv_cache_dtype fp8) and two CI entries that did not pass fp8 (the vLLM-OOT M3 entry in oot_benchmark_models.json, and MiniMax-M3-MXFP4 TP4 in sglang_models_accuracy.json). Anyone else relying on a bf16 or auto KV cache is now refused. recipes/MiniMax-M3.md now says the flag is required rather than a tuning choice.
The three front-ends refuse at different stages. SGLang in _atom_m3_index_dtype (from configure, before weights load); vLLM in _index_cache_torch_dtype (when the sparse layer is constructed, atom/plugin/vllm/attention/minimax_m3_attnetion.py); native ATOM in _index_cache_spec (when the KV pool is allocated, after weights load). They build the message with one shared function, index_cache_must_be_fp8(front_end, got), so each names its own flag (native --kv_cache_dtype fp8 or --index-cache-dtype fp8, vLLM --kv-cache-dtype fp8, SGLang --kv-cache-dtype fp8_e4m3) and none can drift.
Each of the three refusals is pinned by a test that calls the refusing function itself, each checked on CPU at 7d90b2426 by removing the refusal and watching its test fail (for vLLM also by refusing everything and watching the fp8 pass-through test fail), and each also asserts that fp8 still resolves so none can pass by refusing everything. One limit worth stating: those tests live in M3 files that importorskip triton and aiter, and the only CI job that runs tests/ has neither, so they run on a GPU runner or a dev box and not on the pre-checkin gate.
Refusing at startup is the point. The same configuration otherwise starts, serves a prefill, and dies 57 layers into a decode step with ValueError: cache: invalid dtype or rank raised inside aiter, which names neither the flag nor ATOM as the thing refusing. decode_index_score and the CP validator carry the same refusal as a backstop for paths that did not come through the allocator, built by the same function (with no front-end named, it lists all three flags) so they cannot drift.
On any GPU other than gfx950 (gfx942, gfx1250) an fp8 index cache has no scorer after this PR: aiter's FlyDSL scorers are gfx950-only (SUPPORTED_ARCHS = ("gfx950",)) and the Triton decode scorer is deleted. It starts, then fails inside aiter at the first decode (at capture on a graph run), where on main the Triton scorers served those archs. I did not check whether such a deployment exists. A decision is needed before merging: drop those archs for M3, or keep a Triton decode scorer for them.
Not in this PR
The shuffled index-cache layout (ATOM_M3_INDEX_SHUFFLED) and the fp8 index query (ATOM_M3_INDEX_FP8_Q), both deleted along with their env vars.
The shuffled writer measured as a regression in its own right (−12.9% throughput, +17.4% ITL), and in the earlier overnight sweep (its tables are no longer in this description; they were taken at an older head) the shuffled arm came in below the plain one. With aiter no longer gating k_lds on the layout, there is nothing left for it to buy.
The fp8 query goes with it because it changes decode scores: decode's default bf16 MFMA lifts K to Q's precision rather than rounding Q down, so an fp8 Q there is not free, and a tie reordered near the top-k cut changes which blocks get selected. That is an accuracy decision and does not belong in a performance PR. (In prefill an fp8 Q is bit-identical, since fp8_mfma was rounding Q at staging anyway — which is why nothing is lost on that side.) Say the word if you want it kept as an opt-in.
GLM-5.2 DCP sparse prefill — a separate PR, as discussed.
Runs automatically on every eligible PR before approval:
✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests
Heavy model tests:
✅ Run after the PR is approved and Pre Checkin passes
✅ Run immediately when an approval review is submitted
✅ Can be requested before approval with labels
Label
Tests
ci:full
Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom
Run native ATOM model accuracy tests
ci:vllm
Run ATOM vLLM OOT model accuracy tests
ci:sglang
Run ATOM SGLang model accuracy tests
Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2493 --add-label <label>
The exact parity check examines only every 37th query row, so scorer differences in roughly 97% of rows still pass; the preceding tolerance-based score comparison cannot catch a near-cutoff reorder. Compare sorted top-k tensors for all rows on-device to retain order-insensitive set parity without the per-element host synchronization.
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
Exact-selection check skips rows and can miss top-k disagreements
tests/test_minimax_m3_index_topk.py:893
This exact-selection check samples only every 37th query row. A backend disagreement near the top-k cutoff on any skipped row can therefore pass even though the all-score checks use tolerances and cannot detect a changed selected set. Compare the sorted top-k tensors for every head and row.
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
index_score_work_map_size returns the exact grid size for this one (batch, S, H) geometry, but this buffer is reused by other decode geometries. Aiter’s API explicitly requires work_map_capacity for persistent buffers because the exact size is non-monotonic. For example, max_bs=8, max_model_len=8192, spec-7, and one index head allocates 136 rows for S=8, while the Q1 MTP decode path needs 256 rows, so map construction rejects the short buffer. Allocate the capacity for the full envelope instead.
This issue also appears on line 491 of the same file.
Validate selection parity across all rows
tests/test_minimax_m3_index_topk.py:893
This exact parity assertion samples only every 37th row; the four-row q1-tail case checks only row 0, so disagreements for the other requests pass despite the PR’s all-row selection-equivalence claim. Compare sorted top-k indices for the complete tensors on-device instead.
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
Restrict automatic FlyDSL FP8 dispatch to k_lds architectures
atom/model_ops/minimax_m3/index_topk.py:1120
auto routes every FP8 cache to FlyDSL, including gfx942. In the linked aiter kernel, gfx942 forces k128=0, which disables k_lds; that is precisely the register fallback this PR says loses badly to Triton. Thus the default can regress FP8 MiniMax-M3 prefill on MI300X. Gate the automatic FlyDSL choice on both FP8 and an architecture/config that actually enables k_lds (currently gfx950); explicit flydsl can remain literal.
Use architecture-native aiter FP8 dtype in GPU tests
tests/test_minimax_m3_index_topk.py:698
This hardcodes the gfx950 FP8 flavor. On gfx942, aiter's native dtypes.fp8 is float8_e4m3fnuz, and the new scorer rejects float8_e4m3fn, so this GPU test fails before checking decode scores. Use aiter.dtypes.fp8, as the CP parity test already does.
This issue also appears in the following locations of the same file:
line 870
line 1004
Correct PR description for lazy decode import behavior
atom/model_ops/minimax_m3/index_topk.py:65
The PR description still says the decode aiter import is hard and fails at module load, but this accessor deliberately resolves it on first use and raises RuntimeError only when MiniMax-M3 decode is exercised. Update the description so merge/deployment behavior is stated accurately.
Document architecture requirements for the FP8 k_lds path
atom/utils/envs.py:405
This comment documents FP8 as sufficient for the fast k_lds path, but aiter disables that path on gfx942 because k128 uses a CDNA4-only instruction. Update this documentation together with the dispatch so auto does not promise or select FlyDSL on FP8 gfx942.
Correct auto-mode documentation for gfx942 FP8 dispatch
docs/environment_variables.md:120
The auto description is incorrect on gfx942: the linked aiter implementation explicitly disables k128/k_lds there, so an FP8 cache still takes the register fallback rather than the advertised LDS path. Document the architecture condition and keep it consistent with the runtime dispatch.
Replaces the two Triton block-score kernels behind the M3 lightning indexer
with aiter's FlyDSL MFMA ones. Selection is unchanged -- the scorers agree on
every block a row can pick, and `alloc_score` deliberately writes the layout
the Triton kernel wrote -- so this is a pure performance change.
Decode has no choice left: the Triton decode scorer and its CP twin
(`_decode_index_score_tiled_kernel`, `_context_score`) are deleted and the
aiter import is hard, so an aiter without the kernel is an ImportError at load
rather than a silent downgrade found 57 layers into a step.
Prefill keeps both, chosen by ATOM_M3_PREFILL_INDEX_SCORE. "auto" follows the
index cache dtype, which is what decides whether FlyDSL is the faster kernel
at all: its `k_lds` path -- staging K through LDS, a quarter of the K bytes --
is legal only on an fp8 k128 cache. Without it the kernel measured 3.6-4.8x
SLOWER than Triton at 32k-116k context; with it, 36-46% faster over the same
range. The explicit values never fall back, because an arm that asked for
FlyDSL and silently ran Triton is a wrong measurement, not a degraded one.
The decode kernel wants a packed work map so a ragged batch does not pay for
its holes. Building it is ~15 trivial device ops, almost all launch floor:
~120us a call against a score kernel of 17-250us, so per sparse layer it would
cost more than the kernel it feeds. It is built once per decode step in the
metadata (`MiniMaxM3SparseMetadata.index_score_work_map`) and threaded to both
the plain and the indexer-CP decode paths, with a persistent buffer because a
captured decode replays against the pointer baked in at capture. Its bound is
`max_model_len`, never the batch's longest request -- the map's row count IS
the grid, and a bound that tracked the batch would reshape the map under a
grid that cannot move.
Two contracts the new allocator imposes, both now checked rather than assumed:
* `alloc_score` returns a permuted, feature-contiguous view once
`total_q * heads >= 16`, which is every qlen>1 decode at CP world 4.
`local_candidate_keys` rejects a non-contiguous score, so the CP path asks
for the layout it needs via a new `out=`. Both layouts carry bit-identical
values (verified, max abs diff 0.0); this costs the store's coalescing and
nothing else. `_select_with_aiter` declines on the same condition, since its
flattening is a view.
* aiter now rejects a block table narrower than `max_block` rather than
reading past the row. Production never produces that pair -- the engine
sizes tables from `max_model_len`, the same quantity the bound comes from --
so the tests are widened to match production instead of the check relaxed.
Measured on MiniMax-M3-MXFP4, TP4, indexer CP, fp8 KV, eagle3 spec7 / forced
acceptance 5, aiperf agentic traces at concurrency 25, both arms on the same
quad: ~4-6% throughput over the Triton baseline, with the decode scorer
carrying it. Paired per-request over shared turns: ITL -3.7%, latency -4.0%.
tests/test_minimax_m3_index_topk.py and tests/model_ops/test_indexer_cp_parity.py
pass (150 passed, 1 skipped); the wider M3 + attention tracked suite matches an
origin/main control run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four review findings, all confirmed.
**The work map was built with the wrong head count under indexer CP.**
`AiterAttentionMetadataBuilder` sized and built it from `num_head_k`, the
TP-local KV head count, while the sparse layer widens decode `index_q` to one
head per CP rank and the scorer consumes that width
(`attention_mha.py`: `num_idx_heads = indexer_cp_world`). On M3 TP4 those are
1 and 4.
This happened to be harmless: aiter's map is bit-identical across head counts
today, verified directly, so the row counts coincided -- and the assert in
`decode_index_score`, which compares that same head-independent number, could
not have caught it either. That is correctness by coincidence in somebody
else's implementation. A new `index_score_num_idx_heads` carries what the
scorer actually uses, threaded through `make_sparse_decode_metadata` alongside
the bound and the CP pair. It stays separate from `num_idx_heads` because the
column counts really are built for the TP-local heads.
**The plugin decode paths rebuilt the map every sparse layer.**
`index_score_work_map_for_forward` had no callers, while its docstring claimed
the vLLM and SGLang backends used it. Both now hoist it to the forward, like
the `n_valid_column_per_row` sibling beside it. Left unwired this was a real
regression for plugin users rather than a missed optimization: on main the
Triton decode scorer needed no map at all, so they would have started paying
~120us of launch floor per sparse layer that main never charged. Each site
bounds the map by its own block-table width, which is capture-stable.
**`test_triton_dynamic_shapes.py` still named the deleted Triton kernel.**
`one_variant(cp._context_score, ...)` raised `AttributeError` on a GPU runner
before reaching the assertions. The scorer is no longer a Triton kernel and
cannot be instrumented that way; the call below still exercises it at every
shape, so what is dropped is the single-variant assertion, not the coverage.
**Prefill selection parity only ran on bf16.** `auto` resolves to flydsl only
on an fp8 cache, so the exact-selection test -- the only one that can see a tie
resolve to a different block -- never covered the configuration the dispatch
exists to reach. Now parameterized over both dtypes; fp8 agrees exactly.
248 passed, 1 skipped across the M3, CP-parity, dynamic-shape and attention
suites.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fill oracle
Both findings confirmed.
**The work-map row-count check was an `assert`.** It guards against a map
built for a different bound being walked by a right-sized grid -- silently
wrong scores, which is exactly the class the repo already handles with an
explicit raise (`_require_packable`, whose comment says `python -O` must not
be able to turn it into a wrap). The bound here likewise traces back to launch
flags and a captured buffer, so it is now a `ValueError`, and the test that
pinned `AssertionError` pins `ValueError`.
**The prefill oracle modelled decode's rounding.** Prefill rounds Q DOWN to
the cache dtype -- `tl.dot(q.to(k.dtype), k)` under `k.dtype.is_fp8()`, and
flydsl's `fp8_mfma` does the same -- while the oracle lifted K to Q, which is
decode's rule. So on fp8 the "independent definition" was checking different
arithmetic than either scorer.
This was closer to failing than the review suggests. Measured against the
kernels on an fp8 cache:
| oracle | max abs err | max rel err |
|---|---|---|
| old (lift K to Q) | 1.69e-2 | 7.26 |
| new (round Q to fp8) | 1.7e-5 | 1.5e-3 |
The old one passed only because `atol=2e-2` happened to sit 18% above its
error. Both scorers agreed with it equally, which is the shared-defect case
the module docstring says these tests exist to catch.
With the oracle right, the tolerance was three orders of magnitude of slack,
so it is tightened from `atol=2e-2` to `1e-3` -- still ~60x the measured worst
case over every shape, head count and backend, and now tight enough to fail
the wrong rounding. The flydsl-vs-triton comparison takes the same bound.
248 passed, 1 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review asked whether a CP-local bound of `ceil(ceil(max_model_len/128)/world)`
needs the native sparse block table padded to `world * max_block`, on the
grounds that the FlyDSL CP contract wants global coverage.
It does not. aiter validates the table against the LOCAL bound, so the native
widths are already legal. Checked at the exact shape raised, `max_model_len
32769` -> 257 global blocks, `max_block 65`, a table of exactly 257 columns:
all four ranks accept it. The padding in the parity helper is the test being
generous, not the contract.
But the question points at something real one layer down. Round-robin
sharding does not divide evenly: rank 0 owns 65 of those 257 blocks and ranks
1-3 own 64, while every shard is 65 wide, so three of four carry a trailing
slot with no global block behind it. The scorer leaves it unwritten -- the
Triton kernel this path replaced instead stored -inf to every slot, empty
shards included, so that guarantee used to live in the kernel and now lives in
`local_candidate_keys`, which masks from `(rank, world_size, global_blocks)`
rather than from the length alone. Nothing recorded the move.
Verified the masking holds (no out-of-bounds read either: the walk stops at
the length bound and never addresses column 257), and pinned it --
`test_a_shard_that_runs_past_the_global_end_is_not_selectable` poisons the
dead slots with +1e30 / -1e30 / 0 at three non-divisible block counts and
demands one selection. A mask that regressed to bounding by length alone
fails it.
The allocation stays `torch.empty`; the comment and docstring now say why
that is sound and name the test that holds it to it, instead of claiming
`local_candidate_keys` "masks by length".
251 passed, 1 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review is right that the hard imports do not exist on aiter `main` -- the
kernels land in ROCm/aiter#5626, still open. Confirmed against the aiter
remote: `MiniMaxM3IndexScoreConfig`, `minimax_m3_index_score_alloc`,
`minimax_m3_index_score_flydsl`, `minimax_m3_index_score_work_map` and
`score_prefill_flydsl` have ZERO occurrences anywhere in `origin/main`, and
`aiter/ops/flydsl/kernels/` carries no minimax file at all; they exist only on
`m3-index-score-flydsl`. ATOM's image builds `AITER_COMMIT: HEAD` and the CI
workflow defaults to aiter `main`, so a module-scope import fails there.
The blast radius is wider than MiniMax-M3, which is the part worth fixing
regardless of merge order. `sparse_attn` imports this module at module scope;
`atom/plugin/vllm/attention/backend.py` imports `sparse_attn` at module scope
for `SPARSE_BLOCK_SIZE`; and that module carries the MLA prefill backends. So
an ImportError here takes those down too, on an aiter that merely predates one
M3 kernel.
The symbols are now resolved on first use through `_flydsl(name)`, which
raises a `RuntimeError` naming ROCm/aiter#5626. Decode still has no fallback
-- that is the point of the hard dependency -- but the failure is now scoped
to M3 decode and says what to install. Verified against a simulated aiter
main: `index_topk`, `sparse_attn`, `models/minimax_m3/model` and
`aiter_attention` all import, and `index_score_config()` raises the named
error. Prefill was already safe: its import is in a try/except and
`_prefill_score_backend` falls back to Triton.
Two tests pin both halves -- that no module-scope `from aiter... import
minimax_m3_*` comes back, and that using the scorer without the kernel names
the dependency. The stub in the second deliberately drops `__getattr__`:
aiter's flydsl package resolves kernels through a module-level lazy loader,
which otherwise hands the stub the very symbols it is meant to lack.
This does NOT make the PR independent of aiter#5626 -- M3 is still broken
without it, and the merge should still follow. It makes everything else
survive the gap.
253 passed, 1 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review is right, and the bug reproduces. aiter's `work_map_size` docstring
says outright that a caller sizing a persistent `[rows, 2]` buffer for a
cudagraph must use `work_map_capacity` instead, "because the exact grid size
is non-monotonic in the bounds" -- and it is:
max_block=256, S=1, H=1: size(2)=256 > size(3)=195
size(7)=455 > size(8)=256
So a buffer sized at the largest batch is too small for a smaller one. With
the buffer allocated at `size(max_bs=8) = 256` rows, a live batch of 7 fails:
ValueError: work_map: expected packed [at least 455, 2] int32
Both persistent allocations in `AiterAttentionMetadataBuilder` -- the main one
and the per-ubatch one -- now use a new `index_score_work_map_capacity`.
`index_score_work_map_size` stays exactly where it belongs: validating a map
that was handed in, in `decode_index_score`. Only allocation moved.
Cost is a bigger buffer and nothing else. At the widest realistic shape
(`max_model_len` 192k, no indexer CP, `max_bs` 128, qlen 8) capacity is 1.5
MiB against 96 KiB, allocated once at model load.
Verified the half this relies on but does not own: aiter accepts an OVERSIZED
`out` and returns a `[rows, 2]` view of it rather than copying, so the caller
does not slice and the exact-row check in `decode_index_score` still sees the
right shape. Pinned by a GPU test that builds into one capacity buffer at both
batch 8 and batch 7 and asserts the returned map aliases it.
The unit tests are self-validating: one asserts the grid really is
non-monotonic, so if aiter ever makes it monotonic the suite says this class
can go rather than quietly testing nothing.
384 passed, 1 skipped. No new lint: the only ruff codes left on the changed
files (B009, BLE001, I001) are identical to origin/main's and sit outside this
diff.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cache, _cu_count pin, test history)
- indexer_context_scores: the zero-token result has the same width as every other
result (max_block or the local bound).
- n_valid_column_per_row_for_forward and index_score_work_map_for_forward share one
_once_per_forward memo instead of two copies of the getattr/setattr cache.
- the work-map capacity and padded-replay tests pin aiter's _cu_count to 256
(MI355X), so they hold on a machine without a GPU or with another CU count.
- test docstrings describe what they pin, not what the change deleted.
Co-Authored-By: Claude <noreply@anthropic.com>
…he ubatch buffers without allocating
The whole-batch map of a split (TBO) decode is read by nobody -- each micro-batch has its own --
so the FP8 builder skips it as the MXFP4 one already does. The per-ubatch buffer shape comes
from the capacity row count instead of a throwaway allocation.
Co-Authored-By: Claude <noreply@anthropic.com>
…ector stays reachable
Review item 4. minimax_m3_index_topk_decode passes the scorer a contiguous, -inf filled
out= (as the indexer-CP path does), so from max_query_len * num_idx_heads >= 16 -- where
aiter's alloc_score switches to a permuted view -- _select_with_aiter is not declined for
want of score.view(rows, max_block) and the wide-row selector (7 vs 36 us at 1M) serves.
The code is in the merge commit's composition of the decode op; this commit makes the
-inf sentinel alloc_score's contract asks of a caller's own out= explicit and adds the
GPU test (the permuted allocation is asserted, the selector is spied to have served, and
its selection equals the Triton selector's).
Co-Authored-By: Claude <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The test spy returns None instead of the selector’s success flag, triggering the Triton fallback and making the parity assertion compare Triton against itself rather than validate AITER’s output.
The reason will be displayed to describe this comment to others. Learn more.
only dtype narrowed, arch not. on gfx1250/gfx942 this dies at first score call inside aiter. refuse the arch here too, or document gfx950-only and drop the dead gfx1250 launch config?
The reason will be displayed to describe this comment to others. Learn more.
Acknowledged, decision pending, not changed in this PR yet.
You are right, and it is wider than gfx1250. aiter's _validate_arch allows SUPPORTED_ARCHS = ("gfx950",) only (aiter 95d211284, minimax_m3_index_score.py:1488-1507), so gfx942 is affected too, and this PR deletes the Triton decode scorer. An fp8 index cache on gfx942 or gfx1250 therefore has no scorer and fails inside aiter at the first decode (at capture on a graph run).
Two ways out, and I need the M3 owners to pick: drop those archs for M3 (refuse non-gfx950 at startup next to the bf16 refusal in _index_cache_spec, say gfx950-only in the recipe, delete the gfx1250 launch config), or keep a Triton decode scorer for them. I do not know whether anyone serves M3 on gfx942/gfx1250 with fp8. The PR body now states this as the open decision.
On the dead gfx1250 branch: _prefill_score_launch_config is reached only through _triton_prefill_score, i.e. prefill_index_score(backend="triton"); production calls prefill_index_score without backend (index_topk.py:1251). Main's launch-policy tests pin it, so deleting it means rewriting those tests, and it goes together with moving the Triton reference under tests/.
The reason will be displayed to describe this comment to others. Learn more.
shipped vllm recipe passes --kv-cache-dtype auto, so this raises at construction. recipe should pass fp8, and the message should name the front-end flag, not the native one
The reason will be displayed to describe this comment to others. Learn more.
Fixed in b97014235. recipes/atom_vllm/MiniMax-M3.md now passes --kv-cache-dtype fp8 in both commands, and its note says fp8 is required and that auto or bf16 is refused at startup. recipes/mesh/Agentic-MiniMax-M3.md, which passed no KV dtype, now passes --kv_cache_dtype fp8 in both commands.
The refusal text comes from one builder, index_cache_must_be_fp8(front_end, got) (index_topk.py:80), so each front-end names its own flag: native --kv_cache_dtype fp8 or --index-cache-dtype fp8, vLLM --kv-cache-dtype fp8, SGLang --kv-cache-dtype fp8_e4m3. The vLLM message no longer mentions --index-cache-dtype. Not run: the mesh recipe with fp8 KV under Mooncake/LMCache.
The reason will be displayed to describe this comment to others. Learn more.
Fixed in ef503aa4c. You are right: served.append(...) returns None, so _launch_select read "aiter declined", ran the Triton selector over aiter's output, and the test compared Triton with Triton.
The spy now returns the selector's own answer; a counting wrapper on the Triton selector kernel asserts it was not launched in the aiter run (and was launched once on the no-bounds call); the two index sets are compared per row. On one MI355X [RESULTS]: the test passes; with the selector forced to decline it fails (assert served == [True], got [False]), and with the selector run and then reported as declined it fails too. A separate aiter-vs-Triton comparison over the permuted shapes (18 cells, ctx 256K/512K/1M, B 2/8/16, (H,S) = (2,8) and (4,4)): aiter served all 18, no Triton launch in those runs, 0 per-row set mismatches.
The reason will be displayed to describe this comment to others. Learn more.
Fixed in ea53d87bd. The native test is renamed test_the_native_kv_pool_allocator_refuses_bf16_by_name. Two new parametrized tests call the vLLM plugin's _index_cache_torch_dtype itself (loaded from its source, since vLLM is not installed where the file runs): auto, bf16, bfloat16 are refused with --kv-cache-dtype fp8 in the message and without --index-cache-dtype; fp8 and fp8_e4m3 resolve to aiter's fp8 dtype.
Checked on CPU at 7d90b2426 by mutation: making the vLLM function never refuse fails the three refusal cases; making it always refuse fails the two pass-through cases; the same for the native and SGLang refusals (each test fails when its refusal is removed). These tests sit in a file that skips without triton/aiter, and the pre-checks job installs neither, so they run on a GPU runner or a dev box, not on that gate. The PR body no longer says both refusals were pinned before this commit.
…wer and on the fallback
The spy returned None (list.append's value), so _launch_select treated aiter's selector as
declined and Triton overwrote its indices: the test compared Triton with Triton. The spy now
returns the selector's result, the test asserts the Triton selector kernel was not launched in
the aiter run (and was launched once without bounds), and the per-row index sets are compared.
Not run on a GPU: none was available.
Co-Authored-By: Claude <noreply@anthropic.com>
…ach refusal names its own front-end's flag
recipes/atom_vllm/MiniMax-M3.md passed --kv-cache-dtype auto (refused at layer construction)
and recipes/mesh/Agentic-MiniMax-M3.md passed no KV flag, i.e. bf16 (refused at KV-pool
allocation); both now pass fp8, and the vLLM recipe's accuracy table says which rows its
commands reproduce. The refusal text was always the native server's "--index-cache-dtype
fp8", a flag vLLM and SGLang do not have: one builder, index_cache_must_be_fp8(front_end,
got), now names --kv-cache-dtype fp8 (vLLM), --kv-cache-dtype fp8_e4m3 (SGLang) or the native
flags, and all six refusal sites use it.
Co-Authored-By: Claude <noreply@anthropic.com>
…pinned; the native test says it is native
test_the_vllm_pool_allocator_refuses_bf16_by_name built the native AiterAttentionMetadataBuilder
and never touched _index_cache_torch_dtype: renamed test_the_native_kv_pool_allocator_... and
it now also checks the native flag is named. New: auto / bf16 / bfloat16 are refused by
--kv-cache-dtype fp8 (and not by the native flag), fp8 / fp8_e4m3 resolve to aiter's fp8. The
plugin module imports vLLM at the top and vLLM is not installed where this file runs, so the
function is compiled out of the module's source. Each new test fails with its refusal removed
(checked by reverting the raise).
Co-Authored-By: Claude <noreply@anthropic.com>
… whitespace-only docs hunk
test_the_symbols_are_not_imported_at_module_scope filtered text lines that start with
"from aiter" and name minimax_m3_index_score, so a black-formatted
"from aiter.ops.flydsl import (" with the name on the next line could not fail it. It now
parses the module and flags any top-level import of aiter.ops.flydsl (single-line, multi-line
or "from aiter.ops import flydsl"); injecting each of the three makes it fail, the clean
module passes. docs/environment_variables.md goes back to origin/main (the PR only added a
blank line).
Co-Authored-By: Claude <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
⚠️ Action required before merge (5)
Don't drop MiniMax-M3 on gfx1250/gfx942 silently (valar-cop, still open). At head both scorers are FlyDSL only, and aiter refuses any arch but gfx950 (SUPPORTED_ARCHS = ("gfx950",), _validate_arch in score_flydsl and score_prefill_flydsl). An fp8 index cache on gfx1250/gfx942 therefore starts fine, then dies at the first prefill inside aiter. On main those archs ran the Triton scorers. This also leaves _prefill_score_launch_config's gfx1250 branch (index_topk.py:98-111) as dead code in production. Either keep the Triton scorers as the non-gfx950 path, dispatched on get_gfx() (an arch check, not an API probe), or refuse non-gfx950 at startup next to the bf16 refusal (aiter_attention.py:547) and delete the gfx1250 config. Get an explicit decision from the M3 owner either way; the description still says "needs a decision before merging".
Fix the vLLM front-end refusal (valar-cop).recipes/atom_vllm/MiniMax-M3.md:41,67 still passes --kv-cache-dtype auto, and _index_cache_torch_dtype("auto") now raises at layer construction (reproduced by a CPU probe). The message also tells vLLM users to pass --index-cache-dtype fp8, which is not a vLLM flag. Update both recipe commands and the note at :75. Make the message name --kv-cache-dtype fp8, and say in the description that on vLLM this also forces the main KV cache to fp8.
Make the aiter-selector test actually check aiter's output (Copilot + valar-cop). At tests/test_minimax_m3_index_topk.py:677-678 the spy returns served.append(...), which is None. _launch_select therefore runs the Triton fallback and overwrites aiter's indices, so the parity block compares Triton with Triton. Store the result, append it, and return it.
Test the vLLM resolver itself (valar-cop).test_the_vllm_pool_allocator_refuses_bf16_by_name (:1093) builds the native AiterAttentionMetadataBuilder. Nothing tests atom/plugin/vllm/attention/minimax_m3_attnetion.py:_index_cache_torch_dtype. Rename that test, and add one where auto/bf16 are refused and fp8/fp8_e4m3 pass through.
Small cleanups. These docstrings no longer match the code at head: sparse_attn.py:104-107 (the map is also None with an MXFP4 scorer or score_work_map=False), sparse_attn.py:191-193 (the split decode skips the map for fp8 too), minimax_m3_attnetion.py:94 ("anything else is bf16"), and the test comment around tests/test_minimax_m3_index_topk.py:1190, which cites auto/k_lds, neither of which exists. Cut the "Triton kernel this path replaced" history from tests/model_ops/test_indexer_cp_parity.py:441-446, which is the rest of my earlier ask 6. Revert the stray blank line in docs/environment_variables.md:167. Then run heavy CI (ci:atom, ci:vllm, ci:sglang): all three front-ends change, and only light CI has run on this head.
Re-reviewed at d693b2f. Already fixed: the work map is now built over running_bs (via main, pinned by the 7-of-8 case in test_minimax_m3_mxfp4_decode_batch.py); the try/except ImportError and the _flydsl(name) getattr are gone (flydsl_ops() used directly); plain decode passes a contiguous out= (index_topk.py:1554); merged with main (4 behind, merges cleanly) and re-measured (+1.2% throughput, -2.0% ITL, prefill scorer -20%); zero-token width, _cu_count mock and one _once_per_forward helper. All aiter symbols used are on aiter main; this does not depend on aiter #7813/#7814. Light CI is green.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Critical risks in mixed-batch metadata alignment and FP16/FP8 compatibility can abort inference in supported configurations, requiring human assessment.
The reason will be displayed to describe this comment to others. Learn more.
still open: this narrows dtype, not arch. flydsl m3 scorers are gfx950 only, so gfx942/gfx1250 die at first score. refuse non-gfx950 next to the bf16 refusal (get_gfx_runtime()), or keep a triton scorer for those archs. and the gfx1250 launch branch is dead now..
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
A maintainer must resolve the supported-hardware contract because the replacement scorer is gfx950-only, the only Triton decode scorer is removed, and gfx1250 support remains undecided.
Round 5, head e55d2d3cd (comments and docstrings only on top of 7d90b2426; CI on the head: 12 pass, 0 fail, 11 skipping). The PR body is updated to this head.
Review of 2026-11-16 (5 items):
Non-gfx950 archs: acknowledged, decision pending, not changed in this PR yet. Same as my reply on valar-cop's arch thread. The body keeps it as "decision needed".
vLLM refusal: fixed in b97014235. Both recipe commands and the note pass/say --kv-cache-dtype fp8; the message names --kv-cache-dtype fp8; the body now says that on vLLM this also makes the main KV cache fp8 (the index cache follows --kv-cache-dtype).
Selector test: fixed in ef503aa4c (details in my reply on the spy thread).
vLLM resolver test: fixed in ea53d87bd (details in my reply on the vLLM resolver-test thread).
Cleanups: fixed in e55d2d3cd (comments, docstrings only; the two test files it touches give the same counts before and after). sparse_attn.py (the work map is also None with an MXFP4 scorer or score_work_map=False; a split decode skips it for fp8 too), minimax_m3_attnetion.py (fp8 / fp8_e4m3 pass, auto / bf16 are refused), the test comment in tests/test_minimax_m3_index_topk.py (the auto / k_lds reference is gone) and the "replaced" history in tests/model_ops/test_indexer_cp_parity.py. The docs/environment_variables.md hunk was already gone in 7d90b2426 (no diff against main). The heavy CI labels (ci:atom, ci:vllm, ci:sglang) are for a maintainer to add.
Review of 2026-11-12 (6 items), for the record:
running_bs: fixed on the merge, pinned by c0069811a. finish_decode builds the sparse metadata and the work map from context_lens[:index_bs], and for the fp8 index cache index_bs = running_bs (aiter_attention.py:1174-1189); the [:scheduled_bs] call site no longer exists. Tests: a CPU case in test_minimax_m3_mxfp4_decode_batch.py (7 live of 8) and a GPU class that captures at bs=8 / max_block 256 and replays 7 live + 1 pad: the replay map has the captured launch's 256 rows (a map over the 7 live requests alone has 455), scores match the definition, and the 7-row map is refused. Passed on an MI355X at d693b2f0e [RESULTS]; those tests are unchanged at 7d90b2426.
score_prefill_flydsl try/except: removed in the merge 4653a6c8b. Prefill calls flydsl_ops().minimax_m3_index_score_prefill_flydsl (index_topk.py, ~1150), the same accessor the decode symbols use.
_flydsl(name): removed in 4653a6c8b; every call site is flydsl_ops().minimax_m3_index_score_*, no string lookup. I used main's flydsl_ops() accessor instead of a per-function from aiter.ops.flydsl import ... because main's MXFP4 path already goes through it.
Selector on plain decode: done, out= is passed as in the CP path; the test is d693b2f0e, fixed in ef503aa4c. One correction to the premise: aiter's alloc_score permutes from S * H >= 16 (minimax_m3_index_score.py:1774,1798), not total_q * H. At TP4 (H = 1) q_len 1 or 4 stays contiguous, so TP4 plain decode was never declined; the declined shapes were TP2 with 7 draft tokens (8 x 2) and TP1 with 3 (4 x 4). Graph-replayed on one MI355X [RESULTS]: the whole scorer+selector op at a 1M bound is 251 -> 227 us and 256 -> 226 us at B = 8, about the same at B = 32, and +3..+7% at 32K (the microbenchmark's noise floor is about 7%). Kept the aiter selector and the out=.
Rebase and re-measure: merged origin/main32981073c instead of rebasing, so the push is a fast-forward and the threads keep their anchors. Re-measured at d693b2f0e (aiter 95d211284, TP4, indexer CP on, aiperf agentx c25, 900 s, two servers per arm, A/B/A/B, --max-model-len unset so the model default 1,048,576 applied; CI runs 32768, not measured) [RESULTS]: throughput +1.2% (1345.5 vs 1329.7 tok/s; +0.95% and +1.42% by cycle), ITL pooled paired -2.0% (CI -3.7..-0.4) but the main arm's own two servers differ by 8% on ITL, TTFT unchanged. Spec 3 at forced acceptance length 3, because main rejects --num-speculative-tokens 7. Decode scorer alone: 14 -> 17 us at B = 8 and 17 -> 25 us at B = 16 at 32K live context, level at 128K, 1-8% faster from 512K; prefill scorer 20-23% faster on a 32768-token chunk. GSM8K, 600 questions, two servers per arm: 91.08% main vs 91.83% branch (+0.75 points, paired SE 0.66). The old +4.3% / -5.1% figures are gone from the body.
Cleanups: indexer_context_scores zero-token width now max_block or local (indexer_context_parallel.py:94-96); one _once_per_forward memo (index_topk.py:330) behind both hoists; _cu_count pinned (details in my reply on the _cu_count thread). Test-docstring history cut in test_minimax_m3_index_topk.py; the one in test_indexer_cp_parity.py is cut in e55d2d3cd.
@valarLip@valar-cop re-review at e55d2d3cd (cleanups; the non-gfx950 decision is still open and listed in the body).
The reason will be displayed to describe this comment to others. Learn more.
still open: gfx950-only scorer, but we only refuse bf16 here. refuse non-gfx950 at startup too (get_gfx_runtime), or keep a triton fallback dispatched on get_gfx()? and the gfx1250 launch branch in index_topk.py is dead, remove it..
The reason will be displayed to describe this comment to others. Learn more.
⚠️ Action required before merge (3)
Settle the non-gfx950 question before merging (valar-cop + my 11-16 ask 1, still open). The author's reply is "decision pending". At head the FP8 index cache's only scorers are aiter's FlyDSL ones, and aiter accepts only SUPPORTED_ARCHS = ("gfx950",). The Triton decode scorers are deleted. validate_runtime_config (atom/models/minimax_m3/config.py:69-71) returns early for fp8, and the sparse FlyDSL check accepts gfx942/gfx1250 (_FLYDSL_PA_ARCHS). So on gfx942/gfx1250, M3 with an fp8 index cache starts, then fails inside aiter at the first score call. On main those archs ran Triton. Pick one: (a) refuse non-gfx950 for M3 at startup with get_gfx(), in validate_runtime_config and in the vLLM/SGLang resolvers next to the fp8 refusal; say gfx950-only in the three recipes; and delete the now-dead gfx1250 branch of _prefill_score_launch_config (index_topk.py:117-131). Or (b) keep a Triton decode scorer for non-gfx950, dispatched on get_gfx(). Get the M3 owner to sign off and record the choice in the description.
Run heavy CI on the final head. All three front-ends change: native, vLLM (refusal plus recipe/OOT entry) and SGLang (refusal plus the accuracy entry). Only light CI ran on e55d2d3cd, where every Accuracy/GPU job was skipped. Add ci:atom, ci:vllm and ci:sglang once item 1 lands.
Small cleanups. In minimax_m3_bridge.py:279-280, the docstring says SGLang raises "with the same message" as _index_cache_spec. It uses the same builder with a different flag, so say that. In minimax_m3_attnetion.py:94, the docstring says only fp8/fp8_e4m3 pass, but the code is startswith("fp8"), so fp8_e5m2 passes too. Either say "any fp8 variant" or check exactly. Optionally, move _index_cache_torch_dtype (it now needs only torch and aiter dtypes) next to index_cache_must_be_fp8 in index_topk.py. The test could then import it directly instead of using the ast+exec loader at tests/test_minimax_m3_index_topk.py:1166-1184.
Re-reviewed at e55d2d3. Already fixed: the vLLM recipes, the OOT entry and the mesh recipe pass fp8, and the message names --kv-cache-dtype fp8 (b97014235); the selector spy returns aiter's answer and counts Triton launches (ef503aa4c); the vLLM resolver is tested directly for refusal and pass-through (ea53d87bd); stale docstrings, the auto/k_lds test note and the "replaced" history are gone, and the env-doc hunk has no diff vs main; _cu_count is pinned to 256. Earlier: running_bs work map, no aiter import probing (flydsl_ops() used directly), contiguous out= on plain decode. Perf at merged head: +1.2% throughput, -2.0% ITL, prefill scorer -20%; the decode scorer is slower at 32K (14 to 17 us at B=8) and 1-8% faster from 512K. GSM8K +0.75 pt. Merges cleanly with main (8 behind). Light CI is green.
Hi @ganyi1996ppo — #3116 (MiniMax-M3 indexer-CP live blocks + mailbox) also rewrites _context_score: the live-block grid and the gfx1250 WMMA TRANSPOSED orientation. So our two PRs conflict in indexer_context_parallel.py and tests/model_ops/test_indexer_cp_parity.py, and valarLip asked us to agree on an order.
#3116 is close to landing: its review items are addressed and CI is green. So I'd propose #3116 first. Then, when #2493 rebases:
@ganyi1996ppo Following up on the order: we'll go ahead with #3116 first, and #3116's description now says so. Its CI, including the heavy ci:atom accuracy jobs, is green. When #2493 rebases, the two cases from my earlier note still apply:
If FlyDSL scores CP decode everywhere, drop or port test_scores_do_not_depend_on_the_split and test_orientations_agree.
I'm happy to do or review that port.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[MiniMax-M3] Score the lightning indexer with aiter's FlyDSL kernels
Branch:
ganyi/m3-flydsl-index-score-integration, merged withmain(32981073c); heade55d2d3cdc49063febf0244859f1d7d2e88c6b34; 18 files, +1551/−449.What this changes
MiniMax-M3's lightning indexer scores every 128-token block of index keys and keeps the top-k. Both phases did that in Triton. This routes them to aiter's FlyDSL MFMA kernels.
Selection is unchanged. The scorers agree on every block a row can pick, and aiter's
alloc_scorewrites the layout the Triton kernel wrote belowS * H = 16, and above it the decode op asks for that layout without=(see "Two aiter contracts"), so everything downstream is blind to the choice. This is a pure performance change — no accuracy decision is being made here.prefill_index_scorekeeps abackend=parameter that production never passes. It is the entry point forTestPrefillScorersAgree, which is what backs the claim above: the Triton_index_block_score_kernelis the reference the FlyDSL scorer is checked against, block by block. Remove the parameter and that claim loses its oracle._decode_index_score_tiled_kernel(Triton)minimax_m3_index_score(FlyDSL) — the only scorer_context_score(Triton)cp_world/cp_rank_index_block_score_kernel(Triton)minimax_m3_index_score_prefill(FlyDSL). The Triton kernel stays as the tests' reference, reachable by naming it, not by configurationDecode keeps no fallback. The two Triton decode kernels are deleted and the aiter import is hard, so an aiter that lacks the kernel is an
ImportErrorat module load rather than a silent downgrade discovered 57 layers into a decode step.Performance
End-to-end, re-measured at the merged head
ganyi/m3-flydsl-index-score-integrationatd693b2f0e(a merge oforigin/main32981073c) againstorigin/main32981073c. The commits afterd693b2f0eup toe55d2d3cdchange tests, recipes, the text of the index-cache refusal and comments / docstrings, and no scoring code (git diff d693b2f0e e55d2d3cd -- atomtouches only the refusal messages and comments / docstrings). Setup: aiter95d211284, flydsl 0.5.0. MiniMax-M3-MXFP4, TP4 on one quad of MI355X shared by every run, indexer CP on,--kv_cache_dtype fp8,--block-size 128,--max-model-lenunset (the engine takes the model's 1,048,576; CI runs 32768, and a server at that bound was not measured), eagle3 (GQA draft),ATOM_PA_FLYDSL=1, aiperfinferencex-agentx-mvp, concurrency 25, 900 s per run, seed 42. Two servers per arm, order main, branch, main, branch.One deviation: spec 3 at a forced acceptance length of 3, not spec 7 / 5. Current
mainrefuses--num-speculative-tokens 7at config time (num_speculative_tokens must be between 1 and 4), for both arms.Throughput: +1.2% (arm means 1345.5 vs 1329.7; +0.95% and +1.42% in the two cycles from the unrounded values, +1.43% from the table's rounded ones; the arms repeat within 0.3%). ITL: no difference in cycle 1 (-0.4% whole run, -0.0% paired per request), -7.1% / -4.3% in cycle 2, which is the main arm's second server running 8% slower on ITL than its first; the median ITL is lower on the branch in both cycles (-3.2%, -8.9%). Paired on (conversation, turn) and pooled over the runs, ITL is -2.0% (95% CI -3.7..-0.4, request-level) and request latency -1.5% (CI -3.2..+0.3). TTFT is unchanged (+0.4%, CI -3.4..+4.4). This is smaller than the figures this description carried before (+4.3% throughput, -5.1% ITL).
Scorer kernels, re-measured (one MI355X)
Decode scorer, one index head (TP4), bound = 1M tokens, graph-replayed: FlyDSL is slower than the Triton scorer it replaces at 32K (14 -> 17 us at B=8, 17 -> 25 us at B=16), level at 128K, and 1-8% faster from 512K (e.g. B=32: 391 -> 360 us at 512K, 720 -> 699 us at 1M).
Prefill scorer, 32768-token chunk, one index head, fp8 cache: 302 -> 241 us (-20%) at 32768 tokens of prefix, 455 -> 350 us (-23%) at 69632, 633 -> 506 us (-20%) at 118784. This replaces the "-36% / -42% / -46%" table, which was taken on an older aiter against an older Triton scorer.
The earlier finding that FlyDSL is 3.6 to 4.8 times slower than Triton for a bf16 index cache (measured on an older aiter that still had the bf16 path, not re-run) is why the index cache is fp8 only. On M3 the index cache dtype follows
--kv_cache_dtypeunless--index_cache_dtypeis given (atom/config.py:3071-3091), so in practice--kv_cache_dtype fp8selects the supported path.Accuracy
GSM8K, 5-shot chat, temperature 0, real eagle3 (MHA draft, 3 speculative tokens, acceptance not forced), fp8 KV and index, indexer CP off, same prompts for both arms, main
32981073cagainstd693b2f0e: 600 questions, two servers per arm: main 91.08%, branch 91.83%, branch - main +0.75 points (paired SE 0.66, 95% CI -0.55..+2.05). A single-server pair on the first 200 of those questions read 95.0% vs 93.0% (-2.0 points, paired SE 1.6, 7 questions only main right, 3 only the branch), which is inside the noise of 200 questions. No regression is detectable at this scale; one of about half a point is not excluded, and the prompts are short, so long-context selection is not covered (the scorer-against-the-definition GPU tests cover that at kernel level).Earlier accuracy measurement (older head, kept for the scorer-profile and acceptance checks)
The throughput runs above force an acceptance length, which makes the generated text meaningless, so accuracy was measured separately, without forcing acceptance.
Setup. MiniMax-M3-MXFP4, TP4, real eagle3 (
MiniMax-M3-EAGLE3, MHA draft) with 3 speculative tokens,--index-cache-dtype fp8, fp8 KV cache,ATOM_FORCE_ATTN_TRITON=1, indexer CP off. Imagerocm/atom-dev:nightly_202610171506, GPUs 4–7. gsm8k 5-shot, chat template, temperature 0, 1319 docs, three reps per arm, one server launch per arm. The arms are the whole PR: headc0a9913b4(FlyDSL scorers) against its merge-base661ef7661(Triton scorers). Each arm ran from its own source tree, and the import paths were checked.Per-rep stderr is about ±0.0065. FlyDSL minus Triton is −0.07 points flexible and −0.15 points strict; the paired per-doc difference of the 3-rep means (flexible) is −0.08 points, 95% CI [−0.68, +0.53]. The recipe's reference for eagle3 is 0.9469 flexible / 0.9477 strict, about 0.5 points above both arms.
At temperature 0, identical-config reps already differ in about 78% of response texts (eagle3 and batching), so text differences carry no signal. The answer differs on 3.6–4.7% of docs between two reps of the same arm and on 4.3% between the arms, with right→wrong and wrong→right balanced. The 118 docs whose answer is unstable within an arm are the same 118 across arms; no doc disagrees only across arms.
Eagle3 acceptance matches: 73.08% (FlyDSL), 73.31% (Triton), 73.36% (recipe).
Which scorer ran was checked with the torch profiler on all four ranks, not assumed. The FlyDSL arm shows
m3_index_score_prefill_*andm3_index_score_S4_H1_fp8_*kernels and no Triton scorer kernel; the Triton arm shows_index_block_score_kerneland_decode_index_score_tiled_kerneland nom3_index_score*.What this does and does not show.
The work map, and where it is built
The FlyDSL decode kernel takes a packed dispatch map so a ragged batch does not pay for its holes. Building it is ~15 trivial device ops whose cost is almost entirely launch floor: ~120 µs a call, against a score kernel of 17–250 µs. Per sparse layer that costs more than the kernel it feeds.
It is therefore built once per decode step in the metadata builder (
MiniMaxM3SparseMetadata.index_score_work_map) and threaded to both the plain and the indexer-CP decode paths. Two properties this forces:n_valid_column_per_rowalready is.ceil(max_model_len / 128)— never the batch's longest request, which would reshape the map under a fixed grid. Under indexer CP it is this rank's local bound,ceil(that / world), because the shard is what the kernel is dispatched against.Omitting the map is still correct —
decode_index_scorebuilds its own — which is what the test and plugin paths do.Two aiter contracts, now checked instead of assumed
alloc_scorereturns a permuted view. PastS * H >= 16(S = query tokens per request, H = index heads on the rank; aiter'sTRANSPOSE_MIN_F) aiter allocates[max_block, total_q, H]and hands back a feature-contiguous[H, total_q, max_block]permutation, which coalesces the scorer's store. That is a decode with 4 or more query tokens at CP world 4 (H = 4), 8 at TP2 (H = 2), or 16 at TP4 without CP (H = 1).local_candidate_keysrejects a non-contiguous score outright — exactly the check aiter's own docstring asks callers to make. The CP path now requests the layout it needs through a newout=ondecode_index_score.Both layouts carry bit-identical values (verified directly: max abs diff 0.0), so this costs the store's coalescing and nothing else, and it beats copying after the fact.
_select_with_aiterdeclines on the same condition, since its flattening is a.view.Block tables must cover
max_block. A recent aiter rejects a table narrower than the bound rather than reading past the row. Production never produces that pair — the engine sizes block tables frommax_model_len, the same quantity the bound comes from — so the tests were widened to match production rather than the check relaxed. Under CP the widening is byworld * pad, since the bound is local and the table is global.Tests
At
7d90b2426, with no GPU, on the touched M3 / attention / TBO test files (tests/test_minimax_m3_*.py,tests/model_ops/test_indexer_cp_parity.py,tests/model_ops/test_triton_dynamic_shapes.py,tests/test_indexer_cp_gate.py,tests/test_tbo_decode_ubatch_split.py,tests/test_h2d_attention_publication.py,tests/test_mtp_index_share.py,tests/test_mla_index_cache.py,tests/test_paged_attention_dispatch.py,tests/model_ops/test_sparse_indexer_chunk.py) the branch has 2 failed, 853 passed, 269 skipped, againstmain(32981073c) 2 failed, 922 passed, 206 skipped. Both failures are environment ones and appear on both sides:tests/test_paged_attention_dispatch.py::TestDenseDecodeSplits::test_batch_one_asks_for_the_ceiling(needs a GPU), and the subprocess test intests/test_minimax_m3_flydsl_optional.py(namedtest_importing_needs_no_flydsl_...on the branch,test_bf16_and_fp8_need_no_flydsl_...onmain), which on the branch fails withNo module named 'aiter.ops.triton.fused_kv_cache'from the container image's stock aiter. Fewer tests pass on the branch because the PR deletes the Triton decode scorers and their tests.e55d2d3cd(comments and docstrings only) was checked on the two test files it touches (tests/test_minimax_m3_index_topk.py,tests/model_ops/test_indexer_cp_parity.py): 42 passed, 151 skipped, 0 failed at7d90b2426and ate55d2d3cd; the full selection above was not re-run ate55d2d3cd.The GPU run (one MI355X) and the wider tracked-suite comparison were not re-run at this head; the counts quoted earlier in this description are gone.
A user-visible narrowing: the index cache is fp8 only
Before:
index_cache_dtypedefaults tokv_cache_dtypeand then to"bf16", and the Triton decode scorer served it.After: the FlyDSL scorers serve fp8 only (aiter#5626 cut the bf16 cache path; it is on aiter
mainnow, as1305f0c8b), and this PR deletes the Triton decode scorers, so a bf16 index cache is refused at startup, before any request, with a message naming the fp8 flag of the front-end that refuses (below). On vLLM the index cache follows--kv-cache-dtype, so fp8 there is also the main KV cache's dtype.The shipped recipes and CI entries pass fp8 now: this PR changed
recipes/atom_vllm/MiniMax-M3.md(--kv-cache-dtype autotofp8),recipes/mesh/Agentic-MiniMax-M3.md(no KV dtype to--kv_cache_dtype fp8) and two CI entries that did not pass fp8 (the vLLM-OOT M3 entry inoot_benchmark_models.json, andMiniMax-M3-MXFP4 TP4insglang_models_accuracy.json). Anyone else relying on a bf16 orautoKV cache is now refused.recipes/MiniMax-M3.mdnow says the flag is required rather than a tuning choice.The three front-ends refuse at different stages. SGLang in
_atom_m3_index_dtype(fromconfigure, before weights load); vLLM in_index_cache_torch_dtype(when the sparse layer is constructed,atom/plugin/vllm/attention/minimax_m3_attnetion.py); native ATOM in_index_cache_spec(when the KV pool is allocated, after weights load). They build the message with one shared function,index_cache_must_be_fp8(front_end, got), so each names its own flag (native--kv_cache_dtype fp8or--index-cache-dtype fp8, vLLM--kv-cache-dtype fp8, SGLang--kv-cache-dtype fp8_e4m3) and none can drift.Each of the three refusals is pinned by a test that calls the refusing function itself, each checked on CPU at
7d90b2426by removing the refusal and watching its test fail (for vLLM also by refusing everything and watching the fp8 pass-through test fail), and each also asserts that fp8 still resolves so none can pass by refusing everything. One limit worth stating: those tests live in M3 files thatimportorskiptriton and aiter, and the only CI job that runstests/has neither, so they run on a GPU runner or a dev box and not on the pre-checkin gate.Refusing at startup is the point. The same configuration otherwise starts, serves a prefill, and dies 57 layers into a decode step with
ValueError: cache: invalid dtype or rankraised inside aiter, which names neither the flag nor ATOM as the thing refusing.decode_index_scoreand the CP validator carry the same refusal as a backstop for paths that did not come through the allocator, built by the same function (with no front-end named, it lists all three flags) so they cannot drift.On any GPU other than gfx950 (gfx942, gfx1250) an fp8 index cache has no scorer after this PR: aiter's FlyDSL scorers are gfx950-only (
SUPPORTED_ARCHS = ("gfx950",)) and the Triton decode scorer is deleted. It starts, then fails inside aiter at the first decode (at capture on a graph run), where onmainthe Triton scorers served those archs. I did not check whether such a deployment exists. A decision is needed before merging: drop those archs for M3, or keep a Triton decode scorer for them.Not in this PR
The shuffled index-cache layout (
ATOM_M3_INDEX_SHUFFLED) and the fp8 index query (ATOM_M3_INDEX_FP8_Q), both deleted along with their env vars.The shuffled writer measured as a regression in its own right (−12.9% throughput, +17.4% ITL), and in the earlier overnight sweep (its tables are no longer in this description; they were taken at an older head) the shuffled arm came in below the plain one. With aiter no longer gating
k_ldson the layout, there is nothing left for it to buy.The fp8 query goes with it because it changes decode scores: decode's default bf16 MFMA lifts K to Q's precision rather than rounding Q down, so an fp8 Q there is not free, and a tie reordered near the top-k cut changes which blocks get selected. That is an accuracy decision and does not belong in a performance PR. (In prefill an fp8 Q is bit-identical, since
fp8_mfmawas rounding Q at staging anyway — which is why nothing is lost on that side.) Say the word if you want it kept as an opt-in.GLM-5.2 DCP sparse prefill — a separate PR, as discussed.
🤖 Generated with Claude Code