Skip to content

perf(minimax-m3): indexer-CP decode by live blocks, with a one-launch candidate exchange - #3116

Open
lirui927 wants to merge 17 commits into
ROCm:mainfrom
lirui927:lirui/m3-indexer-cp-live-blocks
Open

lirui927 wants to merge 17 commits into
ROCm:mainfrom
lirui927:lirui/m3-indexer-cp-live-blocks

Conversation

@lirui927

@lirui927 lirui927 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Size the MiniMax-M3 indexer-CP scorer and local top-k by each request's live blocks. A shard row up to two 512-block tiles walks narrow tiles; a longer one takes a single tile of up to 2048 blocks. When the launch fits that tile, its loop is not compiled.
  • Use the gfx1250 WMMA orientation with Q as M and K as N.
  • Fuse local top-k, the cross-rank candidate exchange and the merge into one mailbox kernel on gfx950/gfx1250. It emits the same sparse block-table geometry as the collective path, including the MXFP4 KV's (1, 1) from feat(sparse-attn): decode-time indexer context parallelism across TP ranks on the MXFP4 index cache (MiniMax-M3.1 first) #3104.
  • Agree mailbox setup failures across ranks and fall back to the collective path consistently. A delayed peer is waited for, so stale keys are never merged. A missing aiter symbol fails setup instead of falling back.

The fused and fallback paths share the selection, merge and input-validation helpers. Live scores and final selection outputs are bit-identical to the previous path.

Results

Baselines are main before this PR, or the PR's earlier head where noted.

  • MI455X TP4 sparse selection: 46.1-47.3 us -> 27.0-27.9 us.
  • Decode step p50: 11.00 ms -> 9.75 ms (-11.4%).
  • MLPerf agentic: 628.5 -> 667.4 tok/s/GPU (+6.2%).
  • MI355X TP4 decode step: 2.5% faster from the scorer/local-top-k changes.
  • MI355X mailbox fused max-rank p50: 23.2%, 24.0% and 27.9% faster at bs16/32/64.
  • Cold first-decode compile per rank: 2.17 s -> 1.70 s; worst-case variants: 19 -> 14 (scorer LDS: 16 -> 35 KiB).
  • Local top-k on one gfx1250 GPU (local_candidate_keys, graph replay, 2048-block launch = max_model_len 1M at TP4). Main and this PR run interleaved in one process; each figure is the range of per-session medians, main -> this PR:
    • 32 x 194K: 4.98-5.00 -> 2.66-2.67 us.
    • 32 x 524K: 4.98-5.34 -> 4.17-4.18 us.
    • 32 x 900K: 4.97-5.00 -> 4.91-4.93 us.
    • 64 x 900K: 5.14-5.17 -> 5.07-5.09 us (two of six sessions read 5.5 us).
    • 32 x 4 (MTP) x 900K: 6.72-6.76 -> 6.75-6.77 us.

Tests

  • 4 x MI455X: 234 passed, 1 skipped. Mailbox eager/graph parity passed 3/3, including tag and counter wraparound.
  • 4 x MI355X: mailbox passed 3/3, with a 1 s eager/graph rank skew, three 12K-calls/row stress runs, 179 regression tests and 7 dynamic-shape tests. There were zero mismatches, hangs, UECs, spills or scratch use; polling stayed in-loop, and .wt/.cv lowered to sc0 sc1.
  • After the rebase, on one gfx1250 GPU: indexer-CP live-block, parity, mailbox-setup, gate, gather and MiniMax-M3 top-k tests gave 259 passed and 19 skipped (4-GPU and gfx950-only tests). The parity tests include a new 1.1M row that walks two wide tiles, and the live-block tests a new rank with zero live blocks. The 5 failures also fail on main in the same image: two gate tests import a missing test module, and three top-k tests hit a wave32 FlyDSL top-k limit.
  • At this head on 4 x MI455X: the mailbox parity test passes (eager and graph, including an 8 x 900K row). Selection + exchange + merge, max rank, main's collective path -> this PR's mailbox path: 32 x 194K 14.76 -> 6.85 us; 32 x 900K 15.10 -> 9.40 us; 64 x 900K 15.50 -> 9.61 us; 32 x 4 x 900K 17.07 -> 11.68 us. The PR's collective fallback (mailbox unavailable) is within 3% of main at long rows (32 x 4 x 900K 17.50 vs 17.07 us).
  • MiniMax-M3 accuracy at this head, TP4 on 4 x MI455X, MXFP4 + EAGLE3 (3 draft tokens), FP8 KV, indexer CP on (ATOM_M3_INDEXER_CP=1, mailbox exchange in use): GSM8K 5-shot, all 1319, greedy: 95.15% flexible-extract / 94.01% strict-match; EAGLE3 acceptance 85.23%. (aiter main's topk_per_row_small_k was mounted into the test image, whose aiter predates its wave32 support, which ATOM main's M3 index top-k calls on gfx1250.)
  • black and ruff are clean.

Not covered

  • End-to-end decode timing after the rebase and the wide-tile change.
  • The wide tile has run on gfx1250 only, not gfx950.

Merge order

This PR lands before #2493. Both rewrite _context_score and conflict in indexer_context_parallel.py and tests/model_ops/test_indexer_cp_parity.py. When #2493 rebases on top of it:

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 3116 --add-label <label>

@lirui927
lirui927 force-pushed the lirui/m3-indexer-cp-live-blocks branch from 169b6b6 to 789d2b7 Compare November 9, 2026 13:05
@lirui927 lirui927 changed the title perf(minimax-m3): size the indexer-CP decode scorer and local top-k by live blocks perf(minimax-m3): indexer-CP decode by live blocks, with a one-launch candidate exchange Nov 10, 2026
lirui927 pushed a commit that referenced this pull request Nov 10, 2026
… candidate exchange (#3116)

Squash of #3116 at 52f35c2, by @lirui927.
On top of #2873 here, its two CP chunk-policy tests now check the
#3116 score grid instead of _decode_score_chunks (the CP scorer no longer
uses it); they still fail if CP reaches the native chunk policy.
@lirui927
lirui927 marked this pull request as ready for review November 12, 2026 07:24
@lirui927
lirui927 requested a review from valarLip November 12, 2026 07:29

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few small things..

ci Run non-GPU unit tests failed — not sure yet (low): The captured log (.valar/ci/job_131037224269.log) is empty, 0 lines, so nothing attributes the failure. The same job passed on main at 2026-11-12T05:53 and failed at 05:36 the same day, so an infra/flaky origin is possible but unproven.

next: Re-run Pre Checkin and pull the job's raw log; if it reproduces on the head commit, bisect against the CPU test this PR adds (tests/test_indexer_cp_mailbox_setup.py), the only non-GPU test newly in scope. https://github.com/ROCm/ATOM/actions/runs/43175072925/job/131037224269

Comment thread atom/distributed/indexer_cp.py Resolved
Comment thread atom/distributed/indexer_cp.py Resolved
@lirui927
lirui927 requested a review from valar-cop November 12, 2026 08:40
lirui927 added a commit to lirui927/skills that referenced this pull request Nov 12, 2026
…er top-k

ROCm/ATOM#3116 把 indexer CP 的本地 top-k、候选交换和 merge 合成一个
Triton kernel `_select_exchange_merge`,原来的分类规则不认识它,会落进
misc_small。m3_common.py 把它归到 indexer_topk;m3_xlsx.py 的 tab 备注
同步写明这条路径计在 sparse_index_topk 里;test_common.py 加一条分类测试。

Co-authored-by: Cursor <cursoragent@cursor.com>
@lirui927

Copy link
Copy Markdown
Contributor Author

On the Run non-GPU unit tests failure from the review: not this PR. The one failure in run 43175072925 was tests/model_ops/engram/test_host_path.py::test_refused_scale_registration_gives_back_the_values_it_pinned, in a file this PR does not touch. The same head e467dd2 passed 50 minutes earlier, and the job has passed on every later head, including the current one.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (4)

  1. Stop turning a missing private aiter symbol into a quiet fallback. atom/distributed/indexer_cp.py:144 calls _mailbox_runtime() (which imports aiter's private _loaded_hip_runtime / _should_use_vmm) inside except Exception at :163, and tests/test_indexer_cp_mailbox_setup.py asserts that an injected ImportError falls back. That is try/except-ImportError probing of aiter privates: an aiter rename silently drops the 23-28% win with only a warning (same failure class as #3162). Import them directly before the try (or at module level after the arch gate) so a missing name fails at startup. Keep the try for OOM / hipMemset / mapping, and drop the injected-ImportError case from the test. _loaded_hip_runtime().hipMemset can use ATOM's own atom/mono/runtime/hip_ipc.py runtime (hipMemset bound at :91-92). Ask aiter to make _should_use_vmm public.
  2. Own the live-block _local_topk change here, and add long-row numbers. #3104 also rewrites _local_topk / local_candidate_keys in indexer_candidate_exchange.py for the same goal; it has been asked to drop that and agree with you. This PR should keep it because _shard_winners is shared with the fused mailbox kernel and the diff is about 15 lines against #3104's 3-width plus group-max-filter plus scratch design. One gap: LOCAL_TOPK_TILE = 512 (:30) turns main's single 2048 tile for a ~1M row (2048 local blocks at TP4) into a 4-tile walk, and #3104 measured the walk as the bottleneck. Every number here stops at 194K. Post _local_topk µs vs main at near-max context (for example 32 x 900K, TP4). If it regresses, size the tile to the row (one tile up to 2048) as #3104 does.
  3. Test a rank with zero live blocks. tests/model_ops/test_indexer_cp_live_blocks.py:67 bottoms out at 2 live blocks on rank 3. Add a short row (length <= 384 at rank 3) so the live_end == 0 path in _shard_winners / _context_score is exercised. Short decode rows are common.
  4. Fix the warmup_exchange docstring. indexer_cp.py:347 says max_mailbox_rows "comes from the runner's resolved mtp_k + 1", but the runner passes max_num_seqs * (mtp_k + 1) (model_runner.py:4159).

CI: Pre Checkin is green. MiniMax-M3 accuracy has not run at this head; please add ci:atom.


Reviewed at 80f7237. Already addressed from valar-cop: shared validation for the fused and fallback paths, the aiter rev / ROCm floor doc, and the gfx950 gate with MI355X numbers.

@valar-cop

valar-cop commented Nov 13, 2026 •

Copy link
Copy Markdown

wanzhenchn added a commit that referenced this pull request Nov 14, 2026
The one-tile rewrite of `_local_topk` also changed the selector the
BF16/FP8 indexer-CP path runs, and #3116 rewrites the same kernel for the
same goal. Restore main's kernel and its tests so the change lands once,
with #3116. The MXFP4 wiring does not depend on it: main's selector
already masks the lanes past a row's blocks.

Co-authored-by: Cursor <cursoragent@cursor.com>
valarLip pushed a commit that referenced this pull request Nov 14, 2026
…ranks on the MXFP4 index cache (MiniMax-M3.1 first) (#3104)

* minimax_m3: read each indexer-CP shard row in one tile, filtered

The step's local top-k waited on its longest row, which walked its shard
512 blocks at a time. Each row now stops at its own shard length and takes
one tile of the narrowest width covering it (512/1024/2048), and a wide
tile sorts only the keys at or above its group-max bound. Output is
unchanged; batch 24-60 drops from 8.8-15.8 to 6.0-10.0 us a layer.

merge_candidate_keys takes the emitted page geometry (pages_per_block,
block_page_stride), defaulting to the page-16 SHUFFLE cache's 8 / 8 as
before, so a page-128 KV can be merged into its own block table.

Co-authored-by: Cursor <cursoragent@cursor.com>

* minimax_m3: run the indexer-CP decode on the MXFP4 index cache

Under indexer CP each TP rank scores every index head over its 1/W of the
blocks. With the MXFP4 index cache that scorer is aiter's FlyDSL decode
scorer, which already takes the shard (cp_world / cp_rank with k_scale):

- IndexScoreBuffers carries the rank's shard; allocate_index_scorers sizes
  the work map at the shard's bound ceil(blocks / W) over every index head,
  and the workspace for that as well as for a prefill chunk (which stays
  TP). Unsharded, every call is the one it was.
- make_sparse_decode_metadata builds the step's map for the shard.
- indexer_context_scores_mxfp4 scores into a contiguous view of the shared
  workspace on that map; _cp_index_topk_decode runs it for the MXFP4 index
  and merges with the KV's page geometry (1 / 1 for the MXFP4 KV).
- indexer_cp_unsupported_reason no longer refuses the MXFP4 caches.

tests/test_minimax_m3_mxfp4_indexer_cp.py pins the wiring on CPU, and on
gfx950 emulates the 4 ranks against minimax_m3_index_topk_decode: the merged
top-k, sparse block table and context must equal the TP path's bit for bit,
for both page geometries, on a never-filled workspace.

Co-authored-by: Cursor <cursoragent@cursor.com>

* minimax_m3: check the indexer-CP FlyDSL calls against aiter's signatures

The MXFP4 CP decode hands aiter's scorer k_scale and the shard (cp_world /
cp_rank), and its work map and capacity calls the shard. A new GPU-free
test runs those three calls and binds each to the installed aiter's
signature, so an aiter older than ROCm/aiter#7060 (no k_scale) fails at
test time rather than at the first MXFP4 CP decode. It skips where
aiter.ops.flydsl does not import.

Co-authored-by: Cursor <cursoragent@cursor.com>

* minimax_m3: give the local top-k tile choice without its timings

The comment on the shard-row tile widths and the full-grid threshold
quoted per-layer latencies from one deployment. Keep the reasons alone:
one tile of the narrowest covering width, and four warps with an
unfiltered narrow tile below the threshold, two warps and the filter at
or above it.

Co-authored-by: Cursor <cursoragent@cursor.com>

* minimax_m3: leave the indexer-CP shard top-k as on main

The one-tile rewrite of `_local_topk` also changed the selector the
BF16/FP8 indexer-CP path runs, and #3116 rewrites the same kernel for the
same goal. Restore main's kernel and its tests so the change lands once,
with #3116. The MXFP4 wiring does not depend on it: main's selector
already masks the lanes past a row's blocks.

Co-authored-by: Cursor <cursoragent@cursor.com>

* minimax_m3: say what the shard top-k does past a row's blocks

`local_candidate_keys` loads each row up to the shard's block bound and
masks the lanes past the row's blocks, but the MXFP4 scorer's docstring
said it read none of them. The test's workspace docstring said the
workspace was filled and then never filled; it now says what the junk
is for.

Co-authored-by: Cursor <cursoragent@cursor.com>

* minimax_m3: keep the merge's page geometry for the MXFP4 KV

871b053 restored main's whole exchange file, and with it main's
`merge_candidate_keys`, which always emits the page-16 SHUFFLE geometry.
The MXFP4 CP decode passes `pages_per_block` / `block_page_stride` (1, 1),
so it raised a TypeError at its first merge. Bring the two arguments
back; the shard top-k stays main's.

Co-authored-by: Cursor <cursoragent@cursor.com>

* recipe(minimax_m3): say what indexer CP measured on the MXFP4 caches

The CONC >= 15 paragraph and its table are back to main's text: that table
was measured with the FP8 KV of the server command and does not carry over.
A paragraph after it gives the MXFP4 A/B on one aggregated TP4 server (lower
tok/s/GPU in all six cells, median ITL down only at 128K and CONC 64), says
the throughput gain was measured only on 1P2D decode nodes, which need MXFP4
KV under P/D and are not on main yet, and says to leave CP off on an
aggregated MXFP4 server.

Co-authored-by: Cursor <cursoragent@cursor.com>

* recipe(minimax_m3): median ITL fell in all three 128K cells

The MXFP4 paragraph said median ITL fell only at 128K and CONC 64, but
the A/B shows it falling in every 128K cell (0.8-4.7%, most at CONC 64)
and rising at 32K (5.1% at CONC 32, none at 128). The test module's
docstring called its kernel-free tests CPU tests; like every test in the
file they need aiter installed.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
@lirui927
lirui927 force-pushed the lirui/m3-indexer-cp-live-blocks branch from 80f7237 to 1dee409 Compare November 16, 2026 02:52
@lirui927

Copy link
Copy Markdown
Contributor Author

Rebased onto main e9c6ae0. #3104 had added the MXFP4 scorer and the merge's page geometry (pages_per_block / block_page_stride). The fused mailbox kernel now takes the same geometry, and _cp_index_topk_decode passes **self._select_page_kwargs on both paths, so the MXFP4 KV gets its (1, 1) block table from the fused path too. The PR's three original commits keep their content apart from that.

1. Missing private aiter symbol fell back quietly (164b69e).
CandidateMailbox now imports IPCBufferPool / _should_use_vmm before the try. A missing name raises at startup on every rank: _open_candidate_mailbox only catches RuntimeError. The try still covers the real capability and resource failures: ROCm < 7.14 / VMM, OOM, hipMemset and peer mapping. hipMemset now goes through ATOM's atom/mono/runtime/hip_ipc.py (load_runtime / check_hip_error), so _loaded_hip_runtime is no longer used. The setup test no longer has the injected-ImportError case. A world-1 smoke on one gfx1250 GPU opened a real mailbox through that path (_should_use_vmm(True) False). Making _should_use_vmm public in aiter can follow separately.

2. Long-row _local_topk numbers (1dee409).
It did regress. One gfx1250 GPU, local_candidate_keys with graph replay, with a 2048-block launch (max_model_len 1M at TP4), us a call, main / PR before / now:

rows 194K 524K 786K 900K 1M
32 x 1 4.60 / 2.25 / 2.29 4.61 / 3.58 / 3.93 4.62 / 4.94 / 4.70 4.62 / 6.28 / 4.71 4.62 / 6.29 / 4.71
64 x 1 4.74 / 2.31 / 2.36 4.76 / 3.68 / 4.07 4.78 / 5.08 / 4.84 4.78 / 6.46 / 4.85 4.79 / 6.47 / 4.86
32 x 4 6.70 / 2.74 / 2.7-3.5 6.76 / 4.74 / 5.4-5.7 6.78 / 6.73 / 6.9-7.3 6.80 / 8.69 / 6.9 6.80 / 8.71 / 6.9-7.3

The fix is the one you suggested, in a smaller form. A row whose shard part is longer than two 512 tiles now takes one tile of up to 2048 (LOCAL_TOPK_WIDE_TILE). Shorter rows keep the 512 walk. It is a single branch in _shard_winners, so _local_topk and the fused kernel both get it. Near 1M it now matches main to within about 0.1 us (QLEN 4: 6.9 vs 6.8). The trade-off is the 768-1024-block band (393K-524K), which now costs +0.3-0.6 us over the PR before (3.9 vs 3.6 at 32 x 1), but is still below main. The 32 x 4 timings varied between runs, so the table shows a range. Tests: a 600K row in test_indexer_cp_parity.py covers the wide tile against the TP selector. An 8 x 900K case in the 4-GPU test_indexer_cp_mailbox.py covers the fused kernel, and both now ran on 4x MI455X: the mailbox parity test passes (eager and graph), and the fused kernel, max rank, us a call, before / after the wide tile:

rows before after
32 x 194K 6.67 6.95
32 x 900K 10.74 9.40
32 x 4 x 900K 16.16 11.79
64 x 900K 11.12 9.72

3. Zero-live-block rank (f8f2b47).
test_wide_launch_matches_real_length adds a (2, 300) case. Each row has three blocks, so rank 3 has live_end == 0 in both the scorer and the shard top-k. The test also checks that such a rank emits no key (bool(keys.any()) == (real_len > 128 * rank)).

4. warmup_exchange docstring (600afa6).
It now says the runner passes max_num_seqs * (mtp_k + 1), with the drafter's resolved mtp_k (0 without one).

Also 2071685: #3104's indexer_context_scores_mxfp4 docstring said local_candidate_keys reads each row up to max_block. With this PR it reads only the live blocks, and the docstring now says so.

CI: noted the ci:atom request; the label has not been added here.

@lirui927

Copy link
Copy Markdown
Contributor Author

Follow-up on long rows (7545db2): long rows now match or beat main.

The extra ~0.1 us came from the wide-tile path still compiling a loop over further wide tiles, which never runs when the launch fits one tile (up to 2048 local blocks, i.e. 1M at TP4). A host-side constexpr now leaves that loop out, in _local_topk and in the fused kernel. Results stay bit-identical; the long-row parity test now uses a 1.1M row, so it still walks two wide tiles.

One gfx1250 GPU, local_candidate_keys, graph replay, 2048-block launch; range of per-session medians over 3 interleaved sessions, us a call, main / previous head / now:

rows 194K 900K 1M
32 x 1 4.98-5.00 / 2.62-2.63 / 2.66-2.67 4.97-5.00 / 5.05-5.06 / 4.91-4.93 4.99-5.01 / 5.06-5.08 / 4.93-4.95
64 x 1 5.12-5.17 / 2.72-2.76 / 2.75-2.82 5.14-5.17 / 5.20-5.28 / 5.07-5.09 5.15-5.18 / 5.22-5.64 / 5.08-5.11
32 x 4 6.38-6.78 / 2.80-3.20 / 2.77-3.24 6.72-6.76 / 6.86-6.89 / 6.75-6.77 6.75-6.77 / 6.86-6.92 / 6.76-6.79

End to end on 4x MI455X (selection + exchange + merge, max rank, us): main's collective path vs this PR's mailbox path, which gfx1250 takes by default:

rows main this PR
32 x 194K 14.76 6.85
32 x 900K 15.10 9.40
64 x 900K 15.50 9.61
32 x 4 x 900K 17.07 11.68

The 4-GPU mailbox parity test passes at this head (including an 8 x 900K row). The PR's collective fallback, used only where the mailbox is unavailable (e.g. ROCm < 7.14), is within 3% of main at long rows (32 x 4 x 900K: 17.50 vs 17.07 us).

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ci Run non-GPU unit tests failed — not sure yet (low): The captured log is empty (0 lines), so nothing attributes the failure. The same job also failed on main at 2026-11-16T03:32 in the same window and the PR adds no CI config, so a CI-side/pre-existing failure is more likely than this PR, but unproven.

.valar/ci/job_133075964774.log is empty (0 lines captured)
same job on main: 2026-11-16T03:32 failure, 2026-11-16T02:28 success
next: Re-run Pre Checkin and pull the raw job log; if it reproduces at this head, run tests/test_indexer_cp_mailbox_setup.py in isolation (the only CPU test this PR changes) before attributing it. https://github.com/ROCm/ATOM/actions/runs/43799511565/job/133075964774

@lirui927

Copy link
Copy Markdown
Contributor Author

MiniMax-M3 accuracy at head 7545db2, in place of ci:atom: TP4 on 4x MI455X, MXFP4 + EAGLE3 (3 draft tokens), FP8 KV, indexer CP on (ATOM_M3_INDEXER_CP=1; the server log shows indexer_dcp_only enabled and no fallback to the collective exchange, so the mailbox kernel served decode).

  • GSM8K 5-shot, all 1319 questions, greedy: 95.15% flexible-extract / 94.01% strict-match.
  • EAGLE3 acceptance over that run: 85.23% (3.56 tokens per forward).
  • No errors in the server log.

Setup note: the test image's aiter predates wave32 support in topk_per_row_small_k, which ATOM main's M3 index top-k now calls on gfx1250, so aiter main's two topk_per_row_small_k files were mounted. That is independent of this PR.

CI: the one red check, Run non-GPU unit tests (2 failed of 13842), is 'Mxfp4MoEMethod' object has no attribute 'use_flydsl_herd' in atom/model_ops/moe.py, which this PR does not touch. It comes from #2713, merged to main at 03:32; main's own run at 02905b7 fails the same way, while main at e9c6ae0 passed.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (3)

  1. Test the fused mailbox kernel with the MXFP4 KV page geometry. _cp_index_topk_decode passes **self._select_page_kwargs (pages_per_block=1, block_page_stride=1 for kv_cache_dtype == "fp4", attention_mha.py:1362) into select_exchanged_candidates, but tests/model_ops/test_indexer_cp_mailbox.py only runs the page-16 defaults. Your GSM8K run used FP8 KV, and the dispatch test in tests/test_minimax_m3_mxfp4_indexer_cp.py only checks the collective branch. So the fused path never ran with (1, 1). Add a pages_per_block=1 case to _case, compared against merge_candidate_keys(..., pages_per_block=1, block_page_stride=1). Also add a mailbox-present case to the dispatch test that asserts the page kwargs reach select_exchanged_candidates.
  2. Agree the merge order with #2493. #2493 deletes _context_score and scores CP decode with aiter's FlyDSL kernel. This PR rewrites the same kernel (live-block grid plus gfx1250 TRANSPOSED). git merge-tree shows conflicts in indexer_context_parallel.py and tests/model_ops/test_indexer_cp_parity.py. Whichever PR lands second must drop or port the scorer hunk and its tests (test_scores_do_not_depend_on_the_split, test_orientations_agree). If #2493 keeps a Triton scorer for non-gfx950 (its open question), it should keep this PR's version. Say in the description which order you agreed.
  3. Run heavy CI at the final head. Every Accuracy job was skipped at 7545db2d9, and the ci:atom request from the last round is still open. Your manual GSM8K run (95.15 / 94.01) is useful, but it used a mounted aiter topk_per_row_small_k. The Run non-GPU unit tests failure is not from this PR: main also fails at 02905b71a / d2d5a6bbb, and this PR does not touch moe.py.

Minor, non-blocking (full list in result.json): the comment at indexer_candidate_exchange.py:459-467 still says the first tl.topk runs on x[BLOCK_SIZE_K], but it now runs in _walk_tiles on BLOCK_SIZE_K or BLOCK_SIZE_W. _topk_tiles(topk, local_blocks) (:42) takes the launch width, not LOCAL_KEEP, so rename the parameter. Please file the aiter ask to make _should_use_vmm public.


Re-reviewed at 7545db2. Already fixed: aiter symbols are imported before the try, so a rename fails at startup, and hipMemset uses ATOM's hip_ipc (164b69e). Long rows take one wide tile, with 900K/1M numbers at or below main and the fused path 15.1→9.4 us at 32×900K (1dee409, 7545db2). A zero-live-block rank is tested (f8f2b47). The warmup_exchange docstring is fixed (600afa6). Earlier: shared validation, aiter floor doc, gfx950 gate.

lirui927 and others added 4 commits November 16, 2026 08:15
…y live blocks

A decode graph is captured for max_model_len, so the CP scorer and local
top-k were launched for 2048 local blocks a rank (1M, TP4) whatever the
requests' lengths.

_context_score: the grid is sized by the batch, and each request's live
blocks are split over its programs; the slots past them are still
written with -inf. _local_topk: tiles stop at the row's live blocks.
Outputs are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion

On gfx1250 the CP scorer takes Q as WMMA's M operand and the converted K
as its N operand, with streaming K loads, as the gfx1250 TP decode
scorer of ROCm#2982. The D reduction order and FP32 accumulation
are unchanged, so the scores keep their bits. Other architectures keep
the previous orientation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… one launch

On gfx1250 the decode indexer CP ran its shard top-k, aiter's all-gather and
the merge as three launches. One kernel now does all three: each (head, row)
program stores its shard top-k, tagged with the call, straight into the
mailbox of the rank that owns the head (hipIpc-mapped cached memory), and the
programs of this rank's own head poll for every shard's keys and merge them.
_local_topk and _merge_topk share the selection and merge code with it.

The mailbox opens in warmup_exchange, before capture, and is agreed by every
rank; without it (not gfx1250, no graphs, a failed mapping, or a call larger
than it) the all-gather path is unchanged. Outputs are bit-identical to it.

4 x MI455X, graph replay, us a call: 32 x 194K 13.4 -> 7.5, 16 x 60K
12.6 -> 6.9, 64 x 100K 14.8 -> 9.4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wait for every peer instead of merging stale mailbox keys after a poll limit. Agree imports and device allocations before peer handle exchange so a rank-local setup failure sends the whole group to the collective fallback.

Size the mailbox from the resolved decode width, drop score-tail stores that no production consumer reads, and cover setup failures, counter wraparound, and every captured call.

Co-authored-by: Cursor <cursoragent@cursor.com>
lirui927 and others added 10 commits November 16, 2026 08:15
Share the packed-key width with the mailbox tag layout, derive sender and receiver call state through one helper, and make the bank-count invariant explicit.

Reuse the top-k tile calculation, name JIT constexpr arguments, make mailbox eligibility a single checked predicate, and tighten the existing fallback and parity tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
Share score and metadata guards between the collective and mailbox paths so path selection cannot change which layouts are accepted.

Document the private aiter IPC revision, minimum ROCm requirement, and that the gfx1250 gate is validated scope rather than a protocol limitation.

Co-authored-by: Cursor <cursoragent@cursor.com>
MI355X validation passed eager and graph parity, delayed-rank progress, three 12K-call stress runs, regression tests and ISA scope checks. Across the three production shapes the fused exchange lowers max-rank p50 by 23.2-27.9%.

Keep unsupported architectures and ROCm versions without cached hipIPC on the collective fallback.

Co-authored-by: Cursor <cursoragent@cursor.com>
55cde6a0 is on an unmerged aiter branch; also name the first aiter main
commit (de06bd30b, #6974) that has every private name the mailbox uses,
and say the ROCm 7.14 floor is aiter's gfx1250 hipIpc gate applied on
every arch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CandidateMailbox imported aiter's private IPC names inside the try that
turns rank-local setup failures into the collective fallback, so an aiter
rename would silently drop the fused exchange with only a warning. Import
them before the try so a missing name raises at startup; the try keeps the
real capability and resource failures (ROCm < 7.14 / VMM, OOM, hipMemset,
peer mapping). hipMemset now goes through ATOM's own HIP runtime
(atom/mono/runtime/hip_ipc.py), so _loaded_hip_runtime is no longer used.
The setup test drops its injected-ImportError fallback case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runner passes max_num_seqs * (mtp_k + 1), not mtp_k + 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 300-token row has three blocks, so rank 3 owns none: the live_end == 0
path of the scorer and the shard top-k must emit no key there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…their live blocks

After the rebase onto ROCm#3104, its MXFP4 scorer docstring still said
local_candidate_keys loads each row up to max_block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rows

The live-block walk streams 512-block tiles, so a ~1M row (2048 local blocks
at TP4) walked 4 tiles where main took one 2048 tile. A row past two narrow
tiles now takes one tile of up to 2048 blocks, in _local_topk and the fused
mailbox kernel alike.

gfx1250, one GPU, graph replay, 2048-block launch (max_model_len 1M, TP4),
us a call (main / before / after):
  bs32 x 1, 900K:  4.62 / 6.28 / 4.71
  bs32 x 4, 900K:  6.80 / 8.69 / 6.92
  bs32 x 1, 194K:  4.60 / 2.25 / 2.29

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne tile

With the wide tile, long indexer-CP rows were still 0.06-0.15 us behind
main's single 2048 tile. The cost was the wide path's loop: at a launch of
up to 2048 local blocks (1M at TP4) it never runs, but compiling it in cost
about 0.1 us. A host-side constexpr now leaves it out whenever the launch
fits one wide tile. The narrow walk keeps its loop, and results are
bit-identical.

The ISA analysis ruled out the other suspects. VGPRs are 58 vs main's 63
with no spills or LDS. Issuing the wide load before the row length is known
(main's b128 loads ahead of s_wait_kmcnt) was slower: plain loads were sunk
into the branch, and volatile loads were system-scope. Unmasking the wide
tile and dropping the branch did not help either.

gfx1250, one GPU, graph replay, 2048-block launch, interleaved in one
process, median over sessions of 5 rounds each, us a call (main / previous / now):
  32 x 1, 900K:  5.00 / 5.06 / 4.92
  64 x 1, 900K:  5.15 / 5.23 / 5.09
  32 x 4, 900K:  6.74 / 6.88 / 6.75
  32 x 1, 194K:  4.98 / 2.63 / 2.67
Two of six sessions read 5.49-5.50 us at 64 x 1, 900K; the other four did
not reproduce it.

The long-row parity test now runs at 1.1M, so it walks two wide tiles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lirui927
lirui927 force-pushed the lirui/m3-indexer-cp-live-blocks branch from 7545db2 to c7c7cec Compare November 16, 2026 08:16
…th comment

_topk_tiles takes the score row's launch width, not the forced local-block
count, so its parameter is now launch_blocks. The comment in
local_candidate_keys still placed the first tl.topk on x[BLOCK_SIZE_K]; it
runs in _walk_tiles on BLOCK_SIZE_K or BLOCK_SIZE_W.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ge geometry

_cp_index_topk_decode hands the KV's page geometry (pages_per_block=1,
block_page_stride=1 for the MXFP4 KV) to select_exchanged_candidates, but
the tests only ran the page-16 defaults. The 4-GPU mailbox test gains a
one-page-a-block case checked against merge_candidate_keys with the same
geometry. The CPU dispatch test gains an open-mailbox case asserting that
the page kwargs reach select_exchanged_candidates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lirui927

Copy link
Copy Markdown
Contributor Author

1. Fused mailbox kernel with the MXFP4 KV page geometry: added in e2fceb3.

  • tests/model_ops/test_indexer_cp_mailbox.py: _case takes an optional page count. A new (16, 60000, 1, 4, 1) case runs select_exchanged_candidates with pages_per_block=1, block_page_stride=1, eager and graph-replayed, and checks it against merge_candidate_keys(..., pages_per_block=1, block_page_stride=1). On 4x MI455X at this head the mailbox test passes, including this case and the 8 x 900K wide-tile row.
  • tests/test_minimax_m3_mxfp4_indexer_cp.py: the dispatch test is now also parametrized on an open mailbox. With one, _cp_index_topk_decode must call only the scorer and select_exchanged_candidates, passing the mailbox and the KV's page kwargs ({pages_per_block: 1, block_page_stride: 1} for the MXFP4 KV, none otherwise). It passes on CPU (6/6). With the page kwargs removed from the mailbox call, the fp4/fp4 case fails.

Minor items, in 3d6a69a:

  • _topk_tiles takes the launch width, so its parameter is now launch_blocks.
  • The comment in local_candidate_keys now says the first tl.topk runs in _walk_tiles on BLOCK_SIZE_K or BLOCK_SIZE_W.

2. Merge order with #2493. I proposed on #2493 (#2493 (comment)) that this PR lands first, and that #2493 then keeps this PR's Triton scorer for any arch it keeps Triton on, or drops/ports test_scores_do_not_depend_on_the_split and test_orientations_agree if FlyDSL takes CP decode everywhere. I will add the agreed order to the description once its author replies.

3. ci:atom is added; heavy CI runs at this head. Run non-GPU unit tests is green after rebasing onto main 0819cb3, which fixed the use_flydsl_herd failure.

@lirui927
lirui927 requested a review from valarLip November 16, 2026 09:07

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (3)

  1. Validate the wide tile on gfx950 before it ships there. _open_candidate_mailbox enables the fused kernel on ("gfx950", "gfx1250") (indexer_cp.py:98,211), and _shard_winners sends every long row through the 2048-wide _walk_tiles (indexer_candidate_exchange.py:34,138) on both paths. But your "Not covered" section says the wide tile ran only on gfx1250. Run test_indexer_cp_parity.py, test_indexer_cp_live_blocks.py and the 4-GPU test_indexer_cp_mailbox.py (with its 8 x 900K row) on 4 x MI355X at this head. Post the pass counts and a 900K local_candidate_keys time against main. The MI355X numbers in the body predate the wide tile.
  2. Settle the merge order with #2493 and write it in the description. It is still open: git merge-tree of this head with #2493 (e55d2d3cd) still conflicts in indexer_context_parallel.py and tests/model_ops/test_indexer_cp_parity.py, and ganyi1996ppo has not answered your 08:35 proposal yet. Once he agrees, add the order and who ports test_scores_do_not_depend_on_the_split / test_orientations_agree to the body.
  3. Let the ci:atom run finish green at e2fceb3e4. The label is on, and Pre Checkin, non-GPU unit tests and the offline smoke all pass. The MiniMax-M3-MXFP4 Accuracy job and the other heavy jobs are still pending.

Minor, non-blocking (full list in result.json): the fused-kernel test never compiles WIDE_LOOP=True, because MAX_LEN = 1 << 20 gives exactly 2048 local blocks. Use MAX_LEN = (1 << 20) + 128. Also add a row of 384 or fewer tokens: the shortest mailbox row is 1000 tokens, so the mailbox path never sees a rank with zero live blocks. The failure = None at indexer_cp.py:176 is dead (the agreed branch already implies it), so drop it. test_indexer_cp_mailbox.py:21 repeats the arch tuple; import _MAILBOX_ARCHES instead. Please file the aiter ask to make _should_use_vmm public.


Re-reviewed at e2fceb3. Already fixed: the fused kernel is tested with the MXFP4 (1, 1) page geometry against merge_candidate_keys, and the dispatch test checks that the page kwargs reach select_exchanged_candidates (e2fceb3). _topk_tiles renamed to launch_blocks and the tile comment fixed (3d6a69a). Earlier: aiter symbols imported before the try, hip_ipc hipMemset, one wide tile with 900K/1M numbers at or below main, zero-live-block rank test, warmup_exchange docstring.

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed..

ci green.

# tile rather than walking many narrow ones. When the launch fits in one
# wide tile (up to 1M at TP4) its loop is not compiled: carrying the dead
# loop cost the wide path about 0.1 us against main's single tile.
if live_end > 2 * BLOCK_SIZE_K:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wide tile is reachable on gfx950 from ~512K ctx but only ran on gfx1250, please run the parity/live-blocks/mailbox tests on 4x MI355X at this head and post pass counts + a 900K local_candidate_keys time vs main. if slower there, gate BLOCK_SIZE_W by arch

…-block ranks

The mailbox test launched every case exactly 1M wide (2048 local blocks a
shard), so the fused kernel never compiled its wide-tile loop. A case now
runs rows past 1M, widening its launch so every rank walks two wide tiles;
the other cases keep the shipped 1M launch. A 384-token case gives rows of
three blocks and one, so ranks with no live block take the mailbox path.

The test reads the mailbox architectures from indexer_cp, and
CandidateMailbox drops an assignment its agreement already implies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lirui927

Copy link
Copy Markdown
Contributor Author

1. gfx950. Not run yet: I have no MI355X right now. The tile shape itself is not new on gfx950: main already selects a 1M launch (2048 local blocks at TP4) from one 2048-wide tile on every arch. What is new is the bound at each row's live blocks, the narrow/wide split, and the wide-tile loop past 1M. The three tests and a 900K local_candidate_keys timing against main are scripted. I will post the MI355X pass counts and times when I have them. For reference, on 4 x MI455X at 449d939: parity 25 passed, live-blocks 17 passed, mailbox 1 passed; 32 x 900K, graph replay, per-session medians: main 4.75/4.75/4.75 us, this PR 4.81/4.80/4.79 us.

2. Merge order. #3116 lands first; the order and the test-porting rule are in the description; ganyi1996ppo notified on #2493.

3. ci:atom was green at e2fceb3; the label is still on, so it reruns at 449d939.

Minor items, in 449d939:

  • WIDE_LOOP=True: a new case (2, (1 << 20) + 1024, ...) has rows past 1M. Its launch widens to 8200 blocks (2050 a shard), so the fused kernel compiles the wide-tile loop and every rank walks two wide tiles. I kept the other cases at exactly 1M rather than moving MAX_LEN to (1 << 20) + 128, because 1M is the shipped max_model_len launch and its no-loop variant should stay covered. The test now takes 31 s instead of 25 s on 4 x MI455X; the added memory is negligible (about 1 MB of scores).
  • Zero live blocks: a new (2, 384, ...) case gives rows of 3 blocks and 1, so rank 3, and then ranks 1-3, reach the mailbox with no live block.
  • Dead failure = None: removed. The agreement branch now only sets the failure when the peers disagree.
  • Arch tuple: the test imports _MAILBOX_ARCHES from atom.distributed.indexer_cp.
  • _should_use_vmm: I will file the aiter request to make it public, and link it here.

@lirui927

Copy link
Copy Markdown
Contributor Author

On _should_use_vmm: making it public in aiter will be a follow-up after this PR lands. Until then, ATOM imports it before the try, so if it is renamed, startup fails with an error instead of falling back silently.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (2)

  1. Let the ci:atom run finish green at 449d939f. Pre Checkin, non-GPU unit tests and the offline smoke pass; the 14 Accuracy jobs are still pending.
  2. Trim the comments that carry measurement or validation history. Keep the rule they justify and drop the history: the aiter commit pins and "Validated with" at indexer_cp.py:108-110, "2 us slower at 32 x 4 rows" at :130, "Validated on gfx950 and gfx1250..." at :206-208, and "about 0.1 us against main's single tile" at indexer_candidate_exchange.py:136-137. In the same push, log the setup fallback with exc_info=True (indexer_cp.py:221).

Follow-up, not blocking here because indexer CP is off by default (ATOM_M3_INDEXER_CP unset; no CI entry turns it on, so the default path is unchanged): list in the follow-up a 4 x MI355X run of the fused mailbox kernel with the 2048-wide tile (parity, live-blocks and 4-GPU mailbox pass counts, plus a 900K local_candidate_keys time against main). _MAILBOX_ARCHES still includes gfx950, and the fused kernel's gfx950 validation (cb0864d) predates the wide tile. If MI355X turns out slower, gate BLOCK_SIZE_W by arch. Also add a setup test where open_peer_buffers fails on one rank (covers the second group_agree), and link the aiter request to make _should_use_vmm public once it is filed.


Re-reviewed at 449d939. Already fixed: the merge order with #2493 (this PR first, plus the scorer/test porting rule) is in the description and ganyi1996ppo was notified. The fused kernel test now compiles WIDE_LOOP=True with the 1M + 1024 case and has zero-live-block ranks with the 384 case. The dead failure = None is removed, and the test imports _MAILBOX_ARCHES (449d939). Earlier: MXFP4 (1, 1) page geometry on the fused path, launch_blocks rename, aiter symbols imported before the try, hip_ipc hipMemset, one wide tile at or below main on gfx1250, warmup_exchange docstring.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants