Repository navigation
Conversation
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
169b6b6 to
789d2b7
Compare
valar-cop
left a comment
There was a problem hiding this comment.
a few small things..
ci Run non-GPU unit tests failed — not sure yet (low): The captured log (.valar/ci/job_131037224269.log) is empty, 0 lines, so nothing attributes the failure. The same job passed on main at 2026-11-12T05:53 and failed at 05:36 the same day, so an infra/flaky origin is possible but unproven.
next: Re-run Pre Checkin and pull the job's raw log; if it reproduces on the head commit, bisect against the CPU test this PR adds (tests/test_indexer_cp_mailbox_setup.py), the only non-GPU test newly in scope. https://github.com/ROCm/ATOM/actions/runs/43175072925/job/131037224269
…er top-k ROCm/ATOM#3116 把 indexer CP 的本地 top-k、候选交换和 merge 合成一个 Triton kernel `_select_exchange_merge`,原来的分类规则不认识它,会落进 misc_small。m3_common.py 把它归到 indexer_topk;m3_xlsx.py 的 tab 备注 同步写明这条路径计在 sparse_index_topk 里;test_common.py 加一条分类测试。 Co-authored-by: Cursor <cursoragent@cursor.com>
|
On the |
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (4)
- Stop turning a missing private aiter symbol into a quiet fallback.
atom/distributed/indexer_cp.py:144calls_mailbox_runtime()(which imports aiter's private_loaded_hip_runtime/_should_use_vmm) insideexcept Exceptionat :163, andtests/test_indexer_cp_mailbox_setup.pyasserts that an injectedImportErrorfalls back. That is try/except-ImportError probing of aiter privates: an aiter rename silently drops the 23-28% win with only a warning (same failure class as #3162). Import them directly before thetry(or at module level after the arch gate) so a missing name fails at startup. Keep thetryfor OOM /hipMemset/ mapping, and drop the injected-ImportErrorcase from the test._loaded_hip_runtime().hipMemsetcan use ATOM's ownatom/mono/runtime/hip_ipc.pyruntime (hipMemsetbound at :91-92). Ask aiter to make_should_use_vmmpublic. - Own the live-block
_local_topkchange here, and add long-row numbers. #3104 also rewrites_local_topk/local_candidate_keysinindexer_candidate_exchange.pyfor the same goal; it has been asked to drop that and agree with you. This PR should keep it because_shard_winnersis shared with the fused mailbox kernel and the diff is about 15 lines against #3104's 3-width plus group-max-filter plus scratch design. One gap:LOCAL_TOPK_TILE = 512(:30) turns main's single 2048 tile for a ~1M row (2048 local blocks at TP4) into a 4-tile walk, and #3104 measured the walk as the bottleneck. Every number here stops at 194K. Post_local_topkµs vs main at near-max context (for example 32 x 900K, TP4). If it regresses, size the tile to the row (one tile up to 2048) as #3104 does. - Test a rank with zero live blocks.
tests/model_ops/test_indexer_cp_live_blocks.py:67bottoms out at 2 live blocks on rank 3. Add a short row (length <= 384 at rank 3) so thelive_end == 0path in_shard_winners/_context_scoreis exercised. Short decode rows are common. - Fix the
warmup_exchangedocstring.indexer_cp.py:347saysmax_mailbox_rows"comes from the runner's resolvedmtp_k + 1", but the runner passesmax_num_seqs * (mtp_k + 1)(model_runner.py:4159).
CI: Pre Checkin is green. MiniMax-M3 accuracy has not run at this head; please add ci:atom.
Reviewed at 80f7237. Already addressed from valar-cop: shared validation for the fused and fallback paths, the aiter rev / ROCm floor doc, and the gfx950 gate with MI355X numbers.
|
ci triage for 7545db2:
|
The one-tile rewrite of `_local_topk` also changed the selector the BF16/FP8 indexer-CP path runs, and #3116 rewrites the same kernel for the same goal. Restore main's kernel and its tests so the change lands once, with #3116. The MXFP4 wiring does not depend on it: main's selector already masks the lanes past a row's blocks. Co-authored-by: Cursor <cursoragent@cursor.com>
…ranks on the MXFP4 index cache (MiniMax-M3.1 first) (#3104) * minimax_m3: read each indexer-CP shard row in one tile, filtered The step's local top-k waited on its longest row, which walked its shard 512 blocks at a time. Each row now stops at its own shard length and takes one tile of the narrowest width covering it (512/1024/2048), and a wide tile sorts only the keys at or above its group-max bound. Output is unchanged; batch 24-60 drops from 8.8-15.8 to 6.0-10.0 us a layer. merge_candidate_keys takes the emitted page geometry (pages_per_block, block_page_stride), defaulting to the page-16 SHUFFLE cache's 8 / 8 as before, so a page-128 KV can be merged into its own block table. Co-authored-by: Cursor <cursoragent@cursor.com> * minimax_m3: run the indexer-CP decode on the MXFP4 index cache Under indexer CP each TP rank scores every index head over its 1/W of the blocks. With the MXFP4 index cache that scorer is aiter's FlyDSL decode scorer, which already takes the shard (cp_world / cp_rank with k_scale): - IndexScoreBuffers carries the rank's shard; allocate_index_scorers sizes the work map at the shard's bound ceil(blocks / W) over every index head, and the workspace for that as well as for a prefill chunk (which stays TP). Unsharded, every call is the one it was. - make_sparse_decode_metadata builds the step's map for the shard. - indexer_context_scores_mxfp4 scores into a contiguous view of the shared workspace on that map; _cp_index_topk_decode runs it for the MXFP4 index and merges with the KV's page geometry (1 / 1 for the MXFP4 KV). - indexer_cp_unsupported_reason no longer refuses the MXFP4 caches. tests/test_minimax_m3_mxfp4_indexer_cp.py pins the wiring on CPU, and on gfx950 emulates the 4 ranks against minimax_m3_index_topk_decode: the merged top-k, sparse block table and context must equal the TP path's bit for bit, for both page geometries, on a never-filled workspace. Co-authored-by: Cursor <cursoragent@cursor.com> * minimax_m3: check the indexer-CP FlyDSL calls against aiter's signatures The MXFP4 CP decode hands aiter's scorer k_scale and the shard (cp_world / cp_rank), and its work map and capacity calls the shard. A new GPU-free test runs those three calls and binds each to the installed aiter's signature, so an aiter older than ROCm/aiter#7060 (no k_scale) fails at test time rather than at the first MXFP4 CP decode. It skips where aiter.ops.flydsl does not import. Co-authored-by: Cursor <cursoragent@cursor.com> * minimax_m3: give the local top-k tile choice without its timings The comment on the shard-row tile widths and the full-grid threshold quoted per-layer latencies from one deployment. Keep the reasons alone: one tile of the narrowest covering width, and four warps with an unfiltered narrow tile below the threshold, two warps and the filter at or above it. Co-authored-by: Cursor <cursoragent@cursor.com> * minimax_m3: leave the indexer-CP shard top-k as on main The one-tile rewrite of `_local_topk` also changed the selector the BF16/FP8 indexer-CP path runs, and #3116 rewrites the same kernel for the same goal. Restore main's kernel and its tests so the change lands once, with #3116. The MXFP4 wiring does not depend on it: main's selector already masks the lanes past a row's blocks. Co-authored-by: Cursor <cursoragent@cursor.com> * minimax_m3: say what the shard top-k does past a row's blocks `local_candidate_keys` loads each row up to the shard's block bound and masks the lanes past the row's blocks, but the MXFP4 scorer's docstring said it read none of them. The test's workspace docstring said the workspace was filled and then never filled; it now says what the junk is for. Co-authored-by: Cursor <cursoragent@cursor.com> * minimax_m3: keep the merge's page geometry for the MXFP4 KV 871b053 restored main's whole exchange file, and with it main's `merge_candidate_keys`, which always emits the page-16 SHUFFLE geometry. The MXFP4 CP decode passes `pages_per_block` / `block_page_stride` (1, 1), so it raised a TypeError at its first merge. Bring the two arguments back; the shard top-k stays main's. Co-authored-by: Cursor <cursoragent@cursor.com> * recipe(minimax_m3): say what indexer CP measured on the MXFP4 caches The CONC >= 15 paragraph and its table are back to main's text: that table was measured with the FP8 KV of the server command and does not carry over. A paragraph after it gives the MXFP4 A/B on one aggregated TP4 server (lower tok/s/GPU in all six cells, median ITL down only at 128K and CONC 64), says the throughput gain was measured only on 1P2D decode nodes, which need MXFP4 KV under P/D and are not on main yet, and says to leave CP off on an aggregated MXFP4 server. Co-authored-by: Cursor <cursoragent@cursor.com> * recipe(minimax_m3): median ITL fell in all three 128K cells The MXFP4 paragraph said median ITL fell only at 128K and CONC 64, but the A/B shows it falling in every 128K cell (0.8-4.7%, most at CONC 64) and rising at 32K (5.1% at CONC 32, none at 128). The test module's docstring called its kernel-free tests CPU tests; like every test in the file they need aiter installed. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
80f7237 to
1dee409
Compare
|
Rebased onto main e9c6ae0. #3104 had added the MXFP4 scorer and the merge's page geometry ( 1. Missing private aiter symbol fell back quietly (164b69e). 2. Long-row
The fix is the one you suggested, in a smaller form. A row whose shard part is longer than two 512 tiles now takes one tile of up to 2048 (
3. Zero-live-block rank (f8f2b47). 4. Also 2071685: #3104's CI: noted the |
|
Follow-up on long rows (7545db2): long rows now match or beat main. The extra ~0.1 us came from the wide-tile path still compiling a loop over further wide tiles, which never runs when the launch fits one tile (up to 2048 local blocks, i.e. 1M at TP4). A host-side constexpr now leaves that loop out, in One gfx1250 GPU,
End to end on 4x MI455X (selection + exchange + merge, max rank, us): main's collective path vs this PR's mailbox path, which gfx1250 takes by default:
The 4-GPU mailbox parity test passes at this head (including an 8 x 900K row). The PR's collective fallback, used only where the mailbox is unavailable (e.g. ROCm < 7.14), is within 3% of main at long rows (32 x 4 x 900K: 17.50 vs 17.07 us). |
valar-cop
left a comment
There was a problem hiding this comment.
LGTM
ci Run non-GPU unit tests failed — not sure yet (low): The captured log is empty (0 lines), so nothing attributes the failure. The same job also failed on main at 2026-11-16T03:32 in the same window and the PR adds no CI config, so a CI-side/pre-existing failure is more likely than this PR, but unproven.
.valar/ci/job_133075964774.log is empty (0 lines captured)
same job on main: 2026-11-16T03:32 failure, 2026-11-16T02:28 success
next: Re-run Pre Checkin and pull the raw job log; if it reproduces at this head, run tests/test_indexer_cp_mailbox_setup.py in isolation (the only CPU test this PR changes) before attributing it. https://github.com/ROCm/ATOM/actions/runs/43799511565/job/133075964774
|
MiniMax-M3 accuracy at head 7545db2, in place of
Setup note: the test image's aiter predates wave32 support in CI: the one red check, |
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (3)
- Test the fused mailbox kernel with the MXFP4 KV page geometry.
_cp_index_topk_decodepasses**self._select_page_kwargs(pages_per_block=1, block_page_stride=1forkv_cache_dtype == "fp4",attention_mha.py:1362) intoselect_exchanged_candidates, buttests/model_ops/test_indexer_cp_mailbox.pyonly runs the page-16 defaults. Your GSM8K run used FP8 KV, and the dispatch test intests/test_minimax_m3_mxfp4_indexer_cp.pyonly checks the collective branch. So the fused path never ran with (1, 1). Add apages_per_block=1case to_case, compared againstmerge_candidate_keys(..., pages_per_block=1, block_page_stride=1). Also add a mailbox-present case to the dispatch test that asserts the page kwargs reachselect_exchanged_candidates. - Agree the merge order with #2493. #2493 deletes
_context_scoreand scores CP decode with aiter's FlyDSL kernel. This PR rewrites the same kernel (live-block grid plus gfx1250TRANSPOSED).git merge-treeshows conflicts inindexer_context_parallel.pyandtests/model_ops/test_indexer_cp_parity.py. Whichever PR lands second must drop or port the scorer hunk and its tests (test_scores_do_not_depend_on_the_split,test_orientations_agree). If #2493 keeps a Triton scorer for non-gfx950 (its open question), it should keep this PR's version. Say in the description which order you agreed. - Run heavy CI at the final head. Every Accuracy job was skipped at
7545db2d9, and theci:atomrequest from the last round is still open. Your manual GSM8K run (95.15 / 94.01) is useful, but it used a mounted aitertopk_per_row_small_k. TheRun non-GPU unit testsfailure is not from this PR: main also fails at02905b71a/d2d5a6bbb, and this PR does not touchmoe.py.
Minor, non-blocking (full list in result.json): the comment at indexer_candidate_exchange.py:459-467 still says the first tl.topk runs on x[BLOCK_SIZE_K], but it now runs in _walk_tiles on BLOCK_SIZE_K or BLOCK_SIZE_W. _topk_tiles(topk, local_blocks) (:42) takes the launch width, not LOCAL_KEEP, so rename the parameter. Please file the aiter ask to make _should_use_vmm public.
Re-reviewed at 7545db2. Already fixed: aiter symbols are imported before the try, so a rename fails at startup, and hipMemset uses ATOM's hip_ipc (164b69e). Long rows take one wide tile, with 900K/1M numbers at or below main and the fused path 15.1→9.4 us at 32×900K (1dee409, 7545db2). A zero-live-block rank is tested (f8f2b47). The warmup_exchange docstring is fixed (600afa6). Earlier: shared validation, aiter floor doc, gfx950 gate.
…y live blocks A decode graph is captured for max_model_len, so the CP scorer and local top-k were launched for 2048 local blocks a rank (1M, TP4) whatever the requests' lengths. _context_score: the grid is sized by the batch, and each request's live blocks are split over its programs; the slots past them are still written with -inf. _local_topk: tiles stop at the row's live blocks. Outputs are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion On gfx1250 the CP scorer takes Q as WMMA's M operand and the converted K as its N operand, with streaming K loads, as the gfx1250 TP decode scorer of ROCm#2982. The D reduction order and FP32 accumulation are unchanged, so the scores keep their bits. Other architectures keep the previous orientation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… one launch On gfx1250 the decode indexer CP ran its shard top-k, aiter's all-gather and the merge as three launches. One kernel now does all three: each (head, row) program stores its shard top-k, tagged with the call, straight into the mailbox of the rank that owns the head (hipIpc-mapped cached memory), and the programs of this rank's own head poll for every shard's keys and merge them. _local_topk and _merge_topk share the selection and merge code with it. The mailbox opens in warmup_exchange, before capture, and is agreed by every rank; without it (not gfx1250, no graphs, a failed mapping, or a call larger than it) the all-gather path is unchanged. Outputs are bit-identical to it. 4 x MI455X, graph replay, us a call: 32 x 194K 13.4 -> 7.5, 16 x 60K 12.6 -> 6.9, 64 x 100K 14.8 -> 9.4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wait for every peer instead of merging stale mailbox keys after a poll limit. Agree imports and device allocations before peer handle exchange so a rank-local setup failure sends the whole group to the collective fallback. Size the mailbox from the resolved decode width, drop score-tail stores that no production consumer reads, and cover setup failures, counter wraparound, and every captured call. Co-authored-by: Cursor <cursoragent@cursor.com>
Share the packed-key width with the mailbox tag layout, derive sender and receiver call state through one helper, and make the bank-count invariant explicit. Reuse the top-k tile calculation, name JIT constexpr arguments, make mailbox eligibility a single checked predicate, and tighten the existing fallback and parity tests. Co-authored-by: Cursor <cursoragent@cursor.com>
Share score and metadata guards between the collective and mailbox paths so path selection cannot change which layouts are accepted. Document the private aiter IPC revision, minimum ROCm requirement, and that the gfx1250 gate is validated scope rather than a protocol limitation. Co-authored-by: Cursor <cursoragent@cursor.com>
MI355X validation passed eager and graph parity, delayed-rank progress, three 12K-call stress runs, regression tests and ISA scope checks. Across the three production shapes the fused exchange lowers max-rank p50 by 23.2-27.9%. Keep unsupported architectures and ROCm versions without cached hipIPC on the collective fallback. Co-authored-by: Cursor <cursoragent@cursor.com>
55cde6a0 is on an unmerged aiter branch; also name the first aiter main commit (de06bd30b, #6974) that has every private name the mailbox uses, and say the ROCm 7.14 floor is aiter's gfx1250 hipIpc gate applied on every arch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CandidateMailbox imported aiter's private IPC names inside the try that turns rank-local setup failures into the collective fallback, so an aiter rename would silently drop the fused exchange with only a warning. Import them before the try so a missing name raises at startup; the try keeps the real capability and resource failures (ROCm < 7.14 / VMM, OOM, hipMemset, peer mapping). hipMemset now goes through ATOM's own HIP runtime (atom/mono/runtime/hip_ipc.py), so _loaded_hip_runtime is no longer used. The setup test drops its injected-ImportError fallback case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runner passes max_num_seqs * (mtp_k + 1), not mtp_k + 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 300-token row has three blocks, so rank 3 owns none: the live_end == 0 path of the scorer and the shard top-k must emit no key there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…their live blocks After the rebase onto ROCm#3104, its MXFP4 scorer docstring still said local_candidate_keys loads each row up to max_block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rows The live-block walk streams 512-block tiles, so a ~1M row (2048 local blocks at TP4) walked 4 tiles where main took one 2048 tile. A row past two narrow tiles now takes one tile of up to 2048 blocks, in _local_topk and the fused mailbox kernel alike. gfx1250, one GPU, graph replay, 2048-block launch (max_model_len 1M, TP4), us a call (main / before / after): bs32 x 1, 900K: 4.62 / 6.28 / 4.71 bs32 x 4, 900K: 6.80 / 8.69 / 6.92 bs32 x 1, 194K: 4.60 / 2.25 / 2.29 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne tile With the wide tile, long indexer-CP rows were still 0.06-0.15 us behind main's single 2048 tile. The cost was the wide path's loop: at a launch of up to 2048 local blocks (1M at TP4) it never runs, but compiling it in cost about 0.1 us. A host-side constexpr now leaves it out whenever the launch fits one wide tile. The narrow walk keeps its loop, and results are bit-identical. The ISA analysis ruled out the other suspects. VGPRs are 58 vs main's 63 with no spills or LDS. Issuing the wide load before the row length is known (main's b128 loads ahead of s_wait_kmcnt) was slower: plain loads were sunk into the branch, and volatile loads were system-scope. Unmasking the wide tile and dropping the branch did not help either. gfx1250, one GPU, graph replay, 2048-block launch, interleaved in one process, median over sessions of 5 rounds each, us a call (main / previous / now): 32 x 1, 900K: 5.00 / 5.06 / 4.92 64 x 1, 900K: 5.15 / 5.23 / 5.09 32 x 4, 900K: 6.74 / 6.88 / 6.75 32 x 1, 194K: 4.98 / 2.63 / 2.67 Two of six sessions read 5.49-5.50 us at 64 x 1, 900K; the other four did not reproduce it. The long-row parity test now runs at 1.1M, so it walks two wide tiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7545db2 to
c7c7cec
Compare
…th comment _topk_tiles takes the score row's launch width, not the forced local-block count, so its parameter is now launch_blocks. The comment in local_candidate_keys still placed the first tl.topk on x[BLOCK_SIZE_K]; it runs in _walk_tiles on BLOCK_SIZE_K or BLOCK_SIZE_W. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ge geometry _cp_index_topk_decode hands the KV's page geometry (pages_per_block=1, block_page_stride=1 for the MXFP4 KV) to select_exchanged_candidates, but the tests only ran the page-16 defaults. The 4-GPU mailbox test gains a one-page-a-block case checked against merge_candidate_keys with the same geometry. The CPU dispatch test gains an open-mailbox case asserting that the page kwargs reach select_exchanged_candidates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
1. Fused mailbox kernel with the MXFP4 KV page geometry: added in e2fceb3.
Minor items, in 3d6a69a:
2. Merge order with #2493. I proposed on #2493 (#2493 (comment)) that this PR lands first, and that #2493 then keeps this PR's Triton scorer for any arch it keeps Triton on, or drops/ports 3. |
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (3)
- Validate the wide tile on gfx950 before it ships there.
_open_candidate_mailboxenables the fused kernel on("gfx950", "gfx1250")(indexer_cp.py:98,211), and_shard_winnerssends every long row through the 2048-wide_walk_tiles(indexer_candidate_exchange.py:34,138) on both paths. But your "Not covered" section says the wide tile ran only on gfx1250. Runtest_indexer_cp_parity.py,test_indexer_cp_live_blocks.pyand the 4-GPUtest_indexer_cp_mailbox.py(with its 8 x 900K row) on 4 x MI355X at this head. Post the pass counts and a 900Klocal_candidate_keystime against main. The MI355X numbers in the body predate the wide tile. - Settle the merge order with #2493 and write it in the description. It is still open:
git merge-treeof this head with #2493 (e55d2d3cd) still conflicts inindexer_context_parallel.pyandtests/model_ops/test_indexer_cp_parity.py, and ganyi1996ppo has not answered your 08:35 proposal yet. Once he agrees, add the order and who portstest_scores_do_not_depend_on_the_split/test_orientations_agreeto the body. - Let the
ci:atomrun finish green ate2fceb3e4. The label is on, and Pre Checkin, non-GPU unit tests and the offline smoke all pass. The MiniMax-M3-MXFP4 Accuracy job and the other heavy jobs are still pending.
Minor, non-blocking (full list in result.json): the fused-kernel test never compiles WIDE_LOOP=True, because MAX_LEN = 1 << 20 gives exactly 2048 local blocks. Use MAX_LEN = (1 << 20) + 128. Also add a row of 384 or fewer tokens: the shortest mailbox row is 1000 tokens, so the mailbox path never sees a rank with zero live blocks. The failure = None at indexer_cp.py:176 is dead (the agreed branch already implies it), so drop it. test_indexer_cp_mailbox.py:21 repeats the arch tuple; import _MAILBOX_ARCHES instead. Please file the aiter ask to make _should_use_vmm public.
Re-reviewed at e2fceb3. Already fixed: the fused kernel is tested with the MXFP4 (1, 1) page geometry against merge_candidate_keys, and the dispatch test checks that the page kwargs reach select_exchanged_candidates (e2fceb3). _topk_tiles renamed to launch_blocks and the tile comment fixed (3d6a69a). Earlier: aiter symbols imported before the try, hip_ipc hipMemset, one wide tile with 900K/1M numbers at or below main, zero-live-block rank test, warmup_exchange docstring.
| # tile rather than walking many narrow ones. When the launch fits in one | ||
| # wide tile (up to 1M at TP4) its loop is not compiled: carrying the dead | ||
| # loop cost the wide path about 0.1 us against main's single tile. | ||
| if live_end > 2 * BLOCK_SIZE_K: |
There was a problem hiding this comment.
wide tile is reachable on gfx950 from ~512K ctx but only ran on gfx1250, please run the parity/live-blocks/mailbox tests on 4x MI355X at this head and post pass counts + a 900K local_candidate_keys time vs main. if slower there, gate BLOCK_SIZE_W by arch
…-block ranks The mailbox test launched every case exactly 1M wide (2048 local blocks a shard), so the fused kernel never compiled its wide-tile loop. A case now runs rows past 1M, widening its launch so every rank walks two wide tiles; the other cases keep the shipped 1M launch. A 384-token case gives rows of three blocks and one, so ranks with no live block take the mailbox path. The test reads the mailbox architectures from indexer_cp, and CandidateMailbox drops an assignment its agreement already implies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
1. gfx950. Not run yet: I have no MI355X right now. The tile shape itself is not new on gfx950: main already selects a 1M launch (2048 local blocks at TP4) from one 2048-wide tile on every arch. What is new is the bound at each row's live blocks, the narrow/wide split, and the wide-tile loop past 1M. The three tests and a 900K 2. Merge order. #3116 lands first; the order and the test-porting rule are in the description; ganyi1996ppo notified on #2493. 3. Minor items, in 449d939:
|
|
On |
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (2)
- Let the
ci:atomrun finish green at449d939f. Pre Checkin, non-GPU unit tests and the offline smoke pass; the 14 Accuracy jobs are still pending. - Trim the comments that carry measurement or validation history. Keep the rule they justify and drop the history: the aiter commit pins and "Validated with" at
indexer_cp.py:108-110, "2 us slower at 32 x 4 rows" at :130, "Validated on gfx950 and gfx1250..." at :206-208, and "about 0.1 us against main's single tile" atindexer_candidate_exchange.py:136-137. In the same push, log the setup fallback withexc_info=True(indexer_cp.py:221).
Follow-up, not blocking here because indexer CP is off by default (ATOM_M3_INDEXER_CP unset; no CI entry turns it on, so the default path is unchanged): list in the follow-up a 4 x MI355X run of the fused mailbox kernel with the 2048-wide tile (parity, live-blocks and 4-GPU mailbox pass counts, plus a 900K local_candidate_keys time against main). _MAILBOX_ARCHES still includes gfx950, and the fused kernel's gfx950 validation (cb0864d) predates the wide tile. If MI355X turns out slower, gate BLOCK_SIZE_W by arch. Also add a setup test where open_peer_buffers fails on one rank (covers the second group_agree), and link the aiter request to make _should_use_vmm public once it is filed.
Re-reviewed at 449d939. Already fixed: the merge order with #2493 (this PR first, plus the scorer/test porting rule) is in the description and ganyi1996ppo was notified. The fused kernel test now compiles WIDE_LOOP=True with the 1M + 1024 case and has zero-live-block ranks with the 384 case. The dead failure = None is removed, and the test imports _MAILBOX_ARCHES (449d939). Earlier: MXFP4 (1, 1) page geometry on the fused path, launch_blocks rename, aiter symbols imported before the try, hip_ipc hipMemset, one wide tile at or below main on gfx1250, warmup_exchange docstring.
Summary
The fused and fallback paths share the selection, merge and input-validation helpers. Live scores and final selection outputs are bit-identical to the previous path.
Results
Baselines are main before this PR, or the PR's earlier head where noted.
local_candidate_keys, graph replay, 2048-block launch = max_model_len 1M at TP4). Main and this PR run interleaved in one process; each figure is the range of per-session medians, main -> this PR:Tests
.wt/.cvlowered tosc0 sc1.ATOM_M3_INDEXER_CP=1, mailbox exchange in use): GSM8K 5-shot, all 1319, greedy: 95.15% flexible-extract / 94.01% strict-match; EAGLE3 acceptance 85.23%. (aitermain'stopk_per_row_small_kwas mounted into the test image, whose aiter predates its wave32 support, which ATOM main's M3 index top-k calls on gfx1250.)Not covered
Merge order
This PR lands before #2493. Both rewrite
_context_scoreand conflict inindexer_context_parallel.pyandtests/model_ops/test_indexer_cp_parity.py. When #2493 rebases on top of it:test_scores_do_not_depend_on_the_splitandtest_orientations_agree.