Skip to content

[ROCm][Perf] Muse Spark: follow-ups on #7 (o_proj all-reduce in the boundary, token-sharded stream above 256, FP32 router input, skinny GEMM) - #8

Draft
sawsa307 wants to merge 19 commits into
muse-spark-pr7-basefrom
claude/muse-amd-fusion-next
Draft

sawsa307 wants to merge 19 commits into
muse-spark-pr7-basefrom
claude/muse-amd-fusion-next

Conversation

@sawsa307

@sawsa307 sawsa307 commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Overview

Draft, stacked on #7. Base muse-spark-pr7-base is pinned at b2e6c53; #7 has since gained d2c6657, the same test change as 14a2b41 here. To avoid moving SHAs during GPU image preparation, this branch is not rebased yet, so the diff also shows that test file. Not validated on GPU yet.

Commit What Default
58f3342 Attention o_proj all-reduce fused into the residual boundary (≤ 256 tokens). o_proj runs without its all-reduce (quant_method.apply; reduce_results untouched); one launch reduces the partial rows (fp32, rank order, one bf16 RNE rounding) and runs the boundary through the shared norm_residual_norm_tile. TP and TP + EP; own workspace. on (VLLM_ROCM_MUSE_FUSED_ATTN_ALL_REDUCE)
d2c9326 FP32 router input (experiment), 33–128 tokens, as on CUDA. off (VLLM_ROCM_MUSE_FP32_ROUTER_INPUT=1)
28cb4e2 Split-K skinny GEMM for the decode projection shards (from vllm-project#63322). TP8 shards tuned; TP4 untuned until benchmarks/kernels/tune_muse_spark_rocm_skinny_gemm.py runs. tuned shapes only
5821090, f030f92 Token-sharded residual stream above 256 tokens under EP (prefill and mixed prefill+decode steps), the AMD counterpart of the CUDA path's sequence-parallel mode. Ported from vllm-project#63722 + vllm-project#63723 (Mikko Tukiainen) and adapted: o_proj partial via quant_method.apply (monokernel untouched); small batches reduce the expert output exactly once with or without the fused EP finalize; token-sharded batches bypass the decode fusions; per-slot expansion respected; DSpark aux layers gathered. on when EP spans TP with DP1 (VLLM_ROCM_MUSE_TOKEN_SHARDED)
e0eecb4 AITER custom all-gather / reduce-scatter (chunked) for the token-sharded stream; local reset of output_reduced; startup log. off (VLLM_ROCM_MUSE_AITER_TOKEN_COLLECTIVES=1)
14a2b41 Test only: the EP fused-finalize test checks against derived error bounds instead of a fixed tolerance. Validated on MI355X (job 32650: 6/6) and now also on #7 as d2c6657. —
35b3a3f Two-shot variant of the o_proj all-reduce boundary: one-shot moves (W − 1)× a two-shot all-reduce's link bytes and only wins at small batches (cost model: wins at ~32 tokens, loses at 64–128). Above the one-shot limit, each row goes to its owner (t % W), which sums and broadcasts it back; then every rank runs the boundary. Same rounding as one-shot, so the two are bit-identical. Disjoint workspace regions, counters and barrier targets, so any interleaving of calls is safe. Also makes the column-sharding token limit configurable. New benchmark_muse_spark_rocm_attn_all_reduce.py: unfused vs. one-shot vs. two-shot per batch size. one-shot ≤ 32 (VLLM_ROCM_MUSE_ATTN_ALL_REDUCE_ONE_SHOT_MAX_TOKENS), two-shot 33–256; column sharding ≤ 256 (VLLM_ROCM_MUSE_SHARD_PROJECTIONS_MAX_TOKENS)
861a6ad Test only: test_tp_partial_routes_are_reduced_before_norm runs for both VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE settings. It used to require FP32 at the all-reduce, which the recipe's bf16 contradicts (the 40/41 on MI355X). Its mock no longer promotes bf16 to FP32. —

Not implemented, with the reason (see #7 discussion): vLLM-side norm + MXFP4 quant (would split AITER's existing sort + quant fusion for a ~1 MB saving; worthwhile only AITER-side); router aux stream (disabled on ROCm by the CUDA Muse model because it hung; serialized under breakable graphs anyway); o_proj GEMM + reduce-scatter (next step on top of the token-sharded stream; no AMD GEMM-RS kernel yet).

Validation

Done (no GPU in the environment):

  • Triton interpreter, W = 2/4/8: fused o_proj all-reduce + boundary bit-identical to fp32 rank-order sum → bf16 → fused_norm_residual_norm_fwd (output, new stream, FP32 router row), on every rank, for one-shot and two-shot interleaved on one workspace. Covered cases: T < W, T not a multiple of W, T at the one-shot limit and at the workspace maximum. Mutation check: 5/6 injected kernel bugs caught; the miss permutes the slot order, so it only changes the fp32 summation order.
  • Protocol model check of every cross-rank kernel (finalize TP/EP, pooler/unpooler gathers sharing one workspace, o_proj boundary one-shot + two-shot with the switch anywhere in 0–32): 2000 randomized interleavings over 2/4/8 ranks with several blocks per rank, realistic and shuffled call orders, counters starting below the int32 wrap: no deadlock, no stale or future read. Mutation check: every injected protocol bug caught (shared counter/target between variants, missing second barrier, one counter for both phases, region aliasing).
  • AOT gfx942/gfx950, W = 4/8, both variants: all accesses dwordx4, no spills, no peer-pointer waterfalls (the only execnz loops are the barrier spins, one per barrier); fp32→bf16 RNE.
  • 861a6ad on CPU: both parametrizations pass. The old test reproduces the GPU failure under VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE=bf16. The new test fails if the runner ignores the setting. The rest of the file gives the same results under fp32 and bf16; its 4 CPU-only failures need a device and passed on MI355X.
  • pre-commit: passes.

To do on MI355X:

  • tests/kernels/moe/test_muse_spark_rocm_bf16_moe.py with the recipe environment (expect 42/42 with the parametrized test).
  • pytest -v -s tests/models/test_muse_spark_token_sharded.py -m 'distributed(num_gpus=2)'.
  • tests/distributed/test_muse_spark_rocm_column_gather.py -k all_reduce_boundary (now also forces two-shot on tiny batches and captures both variants in one graph), tests/kernels/core/test_muse_spark_norms.py -k router_out, tests/kernels/test_muse_spark_rocm_skinny_gemm.py.
  • torchrun --nproc_per_node=4 benchmarks/kernels/benchmark_muse_spark_rocm_attn_all_reduce.py: set VLLM_ROCM_MUSE_ATTN_ALL_REDUCE_ONE_SHOT_MAX_TOKENS at the one-shot/two-shot crossover. Disable the fusion above any batch size where unfused wins.
  • python benchmarks/kernels/tune_muse_spark_rocm_skinny_gemm.py --tp-sizes 4 → TP4 _CONFIGS; then benchmark_muse_spark_rocm_column_gather.py. Lower VLLM_ROCM_MUSE_SHARD_PROJECTIONS_MAX_TOKENS to the largest batch where shard + gather still wins.
  • A/B, C10 TP4 + EP4 first, then a prefill-heavy point (C28/C36): VLLM_ROCM_MUSE_FUSED_ATTN_ALL_REDUCE=1 vs 0; token sharding on vs. this branch's parent.

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof

…ual boundary

At decode sizes (up to 256 tokens) the decoder now runs o_proj's GEMM
without its TP all-reduce (quant_method.apply; the layer's reduce_results
is left alone, so the stock path, the decode monokernel and every other
caller still reduce exactly once) and hands the partial rows to one
Triton launch that reduces them inside the attention residual boundary:
every rank pushes its partial [T, H] rows into every rank's buffer, and
after one grid barrier sums the W partials in fp32 in rank order, rounds
the sum to bf16 (the unfused all-reduce's output dtype; RNE on gfx942 and
gfx950) and runs the boundary (post-attention norm, gated FP32 residual,
pre-FFN norm) through the shared norm_residual_norm_tile. Every rank
computes the same rows from the same bits, so the replicated FP32 stream
stays identical across ranks. Applies under TP and TP + EP (o_proj is
TP-sharded either way).

The workspace is separate from the MoE gathers' and the fused finalize's,
double-buffered by call parity. VLLM_ROCM_MUSE_FUSED_ATTN_ALL_REDUCE=0
turns it off. The distributed test checks it bit for bit against the
fp32 rank-order sum followed by fused_norm_residual_norm_fwd, against the
production all-reduce within tolerance, rank identity of the stream, and
graph replay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
sawsa307 pushed a commit that referenced this pull request Nov 3, 2026
…he residual boundary"

This reverts commit 58f3342.

It moves to the stacked follow-up PR #8; this branch stays at the content
of b2e6c53, the commit under GPU validation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
claude and others added 2 commits November 3, 2026 21:33
…dary

The AMD router reads the BF16 pre-FFN-norm output; its BF16x3 GEMM keeps
only the router weights in FP32, so the rounding of the activations
remains. The CUDA path can feed the router the unrounded FP32 pre-norm
row instead (for 33-128 tokens), straight from its boundary kernel.

Add the same as an experiment, off by default
(VLLM_ROCM_MUSE_FP32_ROUTER_INPUT=1): the shared boundary helper takes an
optional FP32 store of the pre-norm result (ROUTER_OUT; with it off the
generated code is unchanged apart from the kernel-argument offsets of the
new pointer), fused_norm_residual_norm_fwd and the fused attention
all-reduce boundary expose it as router_out, the unfused fallback
computes it with the pre-norm, and the MoE router then runs its FP32 GEMM
on it. Whether it helps (acceptance length, quality) and what the FP32
router GEMM costs on MI355X still has to be measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…n shards

The column-sharded pooler/unpooler run [M, K] x [N/TP, K] GEMMs at decode
sizes, too narrow for hipBLASLt to fill the GPU. Port the fp32-accumulating
split-K Triton GEMM of vllm-project#63322 (each split accumulates in
fp32, a second kernel adds the splits in fixed order and rounds to BF16
once) and use it for _local_columns when a tuned config exists for the
shape and batch on gfx950; otherwise F.linear as before.

The tuned configs are vllm-project#63322's, for the TP8 shards (512 x 8192, 1024 x
4096). The TP4 shards (1024 x 8192, 2048 x 4096) have none yet, so TP4 is
unchanged until benchmarks/kernels/tune_muse_spark_rocm_skinny_gemm.py,
which times every valid config against F.linear under graph replay with
the weights streaming from HBM and prints _CONFIGS entries, is run on
MI355X. The ported kernel test also covers the TP4 shapes with explicit
configs.

Co-authored-by: fanxingran <xingran.fan@amd.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
@sawsa307 sawsa307 changed the title [ROCm][Perf] Muse Spark: next fusions on top of #7 (attention o_proj all-reduce in the residual boundary, ...) [ROCm][Perf] Muse Spark: follow-ups on #7 (o_proj all-reduce in the boundary, FP32 router input, skinny GEMM for shards) Nov 3, 2026
…kens under EP

Batches past the fused decode collectives' 256-token range (prefill, and
steps that mix a prefill chunk into the decode batch) ran every per-token
op on every rank and all-reduced around attention and the experts. With
EP over the whole TP group and DP1 (the TP4 + EP4 / TP8 + EP8 recipes),
batches of 256+ tokens now keep the residual stream split over tokens, as
the CUDA path does above its threshold: each rank all-gathers the
normalized attention input, reduce-scatters o_proj, runs the boundary,
router, pooler, pre-expert norm and unpooler on its slice, all-gathers
the router logits and pooled rows for the experts and reduce-scatters
their output; the final norm runs on the slice and is gathered once.
Smaller batches keep the replicated stream and the fused decode paths.

Ported from vllm-project#63722 and vllm-project#63723, adapted to this stack:
- The o_proj partial sum comes from quant_method.apply, as for the fused
  boundary, so o_proj.reduce_results (read by the decode monokernel and
  the stock path) is unchanged.
- The experts are built without the runner's final all-reduce; below the
  threshold the block all-reduces their output itself unless the fused
  EP finalize already did (exactly once either way; MuseMoERunner no
  longer reports the finalize's reduction when skip_final_all_reduce).
- Token-sharded batches never take the decode fusions (o_proj all-reduce
  in the boundary, deferred unpooler gather, FP32 router input), which
  would otherwise fire on a slice of <= 256 rows.
- The per-route expansion of the pooled rows follows the per-slot route
  decision (vllm-project#63298) through one _expands helper.
- Aux hidden states (DSpark) no longer turn token sharding off: the aux
  layers' states are all-gathered instead.
- The decode monokernel is not tried for token-sharded batches.

The ported test checks the token-sharded MoE block against its
all-reduce path and the token-sharded decoder layers against full
tokens (TP2 + EP), and adds a check that small batches reduce exactly
once with and without the fused finalize.

Co-authored-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
@sawsa307 sawsa307 changed the title [ROCm][Perf] Muse Spark: follow-ups on #7 (o_proj all-reduce in the boundary, FP32 router input, skinny GEMM for shards) [ROCm][Perf] Muse Spark: follow-ups on #7 (o_proj all-reduce in the boundary, token-sharded stream above 256, FP32 router input, skinny GEMM) Nov 3, 2026
claude and others added 15 commits November 3, 2026 21:50
…harded stream off

For same-image A/B runs; =0 restores the replicated stream and the
runner's all-reduce at every batch size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…d stream (opt-in)

The token-sharded path moves [T, H] activations between ranks two to six
times a layer at batch sizes past the CUDA-graph capture sizes, where
vLLM's TP collectives take RCCL (the AITER AG/RS fast path only runs for
uniform FULL-graph batches). ops/token_collectives.py runs these through
AITER's custom all-gather / reduce-scatter instead, in token chunks that
fit its IPC buffer (as vllm-project#63322 does for its column
all-gather), and falls back to RCCL for anything AITER does not take.
Off by default (VLLM_ROCM_MUSE_AITER_TOKEN_COLLECTIVES=1) until
benchmarks/kernels/benchmark_muse_spark_rocm_token_collectives.py, which
times both on the exact per-layer shapes, shows a win on MI355X.

Also:
- The MoE block clears the fused finalize's "already reduced" flag
  before its experts call, so the decision to all-reduce below the
  token-sharded threshold depends only on this call's finalize.
- Model construction logs whether the residual stream is token-sharded,
  or why not (pipeline parallel, or a non-MoE decoder layer).

Tests: AITER all-gather / reduce-scatter vs. RCCL, single- and
multi-chunk, and the token-sharded layers vs. full tokens with the AITER
collectives on (both skip without an AITER custom all-reduce).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…not a fixed tolerance

On MI355X (job 32646, b2e6c53) the EP cases failed at TP2/4/8 against
the unfused path with a fixed atol=1.5e-2, rtol=1e-2 (max abs diff up
to 0.0625, relative inf where the bf16 all-reduce cancelled to 0), after
passing the fp32 reference check.

The unfused path rounds each rank's partial output to bf16 and
all-reduces in bf16, half an ulp per partial and per add, and cancelling
partials round at their own, larger magnitude; the fused kernel sums the
fp32 partials and rounds once. A CPU simulation of the test reproduces
the failures (max |fused - unfused| = 0.0625 at TP2/4/8) with the fused
error vs. fp64 at ~0.031 (one rounding) for every TP size and the
unfused error growing with it (0.039, 0.058, 0.069).

The EP test now asserts, elementwise against the fp64 finalize:
- fused within one bf16 ulp plus an fp32 accumulation bound;
- unfused within its own bound (half an ulp per partial, per add);
- |fused - unfused| within the sum of the two.
Across 540 simulated checks the bounds hold with the largest observed
difference at 0.77 of the bound, and they reject a dropped or doubled
rank partial, a lost route weight, and partials rounded to bf16 before
the sum. Product code is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…le thresholds

The fused o_proj all-reduce + residual boundary was one-shot only: every rank
pushes its full partial rows to every peer, (W - 1) x the bytes of a two-shot
all-reduce over the links. Past ~32 tokens that loses to the unfused custom
all-reduce + boundary it replaces.

Add a two-shot variant: each partial row goes to its owner (t % W), the owner
sums the W partials in fp32 in rank order, rounds once and pushes the reduced
row to every rank, and after a second barrier every rank runs the boundary.
It rounds exactly as one-shot does, so both give the same bits. One-shot serves
up to VLLM_ROCM_MUSE_ATTN_ALL_REDUCE_ONE_SHOT_MAX_TOKENS (default 32), two-shot
the rest up to 256. The variants use disjoint workspace regions, their own
arrival counters and their own barrier targets, so any interleaving of calls is
safe.

VLLM_ROCM_MUSE_SHARD_PROJECTIONS_MAX_TOKENS (default 256) caps the batch size
the MoE pooler/unpooler are column-sharded at, for when the narrow shard GEMMs
lose to the full ones at the larger decode sizes.

benchmark_muse_spark_rocm_attn_all_reduce.py times unfused vs. one-shot vs.
two-shot per batch size, to set the one-shot threshold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…ed dtype

test_tp_partial_routes_are_reduced_before_norm required the route rows to
reach the all-reduce in FP32, but the runner all-reduces them in
VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE, which the deployment recipe sets to bf16:
on MI355X with the recipe environment the test failed at its mock's FP32
assertion (40/41 in the file). Its mock also promoted a bf16 input to FP32
(x + fp32 peer), unlike an all-reduce, so the norm never saw bf16 rows.

Run the test for both settings: the all-reduce receives the configured dtype,
returns it, and the norm normalizes that result. Runtime code is unchanged;
the test still fails if the runner ignores the setting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
On MI355X TP4 (job 32654, b2e6c53) the column-sharded pooler and
unpooler were slower than the replicated ones at every batch size from 1
to 256 tokens (pool 0.65-0.84x, unpool 0.50-0.62x), which the end-to-end
numbers cannot attribute. --breakdown times each piece on its own: full
and shard GEMM, the unfused norm / boundary and the gather kernels, to
tell a slow narrow GEMM from a slow cross-rank gather. Benchmark only;
it imports nothing newer than b2e6c53.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…rune

The tuner searched BLOCK_K up to 128 and BLOCK_M down to 8 for M <= 8, so
it could not reproduce the TP8 entries in _CONFIGS (all use BLOCK_K 256;
the M <= 8 unpooler entry uses BLOCK_M 16), nor find their TP4
counterparts. Search BLOCK_K up to 512 and splits up to 32, and BLOCK_M
from 16, which puts all nine TP8 entries in the space.

To keep the larger space affordable, --min-grid (default 256, one
workgroup per MI355X CU; every TP8 entry launches at least that) drops
under-filled grids, and --allowlist restricts each shape to a given config
list, e.g. the spill-free configs of an offline AOT compile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…ntract

Two #8 tests failed on MI355X (35b3a3f), both encoding behavior other
than what the code is meant to do:

* small_batch_moe_block_reduces_exactly_once compared the fused EP finalize
  (fp32 sum of the ranks' partials, rounded once) with the unfused path
  (partials rounded to bf16, then a bf16 all-reduce) at atol 0.002; at 7
  tokens 19 of 1792 outputs differed by up to 0.0068 (rel 1.66 where the
  partials cancel). A CPU model of the block reproduces differences up to
  0.0098. The test now counts the reductions (exactly one per forward:
  the fused finalize or the block's all-reduce) and checks both reduced
  outputs against the exact sum of the gathered bf16 partials, and the
  block outputs against each other, within bounds derived from the
  rounding; in the model a doubled or missing reduction breaks them in
  300/300 trials, the real difference stays within 0.98 of them.

* The "unreduced" fail-closed case predates 5821090, which allows
  skip_final_all_reduce when the MoE runs TP1 (EP): its output is then a
  sum of per-rank partials the token-sharded block reduces itself. The case
  now uses TP2, where the reduction must precede the post-expert norm, and
  a new test accepts the EP deployment. Each fails if the validation is
  reverted or removed.

Runtime code unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
On MI355X TP4 (job 32667, b2e6c53) the column gather kernels cost
15-25 us more than the norm / boundary they replace, flat from 1 to 256
tokens, while moving a few KB: a fixed per-launch cost, so the cross-rank
grid_barrier all four peer kernels share is the suspect.

Time a kernel that only runs the barrier, over block counts and waves per
block, in three forms: grid_barrier as used (one counter per rank, every
block arrives on every rank's counter, a system-scope release per wave);
the same without the per-wave release; and one flag per (writer, block)
with no contended atomics. Against an empty launch and a custom
all-reduce of [1, 8192] bf16, to see which part of the barrier the time
goes to before changing the protocol. Not yet run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Signed-off-by: David Cheung <d7cheung@gmail.com>
…lf-contained

Both imported _time_graph from benchmark_muse_spark_rocm_column_gather, which
only resolves with benchmarks/kernels on sys.path; a harness that copies one
script out of the tree fails at import (job 32668 on MI355X: rank 0
ModuleNotFoundError, the other ranks SIGTERM). Inline the graph-replay
timing helper in each, as the other Muse benchmarks do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Signed-off-by: David Cheung <d7cheung@gmail.com>
…ments

Add two flag-barrier variants that keep every wave's stores ordered before
the flag write, as the real kernels need: a per-wave release on a distinct
local word per (block, wave), at system scope and at device scope. With the
contention-free "flags" and the release-free "counter-nowave" already there,
one run separates the counter contention, the per-wave release and its
scope, and shows what a safe replacement would cost.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Signed-off-by: David Cheung <d7cheung@gmail.com>
…e variants

Job 32670 (MI355X TP4): the grid barrier as used costs 28.3 us at 64 blocks
x 16 waves against 4.3 us for a custom all-reduce; without the per-wave
system-scope release it is 11.6 us, and a per-block flag barrier 3.9-6.6 us.
A per-wave release at device scope costs the same as at system scope, so
the price is the atomic round trip to uncached memory, not the L2 writeback.
Add the two forms that avoid it: the per-wave release at workgroup scope on
the uncached word, and on a word of ordinary cached memory (the wait for
this wave's stores, vmcnt(0), plus one L2 atomic).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Signed-off-by: David Cheung <d7cheung@gmail.com>
The fused finalize, pooler/unpooler gathers and o_proj all-reduce boundary
synchronized with one counter per rank: every block of every rank added to
it with a system-scope release per wave. On MI355X TP4 (job 32670) that costs
28.3 us per barrier at 64 blocks x 16 waves, against 4.3 us for a custom
all-reduce; the gather kernels were 15-25 us slower than the norm they
replace and the fused EP finalize 3x slower than unfused (two barriers).
The microbenchmark splits it: 17 us is the per-wave release (1024 atomics on
one uncached word per rank), 6 us the counter contention.

grid_barrier now gives each (writer rank, block) its own flag on every rank:
a block publishes the call's target with one release-xchg per peer, polls all
W * NB flags of its own rank with one vector load, and acquires once. Each
wave still orders its stores before the flag, but with a workgroup-scope
release on a word of its own (vmcnt wait, no system-scope atomic). Workspaces
reserve W * NUM_BLOCKS words per barrier and one wave-scratch region; the
kernels' phase structure, targets and parity buffers are unchanged.

Model check (CPU): 300 random interleavings over 2/4/8 ranks x 1-8 blocks,
targets crossing the int32 wrap, double-buffered data: no stale read, no
deadlock; a flag written only locally deadlocks and a per-block-only wait
reads stale data, as expected. Not yet compiled or run on a GPU: the
benchmark's "grid_barrier" row and the finalize / column_gather /
token_sharded suites are the validation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Signed-off-by: David Cheung <d7cheung@gmail.com>
Job 32730 (MI355X TP4): the new grid_barrier costs 4.8 us at 64 blocks x 8
or 16 waves (the kernels' launches), down from 28.3, against 4.3 us for a
custom all-reduce. At one wave per block the poll loop failed to compile
(TritonGPURemoveLayoutConversions, the loop-carried flag vector); carry the
scalar minimum instead, which compiles at any wave count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Signed-off-by: David Cheung <d7cheung@gmail.com>
…the flag

The ISA of _gather_boundary_kernel (job 32731) shows the workgroup-scope
release atomic followed directly by s_barrier, with no s_waitcnt vmcnt(0):
without tgsplit the compiler omits the wait at workgroup scope, so a wave's
stores to the peers' uncached buffers could still be in flight when another
wave publishes the flag. Replace the atomic with an inline s_waitcnt vmcnt(0)
per wave (GFX9 counts stores in vmcnt), with a memory clobber so no store is
scheduled past it. Expected cost is unchanged from the 4.8 us measured.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Signed-off-by: David Cheung <d7cheung@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants