Repository navigation
[ROCm][Perf] Muse Spark: follow-ups on #7 (o_proj all-reduce in the boundary, token-sharded stream above 256, FP32 router input, skinny GEMM) - #8
Draft
sawsa307 wants to merge 19 commits into
Conversation
…ual boundary At decode sizes (up to 256 tokens) the decoder now runs o_proj's GEMM without its TP all-reduce (quant_method.apply; the layer's reduce_results is left alone, so the stock path, the decode monokernel and every other caller still reduce exactly once) and hands the partial rows to one Triton launch that reduces them inside the attention residual boundary: every rank pushes its partial [T, H] rows into every rank's buffer, and after one grid barrier sums the W partials in fp32 in rank order, rounds the sum to bf16 (the unfused all-reduce's output dtype; RNE on gfx942 and gfx950) and runs the boundary (post-attention norm, gated FP32 residual, pre-FFN norm) through the shared norm_residual_norm_tile. Every rank computes the same rows from the same bits, so the replicated FP32 stream stays identical across ranks. Applies under TP and TP + EP (o_proj is TP-sharded either way). The workspace is separate from the MoE gathers' and the fused finalize's, double-buffered by call parity. VLLM_ROCM_MUSE_FUSED_ATTN_ALL_REDUCE=0 turns it off. The distributed test checks it bit for bit against the fp32 rank-order sum followed by fused_norm_residual_norm_fwd, against the production all-reduce within tolerance, rank identity of the stream, and graph replay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
sawsa307
pushed a commit
that referenced
this pull request
Nov 3, 2026
…he residual boundary" This reverts commit 58f3342. It moves to the stacked follow-up PR #8; this branch stays at the content of b2e6c53, the commit under GPU validation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…dary The AMD router reads the BF16 pre-FFN-norm output; its BF16x3 GEMM keeps only the router weights in FP32, so the rounding of the activations remains. The CUDA path can feed the router the unrounded FP32 pre-norm row instead (for 33-128 tokens), straight from its boundary kernel. Add the same as an experiment, off by default (VLLM_ROCM_MUSE_FP32_ROUTER_INPUT=1): the shared boundary helper takes an optional FP32 store of the pre-norm result (ROUTER_OUT; with it off the generated code is unchanged apart from the kernel-argument offsets of the new pointer), fused_norm_residual_norm_fwd and the fused attention all-reduce boundary expose it as router_out, the unfused fallback computes it with the pre-norm, and the MoE router then runs its FP32 GEMM on it. Whether it helps (acceptance length, quality) and what the FP32 router GEMM costs on MI355X still has to be measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…n shards The column-sharded pooler/unpooler run [M, K] x [N/TP, K] GEMMs at decode sizes, too narrow for hipBLASLt to fill the GPU. Port the fp32-accumulating split-K Triton GEMM of vllm-project#63322 (each split accumulates in fp32, a second kernel adds the splits in fixed order and rounds to BF16 once) and use it for _local_columns when a tuned config exists for the shape and batch on gfx950; otherwise F.linear as before. The tuned configs are vllm-project#63322's, for the TP8 shards (512 x 8192, 1024 x 4096). The TP4 shards (1024 x 8192, 2048 x 4096) have none yet, so TP4 is unchanged until benchmarks/kernels/tune_muse_spark_rocm_skinny_gemm.py, which times every valid config against F.linear under graph replay with the weights streaming from HBM and prints _CONFIGS entries, is run on MI355X. The ported kernel test also covers the TP4 shapes with explicit configs. Co-authored-by: fanxingran <xingran.fan@amd.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…kens under EP Batches past the fused decode collectives' 256-token range (prefill, and steps that mix a prefill chunk into the decode batch) ran every per-token op on every rank and all-reduced around attention and the experts. With EP over the whole TP group and DP1 (the TP4 + EP4 / TP8 + EP8 recipes), batches of 256+ tokens now keep the residual stream split over tokens, as the CUDA path does above its threshold: each rank all-gathers the normalized attention input, reduce-scatters o_proj, runs the boundary, router, pooler, pre-expert norm and unpooler on its slice, all-gathers the router logits and pooled rows for the experts and reduce-scatters their output; the final norm runs on the slice and is gathered once. Smaller batches keep the replicated stream and the fused decode paths. Ported from vllm-project#63722 and vllm-project#63723, adapted to this stack: - The o_proj partial sum comes from quant_method.apply, as for the fused boundary, so o_proj.reduce_results (read by the decode monokernel and the stock path) is unchanged. - The experts are built without the runner's final all-reduce; below the threshold the block all-reduces their output itself unless the fused EP finalize already did (exactly once either way; MuseMoERunner no longer reports the finalize's reduction when skip_final_all_reduce). - Token-sharded batches never take the decode fusions (o_proj all-reduce in the boundary, deferred unpooler gather, FP32 router input), which would otherwise fire on a slice of <= 256 rows. - The per-route expansion of the pooled rows follows the per-slot route decision (vllm-project#63298) through one _expands helper. - Aux hidden states (DSpark) no longer turn token sharding off: the aux layers' states are all-gathered instead. - The decode monokernel is not tried for token-sharded batches. The ported test checks the token-sharded MoE block against its all-reduce path and the token-sharded decoder layers against full tokens (TP2 + EP), and adds a check that small batches reduce exactly once with and without the fused finalize. Co-authored-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…harded stream off For same-image A/B runs; =0 restores the replicated stream and the runner's all-reduce at every batch size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…d stream (opt-in) The token-sharded path moves [T, H] activations between ranks two to six times a layer at batch sizes past the CUDA-graph capture sizes, where vLLM's TP collectives take RCCL (the AITER AG/RS fast path only runs for uniform FULL-graph batches). ops/token_collectives.py runs these through AITER's custom all-gather / reduce-scatter instead, in token chunks that fit its IPC buffer (as vllm-project#63322 does for its column all-gather), and falls back to RCCL for anything AITER does not take. Off by default (VLLM_ROCM_MUSE_AITER_TOKEN_COLLECTIVES=1) until benchmarks/kernels/benchmark_muse_spark_rocm_token_collectives.py, which times both on the exact per-layer shapes, shows a win on MI355X. Also: - The MoE block clears the fused finalize's "already reduced" flag before its experts call, so the decision to all-reduce below the token-sharded threshold depends only on this call's finalize. - Model construction logs whether the residual stream is token-sharded, or why not (pipeline parallel, or a non-MoE decoder layer). Tests: AITER all-gather / reduce-scatter vs. RCCL, single- and multi-chunk, and the token-sharded layers vs. full tokens with the AITER collectives on (both skip without an AITER custom all-reduce). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…not a fixed tolerance On MI355X (job 32646, b2e6c53) the EP cases failed at TP2/4/8 against the unfused path with a fixed atol=1.5e-2, rtol=1e-2 (max abs diff up to 0.0625, relative inf where the bf16 all-reduce cancelled to 0), after passing the fp32 reference check. The unfused path rounds each rank's partial output to bf16 and all-reduces in bf16, half an ulp per partial and per add, and cancelling partials round at their own, larger magnitude; the fused kernel sums the fp32 partials and rounds once. A CPU simulation of the test reproduces the failures (max |fused - unfused| = 0.0625 at TP2/4/8) with the fused error vs. fp64 at ~0.031 (one rounding) for every TP size and the unfused error growing with it (0.039, 0.058, 0.069). The EP test now asserts, elementwise against the fp64 finalize: - fused within one bf16 ulp plus an fp32 accumulation bound; - unfused within its own bound (half an ulp per partial, per add); - |fused - unfused| within the sum of the two. Across 540 simulated checks the bounds hold with the largest observed difference at 0.77 of the bound, and they reject a dropped or doubled rank partial, a lost route weight, and partials rounded to bf16 before the sum. Product code is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…le thresholds The fused o_proj all-reduce + residual boundary was one-shot only: every rank pushes its full partial rows to every peer, (W - 1) x the bytes of a two-shot all-reduce over the links. Past ~32 tokens that loses to the unfused custom all-reduce + boundary it replaces. Add a two-shot variant: each partial row goes to its owner (t % W), the owner sums the W partials in fp32 in rank order, rounds once and pushes the reduced row to every rank, and after a second barrier every rank runs the boundary. It rounds exactly as one-shot does, so both give the same bits. One-shot serves up to VLLM_ROCM_MUSE_ATTN_ALL_REDUCE_ONE_SHOT_MAX_TOKENS (default 32), two-shot the rest up to 256. The variants use disjoint workspace regions, their own arrival counters and their own barrier targets, so any interleaving of calls is safe. VLLM_ROCM_MUSE_SHARD_PROJECTIONS_MAX_TOKENS (default 256) caps the batch size the MoE pooler/unpooler are column-sharded at, for when the narrow shard GEMMs lose to the full ones at the larger decode sizes. benchmark_muse_spark_rocm_attn_all_reduce.py times unfused vs. one-shot vs. two-shot per batch size, to set the one-shot threshold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…ed dtype test_tp_partial_routes_are_reduced_before_norm required the route rows to reach the all-reduce in FP32, but the runner all-reduces them in VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE, which the deployment recipe sets to bf16: on MI355X with the recipe environment the test failed at its mock's FP32 assertion (40/41 in the file). Its mock also promoted a bf16 input to FP32 (x + fp32 peer), unlike an all-reduce, so the norm never saw bf16 rows. Run the test for both settings: the all-reduce receives the configured dtype, returns it, and the norm normalizes that result. Runtime code is unchanged; the test still fails if the runner ignores the setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
On MI355X TP4 (job 32654, b2e6c53) the column-sharded pooler and unpooler were slower than the replicated ones at every batch size from 1 to 256 tokens (pool 0.65-0.84x, unpool 0.50-0.62x), which the end-to-end numbers cannot attribute. --breakdown times each piece on its own: full and shard GEMM, the unfused norm / boundary and the gather kernels, to tell a slow narrow GEMM from a slow cross-rank gather. Benchmark only; it imports nothing newer than b2e6c53. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…rune The tuner searched BLOCK_K up to 128 and BLOCK_M down to 8 for M <= 8, so it could not reproduce the TP8 entries in _CONFIGS (all use BLOCK_K 256; the M <= 8 unpooler entry uses BLOCK_M 16), nor find their TP4 counterparts. Search BLOCK_K up to 512 and splits up to 32, and BLOCK_M from 16, which puts all nine TP8 entries in the space. To keep the larger space affordable, --min-grid (default 256, one workgroup per MI355X CU; every TP8 entry launches at least that) drops under-filled grids, and --allowlist restricts each shape to a given config list, e.g. the spill-free configs of an offline AOT compile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
…ntract Two #8 tests failed on MI355X (35b3a3f), both encoding behavior other than what the code is meant to do: * small_batch_moe_block_reduces_exactly_once compared the fused EP finalize (fp32 sum of the ranks' partials, rounded once) with the unfused path (partials rounded to bf16, then a bf16 all-reduce) at atol 0.002; at 7 tokens 19 of 1792 outputs differed by up to 0.0068 (rel 1.66 where the partials cancel). A CPU model of the block reproduces differences up to 0.0098. The test now counts the reductions (exactly one per forward: the fused finalize or the block's all-reduce) and checks both reduced outputs against the exact sum of the gathered bf16 partials, and the block outputs against each other, within bounds derived from the rounding; in the model a doubled or missing reduction breaks them in 300/300 trials, the real difference stays within 0.98 of them. * The "unreduced" fail-closed case predates 5821090, which allows skip_final_all_reduce when the MoE runs TP1 (EP): its output is then a sum of per-rank partials the token-sharded block reduces itself. The case now uses TP2, where the reduction must precede the post-expert norm, and a new test accepts the EP deployment. Each fails if the validation is reverted or removed. Runtime code unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof
On MI355X TP4 (job 32667, b2e6c53) the column gather kernels cost 15-25 us more than the norm / boundary they replace, flat from 1 to 256 tokens, while moving a few KB: a fixed per-launch cost, so the cross-rank grid_barrier all four peer kernels share is the suspect. Time a kernel that only runs the barrier, over block counts and waves per block, in three forms: grid_barrier as used (one counter per rank, every block arrives on every rank's counter, a system-scope release per wave); the same without the per-wave release; and one flag per (writer, block) with no contended atomics. Against an empty launch and a custom all-reduce of [1, 8192] bf16, to see which part of the barrier the time goes to before changing the protocol. Not yet run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: David Cheung <d7cheung@gmail.com>
…lf-contained Both imported _time_graph from benchmark_muse_spark_rocm_column_gather, which only resolves with benchmarks/kernels on sys.path; a harness that copies one script out of the tree fails at import (job 32668 on MI355X: rank 0 ModuleNotFoundError, the other ranks SIGTERM). Inline the graph-replay timing helper in each, as the other Muse benchmarks do. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: David Cheung <d7cheung@gmail.com>
…ments Add two flag-barrier variants that keep every wave's stores ordered before the flag write, as the real kernels need: a per-wave release on a distinct local word per (block, wave), at system scope and at device scope. With the contention-free "flags" and the release-free "counter-nowave" already there, one run separates the counter contention, the per-wave release and its scope, and shows what a safe replacement would cost. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: David Cheung <d7cheung@gmail.com>
…e variants Job 32670 (MI355X TP4): the grid barrier as used costs 28.3 us at 64 blocks x 16 waves against 4.3 us for a custom all-reduce; without the per-wave system-scope release it is 11.6 us, and a per-block flag barrier 3.9-6.6 us. A per-wave release at device scope costs the same as at system scope, so the price is the atomic round trip to uncached memory, not the L2 writeback. Add the two forms that avoid it: the per-wave release at workgroup scope on the uncached word, and on a word of ordinary cached memory (the wait for this wave's stores, vmcnt(0), plus one L2 atomic). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: David Cheung <d7cheung@gmail.com>
The fused finalize, pooler/unpooler gathers and o_proj all-reduce boundary synchronized with one counter per rank: every block of every rank added to it with a system-scope release per wave. On MI355X TP4 (job 32670) that costs 28.3 us per barrier at 64 blocks x 16 waves, against 4.3 us for a custom all-reduce; the gather kernels were 15-25 us slower than the norm they replace and the fused EP finalize 3x slower than unfused (two barriers). The microbenchmark splits it: 17 us is the per-wave release (1024 atomics on one uncached word per rank), 6 us the counter contention. grid_barrier now gives each (writer rank, block) its own flag on every rank: a block publishes the call's target with one release-xchg per peer, polls all W * NB flags of its own rank with one vector load, and acquires once. Each wave still orders its stores before the flag, but with a workgroup-scope release on a word of its own (vmcnt wait, no system-scope atomic). Workspaces reserve W * NUM_BLOCKS words per barrier and one wave-scratch region; the kernels' phase structure, targets and parity buffers are unchanged. Model check (CPU): 300 random interleavings over 2/4/8 ranks x 1-8 blocks, targets crossing the int32 wrap, double-buffered data: no stale read, no deadlock; a flag written only locally deadlocks and a per-block-only wait reads stale data, as expected. Not yet compiled or run on a GPU: the benchmark's "grid_barrier" row and the finalize / column_gather / token_sharded suites are the validation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: David Cheung <d7cheung@gmail.com>
Job 32730 (MI355X TP4): the new grid_barrier costs 4.8 us at 64 blocks x 8 or 16 waves (the kernels' launches), down from 28.3, against 4.3 us for a custom all-reduce. At one wave per block the poll loop failed to compile (TritonGPURemoveLayoutConversions, the loop-carried flag vector); carry the scalar minimum instead, which compiles at any wave count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: David Cheung <d7cheung@gmail.com>
…the flag The ISA of _gather_boundary_kernel (job 32731) shows the workgroup-scope release atomic followed directly by s_barrier, with no s_waitcnt vmcnt(0): without tgsplit the compiler omits the wait at workgroup scope, so a wave's stores to the peers' uncached buffers could still be in flight when another wave publishes the flag. Replace the atomic with an inline s_waitcnt vmcnt(0) per wave (GFX9 counts stores in vmcnt), with a memory clobber so no store is scheduled past it. Expected cost is unchanged from the 4.8 us measured. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: David Cheung <d7cheung@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
o_projall-reduce fused into the residual boundary (≤ 256 tokens).o_projruns without its all-reduce (quant_method.apply;reduce_resultsuntouched); one launch reduces the partial rows (fp32, rank order, one bf16 RNE rounding) and runs the boundary through the sharednorm_residual_norm_tile. TP and TP + EP; own workspace.VLLM_ROCM_MUSE_FUSED_ATTN_ALL_REDUCE)VLLM_ROCM_MUSE_FP32_ROUTER_INPUT=1)benchmarks/kernels/tune_muse_spark_rocm_skinny_gemm.pyruns.o_projpartial viaquant_method.apply(monokernel untouched); small batches reduce the expert output exactly once with or without the fused EP finalize; token-sharded batches bypass the decode fusions; per-slot expansion respected; DSpark aux layers gathered.VLLM_ROCM_MUSE_TOKEN_SHARDED)output_reduced; startup log.VLLM_ROCM_MUSE_AITER_TOKEN_COLLECTIVES=1)o_projall-reduce boundary: one-shot moves (W − 1)× a two-shot all-reduce's link bytes and only wins at small batches (cost model: wins at ~32 tokens, loses at 64–128). Above the one-shot limit, each row goes to its owner (t % W), which sums and broadcasts it back; then every rank runs the boundary. Same rounding as one-shot, so the two are bit-identical. Disjoint workspace regions, counters and barrier targets, so any interleaving of calls is safe. Also makes the column-sharding token limit configurable. Newbenchmark_muse_spark_rocm_attn_all_reduce.py: unfused vs. one-shot vs. two-shot per batch size.VLLM_ROCM_MUSE_ATTN_ALL_REDUCE_ONE_SHOT_MAX_TOKENS), two-shot 33–256; column sharding ≤ 256 (VLLM_ROCM_MUSE_SHARD_PROJECTIONS_MAX_TOKENS)test_tp_partial_routes_are_reduced_before_normruns for bothVLLM_ROCM_MUSE_MOE_REDUCE_DTYPEsettings. It used to require FP32 at the all-reduce, which the recipe'sbf16contradicts (the 40/41 on MI355X). Its mock no longer promotes bf16 to FP32.Not implemented, with the reason (see #7 discussion): vLLM-side norm + MXFP4 quant (would split AITER's existing sort + quant fusion for a ~1 MB saving; worthwhile only AITER-side); router aux stream (disabled on ROCm by the CUDA Muse model because it hung; serialized under breakable graphs anyway);
o_projGEMM + reduce-scatter (next step on top of the token-sharded stream; no AMD GEMM-RS kernel yet).Validation
Done (no GPU in the environment):
o_projall-reduce + boundary bit-identical to fp32 rank-order sum → bf16 →fused_norm_residual_norm_fwd(output, new stream, FP32 router row), on every rank, for one-shot and two-shot interleaved on one workspace. Covered cases: T < W, T not a multiple of W, T at the one-shot limit and at the workspace maximum. Mutation check: 5/6 injected kernel bugs caught; the miss permutes the slot order, so it only changes the fp32 summation order.o_projboundary one-shot + two-shot with the switch anywhere in 0–32): 2000 randomized interleavings over 2/4/8 ranks with several blocks per rank, realistic and shuffled call orders, counters starting below the int32 wrap: no deadlock, no stale or future read. Mutation check: every injected protocol bug caught (shared counter/target between variants, missing second barrier, one counter for both phases, region aliasing).dwordx4, no spills, no peer-pointer waterfalls (the onlyexecnzloops are the barrier spins, one per barrier); fp32→bf16 RNE.VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE=bf16. The new test fails if the runner ignores the setting. The rest of the file gives the same results underfp32andbf16; its 4 CPU-only failures need a device and passed on MI355X.pre-commit: passes.To do on MI355X:
tests/kernels/moe/test_muse_spark_rocm_bf16_moe.pywith the recipe environment (expect 42/42 with the parametrized test).pytest -v -s tests/models/test_muse_spark_token_sharded.py -m 'distributed(num_gpus=2)'.tests/distributed/test_muse_spark_rocm_column_gather.py -k all_reduce_boundary(now also forces two-shot on tiny batches and captures both variants in one graph),tests/kernels/core/test_muse_spark_norms.py -k router_out,tests/kernels/test_muse_spark_rocm_skinny_gemm.py.torchrun --nproc_per_node=4 benchmarks/kernels/benchmark_muse_spark_rocm_attn_all_reduce.py: setVLLM_ROCM_MUSE_ATTN_ALL_REDUCE_ONE_SHOT_MAX_TOKENSat the one-shot/two-shot crossover. Disable the fusion above any batch size where unfused wins.python benchmarks/kernels/tune_muse_spark_rocm_skinny_gemm.py --tp-sizes 4→ TP4_CONFIGS; thenbenchmark_muse_spark_rocm_column_gather.py. LowerVLLM_ROCM_MUSE_SHARD_PROJECTIONS_MAX_TOKENSto the largest batch where shard + gather still wins.VLLM_ROCM_MUSE_FUSED_ATTN_ALL_REDUCE=1vs0; token sharding on vs. this branch's parent.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
🤖 Generated with Claude Code
https://claude.ai/code/session_01ELU3qEQcTP4Gdn8DYTbEof