Skip to content

[ROCm][Perf] Muse Spark: MXFP4 MoE on native routes and column-sharded at TP prefill, via AITER's two-stage API - #64461

Open
fanxingran wants to merge 4 commits into
vllm-project:mainfrom
fanxingran:fxr/muse-rocm-staged-moe
Open

fanxingran wants to merge 4 commits into
vllm-project:mainfrom
fanxingran:fxr/muse-rocm-staged-moe

Conversation

@fanxingran

@fanxingran fanxingran commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Muse Spark normalizes each expert's output before the top-k weighted sum, so its MoE must return the unweighted per-route outputs [T, top_k, H]. On ROCm the MXFP4 experts get them by repeating the normed input top_k (8) times and calling AITER fused_moe with one route per row, and at TP > 1 every route's partial sums over all H columns are then reduce-scattered before the norm.

This PR runs the experts through AITER's two-stage MoE API (ROCm/aiter#7135):

  • Native routes. One input row per token, on the [T, top_k] routes; stage 2 writes each route to its own row (per_slot). The kernels are the ones AITER selects for today's repeated-input call (config_shape), so the route outputs are bitwise identical.
  • Column-sharded stage 2 at TP prefill. For eager batches of ≥ 1024 tokens, the MXFP4 stage-1 output and its scales are all-gathered along the intermediate dim and stage 2 runs against this rank's H / tp output rows of w2. The norm then needs only a [T, top_k] sum-of-squares all-reduce and an all-gather of the finalized columns, instead of a [T, top_k, H] reduce-scatter.

It replaces #63290 (native routes) and #63263 (column-sharded down projection), which reached into AITER internals (_fused_moe_impl with _metadata_transform and _stage2_override) and could not be combined. With the public API both are one code path. #63290 and #63263 are closed in favour of this PR.

Dependency. Needs ROCm/aiter#7135. With an AITER that lacks the API (including the current ROCm images), nothing changes: every batch takes the existing repeated-input path, and the GPU tests of this PR skip.

Claims

8×MI355X, Muse Spark Open MXFP4 + DSpark (7 speculative tokens), AMD Day 0 recipe; main (77f58e5304) vs this PR, same node and arguments, AITER 0.1.26 + the AITER PR for this PR's arms, stock AITER 0.1.26 for the baseline:

  • TP8 + DSpark: single-request TTFT −13% at 2k and −34% at 8k / 16k / 4 × 8k. Serving 8k in / 1k out: TTFT −31% to −33% at every concurrency, output throughput +17% at c16 and c32.
  • TP8 + EP8 + DSpark (recipe default): TTFT −8% at 2k and −12% at 8k / 16k / 4 × 8k. Output throughput +10% at c16 and +12% at c32.
  • Accuracy. GSM8K TP8 0.9492 → 0.9530 (paired McNemar p = 0.47); TP8 + EP8 0.9477 → 0.9477, the same answer on all 1319 questions. TP8 + EP8 greedy outputs (text and every logprob) are identical to the baseline. At TP8 they are identical too with the column-sharded stage 2 off; with it, the sum is rounded once instead of per rank, see Numerics.
  • Memory. TP8: the second w2 costs 15.9 GiB per GPU, KV cache −9.1%. TP8 + EP8: KV cache +1.0%, the [T * top_k, H] MoE input is gone.
  • No core changes, everything is in vllm/models/muse_spark/amd/, and no switches: with an AITER that has the API, the native routes are always used (their outputs are identical), and the column-sharded stage 2 at TP > 1 without EP.

Validation

Setup. 8×MI355X (g35), image vllm/vllm-openai-rocm:nightly-edad8b03c5285a1c12e7e4f83c1af709b319a01b (AITER 0.1.26, Python code replaced by the tree under test). PR arms: AITER's fused_moe.py replaced by the AITER PR ported to 0.1.26 (the port applies cleanly; the API is the same). Model amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4, drafter Inferact/Muse-Spark-Open-DSpark. Arms ran base, PR, PR, base per config. Commands are in the collapsed block.

Single-request TTFT (median ms; two server starts per arm, mean of the two):

Config prompt base this PR change
TP8 + DSpark 2k 115.4 / 115.7 100.4 / 99.8 −13.4%
TP8 + DSpark 8k 354.8 / 355.7 232.9 / 234.3 −34.2%
TP8 + DSpark 16k 721.1 / 720.1 477.4 / 477.2 −33.8%
TP8 + DSpark 4 × 8k 1394.9 / 1393.1 910.9 / 912.0 −34.6%
TP8 + EP8 + DSpark 2k 101.9 / 101.2 94.4 / 93.1 −7.7%
TP8 + EP8 + DSpark 8k 268.4 / 268.7 235.5 / 234.1 −12.6%
TP8 + EP8 + DSpark 16k 546.9 / 549.7 485.0 / 479.4 −12.1%
TP8 + EP8 + DSpark 4 × 8k 1050.6 / 1051.8 920.1 / 919.2 −12.5%

The arms above ran on AITER 0.1.26, which predates the Muse Spark FMoE tuning, so they use the default configs. AITER ships these rows since 0.1.27 (muse-spark-open-mxfp4_fp4_tuned_fmoe.csv, the same 29 rows in 0.1.28 and 0.1.29) and loads them by default. With them on both sides (TP keys then pick FlyDSL v2 kernels), which matches current AITER, the gains hold:

Config prompt base this PR change
TP8 + DSpark 2k 111.3 / 112.9 97.6 / 97.0 −13.2%
TP8 + DSpark 8k 345.8 / 344.4 229.7 / 231.3 −33.2%
TP8 + DSpark 16k 701.7 / 702.8 470.2 / 473.3 −32.8%
TP8 + DSpark 4 × 8k 1358.0 / 1357.7 899.3 / 900.5 −33.7%
TP8 + EP8 + DSpark 2k 99.2 / 98.9 93.2 / 90.7 −7.2%
TP8 + EP8 + DSpark 8k 256.5 / 253.4 229.1 / 228.1 −10.3%
TP8 + EP8 + DSpark 16k 523.9 / 522.8 473.0 / 468.9 −10.0%
TP8 + EP8 + DSpark 4 × 8k 997.3 / 999.3 897.0 / 896.8 −10.2%

With these rows TP8 + EP8 greedy outputs are again identical to the baseline; at TP8 two baseline runs already differ from each other after the first tokens.

Serving (vllm bench serve, random 8192 in / 1024 out, mean of two runs per concurrency; "step" = TPOT × acceptance length, the decode time per verify step, which does not depend on the DSpark acceptance length):

Config Conc Output tok/s Mean TTFT (ms) Mean TPOT (ms) Step (ms) Accept len
TP8 + DSpark 1 181.9 → 180.4 (−0.8%) 372 → 251 (−32.6%) 5.14 → 5.30 13.57 → 13.55 (−0.1%) 2.64 / 2.55
TP8 + DSpark 4 474.7 → 612.8 (+29.1%) 620 → 429 (−30.8%) 6.62 → 5.67 18.98 → 18.51 (−2.5%) 2.87 / 3.27
TP8 + DSpark 16 1058.4 → 1245.0 (+17.6%) 1262 → 848 (−32.8%) 12.83 → 11.10 35.80 → 31.68 (−11.5%) 2.79 / 2.85
TP8 + DSpark 32 1356.0 → 1589.6 (+17.2%) 2043 → 1378 (−32.5%) 20.44 → 17.27 59.48 → 48.86 (−17.9%) 2.91 / 2.83
TP8 + EP8 + DSpark 1 177.8 → 181.8 (+2.2%) 285 → 253 (−11.5%) 5.35 → 5.26 14.55 → 14.31 (−1.7%) 2.72 / 2.72
TP8 + EP8 + DSpark 4 489.5 → 507.3 (+3.6%) 482 → 465 (−3.6%) 6.67 → 6.36 19.18 → 18.52 (−3.4%) 2.88 / 2.92
TP8 + EP8 + DSpark 16 1153.3 → 1273.7 (+10.4%) 946 → 835 (−11.7%) 11.82 → 10.77 32.08 → 30.63 (−4.5%) 2.71 / 2.84
TP8 + EP8 + DSpark 32 1507.5 → 1681.1 (+11.5%) 1549 → 1388 (−10.4%) 17.75 → 16.29 50.06 → 46.02 (−8.1%) 2.82 / 2.83
  • Prefill gains at every concurrency. Decode steps (< 1024 tokens) at TP8 run the native routes only from 65 tokens up; at c1 the step time is unchanged. From c16 on, prefill chunks overlap decode and the step time drops with them.
  • On random prompts the acceptance length moves between runs (TP8 c4: 2.87 → 3.27), so compare decode by the step column.

Memory (KV cache tokens): TP8 6,905,363 → 6,274,865 (−9.1%; model weights 70.94 → 86.88 GiB per GPU); TP8 + EP8 6,785,908 → 6,851,774 (+1.0%).

Serve and benchmark commands
VLLM_ROCM_USE_AITER=1 VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE=bf16 VLLM_USE_BREAKABLE_CUDAGRAPH=1 \
VLLM_KV_CACHE_LAYOUT=LBHNC AITER_UNIFIED_ATTN_SHORT_QUERY_3D=1 \
vllm serve amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4 --served-model-name musespark \
  --tokenizer-mode hf --config-format auto --load-format safetensors --generation-config vllm \
  --language-model-only --dtype bfloat16 \
  --tensor-parallel-size 8 [--enable-expert-parallel] --attention-backend ROCM_AITER_FA \
  --max-model-len 32768 --max-num-seqs {32 for TP8, 128 for TP8 + EP8} --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.90 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --no-enable-prefix-caching --enable-chunked-prefill \
  --reasoning-parser musespark1x --tool-call-parser musespark1x --enable-auto-tool-choice \
  --speculative-config '{"method":"dspark","model":"Inferact/Muse-Spark-Open-DSpark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false,"attention_backend":"ROCM_AITER_FA"}'

Single-request TTFT: K concurrent /v1/completions requests with fresh random token-id prompts of L tokens, max_tokens=1, temperature=0; wall time until all K finish, median over n bursts (first burst dropped); (K, L, n) = (1, 2048, 12), (1, 8192, 16), (1, 16384, 8), (4, 8192, 6).

Serving, after one warm-up, two runs per concurrency C:

vllm bench serve --backend openai --endpoint /v1/completions --model musespark \
  --dataset-name random --random-input-len 8192 --random-output-len 1024 --random-range-ratio 0.0 \
  --request-rate inf --ignore-eos --temperature 0 \
  --num-prompts $((4 * C > 16 ? 4 * C : 16)) --max-concurrency $C

GSM8K: lm_eval --model local-completions --tasks gsm8k --num_fewshot 5 (1319 questions, 32 concurrent requests), paired exact McNemar test on strict match.

Accuracy

  • GSM8K 5-shot strict, TP8 + DSpark: 0.9492 → 0.9530; 13 questions correct only on the baseline, 18 only with this PR, p = 0.47 (flexible 0.9484 → 0.9530, p = 0.38).
  • GSM8K 5-shot strict, TP8 + EP8 + DSpark: 0.9477 → 0.9477, the same verdict on every question.
  • Greedy outputs with logprobs on 5 fixed prompts (45 to 8192 tokens), each identical across server starts of the same arm:
    • TP8 + EP8: text and every logprob identical to the baseline.
    • TP8 with the column-sharded stage 2 off (an env switch at the time of these runs, removed in review): identical to the baseline, so the native-route path is exact.
    • TP8 with the column-sharded stage 2: the 45-token prompt is identical; the 1024 / 4096 / 8192 / 8192-token prompts differ from the baseline in the first-token logprob by 0.07 / 0.41 / 0.19 / 0.50 nats, the same on every server start. Since the outputs are identical with the column-sharded stage 2 off, these differences come only from where the sum is rounded (see Numerics).

Tests (8×MI355X, nightly image above):

  • New tests/kernels/moe/test_muse_spark_rocm_staged_moe.py (merges the tests of [ROCm][Perf] Muse Spark: run the AITER MXFP4 experts on the native top-k routes #63290 and [ROCm][Perf] Muse Spark: column-shard the MXFP4 MoE down projection at TP prefill #63263) plus the existing Muse ROCm tests (test_muse_spark_rocm_bf16_moe.py, test_muse_spark_moe_finalize.py, test_muse_spark_router.py, test_muse_spark_rocm_rmsnorm_expand.py, test_muse_spark_norms.py, tests/models/test_muse_spark_rocm_loading.py, test_muse_spark_logits_local.py, test_muse_spark_rocm_shared_rope.py): 131 passed, 20 skipped (CUDA-only kernels, and TP batch sizes 1 / 16 / 64 that by design keep the repeated input). The new file covers:
    • native routes against the repeated input, bitwise for the per-route and the finalized outputs, at the TP8 and EP8 rank shapes, T = 16 … 4096 (under EP, non-local routes stay unwritten and the finalize masks them);
    • CUDA graph capture and replay;
    • at TP2 and TP8 with the real shapes (E 256, H 4096, I/TP 512), T = 1024, 1031, 8192: the bit-exact w2 reshard, the column-sharded route outputs within the BF16 rounding bound of the reduce-scatter path, and the final output on every rank;
    • which deployments build the second w2 (CPU).
  • After the review changes (test file trimmed, env switch removed), merged onto main 893d65b0a6, AITER 0.1.27 with the AITER PR: the new file plus test_muse_spark_rocm_bf16_moe.py and tests/models/test_muse_spark_rocm_loading.py, 64 passed, 1 skipped (TP at 16 tokens).
  • Same tests on stock AITER 0.1.26 (no API): the GPU tests of the new file skip, the rest pass.
  • CI: the file runs in the "MoE Kernels" steps (pytest kernels/moe). Until the AITER in the ROCm image has the API, its GPU tests skip; the multi-GPU test also needs 2 or 8 MI355X.

Details

Native routes. staged_fused_moe calls fused_moe_stage1 with the native [T, top_k] routes, unit weights and config_shape=(T * top_k, 1) (+1 slot under EP, matching today's masked-slot key), then fused_moe_stage2(per_slot=True) into the modular kernel's [T, top_k, H] output. The pre-expert norm writes one row per token (fused_rmsnorm_expand(..., 1)) for those batches. A batch uses it when supports_stage2_overrides holds for its config (a lookup, cached per batch size) and, under TP, from 65 tokens up: with 256 experts the native-route sorting does not pay off for smaller TP batches. Under EP every batch size uses it.

Column-sharded stage 2 (TP > 1, no EP, eager, ≥ 1024 tokens):

  • Weights, at load. Before the AITER shuffle, one all-to-all turns this rank's w2[E, H, I/TP] and its scales into w2[E, H/TP, I], shuffled like the regular w2 (kept for decode and small batches).
  • Experts. Between the stages, the MXFP4 stage-1 output and its E8M0 scales are all-gathered along the intermediate dim. Each rank's slice is whole 32-element quant groups and whole scale tiles, so the gathered tensors equal the full-width quantization exactly. fused_moe_stage2(per_slot=True, w2=..., a2=...) then gives each route's exact output on this rank's columns.
  • Finalize. An all-reduce of the per-route sum of squares ([T, top_k] fp32), two Triton kernels for the norm and the weighted top-k sum on the local columns, and an all-gather of [T, H]. At T = 8192 per layer a 512 MiB reduce-scatter becomes about 136 MiB of all-gathers plus 256 KiB of all-reduce.
  • The decision depends only on shapes, the capture state and AITER's tuned configs, so all ranks agree. Graph capture never reaches it (the recipe captures ≤ 512 tokens).

Numerics. Native routes: identical. Column-sharded: both paths compute the same products from the same quantized inputs; the baseline rounds each rank's partial sum to BF16 and reduces in BF16 (VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE=bf16 in the recipe), this path accumulates the full intermediate dim in fp32 and rounds once. The unit test bounds the difference by 2^-8 · (Σ|partial| + |sum|) with a 2× margin.

Memory. Column-sharded: per MXFP4 layer and rank, 256 MiB of FP4 plus 16 MiB of scales, 15.9 GiB over 60 layers at TP8; no extra workspace (the route outputs reuse 1/TP of the output buffer). Native routes: the [T * top_k, H] expanded input is no longer allocated (+1% KV cache under EP).

Fallback. AITER without the API, configs whose stage 2 is not a FlyDSL MXFP4 kernel, padded / non-SiLU / non-BF16 experts, or TP batches < 65 tokens: the existing repeated-input path, unchanged.

Complexity. 746 non-test lines:

  • staged_moe.py (344): the AITER API calls, the support check, the w2 reshard and the activation and scale gathers.
  • ops/column_sharded_finalize.py (199): the two Triton finalize kernels.
  • muse_moe_runner.py (+199 / −16), model.py (+4 / −1): wiring.

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

Run the experts on the native routes, and column-shard stage 2 at TP
prefill. Falls back to the repeated-input path without the API.

Signed-off-by: fanxingran <xingran.fan@amd.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we make this test simpler?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trimmed in 0a64cbf: 687 -> 474 lines, 8 -> 4 tests. Kept the bitwise native-route vs. repeated-input check (TP8 and EP8 ranks), graph replay, the TP2/TP8 column-sharded vs. reduce-scatter check, and the CPU deployment check. Dropped the standalone finalize-kernel test (the multi-GPU test covers that path) and the batch-size / no-API tests, and merged the two config helpers.

Comment thread vllm/envs.py Outdated
# second copy of w2 (about 16 GiB per GPU at TP8), instead of
# reduce-scattering every route's partial sums. Set to 0 to keep the
# reduce-scatter and give that memory to the KV cache.
"VLLM_ROCM_MUSE_MOE_COLUMN_SHARDED_DOWN": lambda: (

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this should be always ON and no need for an env knob here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, removed in 0a64cbf; it is now always on at TP > 1 without EP. The cost is 15.9 GiB per GPU at TP8 (KV cache -9.1%) for -34% TTFT at 8k/16k.

Both paths need the API from ROCm/aiter#7135 (under review). Without it this PR is a no-op, so the merge order does not matter.

@fanxingran fanxingran Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why ROCm/aiter#7135: AITER's public fused_moe only returns the top-k sum, but Muse Spark normalizes each route before summing, so today we repeat the input top_k times. That PR splits fused_moe into fused_moe_stage1 / fused_moe_stage2, which gives us:

  • per-route stage 2 output, so the experts run on the native [T, top_k] routes;
  • a replaceable stage 2 w2 and input, which the column-sharded path needs (the gathered stage 1 output times this rank's w2 rows).

Without it, vLLM would have to call the private _fused_moe_impl(_stage2_override=...) hook and re-derive the FlyDSL kernel arguments itself. That couples vLLM to AITER internals that can change in any release: a renamed hook breaks the model at load, and a change in kernel selection or argument layout can silently pick a different kernel or give wrong outputs. The ROCm CI pins one AITER version, so it would not catch this before an image bump. The public API keeps those internals inside AITER.

Signed-off-by: fanxingran <xingran.fan@amd.com>
@mergify

mergify Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fanxingran.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 7, 2026
Signed-off-by: fanxingran <xingran.fan@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@fanxingran

Copy link
Copy Markdown
Contributor Author

@Fangzhou-Ai Both of your comments are addressed in 0a64cbf: the test is trimmed to 4 cases and the env knob is removed. Could you take another look when you have time? @mawong-amd this pairs with #63254, which you reviewed; the two touch different branches of the finalize. It takes effect once ROCm/aiter#7135 lands. Thanks!

@Fangzhou-Ai

Copy link
Copy Markdown
Collaborator

“AITER 0.1.26 does not ship the Muse Spark FMoE tuning” we are using v0.1.29 now, are we still lacking this config?

@fanxingran

Copy link
Copy Markdown
Contributor Author

@Fangzhou-Ai No, it's there now. AITER ships the Muse Spark FMoE rows since 0.1.27 (29 rows since 0.1.28, ROCm/aiter#6817) and merges model_configs/*tuned_fmoe*.csv by default when AITER_CONFIG_FMOE is unset. The second TTFT table was measured with exactly these rows (the CSV is identical to v0.1.29's), so it reflects current AITER; the first one is the older 0.1.26 image without them. I've reworded that paragraph in the description. What v0.1.29 still lacks is the two-stage API from ROCm/aiter#7135, which this PR needs.

@Fangzhou-Ai Fangzhou-Ai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let’s get aiter pr merged and included in vLLM before merging.

@fanxingran

Copy link
Copy Markdown
Contributor Author

@Fangzhou-Ai Sounds good. ROCm/aiter#7135 has been through one round of review, and those comments are addressed. I've asked the AITER reviewers for another look and for help getting it merged. I'll update this PR once it's merged and available in vLLM's AITER.

@mergify

mergify Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fanxingran.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 17, 2026
Keep both module constants next to vllm-project#63085's token_sharded_moe_output.

Signed-off-by: fanxingran <xingran.fan@amd.com>
@mergify mergify Bot removed the needs-rebase label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rocm Related to AMD ROCm

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

3 participants