Repository navigation
[ROCm][Perf] Muse Spark: MXFP4 MoE on native routes and column-sharded at TP prefill, via AITER's two-stage API - #64461
fanxingran wants to merge 4 commits into
Conversation
Run the experts on the native routes, and column-shard stage 2 at TP prefill. Falls back to the repeated-input path without the API. Signed-off-by: fanxingran <xingran.fan@amd.com>
There was a problem hiding this comment.
can we make this test simpler?
There was a problem hiding this comment.
Trimmed in 0a64cbf: 687 -> 474 lines, 8 -> 4 tests. Kept the bitwise native-route vs. repeated-input check (TP8 and EP8 ranks), graph replay, the TP2/TP8 column-sharded vs. reduce-scatter check, and the CPU deployment check. Dropped the standalone finalize-kernel test (the multi-GPU test covers that path) and the batch-size / no-API tests, and merged the two config helpers.
| # second copy of w2 (about 16 GiB per GPU at TP8), instead of | ||
| # reduce-scattering every route's partial sums. Set to 0 to keep the | ||
| # reduce-scatter and give that memory to the KV cache. | ||
| "VLLM_ROCM_MUSE_MOE_COLUMN_SHARDED_DOWN": lambda: ( |
There was a problem hiding this comment.
I guess this should be always ON and no need for an env knob here?
There was a problem hiding this comment.
Agreed, removed in 0a64cbf; it is now always on at TP > 1 without EP. The cost is 15.9 GiB per GPU at TP8 (KV cache -9.1%) for -34% TTFT at 8k/16k.
Both paths need the API from ROCm/aiter#7135 (under review). Without it this PR is a no-op, so the merge order does not matter.
There was a problem hiding this comment.
Why ROCm/aiter#7135: AITER's public fused_moe only returns the top-k sum, but Muse Spark normalizes each route before summing, so today we repeat the input top_k times. That PR splits fused_moe into fused_moe_stage1 / fused_moe_stage2, which gives us:
- per-route stage 2 output, so the experts run on the native
[T, top_k]routes; - a replaceable stage 2
w2and input, which the column-sharded path needs (the gathered stage 1 output times this rank'sw2rows).
Without it, vLLM would have to call the private _fused_moe_impl(_stage2_override=...) hook and re-derive the FlyDSL kernel arguments itself. That couples vLLM to AITER internals that can change in any release: a renamed hook breaks the model at load, and a change in kernel selection or argument layout can silently pick a different kernel or give wrong outputs. The ROCm CI pins one AITER version, so it would not catch this before an image bump. The public API keeps those internals inside AITER.
Signed-off-by: fanxingran <xingran.fan@amd.com>
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: fanxingran <xingran.fan@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
@Fangzhou-Ai Both of your comments are addressed in 0a64cbf: the test is trimmed to 4 cases and the env knob is removed. Could you take another look when you have time? @mawong-amd this pairs with #63254, which you reviewed; the two touch different branches of the finalize. It takes effect once ROCm/aiter#7135 lands. Thanks! |
|
“AITER 0.1.26 does not ship the Muse Spark FMoE tuning” we are using v0.1.29 now, are we still lacking this config? |
|
@Fangzhou-Ai No, it's there now. AITER ships the Muse Spark FMoE rows since 0.1.27 (29 rows since 0.1.28, ROCm/aiter#6817) and merges |
Fangzhou-Ai
left a comment
There was a problem hiding this comment.
let’s get aiter pr merged and included in vLLM before merging.
|
@Fangzhou-Ai Sounds good. ROCm/aiter#7135 has been through one round of review, and those comments are addressed. I've asked the AITER reviewers for another look and for help getting it merged. I'll update this PR once it's merged and available in vLLM's AITER. |
|
This pull request has merge conflicts that must be resolved before it can be |
Keep both module constants next to vllm-project#63085's token_sharded_moe_output. Signed-off-by: fanxingran <xingran.fan@amd.com>
Overview
Muse Spark normalizes each expert's output before the top-k weighted sum, so its MoE must return the unweighted per-route outputs
[T, top_k, H]. On ROCm the MXFP4 experts get them by repeating the normed inputtop_k(8) times and calling AITERfused_moewith one route per row, and at TP > 1 every route's partial sums over allHcolumns are then reduce-scattered before the norm.This PR runs the experts through AITER's two-stage MoE API (ROCm/aiter#7135):
[T, top_k]routes; stage 2 writes each route to its own row (per_slot). The kernels are the ones AITER selects for today's repeated-input call (config_shape), so the route outputs are bitwise identical.H / tpoutput rows ofw2. The norm then needs only a[T, top_k]sum-of-squares all-reduce and an all-gather of the finalized columns, instead of a[T, top_k, H]reduce-scatter.It replaces #63290 (native routes) and #63263 (column-sharded down projection), which reached into AITER internals (
_fused_moe_implwith_metadata_transformand_stage2_override) and could not be combined. With the public API both are one code path. #63290 and #63263 are closed in favour of this PR.Dependency. Needs ROCm/aiter#7135. With an AITER that lacks the API (including the current ROCm images), nothing changes: every batch takes the existing repeated-input path, and the GPU tests of this PR skip.
Claims
8×MI355X, Muse Spark Open MXFP4 + DSpark (7 speculative tokens), AMD Day 0 recipe;
main(77f58e5304) vs this PR, same node and arguments, AITER 0.1.26 + the AITER PR for this PR's arms, stock AITER 0.1.26 for the baseline:w2costs 15.9 GiB per GPU, KV cache −9.1%. TP8 + EP8: KV cache +1.0%, the[T * top_k, H]MoE input is gone.vllm/models/muse_spark/amd/, and no switches: with an AITER that has the API, the native routes are always used (their outputs are identical), and the column-sharded stage 2 at TP > 1 without EP.Validation
Setup. 8×MI355X (g35), image
vllm/vllm-openai-rocm:nightly-edad8b03c5285a1c12e7e4f83c1af709b319a01b(AITER 0.1.26, Python code replaced by the tree under test). PR arms: AITER'sfused_moe.pyreplaced by the AITER PR ported to 0.1.26 (the port applies cleanly; the API is the same). Modelamd/Muse-Spark-Open-816B-A42B-Quark-MXFP4, drafterInferact/Muse-Spark-Open-DSpark. Arms ran base, PR, PR, base per config. Commands are in the collapsed block.Single-request TTFT (median ms; two server starts per arm, mean of the two):
The arms above ran on AITER 0.1.26, which predates the Muse Spark FMoE tuning, so they use the default configs. AITER ships these rows since 0.1.27 (
muse-spark-open-mxfp4_fp4_tuned_fmoe.csv, the same 29 rows in 0.1.28 and 0.1.29) and loads them by default. With them on both sides (TP keys then pick FlyDSL v2 kernels), which matches current AITER, the gains hold:With these rows TP8 + EP8 greedy outputs are again identical to the baseline; at TP8 two baseline runs already differ from each other after the first tokens.
Serving (
vllm bench serve, random 8192 in / 1024 out, mean of two runs per concurrency; "step" = TPOT × acceptance length, the decode time per verify step, which does not depend on the DSpark acceptance length):Memory (KV cache tokens): TP8 6,905,363 → 6,274,865 (−9.1%; model weights 70.94 → 86.88 GiB per GPU); TP8 + EP8 6,785,908 → 6,851,774 (+1.0%).
Serve and benchmark commands
VLLM_ROCM_USE_AITER=1 VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE=bf16 VLLM_USE_BREAKABLE_CUDAGRAPH=1 \ VLLM_KV_CACHE_LAYOUT=LBHNC AITER_UNIFIED_ATTN_SHORT_QUERY_3D=1 \ vllm serve amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4 --served-model-name musespark \ --tokenizer-mode hf --config-format auto --load-format safetensors --generation-config vllm \ --language-model-only --dtype bfloat16 \ --tensor-parallel-size 8 [--enable-expert-parallel] --attention-backend ROCM_AITER_FA \ --max-model-len 32768 --max-num-seqs {32 for TP8, 128 for TP8 + EP8} --max-num-batched-tokens 8192 \ --gpu-memory-utilization 0.90 \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ --no-enable-prefix-caching --enable-chunked-prefill \ --reasoning-parser musespark1x --tool-call-parser musespark1x --enable-auto-tool-choice \ --speculative-config '{"method":"dspark","model":"Inferact/Muse-Spark-Open-DSpark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false,"attention_backend":"ROCM_AITER_FA"}'Single-request TTFT: K concurrent
/v1/completionsrequests with fresh random token-id prompts of L tokens,max_tokens=1,temperature=0; wall time until all K finish, median over n bursts (first burst dropped); (K, L, n) = (1, 2048, 12), (1, 8192, 16), (1, 16384, 8), (4, 8192, 6).Serving, after one warm-up, two runs per concurrency C:
GSM8K:
lm_eval --model local-completions --tasks gsm8k --num_fewshot 5(1319 questions, 32 concurrent requests), paired exact McNemar test on strict match.Accuracy
Tests (8×MI355X, nightly image above):
tests/kernels/moe/test_muse_spark_rocm_staged_moe.py(merges the tests of [ROCm][Perf] Muse Spark: run the AITER MXFP4 experts on the native top-k routes #63290 and [ROCm][Perf] Muse Spark: column-shard the MXFP4 MoE down projection at TP prefill #63263) plus the existing Muse ROCm tests (test_muse_spark_rocm_bf16_moe.py,test_muse_spark_moe_finalize.py,test_muse_spark_router.py,test_muse_spark_rocm_rmsnorm_expand.py,test_muse_spark_norms.py,tests/models/test_muse_spark_rocm_loading.py,test_muse_spark_logits_local.py,test_muse_spark_rocm_shared_rope.py): 131 passed, 20 skipped (CUDA-only kernels, and TP batch sizes 1 / 16 / 64 that by design keep the repeated input). The new file covers:w2reshard, the column-sharded route outputs within the BF16 rounding bound of the reduce-scatter path, and the final output on every rank;w2(CPU).main893d65b0a6, AITER 0.1.27 with the AITER PR: the new file plustest_muse_spark_rocm_bf16_moe.pyandtests/models/test_muse_spark_rocm_loading.py, 64 passed, 1 skipped (TP at 16 tokens).pytest kernels/moe). Until the AITER in the ROCm image has the API, its GPU tests skip; the multi-GPU test also needs 2 or 8 MI355X.Details
Native routes.
staged_fused_moecallsfused_moe_stage1with the native[T, top_k]routes, unit weights andconfig_shape=(T * top_k, 1)(+1slot under EP, matching today's masked-slot key), thenfused_moe_stage2(per_slot=True)into the modular kernel's[T, top_k, H]output. The pre-expert norm writes one row per token (fused_rmsnorm_expand(..., 1)) for those batches. A batch uses it whensupports_stage2_overridesholds for its config (a lookup, cached per batch size) and, under TP, from 65 tokens up: with 256 experts the native-route sorting does not pay off for smaller TP batches. Under EP every batch size uses it.Column-sharded stage 2 (TP > 1, no EP, eager, ≥ 1024 tokens):
w2[E, H, I/TP]and its scales intow2[E, H/TP, I], shuffled like the regularw2(kept for decode and small batches).fused_moe_stage2(per_slot=True, w2=..., a2=...)then gives each route's exact output on this rank's columns.[T, top_k]fp32), two Triton kernels for the norm and the weighted top-k sum on the local columns, and an all-gather of[T, H]. At T = 8192 per layer a 512 MiB reduce-scatter becomes about 136 MiB of all-gathers plus 256 KiB of all-reduce.Numerics. Native routes: identical. Column-sharded: both paths compute the same products from the same quantized inputs; the baseline rounds each rank's partial sum to BF16 and reduces in BF16 (
VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE=bf16in the recipe), this path accumulates the full intermediate dim in fp32 and rounds once. The unit test bounds the difference by2^-8 · (Σ|partial| + |sum|)with a 2× margin.Memory. Column-sharded: per MXFP4 layer and rank, 256 MiB of FP4 plus 16 MiB of scales, 15.9 GiB over 60 layers at TP8; no extra workspace (the route outputs reuse 1/TP of the output buffer). Native routes: the
[T * top_k, H]expanded input is no longer allocated (+1% KV cache under EP).Fallback. AITER without the API, configs whose stage 2 is not a FlyDSL MXFP4 kernel, padded / non-SiLU / non-BF16 experts, or TP batches < 65 tokens: the existing repeated-input path, unchanged.
Complexity. 746 non-test lines:
staged_moe.py(344): the AITER API calls, the support check, thew2reshard and the activation and scale gathers.ops/column_sharded_finalize.py(199): the two Triton finalize kernels.muse_moe_runner.py(+199 / −16),model.py(+4 / −1): wiring.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.