Repository navigation
[ROCm][Muse Spark] Support data parallelism in the MoE finalize - #30
Open
amd-mghanimi wants to merge 2 commits into
Open
amd-mghanimi wants to merge 2 commits into
amd-mghanimi wants to merge 2 commits into
Conversation
With EP, every route is computed whole on its owner rank, so the post-expert norm is local and the DP/SP combine plus TP all-reduce that follow are linear: lifting the DP1 check, and the sequence-parallel check under EP (vLLM turns SP MoE on for DP x TP x EP), is enough. Without EP, MoE TP spans DP x TP ranks over DP-gathered tokens. Reduce-scatter the routes over DP first, so each rank holds its own tokens' TP partial sums exactly as without DP, then finalize as before; zero the other DP ranks' rows so the DP combine that follows passes this rank's tokens through. Co-authored-by: Cursor <cursoragent@cursor.com>
… group DP without expert parallelism flattens MoE TP across DP×TP, so the column-sharded down projection has to gather on that group rather than the attention TP group. Choose the group by comparing EP and TP world sizes. Reading the vLLM config in the forward path asserts during warm-up. Co-authored-by: Cursor <cursoragent@cursor.com>
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Muse Spark on ROCm rejects
--data-parallel-size > 1at startup. The post-expert RMSNorm needs each route's complete expert output, and DP changes the expert sharding. This PR accepts DP and keeps that norm correct, for DP without expert parallelism (DP2×TP4) and with it (DP2×TEP4).This PR is stacked on
muse-rocm-future-prs(vllm-project#63322, vllm-project#63254, vllm-project#64461, vllm-project#63085, vllm-project#63729). It supersedes #28, which targetedmainfrommuse-rocm-dp-moe.Claims
Validation
Arms
be47189f37f6162991366ff6b91539f6e9d1f432with AITER9f30ad8, plus [ROCm][Perf] Muse Spark: finalize the token-sharded MoE reduction in one reduce-scatter vllm-project/vllm#63254, [ROCm][Perf] Muse Spark: MXFP4 MoE on native routes and column-sharded at TP prefill, via AITER's two-stage API vllm-project/vllm#64461, and [ROCm][Perf] Muse Spark Open Shard residual stream across TP vllm-project/vllm#63085. Taglocal/vllm-rocm:nightly-be47189-aiter9f30ad8-mmoe-base-730744. Used for the TP4 and TEP4 single-replica runs.patches/dp-moe.diff. Tags…-dp-63cb98(DP2×TP4) and…-dp-400fca(DP2×TEP4).The measured arms are not this branch. This branch is on nightly
96710245and also includes vllm-project#63322 and vllm-project#63729, and its DP code is the commit series onmuse-rocm-dp-after-stack. The numbers below are Arm A vs Arm B, and they apply to this branch only after it is rebuilt and rerun.The DP2×TEP4 run used
…-dp-400fca. The DP2×TP4 run that finished used…-dp-63cb98, which differs only incolumn_shard_group(see Details).Serving comparison
Muse Spark Open 816B MXFP4, random prompts with range ratio 0.8, 90 sweep points per arm. Each row is the mean of that workload's repeats. Arm A is one TP4 or TEP4 replica; Arm B is two replicas (DP2×TP4 or DP2×TEP4) on 8 GPUs. Output throughput is tok/s; median TTFT and mean TPOT are milliseconds.
DP2×TP4 (Arm B) against TP4 (Arm A)
DP2×TEP4 (Arm B) against TEP4 (Arm A)
Accuracy
Required for this PR. Not measured on this branch yet. Earlier GSM8K 5-shot results were from an older revision of this DP finalize on nightly
96710245, and they don't cover the column-group change: TP4 0.9462, TEP4 0.950, DP2×TP4 0.9507, DP2×TEP4 0.9409. These will be rerun on this branch's image before merge.Tests
tests/kernels/moe/test_muse_spark_rocm_bf16_moe.pycovers the DP finalize against the single-rank result, andtests/kernels/moe/test_muse_spark_rocm_staged_moe.pycovers the DP case of the column-shard gate. Neither was run on GPU, and torch is not installed in the local environment, so they have not been run at all on the current commits.Details
With DP and no expert parallelism, MoE TP is flattened across DP×TP. The pre-norm all-reduce only covers the attention TP group, so route outputs stay partial over DP. This PR reduce-scatters the gathered route partials over the DP group before the existing TP finalize, and zeroes the other ranks' rows so the DP combine returns this rank's tokens unchanged.
With expert parallelism, each route is complete on its owner rank, so the norm is local. Sequence-parallel MoE is accepted in that case.
The fused norm-expand handoff stays off when
dp_size > 1or sequence-parallel MoE is on. It returns a strided first-route view, and the DP all-gather reads the wrong rows from that view.vllm-project#64461's column gathers run on the flattened MoE group under DP without EP.
column_sharded_down_supportedno longer requiresdp_size == 1. The group is chosen by comparing the expert-parallel world size with the attention TP world size, not by reading the vLLM config, because the column path runs in forward and during warm-up. An earlier revision read the config there, and warm-up asserted. That is the only code difference between the two DP image tags above.PCP>1, and sequence-parallel MoE without EP, still fail closed. vllm-project#63085's token-sharded residual still requires DP = 1, so these runs do not exercise it.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)