Repository navigation
[ROCm][Muse Spark] Support data parallelism in the MoE finalize - #28
Closed
amd-mghanimi wants to merge 1 commit into
Closed
amd-mghanimi wants to merge 1 commit into
amd-mghanimi wants to merge 1 commit into
Conversation
With EP, every route is computed whole on its owner rank, so the post-expert norm is local and the DP/SP combine plus TP all-reduce that follow are linear: lifting the DP1 check, and the sequence-parallel check under EP (vLLM turns SP MoE on for DP x TP x EP), is enough. Without EP, MoE TP spans DP x TP ranks over DP-gathered tokens. Reduce-scatter the routes over DP first, so each rank holds its own tokens' TP partial sums exactly as without DP, then finalize as before; zero the other DP ranks' rows so the DP combine that follows passes this rank's tokens through. Co-authored-by: Cursor <cursoragent@cursor.com>
1 of 6 tasks
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Muse Spark on ROCm rejects
--data-parallel-size > 1at startup. The post-expert RMSNorm needs each route's complete expert output, and DP changes the expert sharding. This PR accepts DP, and keeps that norm correct for DP without expert parallelism and for DP with it.Claims
Validation
Serving comparison on Muse Spark Open 816B MXFP4, random prompts with range ratio 0.8. Each cell is the mean of that point's repeats. Output throughput is tok/s. Median TTFT and mean TPOT are milliseconds.
Arm A is one TP4 or TEP4 replica. Arm B is DP2×TP4 or DP2×TEP4, two replicas.
Arm A image, both layouts: nightly
vllm/vllm-openai-rocm:nightly-be47189f37f6162991366ff6b91539f6e9d1f432, AITER9f30ad8, plus vllm-project#63254, vllm-project#64461, and vllm-project#63085. Taglocal/vllm-rocm:nightly-be47189-aiter9f30ad8-mmoe-base-730744.Arm B, DP2×TP4: the same image plus this DP change, with column gathers routed on the flattened MoE group. Tag
local/vllm-rocm:nightly-be47189-aiter9f30ad8-dp-63cb98.Arm B, DP2×TEP4: the same base plus the previous revision of that patch. Tag
local/vllm-rocm:nightly-be47189-aiter9f30ad8-dp-400fca. The only code difference iscolumn_shard_group. The previous revision calledget_current_vllm_config()in the forward path, and warm-up asserts there because the config context is unset. The measured revision compares the expert-parallel world size with the attention TP world size. Expert parallelism does not build the column-sharded down projection, so DP2×TEP4 never calls that function. Its log has no column-sharded line.DP2×TP4 against TP4
DP2×TEP4 against TEP4
Accuracy
This build was not run through GSM8K or lm-eval. On nightly
96710245, for the finalize without the column-group routing, GSM8K 5-shot exact match was TP4 0.9462, TEP4 0.950, DP2×TP4 0.9507, DP2×TEP4 0.9409. A 48-token greedy smoke matched TP4 on all four prompts for DP2×TP4, and on three of four for DP2×TEP4.CPU:
tests/kernels/moe/test_muse_spark_rocm_bf16_moe.pycovers the DP finalize against the single-rank result. The column-shard gate test istests/kernels/moe/test_muse_spark_rocm_staged_moe.pyon the stack branch below.Details
With DP and no expert parallelism, MoE TP is flattened across DP×TP. The pre-norm all-reduce only covers the attention TP group, so route outputs stay partial over DP. This PR reduce-scatters the gathered route partials over the DP group before the existing TP finalize, and zeroes the other ranks' rows so the DP combine returns this rank's tokens unchanged.
With expert parallelism, each route is complete on its owner rank, so the norm is local. Sequence-parallel MoE is accepted in that case. vLLM turns it on for DP×TP with expert parallelism.
The fused norm-expand handoff stays off when
dp_size > 1or sequence-parallel MoE is on. It returns a strided first-route view, and the DP all-gather reads the wrong rows from that view.vllm-project#64461's column gathers have the same shape of bug: they ran on the attention TP group, while the expert shards span the flattened group.
column_sharded_down_supportedno longer requiresdp_size == 1. The gathers, the sum-of-squares all-reduce, and the column all-gather use the expert-parallel group when it is larger than the attention TP group, which is the flattened rank order. The group is chosen by world size, not by reading the vLLM config, because this runs in forward and during warm-up. That routing is6992c02022onmuse-rocm-dp-after-stack(this change on top of vllm-project#63254, vllm-project#64461, and vllm-project#63085). This PR's base does not contain the column-sharded path, so that commit is not in this diff.PCP>1, and sequence-parallel MoE without expert parallelism, still fail closed. vllm-project#63085's token-sharded residual still requires DP 1, so these runs do not exercise it.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)