Skip to content

[ROCm][Muse Spark] Support data parallelism in the MoE finalize - #28

Closed
amd-mghanimi wants to merge 1 commit into
mainfrom
muse-rocm-dp-moe
Closed

amd-mghanimi wants to merge 1 commit into
mainfrom
muse-rocm-dp-moe

Conversation

@amd-mghanimi

@amd-mghanimi amd-mghanimi commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Overview

Muse Spark on ROCm rejects --data-parallel-size > 1 at startup. The post-expert RMSNorm needs each route's complete expert output, and DP changes the expert sharding. This PR accepts DP, and keeps that norm correct for DP without expert parallelism and for DP with it.

Claims

Validation

Serving comparison on Muse Spark Open 816B MXFP4, random prompts with range ratio 0.8. Each cell is the mean of that point's repeats. Output throughput is tok/s. Median TTFT and mean TPOT are milliseconds.

Arm A is one TP4 or TEP4 replica. Arm B is DP2×TP4 or DP2×TEP4, two replicas.

Arm A image, both layouts: nightly vllm/vllm-openai-rocm:nightly-be47189f37f6162991366ff6b91539f6e9d1f432, AITER 9f30ad8, plus vllm-project#63254, vllm-project#64461, and vllm-project#63085. Tag local/vllm-rocm:nightly-be47189-aiter9f30ad8-mmoe-base-730744.

Arm B, DP2×TP4: the same image plus this DP change, with column gathers routed on the flattened MoE group. Tag local/vllm-rocm:nightly-be47189-aiter9f30ad8-dp-63cb98.

Arm B, DP2×TEP4: the same base plus the previous revision of that patch. Tag local/vllm-rocm:nightly-be47189-aiter9f30ad8-dp-400fca. The only code difference is column_shard_group. The previous revision called get_current_vllm_config() in the forward path, and warm-up asserts there because the config context is unset. The measured revision compares the expert-parallel world size with the attention TP world size. Expert parallelism does not build the column-sharded down projection, so DP2×TEP4 never calls that function. Its log has no column-sharded line.

DP2×TP4 against TP4

ISL/OSL/MC n A tok/s B tok/s A median TTFT B median TTFT A TPOT B TPOT
8k/1k C1 5 261 222 330 498 3.48 3.97
8k/1k C4 5 623 597 365 505 5.79 5.88
8k/1k C16 5 1124 1080 561 670 12.72 13.24
8k/1k C32 5 1411 1352 929 986 20.05 21.16
8k/1k C64 5 1711 1619 1430 1493 32.97 35.44
8k/1k C128 5 1997 2017 4004 3099 52.70 54.27
8k/64 C1 9 103 78 340 481 3.61 4.13
8k/64 C16 9 176 185 1713 2041 55.00 48.85
8k/64 C64 9 184 219 14924 8365 80.89 124.44
8k/64 C128 9 185 224 34513 24471 83.43 132.53
8k/16 C16 5 45 52 3709 2205 87.88 138.38
8k/16 C64 5 46 56 18175 13865 91.37 149.40
60k/16 C4 9 6 7 8025 5799 83.60 120.29
60k/600 C4 5 175 158 3019 4351 14.99 15.75

DP2×TEP4 against TEP4

ISL/OSL/MC n A tok/s B tok/s A median TTFT B median TTFT A TPOT B TPOT
8k/1k C1 5 256 226 299 331 3.59 4.07
8k/1k C4 5 632 676 326 338 5.77 5.36
8k/1k C16 5 1221 1510 520 528 11.73 9.45
8k/1k C32 5 1572 2047 878 662 17.98 13.91
8k/1k C64 5 1930 2614 1240 979 29.15 21.84
8k/1k C128 5 2184 3188 3760 2086 48.20 34.17
8k/64 C1 9 110 99 297 334 3.74 4.22
8k/64 C16 9 194 279 1507 1335 50.19 32.52
8k/64 C64 9 202 340 13553 5397 73.43 80.41
8k/64 C128 9 204 347 31370 15602 75.70 85.59
8k/16 C16 5 50 78 3356 1441 79.69 90.76
8k/16 C64 5 51 87 16461 8901 82.53 95.92
60k/16 C4 9 6 10 7394 3846 77.26 82.60
60k/600 C4 5 186 216 2671 3023 14.34 11.75

Accuracy

This build was not run through GSM8K or lm-eval. On nightly 96710245, for the finalize without the column-group routing, GSM8K 5-shot exact match was TP4 0.9462, TEP4 0.950, DP2×TP4 0.9507, DP2×TEP4 0.9409. A 48-token greedy smoke matched TP4 on all four prompts for DP2×TP4, and on three of four for DP2×TEP4.

CPU: tests/kernels/moe/test_muse_spark_rocm_bf16_moe.py covers the DP finalize against the single-rank result. The column-shard gate test is tests/kernels/moe/test_muse_spark_rocm_staged_moe.py on the stack branch below.

Details

With DP and no expert parallelism, MoE TP is flattened across DP×TP. The pre-norm all-reduce only covers the attention TP group, so route outputs stay partial over DP. This PR reduce-scatters the gathered route partials over the DP group before the existing TP finalize, and zeroes the other ranks' rows so the DP combine returns this rank's tokens unchanged.

With expert parallelism, each route is complete on its owner rank, so the norm is local. Sequence-parallel MoE is accepted in that case. vLLM turns it on for DP×TP with expert parallelism.

The fused norm-expand handoff stays off when dp_size > 1 or sequence-parallel MoE is on. It returns a strided first-route view, and the DP all-gather reads the wrong rows from that view.

vllm-project#64461's column gathers have the same shape of bug: they ran on the attention TP group, while the expert shards span the flattened group. column_sharded_down_supported no longer requires dp_size == 1. The gathers, the sum-of-squares all-reduce, and the column all-gather use the expert-parallel group when it is larger than the attention TP group, which is the flattened rank order. The group is chosen by world size, not by reading the vLLM config, because this runs in forward and during warm-up. That routing is 6992c02022 on muse-rocm-dp-after-stack (this change on top of vllm-project#63254, vllm-project#64461, and vllm-project#63085). This PR's base does not contain the column-sharded path, so that commit is not in this diff.

PCP>1, and sequence-parallel MoE without expert parallelism, still fail closed. vllm-project#63085's token-sharded residual still requires DP 1, so these runs do not exercise it.


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

With EP, every route is computed whole on its owner rank, so the post-expert
norm is local and the DP/SP combine plus TP all-reduce that follow are linear:
lifting the DP1 check, and the sequence-parallel check under EP (vLLM turns
SP MoE on for DP x TP x EP), is enough.

Without EP, MoE TP spans DP x TP ranks over DP-gathered tokens. Reduce-scatter
the routes over DP first, so each rank holds its own tokens' TP partial sums
exactly as without DP, then finalize as before; zero the other DP ranks' rows
so the DP combine that follows passes this rank's tokens through.

Co-authored-by: Cursor <cursoragent@cursor.com>
@amd-mghanimi

Copy link
Copy Markdown
Owner Author

Superseded by #30, which targets muse-rocm-future-prs from muse-rocm-dp-after-stack. This PR's head branch (muse-rocm-dp-moe) is on main, so its diff does not match the stacked design. Numbers and accuracy in #30 are for the stated arms and must be rerun on the new branch's image before merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant