Skip to content

[ROCm][Muse Spark] Support data parallelism in the MoE finalize - #30

Open
amd-mghanimi wants to merge 2 commits into
muse-rocm-future-prsfrom
muse-rocm-dp-after-stack
Open

amd-mghanimi wants to merge 2 commits into
muse-rocm-future-prsfrom
muse-rocm-dp-after-stack

Conversation

@amd-mghanimi

Copy link
Copy Markdown
Owner

Overview

Muse Spark on ROCm rejects --data-parallel-size > 1 at startup. The post-expert RMSNorm needs each route's complete expert output, and DP changes the expert sharding. This PR accepts DP and keeps that norm correct, for DP without expert parallelism (DP2×TP4) and with it (DP2×TEP4).

This PR is stacked on muse-rocm-future-prs (vllm-project#63322, vllm-project#63254, vllm-project#64461, vllm-project#63085, vllm-project#63729). It supersedes #28, which targeted main from muse-rocm-dp-moe.

Claims

Validation

Arms

The measured arms are not this branch. This branch is on nightly 96710245 and also includes vllm-project#63322 and vllm-project#63729, and its DP code is the commit series on muse-rocm-dp-after-stack. The numbers below are Arm A vs Arm B, and they apply to this branch only after it is rebuilt and rerun.

The DP2×TEP4 run used …-dp-400fca. The DP2×TP4 run that finished used …-dp-63cb98, which differs only in column_shard_group (see Details).

Serving comparison

Muse Spark Open 816B MXFP4, random prompts with range ratio 0.8, 90 sweep points per arm. Each row is the mean of that workload's repeats. Arm A is one TP4 or TEP4 replica; Arm B is two replicas (DP2×TP4 or DP2×TEP4) on 8 GPUs. Output throughput is tok/s; median TTFT and mean TPOT are milliseconds.

DP2×TP4 (Arm B) against TP4 (Arm A)

ISL/OSL/MC n A tok/s B tok/s A median TTFT B median TTFT A TPOT B TPOT
8k/1k C1 5 261 222 330 498 3.48 3.97
8k/1k C4 5 623 597 365 505 5.79 5.88
8k/1k C16 5 1124 1080 561 670 12.72 13.24
8k/1k C32 5 1411 1352 929 986 20.05 21.16
8k/1k C64 5 1711 1619 1430 1493 32.97 35.44
8k/1k C128 5 1997 2017 4004 3099 52.70 54.27
8k/64 C1 9 103 78 340 481 3.61 4.13
8k/64 C16 9 176 185 1713 2041 55.00 48.85
8k/64 C64 9 184 219 14924 8365 80.89 124.44
8k/64 C128 9 185 224 34513 24471 83.43 132.53
8k/16 C16 5 45 52 3709 2205 87.88 138.38
8k/16 C64 5 46 56 18175 13865 91.37 149.40
60k/16 C4 9 6 7 8025 5799 83.60 120.29
60k/600 C4 5 175 158 3019 4351 14.99 15.75

DP2×TEP4 (Arm B) against TEP4 (Arm A)

ISL/OSL/MC n A tok/s B tok/s A median TTFT B median TTFT A TPOT B TPOT
8k/1k C1 5 256 226 299 331 3.59 4.07
8k/1k C4 5 632 676 326 338 5.77 5.36
8k/1k C16 5 1221 1510 520 528 11.73 9.45
8k/1k C32 5 1572 2047 878 662 17.98 13.91
8k/1k C64 5 1930 2614 1240 979 29.15 21.84
8k/1k C128 5 2184 3188 3760 2086 48.20 34.17
8k/64 C1 9 110 99 297 334 3.74 4.22
8k/64 C16 9 194 279 1507 1335 50.19 32.52
8k/64 C64 9 202 340 13553 5397 73.43 80.41
8k/64 C128 9 204 347 31370 15602 75.70 85.59
8k/16 C16 5 50 78 3356 1441 79.69 90.76
8k/16 C64 5 51 87 16461 8901 82.53 95.92
60k/16 C4 9 6 10 7394 3846 77.26 82.60
60k/600 C4 5 186 216 2671 3023 14.34 11.75

Accuracy

Required for this PR. Not measured on this branch yet. Earlier GSM8K 5-shot results were from an older revision of this DP finalize on nightly 96710245, and they don't cover the column-group change: TP4 0.9462, TEP4 0.950, DP2×TP4 0.9507, DP2×TEP4 0.9409. These will be rerun on this branch's image before merge.

Tests

  • CPU: tests/kernels/moe/test_muse_spark_rocm_bf16_moe.py covers the DP finalize against the single-rank result, and tests/kernels/moe/test_muse_spark_rocm_staged_moe.py covers the DP case of the column-shard gate. Neither was run on GPU, and torch is not installed in the local environment, so they have not been run at all on the current commits.

Details

With DP and no expert parallelism, MoE TP is flattened across DP×TP. The pre-norm all-reduce only covers the attention TP group, so route outputs stay partial over DP. This PR reduce-scatters the gathered route partials over the DP group before the existing TP finalize, and zeroes the other ranks' rows so the DP combine returns this rank's tokens unchanged.

With expert parallelism, each route is complete on its owner rank, so the norm is local. Sequence-parallel MoE is accepted in that case.

The fused norm-expand handoff stays off when dp_size > 1 or sequence-parallel MoE is on. It returns a strided first-route view, and the DP all-gather reads the wrong rows from that view.

vllm-project#64461's column gathers run on the flattened MoE group under DP without EP. column_sharded_down_supported no longer requires dp_size == 1. The group is chosen by comparing the expert-parallel world size with the attention TP world size, not by reading the vLLM config, because the column path runs in forward and during warm-up. An earlier revision read the config there, and warm-up asserted. That is the only code difference between the two DP image tags above.

PCP>1, and sequence-parallel MoE without EP, still fail closed. vllm-project#63085's token-sharded residual still requires DP = 1, so these runs do not exercise it.


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

amd-mghanimi and others added 2 commits November 11, 2026 15:20
With EP, every route is computed whole on its owner rank, so the post-expert
norm is local and the DP/SP combine plus TP all-reduce that follow are linear:
lifting the DP1 check, and the sequence-parallel check under EP (vLLM turns
SP MoE on for DP x TP x EP), is enough.

Without EP, MoE TP spans DP x TP ranks over DP-gathered tokens. Reduce-scatter
the routes over DP first, so each rank holds its own tokens' TP partial sums
exactly as without DP, then finalize as before; zero the other DP ranks' rows
so the DP combine that follows passes this rank's tokens through.

Co-authored-by: Cursor <cursoragent@cursor.com>
… group

DP without expert parallelism flattens MoE TP across DP×TP, so the
column-sharded down projection has to gather on that group rather than the
attention TP group. Choose the group by comparing EP and TP world sizes.
Reading the vLLM config in the forward path asserts during warm-up.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant