Skip to content

[ROCm][Muse Spark] Support EPLB with FP8 and BF16 experts - #63729

Open
mjkvaak-amd wants to merge 4 commits into
vllm-project:mainfrom
mjkvaak-amd:rocm/muse-spark-eplb
Open

mjkvaak-amd wants to merge 4 commits into
vllm-project:mainfrom
mjkvaak-amd:rocm/muse-spark-eplb

Conversation

@mjkvaak-amd

@mjkvaak-amd mjkvaak-amd commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Lets Muse Spark run with --enable-eplb on ROCm with Triton FP8 and BF16 experts, where it currently fails at init (Muse Spark FP8 on ROCm does not support EPLB). This is enablement work, not a speedup: on 8× MI300X, EPLB gives +2.2–6.5% throughput at 8k–16k input and concurrency 32–64, within ±2% at lower concurrency, and 3.5–5.2% slower at 1k input. Median TTFT is 11–27% lower at every 8k and 16k point. MXFP4 still rejects EPLB.

Changes:

  • The MoE block passes enable_eplb and num_redundant_experts to the experts and gives the Muse router an EplbLayerState (FusedMoEFactory only attaches one to routers it builds), so the router maps logical to physical experts and records load.
  • The Quark FP8 Triton MoE method declares supports_eplb: the expert dim is outermost in every weight and scale, and the post-expert norm is shared by all experts.
  • The consolidated expert loader fills every initial physical replica, not only the first num_logical_experts.

On ROCm, use synchronous EPLB: "communicator":"pynccl","use_async":false. With the default async NIXL communicator, rearrangements did not complete before the next one started, and serving was 9–68% slower than without EPLB, so the model logs a warning when EPLB is async or uses NIXL. Synchronous rearrangement steps on main do up to five GPU→CPU syncs in the ROCm skip-rearrange heuristic, and fail under VLLM_GPU_SYNC_CHECK=error; #66399 removes those syncs. The numbers below were measured without #66399 (21 rearrangements of about 0.02 s each over the sweep).

Validation

amd/Muse-Spark-Open-816B-A42B-Quark-PTPC-FP8, TP8 + EP, 8× MI300X, vllm/vllm-openai-rocm:nightly-83c078d955 with this branch's files mounted. Both arms ran on the same node; the EPLB arm adds the last two flags.

VLLM_ROCM_USE_AITER=1 VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE=bf16 VLLM_USE_BREAKABLE_CUDAGRAPH=1 \
vllm serve amd/Muse-Spark-Open-816B-A42B-Quark-PTPC-FP8 \
  --tensor-parallel-size 8 --enable-expert-parallel --moe-backend triton \
  --dtype bfloat16 --distributed-executor-backend mp --language-model-only \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' --no-enable-prefix-caching \
  --enable-chunked-prefill --attention-backend ROCM_AITER_FA --max-model-len 32768 \
  --max-num-seqs 128 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.90 \
  --reasoning-parser muse \
  --enable-eplb --eplb-config '{"num_redundant_experts":32,"communicator":"pynccl","use_async":false}'

vllm bench serve --backend vllm --model amd/Muse-Spark-Open-816B-A42B-Quark-PTPC-FP8 \
  --dataset-name random --random-input-len $ISL --random-output-len 1024 --random-range-ratio 0 \
  --ignore-eos --max-concurrency $CONC --num-prompts $((CONC * 3)) --seed 6   # 4 prompts at c1
ISL / OSL Conc No EPLB tok/s EPLB tok/s Δ Median TTFT Median TPOT
1k / 1k 1 146 141 -3.5% 122 → 120 ms 13.5 → 13.8 ms
1k / 1k 16 1,180 1,132 -4.1% 767 → 682 ms 26.5 → 27.4 ms
1k / 1k 64 2,455 2,328 -5.2% 1.82 → 1.56 s 50.2 → 53.5 ms
8k / 1k 1 636 625 -1.8% 632 → 544 ms 13.6 → 13.9 ms
8k / 1k 4 1,843 1,836 -0.4% 1.85 → 1.62 s 17.8 → 18.1 ms
8k / 1k 8 2,859 2,835 -0.8% 2.48 → 2.17 s 22.7 → 23.3 ms
8k / 1k 16 4,055 4,070 +0.4% 2.50 → 2.17 s 32.5 → 32.9 ms
8k / 1k 32 5,461 5,581 +2.2% 2.51 → 2.17 s 50.1 → 48.9 ms
8k / 1k 64 6,700 6,909 +3.1% 2.52 → 2.18 s 82.5 → 80.6 ms
16k / 1k 1 1,148 1,137 -0.9% 1.28 → 1.11 s 13.6 → 13.9 ms
16k / 1k 4 3,080 3,104 +0.8% 3.15 → 2.75 s 18.9 → 19.2 ms
16k / 1k 8 4,442 4,507 +1.5% 3.81 → 2.80 s 26.9 → 27.3 ms
16k / 1k 16 6,040 6,089 +0.8% 3.78 → 3.32 s 40.7 → 41.5 ms
16k / 1k 32 7,547 7,852 +4.0% 3.80 → 2.77 s 68.0 → 65.9 ms
16k / 1k 64 8,768 9,335 +6.5% 3.19 → 2.76 s 118.9 → 111.5 ms

With 32 redundant experts each rank holds 36 experts, which has no tuned MI300X fused-MoE config, so part of the 1k-input loss is default tiles.

GSM8K 5-shot, 1,319 questions:

lm_eval --model local-completions --tasks gsm8k --num_fewshot 5 --seed 4321 \
  --model_args model=amd/Muse-Spark-Open-816B-A42B-Quark-PTPC-FP8,base_url=http://localhost:8000/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False
flexible-extract strict-match
No EPLB 0.9507 ± 0.0060 0.9492 ± 0.0060
EPLB 0.9545 ± 0.0057 0.9538 ± 0.0058

Tests. New test_consolidated_loader_fills_every_initial_redundant_replica (4 logical, 6 physical experts) fails on main and passes with this PR. test_muse_spark_rocm_loading.py, test_muse_spark_logits_local.py and test_muse_spark_rocm_bf16_moe.py on MI325X after the rebase onto 821851eab0: 46 passed, 3 skipped (main: 40 passed, 3 skipped; the difference is the new loader test and the 5 warning cases).

gfx950. Smoke test only (--load-format dummy, TP4 + EP on 4× MI355X, 16 redundant experts, 64-step interval): served all requests across 244 rearrangements.

Not tested together with the token-sharded MoE of #63085.


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
  • AI assistance was used during the creation of this PR.
  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

Co-authored-by: Cursor cursoragent@cursor.com

@mergify

mergify Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mjkvaak-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 8, 2026
mjkvaak-amd and others added 2 commits November 9, 2026 07:43
The ROCm Muse Spark runners rejected --enable-eplb. Its post-expert norm is
shared by all experts and the Triton experts keep the expert dim outermost,
so EPLB's weight moves are layout-safe. Wire the redundant expert count into
the MoE block, declare EPLB support for the Quark FP8 Triton method, and load
consolidated expert tensors into every initial physical replica. MXFP4 keeps
rejecting EPLB.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ant replicas

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
@mergify mergify Bot removed the needs-rebase label Nov 9, 2026

@akii96 akii96 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this on MI300X with the latest nightly. The Muse Spark tests pass, and the new test fails without this PR and passes with it. I also checked that moving experts works for every FP8 checkpoint type, not only PTPC.

Just wondering a bit if the bit about using "communicator":"pynccl","use_async":false on ROCm should also go in some kind of warning in some kind of logs or recipe as well so folks running this will find that ? Just something to ponder and non blocking!

Async EPLB and the NIXL communicator stalled Muse Spark on MI300X, and
leaving the communicator unset selects NIXL. Log the synchronous pynccl
config to use instead, and reword the supports_eplb comment.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
@mjkvaak-amd

Copy link
Copy Markdown
Contributor Author

I tested this on MI300X with the latest nightly. The Muse Spark tests pass, and the new test fails without this PR and passes with it. I also checked that moving experts works for every FP8 checkpoint type, not only PTPC.

Just wondering a bit if the bit about using "communicator":"pynccl","use_async":false on ROCm should also go in some kind of warning in some kind of logs or recipe as well so folks running this will find that ? Just something to ponder and non blocking!

Thanks @akii96 🙏 . I switched to your wording in the supports_eplb comment. Also, the model surfaces the issue by logging a warning with the pynccl / use_async: false config when EPLB runs async or over NIXL (also covered in test_eplb_warns_on_async_or_nixl).

FWIW, making ROCm auto-select synchronous pynccl would also be possible, but changes EPLB for every MoE model on ROCm. Therefore it's out of scope here.

@mjkvaak-amd
mjkvaak-amd marked this pull request as ready for review November 10, 2026 06:59

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify

mergify Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mjkvaak-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 17, 2026

@simondanielsson simondanielsson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work!

Suggestion: Can we run the accuracy validation (preferably gpqa) on both cdna3+4 while also forcing expert rearrangements? There's a risk that the existing gsm8k score never triggered a rearrangement and hence didn't test the correctness. I think you can set something like step_interval in the eplb config to a very low value


def warn_on_async_eplb(eplb_config: EPLBConfig) -> None:
if eplb_config.use_async or eplb_config.communicator == "nixl":
logger.warning_once(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought: should we include an explanation of the consequence of this stall, i.e. performance might be significantly worse? If the stall would have been persisted indefinitely I'd recommend raising an error here or even overriding the config value



@pytest.mark.parametrize(
"use_async,communicator,warns",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be fine without this test

@@ -9,6 +9,7 @@
import torch

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: I'm not an eplb expert but can we check if we can add a more e2e-style test? Let's do some research on how other models do this

expert_id: int,
) -> None:
"""Load one logical expert into each of its initial physical replicas."""
for physical_id in range(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: is it worth re-using EplbState.build_initial_global_physical_to_logical_map here?

router_output_multiplier=self.router_output_multiplier,
router_score_normalization="sigmoid",
router_score_division_eps=1e-15,
# FusedMoEFactory only attaches EPLB state to routers it creates.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: I suppose we could fix this if we want:

# in FusedMoEFactory 
if router is None:
    router = create_fused_moe_router(..., eplb_state=eplb_state, ...)
elif eplb_state is not None and router.eplb_state is None:
    router.eplb_state = eplb_state

wdyt?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-rebase rocm Related to AMD ROCm

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

3 participants