Repository navigation
[ROCm][Muse Spark] Support EPLB with FP8 and BF16 experts - #63729
mjkvaak-amd wants to merge 4 commits into
Conversation
0192f74 to
6c7aa41
Compare
|
This pull request has merge conflicts that must be resolved before it can be |
The ROCm Muse Spark runners rejected --enable-eplb. Its post-expert norm is shared by all experts and the Triton experts keep the expert dim outermost, so EPLB's weight moves are layout-safe. Wire the redundant expert count into the MoE block, declare EPLB support for the Quark FP8 Triton method, and load consolidated expert tensors into every initial physical replica. MXFP4 keeps rejecting EPLB. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…ant replicas Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
6c7aa41 to
1510f9e
Compare
akii96
left a comment
There was a problem hiding this comment.
I tested this on MI300X with the latest nightly. The Muse Spark tests pass, and the new test fails without this PR and passes with it. I also checked that moving experts works for every FP8 checkpoint type, not only PTPC.
Just wondering a bit if the bit about using "communicator":"pynccl","use_async":false on ROCm should also go in some kind of warning in some kind of logs or recipe as well so folks running this will find that ? Just something to ponder and non blocking!
Async EPLB and the NIXL communicator stalled Muse Spark on MI300X, and leaving the communicator unset selects NIXL. Log the synchronous pynccl config to use instead, and reword the supports_eplb comment. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Thanks @akii96 🙏 . I switched to your wording in the FWIW, making ROCm auto-select synchronous |
|
This pull request has merge conflicts that must be resolved before it can be |
simondanielsson
left a comment
There was a problem hiding this comment.
Thanks for the work!
Suggestion: Can we run the accuracy validation (preferably gpqa) on both cdna3+4 while also forcing expert rearrangements? There's a risk that the existing gsm8k score never triggered a rearrangement and hence didn't test the correctness. I think you can set something like step_interval in the eplb config to a very low value
|
|
||
| def warn_on_async_eplb(eplb_config: EPLBConfig) -> None: | ||
| if eplb_config.use_async or eplb_config.communicator == "nixl": | ||
| logger.warning_once( |
There was a problem hiding this comment.
Thought: should we include an explanation of the consequence of this stall, i.e. performance might be significantly worse? If the stall would have been persisted indefinitely I'd recommend raising an error here or even overriding the config value
|
|
||
|
|
||
| @pytest.mark.parametrize( | ||
| "use_async,communicator,warns", |
There was a problem hiding this comment.
I'd be fine without this test
| @@ -9,6 +9,7 @@ | |||
| import torch | |||
There was a problem hiding this comment.
Suggestion: I'm not an eplb expert but can we check if we can add a more e2e-style test? Let's do some research on how other models do this
| expert_id: int, | ||
| ) -> None: | ||
| """Load one logical expert into each of its initial physical replicas.""" | ||
| for physical_id in range( |
There was a problem hiding this comment.
Nit: is it worth re-using EplbState.build_initial_global_physical_to_logical_map here?
| router_output_multiplier=self.router_output_multiplier, | ||
| router_score_normalization="sigmoid", | ||
| router_score_division_eps=1e-15, | ||
| # FusedMoEFactory only attaches EPLB state to routers it creates. |
There was a problem hiding this comment.
Optional: I suppose we could fix this if we want:
# in FusedMoEFactory
if router is None:
router = create_fused_moe_router(..., eplb_state=eplb_state, ...)
elif eplb_state is not None and router.eplb_state is None:
router.eplb_state = eplb_statewdyt?
Overview
Lets Muse Spark run with
--enable-eplbon ROCm with Triton FP8 and BF16 experts, where it currently fails at init (Muse Spark FP8 on ROCm does not support EPLB). This is enablement work, not a speedup: on 8× MI300X, EPLB gives +2.2–6.5% throughput at 8k–16k input and concurrency 32–64, within ±2% at lower concurrency, and 3.5–5.2% slower at 1k input. Median TTFT is 11–27% lower at every 8k and 16k point. MXFP4 still rejects EPLB.Changes:
enable_eplbandnum_redundant_expertsto the experts and gives the Muse router anEplbLayerState(FusedMoEFactoryonly attaches one to routers it builds), so the router maps logical to physical experts and records load.supports_eplb: the expert dim is outermost in every weight and scale, and the post-expert norm is shared by all experts.num_logical_experts.On ROCm, use synchronous EPLB:
"communicator":"pynccl","use_async":false. With the default async NIXL communicator, rearrangements did not complete before the next one started, and serving was 9–68% slower than without EPLB, so the model logs a warning when EPLB is async or uses NIXL. Synchronous rearrangement steps onmaindo up to five GPU→CPU syncs in the ROCm skip-rearrange heuristic, and fail underVLLM_GPU_SYNC_CHECK=error; #66399 removes those syncs. The numbers below were measured without #66399 (21 rearrangements of about 0.02 s each over the sweep).Validation
amd/Muse-Spark-Open-816B-A42B-Quark-PTPC-FP8, TP8 + EP, 8× MI300X,vllm/vllm-openai-rocm:nightly-83c078d955with this branch's files mounted. Both arms ran on the same node; the EPLB arm adds the last two flags.With 32 redundant experts each rank holds 36 experts, which has no tuned MI300X fused-MoE config, so part of the 1k-input loss is default tiles.
GSM8K 5-shot, 1,319 questions:
Tests. New
test_consolidated_loader_fills_every_initial_redundant_replica(4 logical, 6 physical experts) fails onmainand passes with this PR.test_muse_spark_rocm_loading.py,test_muse_spark_logits_local.pyandtest_muse_spark_rocm_bf16_moe.pyon MI325X after the rebase onto821851eab0: 46 passed, 3 skipped (main: 40 passed, 3 skipped; the difference is the new loader test and the 5 warning cases).gfx950. Smoke test only (
--load-format dummy, TP4 + EP on 4× MI355X, 16 redundant experts, 64-step interval): served all requests across 244 rearrangements.Not tested together with the token-sharded MoE of #63085.
Pull Request Checklist
/pr-checklistskill. (Mandatory for agents, optional for humans).Co-authored-by: Cursor cursoragent@cursor.com