Skip to content

[MORI] Run MORI EPv2 on Quark MXFP4 MoE models; stop passing the received-row count - #66733

Open
ChengYM5685 wants to merge 2 commits into
vllm-project:mainfrom
ChengYM5685:mori-epv2-mxfp4-silu-num-local
Open

ChengYM5685 wants to merge 2 commits into
vllm-project:mainfrom
ChengYM5685:mori-epv2-mxfp4-silu-num-local

Conversation

@ChengYM5685

@ChengYM5685 ChengYM5685 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Overview

MORI EPv2 (--all2all-backend mori, from #61361) refuses every Quark MXFP4 (OCP MX W4A4) MoE model at startup, and once that check is fixed, MXFP4 models (e.g. GLM-5.3-MXFP4) crash in AITER's fused MoE at the first decode warmup step because they are handed the received-row count. This PR fixes both. The second fix is not a per-format exemption: MORI EPv2 no longer passes the received-row count to any experts kernel, since the receive rows it hands the experts are already bounded and the unused ones already route to expert -1.

Claims

  • MORI EPv2 accepts OCP MX activations: they were counted as per-tensor quantization, which EPv2 rejects.
  • MORI EPv2 passes no received-row count (expert_tokens_meta) to the experts and always clears the unused receive rows. No per-format special case remains.
  • GLM-5.3-MXFP4 runs with MORI EPv2 (DP8 + EP8, gfx950): GSM8K 0.98, where it previously faulted at warmup.
  • Same accuracy for BF16 and FP8, with faster BF16 decode: Qwen3.5-35B-A3B BF16 decode at concurrency 64 is about 40% faster; GLM-5.3 FP8 decode is unchanged.
  • No new flags or environment variables.

Validation

End to end, 8x gfx950, one node, --all2all-backend mori, VLLM_MORI_EP_V2=1, TP1 x DP8 + EP8.

Accuracy: GSM8K, full 1319 questions, 5-shot, chat completions, temperature=0, max_tokens=12288.

Performance: vllm bench serve, random dataset, seed 0, --ignore-eos. Three settings: 1024 in / 512 out at concurrency 64 (n=320), 4096 in / 16 out at concurrency 32 (n=160), and 1024 in / 512 out at concurrency 8 (n=48). "Before" is main's behavior for these models (the count is passed); "After" is this PR. Runs are interleaved Before / After / After / Before, so the tables show the mean of two runs:

Model Run GSM8K c64 decode: output tok/s / TPOT 4k prefill c32: total tok/s / TTFT c8 decode: TPOT
Qwen3.5-35B-A3B BF16 Before 0.964 5145 / 11.8 ms 106521 / 718 ms 8.75 ms
After 0.964 7194 / 8.2 ms 109072 / 710 ms 7.50 ms
GLM-5.3 FP8 (MTP 1) Before 0.970 1208 / 48.1 ms 17174 / 4517 ms 26.3 ms
After 0.971 1222 / 47.5 ms 15939 / 4176 ms 26.0 ms
GLM-5.2-MXFP4 (After only; main rejects it at startup) After 0.965 2289 / 25.4 ms 20838 / 3924 ms 21.7 ms

All runs had no Traceback and no memory fault.

GLM-5.3 FP8's 4k-prefill setting trades total throughput (-7%) for TTFT (-7.5%). Its two Before runs alone differ by 9% (17945 vs 16403 tok/s), and an earlier single-run comparison on another node went the other way (+10% total throughput without the count). Further repeats of this setting are in progress.

Other MXFP4 models with the count withheld (one node, GSM8K 200 questions):

Model Activation Result
GLM-5.3-MXFP4 SiLU starts, GSM8K 0.99; with only the first commit, memory fault at decode warmup
DeepSeek-R1-0528-MXFP4 SiLU GSM8K 0.94
MiniMax-M3-MXFP4 SwiGLU-OAI GSM8K 0.935 (allgather_reducescatter: 0.95)
MiniMax-M3.1-Flash-Quark-MXFP4 SwiGLU-OAI first commit only: GSM8K 0.961 / 0.963 (c64 / c128)
Server command (GLM-5.3-MXFP4)
VLLM_MORI_EP_V2=1 VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 \
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=0 \
vllm serve /models/GLM-5.3-MXFP4 --trust-remote-code \
  --tensor-parallel-size 1 --data-parallel-size 8 --enable-expert-parallel \
  --all2all-backend mori --moe-backend aiter --block-size 64 \
  --gpu-memory-utilization 0.85 --max-model-len 262144 \
  --max-num-batched-tokens 16384 --max-num-seqs 32 \
  --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

Unit tests (CPU): pytest tests/kernels/moe/test_mori_epv2_prepare_finalize.py passes. The MXFP4 case of test_experts_it_cannot_feed_are_rejected fails on main, and test_receive_rows_are_bounded_by_dp_tokens now checks that no row count is passed and that the unused rows are cleared. (On a CPU-only host the shared tests/conftest.py fixture raises Cannot access accelerator device around the first test; that test passes when run after another.)

CI does not run MORI EPv2 on gfx950.

Details

OCP MX is not per-tensor. OCP MX quant configs are built without per-token or block flags, so their activation group shape is per tensor and is_per_tensor is true, although each 1x32 block of a row has its own scale. EPv2 rejects per-tensor activations because the experts quantize the received rows and one scale would be taken over padding. Only FP8 / INT8 activations now count as per tensor, the same test FusedMoEQuantConfig.act_scale_is_tensor_wide uses.

The fault. EPv2 slices the receive buffers it hands the experts to the rows this step can receive (the DP batch sum, rounded up to 32) and routes the unused rows to expert -1 with zero weight. It also passed the received-row count, which AITER takes as num_local_tokens. On a few hundred rows (512 and 1024 for GLM-5.3 decode), AITER's MXFP4 W4A4 fused MoE picks small-tile FlyDSL kernels (t32x128x256) that fault in moe_sorting when given the count. On GLM-5.3-MXFP4, DP8 + EP8:

Variant Result
allgather_reducescatter (passes no row count) starts, GSM8K 0.975
EPv2, row count passed HSA_STATUS_ERROR_MEMORY_FAULT in AITER fused MoE at decode warmup
same, AITER_FLYDSL_FORCE=0 same fault
same, unused receive rows zeroed same fault
same, unused rows routed to a local expert instead of -1 same fault
EPv2, row count passed, receive rows not sliced (M = capacity, larger tiles; measured with MXFP4-quantized dispatch) starts, but decode is 2.7x to 4.5x slower
EPv2, row count withheld starts, GSM8K 0.98

Pre-quantizing the activations to MXFP4 before the dispatch does not avoid the fault either: the same kernels see the same rows and count.

Why no count at all. With the rows already bounded and the unused ones routed to expert -1, the count adds little for AITER: the sort drops expert -1 rows. Among Standard-format experts, only AITER reads this scalar count; the others either ignore it or expect per-expert counts, for which a single scalar does not apply. Dropping it avoids a per-format exemption and, on BF16, makes decode faster (table above; why AITER is slower with the count was not investigated). If AITER's kernels are fixed, passing the count can be reconsidered on measured benefit.

Cost. The experts sort every bounded receive row instead of only the received ones; rows with expert id -1 are dropped by the sort. The tables above show no decode cost.

AI assistance (Claude) was used to write this change and this description. The submitter reviewed the changes and ran the validation above.


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

@ChengYM5685 ChengYM5685 changed the title [MORI] Run MORI EPv2 on Quark MXFP4 MoE models [MORI] Run MORI EPv2 on Quark MXFP4 MoE models; stop passing the received-row count Nov 11, 2026
@ChengYM5685
ChengYM5685 marked this pull request as ready for review November 11, 2026 01:50

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@okorzh-amd

Copy link
Copy Markdown
Contributor

PR description is not aligned wit the current code state.

@okorzh-amd

Copy link
Copy Markdown
Contributor

Related PRs:

ChengYM5685 added a commit to ChengYM5685/vllm that referenced this pull request Nov 11, 2026
On main MORI_ENABLE_SDMA=1 always selects EPv1, which is how Quark MXFP4
(OCP MX) MoE models run with --all2all-backend mori. Selecting EPv2 AsyncLL
for it by default rejected those models at startup until vllm-project#66733 and vllm-project#66734
land. Keep EPv1 for MORI_ENABLE_SDMA=1 by default and run EPv2's AsyncLL
kernels only with an explicit VLLM_MORI_EP_V2=1.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
MORI EPv2 rejects experts that quantize activations per tensor, since the
experts quantize the received rows and one scale would be taken over padding.
OCP MX quant configs (e.g. Quark MXFP4) are built without per-token or block
flags, so their activation group shape is per tensor and `is_per_tensor` is
true, although each 1x32 block of a row has its own scale. EPv2 therefore
refused every MXFP4 model at kernel setup.

Count only FP8 / INT8 activations as per tensor, as
FusedMoEQuantConfig.act_scale_is_tensor_wide does, both in that check and in
the one that clears unused receive rows.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
MORI EPv2 slices the receive buffers it hands the experts to the rows this
step can receive (the DP batches, rounded up to 32 rows) and routes the
unused rows to expert -1 with zero weight. It also passed the received-row
count, which AITER's fused MoE takes as num_local_tokens. With the rows
already bounded, the count saves little, and AITER's MXFP4 W4A4 path faults
with it: on a few hundred rows (GLM-5.3-MXFP4 decode warmup) it picks
small-tile FlyDSL kernels that read out of bounds.

Rather than exempting a quant format, pass no count to any experts kernel
and always clear the unused receive rows.

On gfx950, single-node DEP8 (ABBA, gsm8k unchanged): Qwen3.5-35B-A3B BF16
decode at concurrency 64 runs about 40% faster (TPOT 11.8 -> 8.2 ms); GLM-5.3
FP8 decode is unchanged, and its 4k-prefill run trades total throughput
(-7%) for TTFT (-7.5%) within run-to-run noise.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
@ChengYM5685
ChengYM5685 force-pushed the mori-epv2-mxfp4-silu-num-local branch from f980bbe to 14acbc3 Compare November 11, 2026 07:03
ChengYM5685 added a commit to ChengYM5685/vllm that referenced this pull request Nov 11, 2026
In a segmented round the received-row count is world_size * stride, the
row count the experts already get, so it carries no information; and AITER's
MXFP4 W4A4 fused MoE faults on small batches given a count (vllm-project#66733). Hand the
experts no count, as vllm-project#66733 does for the compact receive.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
@ChengYM5685

Copy link
Copy Markdown
Contributor Author

PR description is not aligned wit the current code state.

Thanks, fixed. The description still described an intermediate state: the unused receive rows are now always cleared, and "Before" is main's behavior for the BF16 and FP8 models. I also squashed the intermediate MXFP4-only commit, so the history matches the description.

@ChengYM5685

Copy link
Copy Markdown
Contributor Author

Related PRs:

Thanks. #65128 and #66697 are mine, so whichever of them and this PR lands later will drop the count in _finish_dispatch and in the segmented path. For #65759, whichever lands second takes the quant_schema rename.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants