Repository navigation
[MORI] Run MORI EPv2 on Quark MXFP4 MoE models; stop passing the received-row count - #66733
Open
ChengYM5685 wants to merge 2 commits into
Open
ChengYM5685 wants to merge 2 commits into
ChengYM5685 wants to merge 2 commits into
Conversation
ChengYM5685
marked this pull request as ready for review
November 11, 2026 01:50
ChengYM5685
requested review from
bnellnm,
mgoin,
pavanimajety,
wzhao18 and
zyongye
as code owners
November 11, 2026 01:50
ChengYM5685
force-pushed
the
mori-epv2-mxfp4-silu-num-local
branch
from
November 11, 2026 02:24
5a19942 to
f980bbe
Compare
6 tasks done
okorzh-amd
reviewed
Nov 11, 2026
okorzh-amd
reviewed
Nov 11, 2026
Contributor
|
PR description is not aligned wit the current code state. |
Contributor
|
Related PRs:
|
ChengYM5685
added a commit
to ChengYM5685/vllm
that referenced
this pull request
Nov 11, 2026
On main MORI_ENABLE_SDMA=1 always selects EPv1, which is how Quark MXFP4 (OCP MX) MoE models run with --all2all-backend mori. Selecting EPv2 AsyncLL for it by default rejected those models at startup until vllm-project#66733 and vllm-project#66734 land. Keep EPv1 for MORI_ENABLE_SDMA=1 by default and run EPv2's AsyncLL kernels only with an explicit VLLM_MORI_EP_V2=1. Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
MORI EPv2 rejects experts that quantize activations per tensor, since the experts quantize the received rows and one scale would be taken over padding. OCP MX quant configs (e.g. Quark MXFP4) are built without per-token or block flags, so their activation group shape is per tensor and `is_per_tensor` is true, although each 1x32 block of a row has its own scale. EPv2 therefore refused every MXFP4 model at kernel setup. Count only FP8 / INT8 activations as per tensor, as FusedMoEQuantConfig.act_scale_is_tensor_wide does, both in that check and in the one that clears unused receive rows. Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
MORI EPv2 slices the receive buffers it hands the experts to the rows this step can receive (the DP batches, rounded up to 32 rows) and routes the unused rows to expert -1 with zero weight. It also passed the received-row count, which AITER's fused MoE takes as num_local_tokens. With the rows already bounded, the count saves little, and AITER's MXFP4 W4A4 path faults with it: on a few hundred rows (GLM-5.3-MXFP4 decode warmup) it picks small-tile FlyDSL kernels that read out of bounds. Rather than exempting a quant format, pass no count to any experts kernel and always clear the unused receive rows. On gfx950, single-node DEP8 (ABBA, gsm8k unchanged): Qwen3.5-35B-A3B BF16 decode at concurrency 64 runs about 40% faster (TPOT 11.8 -> 8.2 ms); GLM-5.3 FP8 decode is unchanged, and its 4k-prefill run trades total throughput (-7%) for TTFT (-7.5%) within run-to-run noise. Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
ChengYM5685
force-pushed
the
mori-epv2-mxfp4-silu-num-local
branch
from
November 11, 2026 07:03
f980bbe to
14acbc3
Compare
ChengYM5685
added a commit
to ChengYM5685/vllm
that referenced
this pull request
Nov 11, 2026
In a segmented round the received-row count is world_size * stride, the row count the experts already get, so it carries no information; and AITER's MXFP4 W4A4 fused MoE faults on small batches given a count (vllm-project#66733). Hand the experts no count, as vllm-project#66733 does for the compact receive. Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
6 tasks done
Contributor
Author
Thanks, fixed. The description still described an intermediate state: the unused receive rows are now always cleared, and "Before" is main's behavior for the BF16 and FP8 models. I also squashed the intermediate MXFP4-only commit, so the history matches the description. |
Contributor
Author
Thanks. #65128 and #66697 are mine, so whichever of them and this PR lands later will drop the count in _finish_dispatch and in the segmented path. For #65759, whichever lands second takes the quant_schema rename. |
6 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
MORI EPv2 (
--all2all-backend mori, from #61361) refuses every Quark MXFP4 (OCP MX W4A4) MoE model at startup, and once that check is fixed, MXFP4 models (e.g. GLM-5.3-MXFP4) crash in AITER's fused MoE at the first decode warmup step because they are handed the received-row count. This PR fixes both. The second fix is not a per-format exemption: MORI EPv2 no longer passes the received-row count to any experts kernel, since the receive rows it hands the experts are already bounded and the unused ones already route to expert -1.Claims
expert_tokens_meta) to the experts and always clears the unused receive rows. No per-format special case remains.Validation
End to end, 8x gfx950, one node,
--all2all-backend mori,VLLM_MORI_EP_V2=1, TP1 x DP8 + EP8.Accuracy: GSM8K, full 1319 questions, 5-shot, chat completions,
temperature=0,max_tokens=12288.Performance:
vllm bench serve, random dataset, seed 0,--ignore-eos. Three settings: 1024 in / 512 out at concurrency 64 (n=320), 4096 in / 16 out at concurrency 32 (n=160), and 1024 in / 512 out at concurrency 8 (n=48). "Before" is main's behavior for these models (the count is passed); "After" is this PR. Runs are interleaved Before / After / After / Before, so the tables show the mean of two runs:All runs had no Traceback and no memory fault.
GLM-5.3 FP8's 4k-prefill setting trades total throughput (-7%) for TTFT (-7.5%). Its two Before runs alone differ by 9% (17945 vs 16403 tok/s), and an earlier single-run comparison on another node went the other way (+10% total throughput without the count). Further repeats of this setting are in progress.
Other MXFP4 models with the count withheld (one node, GSM8K 200 questions):
Server command (GLM-5.3-MXFP4)
VLLM_MORI_EP_V2=1 VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 \ VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=0 \ vllm serve /models/GLM-5.3-MXFP4 --trust-remote-code \ --tensor-parallel-size 1 --data-parallel-size 8 --enable-expert-parallel \ --all2all-backend mori --moe-backend aiter --block-size 64 \ --gpu-memory-utilization 0.85 --max-model-len 262144 \ --max-num-batched-tokens 16384 --max-num-seqs 32 \ --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \ --speculative-config '{"method":"mtp","num_speculative_tokens":1}'Unit tests (CPU):
pytest tests/kernels/moe/test_mori_epv2_prepare_finalize.pypasses. The MXFP4 case oftest_experts_it_cannot_feed_are_rejectedfails on main, andtest_receive_rows_are_bounded_by_dp_tokensnow checks that no row count is passed and that the unused rows are cleared. (On a CPU-only host the sharedtests/conftest.pyfixture raisesCannot access accelerator devicearound the first test; that test passes when run after another.)CI does not run MORI EPv2 on gfx950.
Details
OCP MX is not per-tensor. OCP MX quant configs are built without per-token or block flags, so their activation group shape is per tensor and
is_per_tensoris true, although each 1x32 block of a row has its own scale. EPv2 rejects per-tensor activations because the experts quantize the received rows and one scale would be taken over padding. Only FP8 / INT8 activations now count as per tensor, the same testFusedMoEQuantConfig.act_scale_is_tensor_wideuses.The fault. EPv2 slices the receive buffers it hands the experts to the rows this step can receive (the DP batch sum, rounded up to 32) and routes the unused rows to expert -1 with zero weight. It also passed the received-row count, which AITER takes as
num_local_tokens. On a few hundred rows (512 and 1024 for GLM-5.3 decode), AITER's MXFP4 W4A4 fused MoE picks small-tile FlyDSL kernels (t32x128x256) that fault inmoe_sortingwhen given the count. On GLM-5.3-MXFP4, DP8 + EP8:allgather_reducescatter(passes no row count)HSA_STATUS_ERROR_MEMORY_FAULTin AITER fused MoE at decode warmupAITER_FLYDSL_FORCE=0Pre-quantizing the activations to MXFP4 before the dispatch does not avoid the fault either: the same kernels see the same rows and count.
Why no count at all. With the rows already bounded and the unused ones routed to expert -1, the count adds little for AITER: the sort drops expert -1 rows. Among Standard-format experts, only AITER reads this scalar count; the others either ignore it or expect per-expert counts, for which a single scalar does not apply. Dropping it avoids a per-format exemption and, on BF16, makes decode faster (table above; why AITER is slower with the count was not investigated). If AITER's kernels are fixed, passing the count can be reconsidered on measured benefit.
Cost. The experts sort every bounded receive row instead of only the received ones; rows with expert id -1 are dropped by the sort. The tables above show no decode cost.
AI assistance (Claude) was used to write this change and this description. The submitter reviewed the changes and ran the validation above.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.