Repository navigation
[ROCm][MORI] Dispatch the activation format AITER's fused MoE consumes in MORI EPv2 - #66964
Draft
ChengYM5685 wants to merge 3 commits into
Draft
ChengYM5685 wants to merge 3 commits into
ChengYM5685 wants to merge 3 commits into
Conversation
MORI EPv2 rejects experts that quantize activations per tensor, since the experts quantize the received rows and one scale would be taken over padding. OCP MX quant configs (e.g. Quark MXFP4) are built without per-token or block flags, so their activation group shape is per tensor and `is_per_tensor` is true, although each 1x32 block of a row has its own scale. EPv2 therefore refused every MXFP4 model at kernel setup. Count only FP8 / INT8 activations as per tensor, as FusedMoEQuantConfig.act_scale_is_tensor_wide does, both in that check and in the one that clears unused receive rows. Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
MORI EPv2 slices the receive buffers it hands the experts to the rows this step can receive (the DP batches, rounded up to 32 rows) and routes the unused rows to expert -1 with zero weight. It also passed the received-row count, which AITER's fused MoE takes as num_local_tokens. With the rows already bounded, the count saves little, and AITER's MXFP4 W4A4 path faults with it: on a few hundred rows (GLM-5.3-MXFP4 decode warmup) it picks small-tile FlyDSL kernels that read out of bounds. Rather than exempting a quant format, pass no count to any experts kernel and always clear the unused receive rows. On gfx950, single-node DEP8 (ABBA, gsm8k unchanged): Qwen3.5-35B-A3B BF16 decode at concurrency 64 runs about 40% faster (TPOT 11.8 -> 8.2 ms); GLM-5.3 FP8 decode is unchanged, and its 4k-prefill run trades total throughput (-7%) for TTFT (-7.5%) within run-to-run noise. Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
This was referenced Nov 12, 2026
MORI EPv2 dispatched BF16 for every model and left quantization to AITER's fused MoE on the receiving rank. Ask AITER instead which activation dtype its fused MoE consumes for the layer (resolve_activation_dtype, without the token count), and dispatch that, quantized before the all2all: FP8 with FP32 per-128 scales for FP8 block-128 weights, MXFP8 or packed MXFP4 with e8m0 per-32 scales for MXFP4 weights. Less crosses the fabric and AITER skips quantizing the received rows. Combine stays BF16. This ports SGLang vllm-project#37367. BF16 stays where AITER's choice depends on the batch size (AITER answers None; the log names the bound env var that pins it), where it consumes BF16, for per-token / per-tensor FP8, and for experts other than AITER fused MoE. MXFP8 needs an AITER that both emits e8m0 scales and consumes MXFP8 input (ROCm/aiter#4954), probed at startup, else BF16 with a warning. AsyncLL (MORI_ENABLE_SDMA=1) and the internode kernels carry no packed FP4, so MXFP4 goes as BF16 there: MXFP8 would need AITER's A8W4 kernels, which exist only for gate/up-interleaved weights, and AITER answers MXFP4 only for separated ones (its CK stage 1 rejects A8W4 SiLU with separated gate/up). One function, mori_epv2_dispatch_format, sizes the MORI op and decides what AiterExperts expect, so the two cannot disagree; each op logs "MORI EPv2 dispatch dtype=<fmt> (<reason>)" once. The received scales are taken as the leading columns of MORI's possibly padded scale rows, and quantized receive rows are cleared bytewise along with their scales. AITER transposes the scales of pre-quantized FP8 block input over a row count it requires, so with no count from MORI EPv2 the experts give it every received row. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
ChengYM5685
force-pushed
the
mori-epv2-aiter-dispatch-dtype
branch
from
November 13, 2026 04:12
6c97e1f to
d5b4150
Compare
1 of 2 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Claims
Validation
Details
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.