Skip to content

[ROCm][MORI] Dispatch the activation format AITER's fused MoE consumes in MORI EPv2 - #66964

Draft
ChengYM5685 wants to merge 3 commits into
vllm-project:mainfrom
ChengYM5685:mori-epv2-aiter-dispatch-dtype
Draft

ChengYM5685 wants to merge 3 commits into
vllm-project:mainfrom
ChengYM5685:mori-epv2-aiter-dispatch-dtype

Conversation

@ChengYM5685

@ChengYM5685 ChengYM5685 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Claims

Validation

Details


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

@mergify mergify Bot added the rocm Related to AMD ROCm label Nov 11, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Nov 11, 2026
MORI EPv2 rejects experts that quantize activations per tensor, since the
experts quantize the received rows and one scale would be taken over padding.
OCP MX quant configs (e.g. Quark MXFP4) are built without per-token or block
flags, so their activation group shape is per tensor and `is_per_tensor` is
true, although each 1x32 block of a row has its own scale. EPv2 therefore
refused every MXFP4 model at kernel setup.

Count only FP8 / INT8 activations as per tensor, as
FusedMoEQuantConfig.act_scale_is_tensor_wide does, both in that check and in
the one that clears unused receive rows.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
MORI EPv2 slices the receive buffers it hands the experts to the rows this
step can receive (the DP batches, rounded up to 32 rows) and routes the
unused rows to expert -1 with zero weight. It also passed the received-row
count, which AITER's fused MoE takes as num_local_tokens. With the rows
already bounded, the count saves little, and AITER's MXFP4 W4A4 path faults
with it: on a few hundred rows (GLM-5.3-MXFP4 decode warmup) it picks
small-tile FlyDSL kernels that read out of bounds.

Rather than exempting a quant format, pass no count to any experts kernel
and always clear the unused receive rows.

On gfx950, single-node DEP8 (ABBA, gsm8k unchanged): Qwen3.5-35B-A3B BF16
decode at concurrency 64 runs about 40% faster (TPOT 11.8 -> 8.2 ms); GLM-5.3
FP8 decode is unchanged, and its 4k-prefill run trades total throughput
(-7%) for TTFT (-7.5%) within run-to-run noise.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
@ChengYM5685 ChengYM5685 changed the title Mori epv2 aiter dispatch dtype [ROCm][MORI] Dispatch the activation format AITER's fused MoE consumes in MORI EPv2 Nov 11, 2026
MORI EPv2 dispatched BF16 for every model and left quantization to AITER's
fused MoE on the receiving rank. Ask AITER instead which activation dtype its
fused MoE consumes for the layer (resolve_activation_dtype, without the token
count), and dispatch that, quantized before the all2all: FP8 with FP32
per-128 scales for FP8 block-128 weights, MXFP8 or packed MXFP4 with e8m0
per-32 scales for MXFP4 weights. Less crosses the fabric and AITER skips
quantizing the received rows. Combine stays BF16. This ports SGLang vllm-project#37367.

BF16 stays where AITER's choice depends on the batch size (AITER answers
None; the log names the bound env var that pins it), where it consumes BF16,
for per-token / per-tensor FP8, and for experts other than AITER fused MoE.
MXFP8 needs an AITER that both emits e8m0 scales and consumes MXFP8 input
(ROCm/aiter#4954), probed at startup, else BF16 with a warning. AsyncLL
(MORI_ENABLE_SDMA=1) and the internode kernels carry no packed FP4, so MXFP4
goes as BF16 there: MXFP8 would need AITER's A8W4 kernels, which exist only
for gate/up-interleaved weights, and AITER answers MXFP4 only for separated
ones (its CK stage 1 rejects A8W4 SiLU with separated gate/up).

One function, mori_epv2_dispatch_format, sizes the MORI op and decides what
AiterExperts expect, so the two cannot disagree; each op logs
"MORI EPv2 dispatch dtype=<fmt> (<reason>)" once. The received scales are
taken as the leading columns of MORI's possibly padded scale rows, and
quantized receive rows are cleared bytewise along with their scales.
AITER transposes the scales of pre-quantized FP8 block input over a row
count it requires, so with no count from MORI EPv2 the experts give it every
received row.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rocm Related to AMD ROCm

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

1 participant