Skip to content

[ROCm][MoE] Enable MoRI FP4 dispatch for DeepSeek V4.1 a4w4 - #59852

Closed
Fangzhou-Ai wants to merge 4 commits into
vllm-project:mainfrom
Fangzhou-Ai:rocm-mori-fp4-dispatch-dsv41
Closed

Fangzhou-Ai wants to merge 4 commits into
vllm-project:mainfrom
Fangzhou-Ai:rocm-mori-fp4-dispatch-dsv41

Conversation

@Fangzhou-Ai

@Fangzhou-Ai Fangzhou-Ai commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

Overview

Enable MoRI FP4 dispatch for DeepSeek V4.1 a4w4 (VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1) and trim MoRI's dispatch output to the valid token count. Stacked on #59596, which adds MoRI FP4 dispatch for MXFP4 W4A4 checkpoints. Only the last two commits are new here; this PR will be rebased once #59596 lands.

Claims

  • DeepSeek V4.1 a4w4 under --all2all-backend mori_* now dispatches packed FP4 with e8m0 block scales instead of BF16. Before this change, [ROCm][DI] Enable MoRI FP4 dispatch for mxfp4 w4a4 moe  #59596's W4A4-only gate kept it on BF16 dispatch, because this mode runs on a W4A16 quant config with MXFP4 activations in AITER.
  • The two copies of the "MoRI dispatches FP4" check (prepare/finalize selection and AiterExperts.workspace_shapes) are now a single FusedMoEConfig.use_mori_fp4_dispatch().
  • MoRI's dispatch outputs are sliced to num_dispatchers * max(num_tokens_across_dp) rows instead of the full world_size * max_num_tokens receive buffer, so AITER picks its MoE kernel config for the real M rather than the worst case.
  • No accuracy change on DeepSeek-V4.1-Flash DEP2: GSM8K is 0.974 with MoRI FP4 dispatch and trimming, against 0.977 on the default path.
  • On MI355X DEP2 agentic-coding traces, output throughput per GPU is 1.8–11.0% higher than allgather/reducescatter at c32–c192. One MoE layer's dispatch+combine takes 1.33–1.49× less time than MoRI BF16 at 384–8,192 tokens/rank.

Validation

Unit test

tests/kernels/moe/test_mori_moe.py gets an mxfp4_dsv4 scheme. It builds W4A16 weights in the SEPARATED gate/up layout, sets use_mxfp4_w4a4_dsv4, asserts FP4 dispatch is selected, and compares two-rank MoRI against AITER's non-EP path, including CUDA graph replay at m=32. Every case also asserts that the experts receive 2 * m rows, the trimmed prefix, rather than MoRI's max-sized buffer.

FLYDSL_RUNTIME_CACHE_DIR=<writable dir> pytest tests/kernels/moe/test_mori_moe.py

On 2x MI355X, all 28 cases pass: 7 shapes each for bf16, fp8_token_channel, mxfp4 and mxfp4_dsv4. Without a writable FLYDSL_RUNTIME_CACHE_DIR, both MXFP4 schemes fail with a PermissionError on AITER's default FlyDSL cache dir in our image. That is unrelated to this change.

E2E: DeepSeek-V4.1-Flash DEP2 on MI355X

Setup: 1 node, 2 GPUs, --tensor-parallel-size 1 --data-parallel-size 2 --enable-expert-parallel --moe-backend aiter --kv-cache-dtype fp8 --block-size 128 --max-num-seqs 64 --max-num-batched-tokens 8192, CUDA graphs on. Validated on a local stack of main@0cbac6cd13 + #58014 + #59596 + this change.

Accuracy. InferenceX GSM8K eval: lm_eval --model local-chat-completions --apply_chat_template, 5-shot, all 1319 questions, max_tokens=16384, temperature=0, num_concurrent=64, no speculative decoding.

config strict-match flexible-extract
default path: a8w4 MoE, allgather_reducescatter 0.9765 ± 0.0042 0.9757 ± 0.0042
a4w4 MoE, MoRI FP4 dispatch + trim (this PR) 0.9742 ± 0.0044 0.9735 ± 0.0044

The difference is within one standard error.

Throughput. InferenceX agentic-coding sweep (semianalysisai/cc-traces-weka-062126, aiperf through vllm-router, 1 h per point), a4w4 MoE on both sides (VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1), DSpark speculative decoding with 5 tokens and synthetic acceptance (3.51). The only difference between the two sides is --all2all-backend.

conc output tok/s/GPU, agrs → MoRI FP4 Δ p50 TPOT (ms) p50 TTFT (s) p50 interactivity (tok/s/user)
32 582 → 593 +1.8% 7.75 → 7.46 0.253 → 0.241 129.0 → 134.0
64 1015 → 1060 +4.5% 16.9 → 16.0 0.363 → 0.351 59.4 → 62.6
128 1237 → 1278 +3.4% 45.8 → 42.8 0.689 → 0.633 21.9 → 23.4
192 1091 → 1210 +11.0% 82.5 → 74.2 1.19 → 1.08 12.1 → 13.5

Kernel. EP2 dispatch+combine, hidden 5120, 384 experts, top-6, HIP-graph replay. FP4 times include the MXFP4 quantize before dispatch.

tokens/rank allgather + reducescatter MoRI BF16 MoRI FP4 dispatch
64 58 µs 46 µs 47 µs
384 182 µs 164 µs 123 µs
1152 474 µs 449 µs 316 µs
8192 2.93 ms 3.02 ms 2.03 ms

Details

  • The DSv4.1 a4w4 expert path already sets q_dtype_a=fp4x2 and forwards a1q_scale, so [ROCm][DI] Enable MoRI FP4 dispatch for mxfp4 w4a4 moe  #59596's pre-quantized FP4 handling in rocm_aiter_fused_experts applies unchanged. Only the gate needed widening.
  • Trimming: the bound comes from dp_metadata.num_tokens_across_dp_cpu, so it is known on the host, needs no device sync, and is fixed for a given CUDA graph capture size. Each source rank sends a token at most once, so every valid received row is inside that prefix.
  • The e2e gain is modest on EP2. With top-6 over 2 ranks, almost every token goes to both ranks, so dispatch volume is not reduced, and a single xGMI link between the two GPUs (~64 GB/s) carries all traffic. FP4 shrinks dispatch bytes about 3.6×, while combine stays BF16.
  • I also tried MoRI's combine quantization on this model (EP2) without wiring it in. fp8_direct_cast was about 1.3× faster on combine but lowered accuracy (gsm8k 0.869 vs 0.897 on the repo's gsm8k_eval.py). The fp8_blockwise and fp4_blockwise modes were slower than BF16 combine at EP2. Combine is left BF16. [AMD] Support fp4 dispatch and fp8 combine for MORI EP #44323 covers combine dtype selection more broadly.
  • Not a duplicate: [ROCm][DI] Enable MoRI FP4 dispatch for mxfp4 w4a4 moe  #59596 only gates on quant_config.use_mxfp4_w4a4, and [AMD] Support fp4 dispatch and fp8 combine for MORI EP #44323 (MoRI dispatch/combine dtype refactor) has no DSv4.1 a4w4 handling. I'm happy to fold this into [ROCm][DI] Enable MoRI FP4 dispatch for mxfp4 w4a4 moe  #59596 if the author prefers.

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

lcskrishna and others added 3 commits October 1, 2026 11:46
Signed-off-by: lcskrishna <lollachaitanya@gmail.com>
DeepSeek V4.1 a4w4 (VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1) runs MXFP4
activations in AITER under a W4A16 quant config, so the W4A4-only gate
kept MoRI on BF16 dispatch. Route it through the same packed FP4 + e8m0
dispatch and add an mxfp4_dsv4 case to the MoRI MoE numerics test.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
@mergify mergify Bot added deepseek Related to DeepSeek models DSv4.1 Related to DeepSeek-V4.1 models rocm Related to AMD ROCm labels Oct 3, 2026
MoRI returns its full receive buffer (world_size * max_num_tokens rows), so
AITER sized its MoE kernels for the worst case on every step. Slice the
dispatch outputs to world_size * max(num_tokens_across_dp), which is known on
the host, needs no sync, and is fixed under CUDA graph capture.

Co-authored-by: Cursor Agent
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
@mergify

mergify Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Fangzhou-Ai.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Oct 13, 2026
@Fangzhou-Ai

Copy link
Copy Markdown
Collaborator Author

We experience precision issue with a4w4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek Related to DeepSeek models DSv4.1 Related to DeepSeek-V4.1 models needs-rebase rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants