Repository navigation
[ROCm][MoE] Enable MoRI FP4 dispatch for DeepSeek V4.1 a4w4 - #59852
Closed
Fangzhou-Ai wants to merge 4 commits into
Closed
Fangzhou-Ai wants to merge 4 commits into
Fangzhou-Ai wants to merge 4 commits into
Conversation
Signed-off-by: lcskrishna <lollachaitanya@gmail.com>
DeepSeek V4.1 a4w4 (VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1) runs MXFP4 activations in AITER under a W4A16 quant config, so the W4A4-only gate kept MoRI on BF16 dispatch. Route it through the same packed FP4 + e8m0 dispatch and add an mxfp4_dsv4 case to the MoRI MoE numerics test. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
MoRI returns its full receive buffer (world_size * max_num_tokens rows), so AITER sized its MoE kernels for the worst case on every step. Slice the dispatch outputs to world_size * max(num_tokens_across_dp), which is known on the host, needs no sync, and is fixed under CUDA graph capture. Co-authored-by: Cursor Agent Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Collaborator
Author
|
We experience precision issue with a4w4 |
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Enable MoRI FP4 dispatch for DeepSeek V4.1 a4w4 (
VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1) and trim MoRI's dispatch output to the valid token count. Stacked on #59596, which adds MoRI FP4 dispatch for MXFP4 W4A4 checkpoints. Only the last two commits are new here; this PR will be rebased once #59596 lands.Claims
--all2all-backend mori_*now dispatches packed FP4 with e8m0 block scales instead of BF16. Before this change, [ROCm][DI] Enable MoRI FP4 dispatch for mxfp4 w4a4 moe #59596's W4A4-only gate kept it on BF16 dispatch, because this mode runs on a W4A16 quant config with MXFP4 activations in AITER.AiterExperts.workspace_shapes) are now a singleFusedMoEConfig.use_mori_fp4_dispatch().num_dispatchers * max(num_tokens_across_dp)rows instead of the fullworld_size * max_num_tokensreceive buffer, so AITER picks its MoE kernel config for the real M rather than the worst case.Validation
Unit test
tests/kernels/moe/test_mori_moe.pygets anmxfp4_dsv4scheme. It builds W4A16 weights in the SEPARATED gate/up layout, setsuse_mxfp4_w4a4_dsv4, asserts FP4 dispatch is selected, and compares two-rank MoRI against AITER's non-EP path, including CUDA graph replay at m=32. Every case also asserts that the experts receive2 * mrows, the trimmed prefix, rather than MoRI's max-sized buffer.On 2x MI355X, all 28 cases pass: 7 shapes each for bf16, fp8_token_channel, mxfp4 and mxfp4_dsv4. Without a writable
FLYDSL_RUNTIME_CACHE_DIR, both MXFP4 schemes fail with aPermissionErroron AITER's default FlyDSL cache dir in our image. That is unrelated to this change.E2E: DeepSeek-V4.1-Flash DEP2 on MI355X
Setup: 1 node, 2 GPUs,
--tensor-parallel-size 1 --data-parallel-size 2 --enable-expert-parallel --moe-backend aiter --kv-cache-dtype fp8 --block-size 128 --max-num-seqs 64 --max-num-batched-tokens 8192, CUDA graphs on. Validated on a local stack ofmain@0cbac6cd13+ #58014 + #59596 + this change.Accuracy. InferenceX GSM8K eval:
lm_eval --model local-chat-completions --apply_chat_template, 5-shot, all 1319 questions,max_tokens=16384,temperature=0,num_concurrent=64, no speculative decoding.The difference is within one standard error.
Throughput. InferenceX agentic-coding sweep (
semianalysisai/cc-traces-weka-062126, aiperf throughvllm-router, 1 h per point), a4w4 MoE on both sides (VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1), DSpark speculative decoding with 5 tokens and synthetic acceptance (3.51). The only difference between the two sides is--all2all-backend.Kernel. EP2 dispatch+combine, hidden 5120, 384 experts, top-6, HIP-graph replay. FP4 times include the MXFP4 quantize before dispatch.
Details
q_dtype_a=fp4x2and forwardsa1q_scale, so [ROCm][DI] Enable MoRI FP4 dispatch for mxfp4 w4a4 moe #59596's pre-quantized FP4 handling inrocm_aiter_fused_expertsapplies unchanged. Only the gate needed widening.dp_metadata.num_tokens_across_dp_cpu, so it is known on the host, needs no device sync, and is fixed for a given CUDA graph capture size. Each source rank sends a token at most once, so every valid received row is inside that prefix.fp8_direct_castwas about 1.3× faster on combine but lowered accuracy (gsm8k 0.869 vs 0.897 on the repo'sgsm8k_eval.py). Thefp8_blockwiseandfp4_blockwisemodes were slower than BF16 combine at EP2. Combine is left BF16. [AMD] Support fp4 dispatch and fp8 combine for MORI EP #44323 covers combine dtype selection more broadly.quant_config.use_mxfp4_w4a4, and [AMD] Support fp4 dispatch and fp8 combine for MORI EP #44323 (MoRI dispatch/combine dtype refactor) has no DSv4.1 a4w4 handling. I'm happy to fold this into [ROCm][DI] Enable MoRI FP4 dispatch for mxfp4 w4a4 moe #59596 if the author prefers.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.