Repository navigation
Conversation
MXFP4 W4A16 experts declare BF16 activations, so MoRI dispatches raw BF16 and AITER quantizes to group-32 mxfp8 on arrival. A token routed to k distinct ranks is therefore quantized k times (expected ~4.4 of 8 ranks at topk=6), and the dispatch carries 2 bytes/element instead of 1 + 1/32. Quantizing in prepare() moves that to once per token on the rank that owns it. AITER already consumes the result: with a pre-quantized fp8 operand and E8M0 scales its stage-1 path sorts the scales (mxfp4_moe_sort_fwd) instead of running its own quant. This mirrors the MXFP4 W4A4 dispatch added in vllm-project#59596, which takes the same pre-quantized route for a4w4. The MoRI side needs only config: dispatch infers the payload dtype from the input tensor and transports scale_dim scales of scale_type_size bytes per token, so this sets scale_dim = hidden_dim/32 and a 1-byte scale element. E8M0 exponents are bytes, not the fp32 the per-token/per-tensor FP8 paths use. Gating is deliberately narrow. The pre-quantized operand is only safe when AITER would have produced mxfp8 itself: a BF16-activation kernel would cast the fp8 payload back to BF16 and never apply the scales, silently losing the scaling. mori_use_mxfp8_dispatch() therefore asks AITER directly via resolve_activation_dtype(), which is documented for exactly this ("lets a caller that produces the activations itself -- an all-to-all dispatch, a fused epilogue -- emit that dtype directly") and returns None rather than guessing when the answer would have depended on a token count that does not exist yet. Signed-off-by: Hemanth Acharya <heachary@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
Hi @heachary Thanks for the PR! Due to the relative smaller size of dsv4.1 flash, DEP8 is not a realistic config when deploying for production, can you also test the perf with DEP2 and DEP4? |
Cherry-picked from vllm-project#61962. MXFP4 W4A16 experts declare BF16 activations, so MoRI dispatches raw BF16 and AITER quantizes to group-32 mxfp8 on arrival. Quantizing in prepare() moves that to once per token on the rank that owns it and roughly halves the dispatch payload. AITER's stage-1 path then sorts the E8M0 scales instead of running its own quant. Signed-off-by: sabreshao <sabre.shao@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
The gate asked resolve_activation_dtype() with a fixed Silu + INTERLEAVE, but AiterExperts picks activation and gate_mode per model and the two can disagree while the quant config is still w4a16: - VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1 passes q_dtype_a=fp4x2 with SEPARATED (experts/rocm_aiter_moe.py), which resolve_activation_dtype answers fp4x2. - SiTU takes its own branch keyed on AITER_SITUV2_*, which defaults to a4w4 and also answers fp4x2. Both pin AITER_BF16_FP8_MOE_BOUND=0, the one input the Silu + INTERLEAVE query depends on, so the gate returned True and MoRI shipped fp8 + e8m0. AITER then took the per_1x32 fp4 branch, dropped the dispatched scales and handed the fp8 tensor to fused_dynamic_mxfp4_quant_moe_sort, whose kernel dispatches fp16/bf16 only -- a startup failure at the profile run. Both configs sent bf16 before this series. Restrict the dispatch to the expert configurations that resolve the same way the query does. DeepSeek-V4.1-Flash w4a16 is unaffected: deepseek_v2.py rejects any hidden_act other than silu. Reported-by: okorzh-amd <okorzh-amd@users.noreply.github.com> Signed-off-by: Hemanth Acharya <heachary@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
@Fangzhou-Ai, thanks for the review. Tested DEP2 and DEP4, here are the results: DEP4 (TP1 x DP4 x EP4)
DEP2 (TP1 x DP2 x EP2)
Also updated these details in the PR desc. |
The previous commit restricted the dispatch to SILU, which also turned off the one SiTU configuration that does want MXFP8: AITER_SITUV2_A8W4 resolves to fp8 and has its own tuned table (kimik3_a8w4_tuned_fmoe.csv). Pass the activation and gate mode AiterExperts will pass instead. The activation selects which input decides the dtype -- SiTUv2 reads AITER_SITUV2_*, Silu/Swiglu on INTERLEAVE reads AITER_BF16_FP8_MOE_BOUND -- so querying with a stand-in reads a knob that may say nothing about the model, which is what let SiTU a4w4 through. Resolved against AITER: SILU + INTERLEAVE, bound=0 -> fp8 dispatch Swiglu + INTERLEAVE, bound=0 -> fp8 dispatch Situv2 + INTERLEAVE, SITUV2_A8W4=1 -> fp8 dispatch Situv2 + SEPARATED, SITUV2_A4W4=1 -> fp4x2 bf16 Situv2 + SEPARATED, a16w4 -> bf16 bf16 SILU + SEPARATED -> fp4x2 bf16 SILU + INTERLEAVE, default bound -> None bf16 Activations outside the set AiterExperts accepts, and the DeepSeek-V4.1 a4w4 opt-in that overrides q_dtype_a on the fused_moe call, still fail closed. Signed-off-by: Hemanth Acharya <heachary@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
bnellnm
left a comment
There was a problem hiding this comment.
LGTM but a pytest level test would be nice if there are not already existing tests in tests/kernels/moe
The dispatch is only safe where AITER would have produced mxfp8 itself, so the gate asks resolve_activation_dtype() with the activation and gate mode AiterExperts will pass. Pin that against AITER directly: each case asserts both the dtype AITER resolves and that the gate follows it, which keeps AITER changing its mind separable from the gate drifting away from it. The table covers silu/gelu/swiglu on the pinned bound, the unpinned bound where AITER defers for want of a token count, and all three SiTUv2 modes -- the disagreement that reached the fp4 branch before. The rest covers what follows from the gate: each guard turning the DeepSeek-V4.1-Flash config off on its own, the mori buffer args (fp8 payload, hidden/32 scales, one byte each rather than fp32), and prepare() emitting fp8 with e8m0 scales that dequantize back to within one e4m3 step of the group scale. GPU-free except for the quant itself, which runs on gfx950 where the resolver can answer fp8 at all. Requested-by: bnellnm <bnellnm@users.noreply.github.com> Signed-off-by: Hemanth Acharya <heachary@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
@bnellnm thanks — added The core of it pins the gate against AITER itself: each case asserts both the dtype The rest covers what follows from the gate:
|
Co-authored-by: Cursor <cursoragent@cursor.com>
3cf4ff7 to
63ee39d
Compare
|
✅ @heachary, CI is now available for this PR.
|
Signed-off-by: Hemanth Acharya <heachary@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Hemanth Acharya <heachary@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Hemanth Acharya <heachary@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
/ci run |
|
✅ Triggered Buildkite CI #102127 for commit ℹ️ This PR is 91 commits behind upstream |
|
/ci retry |
|
✅ Queued 2 failed job(s) for retry in Buildkite CI #102127. |
|
/ci-retry |
fxmarty-amd
left a comment
There was a problem hiding this comment.
Some comments, this is much nicer! Almost LGTM
My main remaining concerns are adding deferred_activation_key, adding checkpoint-independent logic in quantization/mxfp4.py that should rather be in the oracle and/or shared for all model producers, and --quantization-config.moe.activation null that should either work (see https://github.com/vllm-project/vllm/pull/67529/changes#diff-7626acc01163a9549f321f672f7b4d6395d555f5f3cdb9c2c3a6a9983b9a7246), either raise.
| self.w13_precision_config = None | ||
| self.w2_precision_config = None | ||
|
|
||
| def _aiter_deferred_activation_key(self) -> QuantKey | None: |
There was a problem hiding this comment.
This does not belong to vllm/model_executor/layers/quantization/mxfp4.py but to the MXFP4 oracle. There are different model producers that should strive to use a shared logic for activation quant key:
- https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/mxfp4.py
- https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_w4a4_mxfp4.py
See the changes in #67529 to vllm/model_executor/layers/fused_moe/oracle/mxfp4.py.
There was a problem hiding this comment.
Refactored this : The decision now lives in the oracle: make_mxfp4_moe_quant_config() sets deferred_activation_key from the MoE config and the activation key passed to backend selection, applying the user override the same way select_mxfp4_moe_backend() does. All MXFP4 MoE methods (Mxfp4MoEMethod, gpt-oss, Quark, online, compressed-tensors, INC) pass these through, so none of them carries its own copy of the logic.
| if _user_moe_activation_override_is_set(): | ||
| logger.warning_once( | ||
| "%s experts cannot run MoE activation=%s in this configuration; " | ||
| "it is ignored and their inputs are not quantized before " | ||
| "dispatch.", | ||
| self.mxfp4_backend.value, | ||
| self.activation_key, | ||
| ) |
There was a problem hiding this comment.
This should raise an error. If what users request can't run, it should not go through.
There was a problem hiding this comment.
Shifted to an error instead of warning.
| # Quantization the experts apply to their own unquantized (``_a1``) input, | ||
| # e.g. MXFP8 for AITER's MXFP4 w4a16 kernels. An all-to-all may apply it | ||
| # before dispatch instead, which the experts then consume as is. | ||
| deferred_activation_key: QuantKey | None = None |
|
This pull request has merge conflicts that must be resolved before it can be |
Cherry-picked from vllm-project#61962. MXFP4 W4A16 experts declare BF16 activations, so MoRI dispatches raw BF16 and AITER quantizes to group-32 mxfp8 on arrival. Quantizing in prepare() moves that to once per token on the rank that owns it and roughly halves the dispatch payload. AITER's stage-1 path then sorts the E8M0 scales instead of running its own quant. Signed-off-by: sabreshao <sabre.shao@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 75e330e)
…mori-mxfp8-dispatch Signed-off-by: Hemanth Acharya <heachary@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # vllm/models/deepseek_v41/quant_config.py
There was a problem hiding this comment.
LGTM if you can address the above
Please see #68160 for context about my pushback, that can also clean things up afterwards!
|
Hi @heachary thanks for the PR. A quick question as you PR mentioend "MXFP4 W4A16 experts declare BF16 activations" why don't we just use native |
Which activation quantization AITER_MXFP4_BF16 experts run depends on the backend, the MoE config and the requested activation, not on the checkpoint format, so it now lives in oracle/mxfp4.py instead of Mxfp4MoEMethod. make_mxfp4_moe_quant_config sets deferred_activation_key when given the MoE config and the activation key passed to backend selection, applying the user override as select_mxfp4_moe_backend does. Every MXFP4 MoE method passes them: Mxfp4MoEMethod, GptOssMxfp4MoEMethod, Quark OCP MX, online MXFP4 (and FP8 re-quant through it), compressed-tensors and INC W4A4. Only DeepSeek-V4/V4.1 default to MXFP8; the others dispatch MXFP8 when the user requests moe.activation=mxfp8 on those experts. Requested-by: fxmarty-amd <fxmarty-amd@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Hemanth Acharya <heachary@amd.com>
An explicit quantization_config.moe.activation that the experts do not compute with now raises instead of being ignored with a warning: MXFP8 is what the interleaved w4a16 experts compute with, MXFP4 the DeepSeek-V4.1 a4w4 opt-in, and SiTU whatever VLLM_ROCM_USE_AITER_MOE_SITUV2 selects. A model default that does not apply still falls back to BF16 dispatch. Requested-by: fxmarty-amd <fxmarty-amd@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Hemanth Acharya <heachary@amd.com>
|
afaik #66564 adds |
|
/ci run |
|
✅ Triggered Buildkite CI #103744 for commit ℹ️ This PR is 43 commits behind upstream |
Motivation
MXFP4 W4A16 experts declare BF16 activations, so MoRI dispatches raw BF16 and
AITER quantizes to group-32 MXFP8 on arrival. A token routed to
kdistinctranks is therefore quantized
ktimes — expected ~4.4 of 8 ranks at topk=6 forDeepSeek-V4.1-Flash — and the dispatch carries 2 bytes/element instead of
1 + 1/32.
Quantizing in
prepare()moves that to once per token on the rank that owns it,and roughly halves the dispatch payload.
AITER already consumes the result: with a pre-quantized FP8 operand and E8M0
scales, its stage-1 path sorts the scales (
mxfp4_moe_sort_fwd) instead ofrunning its own quant. This mirrors the MXFP4 W4A4 dispatch added in #59596,
which takes the same pre-quantized route for a4w4.
Summary of changes
all2all_utils.py— wires a third dispatch mode intomaybe_make_prepare_finalize(), taken when the experts' quant config carriesdeferred_activation_key == kMxfp8Dynamic(see below). MoRI infers the payloaddtype from the input tensor and transports
scale_dimscales ofscale_type_sizebytes per token, so the new branch setsscale_dim = hidden_dim / 32and a 1-byte scale element. E8M0 exponents arebytes, not the fp32 the per-token/per-tensor FP8 paths use.
prepare_finalize/mori.py— adds theuse_mxfp8_dispatchflag and thecorresponding quant in
prepare(), viaget_hip_quant(QuantType.per_1x32)withscale_type=dtypes.fp8_e8m0, matching the FP4 branch directly above it. Themutual-exclusion assert becomes
sum(...) <= 1now that there are three modes.The dispatch dtype comes from the MoE activation key. A pre-quantized
operand is only safe when the experts consume MXFP8: a BF16-activation kernel
would cast the FP8 payload back to BF16 and never apply the scales, silently
losing the scaling. Rather than inferring that from AITER, the choice follows
the
QuantKeythe quant method already carries:kMxfp8DynamictoMxfp4MoEMethodas a default through the newresolve_default_moe_activation_key()(oracle/mxfp4.py). The checkpointserializes no activation scheme for its experts, so the user's
quantization_config.moe.activationtakes precedence, an explicitnull(BF16) included. For DeepSeek-V4.1 this applies off CUDA; NVIDIA GPUs keep
the method
nvidia/quantization.pychooses.FusedMoEQuantConfig.deferred_activation_key(config.py) records aquantization the experts apply to their own unquantized input, which an
all-to-all may apply before dispatch instead.
MoE config and the requested activation, not on the checkpoint format, so
make_mxfp4_moe_quant_config()(oracle/mxfp4.py) sets the key, given theMoE config and the activation key passed to backend selection. It applies the
user override to that key as
select_mxfp4_moe_backend()does, and setskMxfp8Dynamiconly forAITER_MXFP4_BF16experts with an MXFP8 key,outside SiTU and the opt-in DSv4.1 a4w4 path, on an AITER build whose
fused_moeacceptsquant_dtype_a. A user-requested key those experts donot compute with raises: MXFP8 on the interleaved path, MXFP4 with the DSv4.1
a4w4 opt-in, and SiTU whatever
VLLM_ROCM_USE_AITER_MOE_SITUV2selects. Amodel default that does not apply falls back to BF16 dispatch.
Mxfp4MoEMethod,GptOssMxfp4MoEMethod, Quark OCP MX, online MXFP4 (and FP8 re-quant throughit), compressed-tensors and INC W4A4. Only DeepSeek-V4 / V4.1 default to
MXFP8; on the others MXFP8 dispatch takes an explicit
quantization_config.moe.activation: mxfp8onAITER_MXFP4_BF16experts(gpt-oss, Quark W4A16). Compressed-tensors and INC do not select that backend
today, and online MXFP4 rejects
mxfp8on it until [Quantization][ROCm] Honor activation requests for AITER MXFP4 BF16 MoE #68160.rocm_aiter_moe.pypassesq_dtype_a=fp8explicitly when the key is set,so the experts expect exactly what the MXFP8 dispatch hands over instead of
relying on AITER's internal BF16/FP8 heuristic.
The previous
mori_use_mxfp8_dispatch()/resolve_activation_dtype()gate isremoved.
Tests —
tests/kernels/moe/test_mori_mxfp8_dispatch.pychecks thatDeepSeek-V4.1-Flash picks MXFP8 dispatch by default, that a user override
(
mxfp8, ornull) takes precedence, and that an override the experts cannotrun (
mxfp4on the interleaved path,mxfp8with a4w4) raises, along with theMoRI buffer sizing and the quant
prepare()applies. It runs on MI355 in theMoRI EP Numerics job.
Results
8x MI355X (gfx950), DeepSeek-V4.1-Flash, DEP8, DEP4, DEP2 with
mori_high_throughput,speculative decoding off. 8192 in / 1024 out, 256 prompts at concurrency 64,
against a stock nightly (
0.32.0rc1.dev190+g3f1bcfd82) with only this changeapplied — this PR was measured on its own, with no AITER-side patches, so the
numbers are attributable to the dispatch change alone.
Engagement was confirmed rather than assumed: the dispatch kernel switches from
EpDispatchIntraNodeKernel_bf16toEpDispatchIntraNodeKernel_fp8_ocp. Thesenumbers were taken with the earlier
resolve_activation_dtype()gate, whichselected MXFP8 dispatch for this configuration just as the activation-key
default does now.
Performance
DEP8
DEP4
DEP2
Accuracy
gsm8k, 5-shot, all 1319 questions.
Within ~1 sigma of baseline. The same mxfp8 quantization is applied either way;
this only changes which rank performs it and how many times.
Reproduction
Server:
Benchmark:
Accuracy: