Skip to content

[AMD][DSV4.1Flash] Quantize MoE activations to MXFP8 before the MoRI dispatch - #61962

Open
heachary wants to merge 23 commits into
vllm-project:mainfrom
heachary:heachary/dsv41_flash/mori-mxfp8-dispatch
Open

heachary wants to merge 23 commits into
vllm-project:mainfrom
heachary:heachary/dsv41_flash/mori-mxfp8-dispatch

Conversation

@heachary

@heachary heachary commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

MXFP4 W4A16 experts declare BF16 activations, so MoRI dispatches raw BF16 and
AITER quantizes to group-32 MXFP8 on arrival. A token routed to k distinct
ranks is therefore quantized k times — expected ~4.4 of 8 ranks at topk=6 for
DeepSeek-V4.1-Flash — and the dispatch carries 2 bytes/element instead of
1 + 1/32.

Quantizing in prepare() moves that to once per token on the rank that owns it,
and roughly halves the dispatch payload.

AITER already consumes the result: with a pre-quantized FP8 operand and E8M0
scales, its stage-1 path sorts the scales (mxfp4_moe_sort_fwd) instead of
running its own quant. This mirrors the MXFP4 W4A4 dispatch added in #59596,
which takes the same pre-quantized route for a4w4.

Summary of changes

all2all_utils.py — wires a third dispatch mode into
maybe_make_prepare_finalize(), taken when the experts' quant config carries
deferred_activation_key == kMxfp8Dynamic (see below). MoRI infers the payload
dtype from the input tensor and transports scale_dim scales of
scale_type_size bytes per token, so the new branch sets
scale_dim = hidden_dim / 32 and a 1-byte scale element. E8M0 exponents are
bytes, not the fp32 the per-token/per-tensor FP8 paths use.

prepare_finalize/mori.py — adds the use_mxfp8_dispatch flag and the
corresponding quant in prepare(), via get_hip_quant(QuantType.per_1x32) with
scale_type=dtypes.fp8_e8m0, matching the FP4 branch directly above it. The
mutual-exclusion assert becomes sum(...) <= 1 now that there are three modes.

The dispatch dtype comes from the MoE activation key. A pre-quantized
operand is only safe when the experts consume MXFP8: a BF16-activation kernel
would cast the FP8 payload back to BF16 and never apply the scales, silently
losing the scaling. Rather than inferring that from AITER, the choice follows
the QuantKey the quant method already carries:

  • Model default, user override. The DeepSeek-V4 / V4.1 quant configs pass
    kMxfp8Dynamic to Mxfp4MoEMethod as a default through the new
    resolve_default_moe_activation_key() (oracle/mxfp4.py). The checkpoint
    serializes no activation scheme for its experts, so the user's
    quantization_config.moe.activation takes precedence, an explicit null
    (BF16) included. For DeepSeek-V4.1 this applies off CUDA; NVIDIA GPUs keep
    the method nvidia/quantization.py chooses.
  • FusedMoEQuantConfig.deferred_activation_key (config.py) records a
    quantization the experts apply to their own unquantized input, which an
    all-to-all may apply before dispatch instead.
  • The decision lives in the MXFP4 oracle. It depends on the backend, the
    MoE config and the requested activation, not on the checkpoint format, so
    make_mxfp4_moe_quant_config() (oracle/mxfp4.py) sets the key, given the
    MoE config and the activation key passed to backend selection. It applies the
    user override to that key as select_mxfp4_moe_backend() does, and sets
    kMxfp8Dynamic only for AITER_MXFP4_BF16 experts with an MXFP8 key,
    outside SiTU and the opt-in DSv4.1 a4w4 path, on an AITER build whose
    fused_moe accepts quant_dtype_a. A user-requested key those experts do
    not compute with raises: MXFP8 on the interleaved path, MXFP4 with the DSv4.1
    a4w4 opt-in, and SiTU whatever VLLM_ROCM_USE_AITER_MOE_SITUV2 selects. A
    model default that does not apply falls back to BF16 dispatch.
  • Every MXFP4 MoE method passes them: Mxfp4MoEMethod,
    GptOssMxfp4MoEMethod, Quark OCP MX, online MXFP4 (and FP8 re-quant through
    it), compressed-tensors and INC W4A4. Only DeepSeek-V4 / V4.1 default to
    MXFP8; on the others MXFP8 dispatch takes an explicit
    quantization_config.moe.activation: mxfp8 on AITER_MXFP4_BF16 experts
    (gpt-oss, Quark W4A16). Compressed-tensors and INC do not select that backend
    today, and online MXFP4 rejects mxfp8 on it until [Quantization][ROCm] Honor activation requests for AITER MXFP4 BF16 MoE #68160.
  • rocm_aiter_moe.py passes q_dtype_a=fp8 explicitly when the key is set,
    so the experts expect exactly what the MXFP8 dispatch hands over instead of
    relying on AITER's internal BF16/FP8 heuristic.

The previous mori_use_mxfp8_dispatch() / resolve_activation_dtype() gate is
removed.

Tests — tests/kernels/moe/test_mori_mxfp8_dispatch.py checks that
DeepSeek-V4.1-Flash picks MXFP8 dispatch by default, that a user override
(mxfp8, or null) takes precedence, and that an override the experts cannot
run (mxfp4 on the interleaved path, mxfp8 with a4w4) raises, along with the
MoRI buffer sizing and the quant prepare() applies. It runs on MI355 in the
MoRI EP Numerics job.

Results

8x MI355X (gfx950), DeepSeek-V4.1-Flash, DEP8, DEP4, DEP2 with mori_high_throughput,
speculative decoding off. 8192 in / 1024 out, 256 prompts at concurrency 64,
against a stock nightly (0.32.0rc1.dev190+g3f1bcfd82) with only this change
applied
— this PR was measured on its own, with no AITER-side patches, so the
numbers are attributable to the dispatch change alone.

Engagement was confirmed rather than assumed: the dispatch kernel switches from
EpDispatchIntraNodeKernel_bf16 to EpDispatchIntraNodeKernel_fp8_ocp. These
numbers were taken with the earlier resolve_activation_dtype() gate, which
selected MXFP8 dispatch for this configuration just as the activation-key
default does now.

Performance

DEP8

Metric Baseline With patch Delta
Output token throughput (tok/s) 1103.58 1309.33 +18.6%
Total token throughput (tok/s) 9932.18 11783.95 +18.6%
Median TPOT (ms) 54.16 46.83 -13.5%
Mean TPOT (ms) 54.68 46.69 -14.6%
Median ITL (ms) 51.40 44.29 -13.8%
Mean TTFT (ms) 3392.14 2248.40 -33.7%
Benchmark duration (s) 237.54 200.21 -15.7%

DEP4

Metric Baseline With patch Delta
Output token throughput (tok/s) 1511.38 1689.81 +11.8%
Total token throughput (tok/s) 13602.43 15208.28 +11.8%
Median TPOT (ms) 40.29 36.11 -10.4%
Mean TPOT (ms) 39.68 35.44 -10.7%
Median ITL (ms) 33.86 30.40 -10.2%
Mean TTFT (ms) 2706.62 2474.43 -8.6%
Median TTFT (ms) 1990.54 1840.41 -7.5%
Median E2EL (ms) 43199.28 38666.46 -10.5%
Benchmark duration (s) 173.45 155.13 -10.6%

DEP2

Metric Baseline With patch Delta
Output token throughput (tok/s) 1515.65 1645.83 +8.6%
Total token throughput (tok/s) 13640.83 14812.48 +8.6%
Median TPOT (ms) 40.18 36.97 -8.0%
Mean TPOT (ms) 38.78 35.72 -7.9%
Median ITL (ms) 27.37 25.28 -7.7%
Mean TTFT (ms) 3467.17 3194.15 -7.9%
Median TTFT (ms) 1872.13 1725.74 -7.8%
Median E2EL (ms) 42983.44 39589.57 -7.9%
Benchmark duration (s) 172.96 159.28 -7.9%

Accuracy

gsm8k, 5-shot, all 1319 questions.

Arm strict-match stderr Delta flexible-extract
Baseline 0.9196 0.0075 — 0.9196
With patch 0.9280 0.0071 +0.0084 0.9287

Within ~1 sigma of baseline. The same mxfp8 quantization is applied either way;
this only changes which rank performs it and how many times.

Reproduction

Server:

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=1
export VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=True
export VLLM_USE_BREAKABLE_CUDAGRAPH=1
export VLLM_USE_RUST_FRONTEND=1

vllm serve /model/DeepSeek-V4.1-Flash \
  --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
  --tensor-parallel-size 1 \
  --data-parallel-size 8 \
  --enable-expert-parallel \
  --all2all-backend mori_high_throughput \
  --moe-backend aiter \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.9 \
  --max-num-batched-tokens 8192 \
  --max-cudagraph-capture-size 1024 \
  --block-size 128 \
  --tokenizer-mode deepseek_v41 \
  --tool-call-parser deepseek_v41 \
  --reasoning-parser deepseek_v41 \
  --enable-auto-tool-choice \
  --language-model-only \
  --no-swa-bounded-replay \
  --no-enable-prefix-caching \
  --disable-uvicorn-access-log \
  --attention-config '{"indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'

Benchmark:

vllm bench serve \
  --model deepseek-ai/DeepSeek-V4.1-Flash \
  --tokenizer /model/DeepSeek-V4.1-Flash \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --ignore-eos \
  --num-prompts 256 \
  --max-concurrency 64 \
  --temperature 0 \
  --seed 0 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,99

Accuracy:

lm_eval --model local-completions --tasks gsm8k --num_fewshot 5 \
  --model_args base_url=http://localhost:8000/v1/completions,model=deepseek-ai/DeepSeek-V4.1-Flash,tokenizer_backend=None,num_concurrent=64,max_retries=3,timeout=3600

MXFP4 W4A16 experts declare BF16 activations, so MoRI dispatches raw BF16 and
AITER quantizes to group-32 mxfp8 on arrival. A token routed to k distinct
ranks is therefore quantized k times (expected ~4.4 of 8 ranks at topk=6), and
the dispatch carries 2 bytes/element instead of 1 + 1/32.

Quantizing in prepare() moves that to once per token on the rank that owns it.
AITER already consumes the result: with a pre-quantized fp8 operand and E8M0
scales its stage-1 path sorts the scales (mxfp4_moe_sort_fwd) instead of
running its own quant. This mirrors the MXFP4 W4A4 dispatch added in vllm-project#59596,
which takes the same pre-quantized route for a4w4.

The MoRI side needs only config: dispatch infers the payload dtype from the
input tensor and transports scale_dim scales of scale_type_size bytes per
token, so this sets scale_dim = hidden_dim/32 and a 1-byte scale element.
E8M0 exponents are bytes, not the fp32 the per-token/per-tensor FP8 paths use.

Gating is deliberately narrow. The pre-quantized operand is only safe when
AITER would have produced mxfp8 itself: a BF16-activation kernel would cast the
fp8 payload back to BF16 and never apply the scales, silently losing the
scaling. mori_use_mxfp8_dispatch() therefore asks AITER directly via
resolve_activation_dtype(), which is documented for exactly this ("lets a
caller that produces the activations itself -- an all-to-all dispatch, a fused
epilogue -- emit that dtype directly") and returns None rather than guessing
when the answer would have depended on a token count that does not exist yet.

Signed-off-by: Hemanth Acharya <heachary@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added quantization rocm Related to AMD ROCm labels Oct 16, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Oct 16, 2026
@Fangzhou-Ai

Copy link
Copy Markdown
Collaborator

Hi @heachary Thanks for the PR! Due to the relative smaller size of dsv4.1 flash, DEP8 is not a realistic config when deploying for production, can you also test the perf with DEP2 and DEP4?

sabreshao pushed a commit to sabreshao/vllm that referenced this pull request Oct 21, 2026
Cherry-picked from vllm-project#61962.

MXFP4 W4A16 experts declare BF16 activations, so MoRI dispatches raw BF16 and
AITER quantizes to group-32 mxfp8 on arrival. Quantizing in prepare() moves
that to once per token on the rank that owns it and roughly halves the
dispatch payload. AITER's stage-1 path then sorts the E8M0 scales instead of
running its own quant.

Signed-off-by: sabreshao <sabre.shao@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The gate asked resolve_activation_dtype() with a fixed Silu + INTERLEAVE,
but AiterExperts picks activation and gate_mode per model and the two can
disagree while the quant config is still w4a16:

- VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1 passes q_dtype_a=fp4x2 with SEPARATED
  (experts/rocm_aiter_moe.py), which resolve_activation_dtype answers fp4x2.
- SiTU takes its own branch keyed on AITER_SITUV2_*, which defaults to a4w4
  and also answers fp4x2.

Both pin AITER_BF16_FP8_MOE_BOUND=0, the one input the Silu + INTERLEAVE
query depends on, so the gate returned True and MoRI shipped fp8 + e8m0.
AITER then took the per_1x32 fp4 branch, dropped the dispatched scales and
handed the fp8 tensor to fused_dynamic_mxfp4_quant_moe_sort, whose kernel
dispatches fp16/bf16 only -- a startup failure at the profile run. Both
configs sent bf16 before this series.

Restrict the dispatch to the expert configurations that resolve the same way
the query does. DeepSeek-V4.1-Flash w4a16 is unaffected: deepseek_v2.py
rejects any hidden_act other than silu.

Reported-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Signed-off-by: Hemanth Acharya <heachary@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@heachary
heachary requested a review from bnellnm as a code owner October 21, 2026 06:22
@heachary

Copy link
Copy Markdown
Contributor Author

Hi @heachary Thanks for the PR! Due to the relative smaller size of dsv4.1 flash, DEP8 is not a realistic config when deploying for production, can you also test the perf with DEP2 and DEP4?

@Fangzhou-Ai, thanks for the review. Tested DEP2 and DEP4, here are the results:

DEP4 (TP1 x DP4 x EP4)

Metric Baseline With patch Delta
Output token throughput (tok/s) 1511.38 1689.81 +11.8%
Total token throughput (tok/s) 13602.43 15208.28 +11.8%
Median TPOT (ms) 40.29 36.11 -10.4%
Mean TPOT (ms) 39.68 35.44 -10.7%
Median ITL (ms) 33.86 30.40 -10.2%
Mean TTFT (ms) 2706.62 2474.43 -8.6%
Median TTFT (ms) 1990.54 1840.41 -7.5%
Median E2EL (ms) 43199.28 38666.46 -10.5%
Benchmark duration (s) 173.45 155.13 -10.6%

DEP2 (TP1 x DP2 x EP2)

Metric Baseline With patch Delta
Output token throughput (tok/s) 1515.65 1645.83 +8.6%
Total token throughput (tok/s) 13640.83 14812.48 +8.6%
Median TPOT (ms) 40.18 36.97 -8.0%
Mean TPOT (ms) 38.78 35.72 -7.9%
Median ITL (ms) 27.37 25.28 -7.7%
Mean TTFT (ms) 3467.17 3194.15 -7.9%
Median TTFT (ms) 1872.13 1725.74 -7.8%
Median E2EL (ms) 42983.44 39589.57 -7.9%
Benchmark duration (s) 172.96 159.28 -7.9%

Also updated these details in the PR desc.

The previous commit restricted the dispatch to SILU, which also turned off
the one SiTU configuration that does want MXFP8: AITER_SITUV2_A8W4 resolves
to fp8 and has its own tuned table (kimik3_a8w4_tuned_fmoe.csv).

Pass the activation and gate mode AiterExperts will pass instead. The
activation selects which input decides the dtype -- SiTUv2 reads
AITER_SITUV2_*, Silu/Swiglu on INTERLEAVE reads AITER_BF16_FP8_MOE_BOUND --
so querying with a stand-in reads a knob that may say nothing about the
model, which is what let SiTU a4w4 through. Resolved against AITER:

  SILU   + INTERLEAVE, bound=0        -> fp8     dispatch
  Swiglu + INTERLEAVE, bound=0        -> fp8     dispatch
  Situv2 + INTERLEAVE, SITUV2_A8W4=1  -> fp8     dispatch
  Situv2 + SEPARATED,  SITUV2_A4W4=1  -> fp4x2   bf16
  Situv2 + SEPARATED,  a16w4          -> bf16    bf16
  SILU   + SEPARATED                  -> fp4x2   bf16
  SILU   + INTERLEAVE, default bound   -> None    bf16

Activations outside the set AiterExperts accepts, and the DeepSeek-V4.1 a4w4
opt-in that overrides q_dtype_a on the fused_moe call, still fail closed.

Signed-off-by: Hemanth Acharya <heachary@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

@bnellnm bnellnm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but a pytest level test would be nice if there are not already existing tests in tests/kernels/moe

The dispatch is only safe where AITER would have produced mxfp8 itself,
so the gate asks resolve_activation_dtype() with the activation and gate
mode AiterExperts will pass. Pin that against AITER directly: each case
asserts both the dtype AITER resolves and that the gate follows it, which
keeps AITER changing its mind separable from the gate drifting away from
it. The table covers silu/gelu/swiglu on the pinned bound, the unpinned
bound where AITER defers for want of a token count, and all three SiTUv2
modes -- the disagreement that reached the fp4 branch before.

The rest covers what follows from the gate: each guard turning the
DeepSeek-V4.1-Flash config off on its own, the mori buffer args (fp8
payload, hidden/32 scales, one byte each rather than fp32), and prepare()
emitting fp8 with e8m0 scales that dequantize back to within one e4m3
step of the group scale.

GPU-free except for the quant itself, which runs on gfx950 where the
resolver can answer fp8 at all.

Requested-by: bnellnm <bnellnm@users.noreply.github.com>
Signed-off-by: Hemanth Acharya <heachary@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@heachary

Copy link
Copy Markdown
Contributor Author

LGTM but a pytest level test would be nice if there are not already existing tests in tests/kernels/moe

@bnellnm thanks — added tests/kernels/moe/test_mori_mxfp8_dispatch.py (19 tests) in the latest commit.

The core of it pins the gate against AITER itself: each case asserts both the dtype resolve_activation_dtype() returns and that mori_use_mxfp8_dispatch() follows it, so AITER changing its mind stays distinguishable from the gate drifting away from it. The table covers silu/gelu/swiglu on the pinned FP8 bound, the unpinned bound where AITER defers for want of a token count, and all three SiTUv2 modes — the a4w4/a16w4 disagreement @okorzh-amd caught above is now a regression test.

The rest covers what follows from the gate:

  • each guard (w4a4, fp8-block, the DSv4 a4w4 opt-in, a hidden dim not divisible by 32, an activation AiterExperts rejects) turning the DeepSeek-V4.1-Flash config off on its own;
  • the MoRI buffer args — FP8 payload, hidden/32 scales, one byte each rather than the fp32 the per-token/per-tensor paths use — plus the BF16 fallback carrying no scales;
  • prepare() emitting FP8 with E8M0 scales that dequantize back to within one E4M3 step of the group scale, defer_input_quant passing BF16 through untouched, and the three-way mutual-exclusion assert.

Co-authored-by: Cursor <cursoragent@cursor.com>
@heachary
heachary force-pushed the heachary/dsv41_flash/mori-mxfp8-dispatch branch from 3cf4ff7 to 63ee39d Compare October 26, 2026 04:56
@heachary
heachary requested a review from bnellnm October 26, 2026 05:11
@Rohan138 Rohan138 added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 27, 2026
@github-actions

Copy link
Copy Markdown

✅ @heachary, CI is now available for this PR.

  • /ci run starts upstream CI; /amd-ci run starts AMD CI only.
  • Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
  • /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

heachary and others added 3 commits November 12, 2026 11:17
Signed-off-by: Hemanth Acharya <heachary@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Hemanth Acharya <heachary@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Hemanth Acharya <heachary@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@heachary
heachary requested a review from a team November 12, 2026 12:02
@mergify mergify Bot added the ci/build label Nov 12, 2026
@heachary

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #102127 for commit 5667a28cf9e8.

ℹ️ This PR is 91 commits behind upstream main. Merged upstream main into this branch as 5667a28cf9e8 before starting CI. Pull the branch before pushing more changes.

@heachary

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Queued 2 failed job(s) for retry in Buildkite CI #102127.

@heachary

Copy link
Copy Markdown
Contributor Author

/ci-retry

@heachary
heachary requested a review from okorzh-amd November 16, 2026 11:22

@fxmarty-amd fxmarty-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments, this is much nicer! Almost LGTM

My main remaining concerns are adding deferred_activation_key, adding checkpoint-independent logic in quantization/mxfp4.py that should rather be in the oracle and/or shared for all model producers, and --quantization-config.moe.activation null that should either work (see https://github.com/vllm-project/vllm/pull/67529/changes#diff-7626acc01163a9549f321f672f7b4d6395d555f5f3cdb9c2c3a6a9983b9a7246), either raise.

self.w13_precision_config = None
self.w2_precision_config = None

def _aiter_deferred_activation_key(self) -> QuantKey | None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not belong to vllm/model_executor/layers/quantization/mxfp4.py but to the MXFP4 oracle. There are different model producers that should strive to use a shared logic for activation quant key:

See the changes in #67529 to vllm/model_executor/layers/fused_moe/oracle/mxfp4.py.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactored this : The decision now lives in the oracle: make_mxfp4_moe_quant_config() sets deferred_activation_key from the MoE config and the activation key passed to backend selection, applying the user override the same way select_mxfp4_moe_backend() does. All MXFP4 MoE methods (Mxfp4MoEMethod, gpt-oss, Quark, online, compressed-tensors, INC) pass these through, so none of them carries its own copy of the logic.

Comment on lines +727 to +734
if _user_moe_activation_override_is_set():
logger.warning_once(
"%s experts cannot run MoE activation=%s in this configuration; "
"it is ignored and their inputs are not quantized before "
"dispatch.",
self.mxfp4_backend.value,
self.activation_key,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should raise an error. If what users request can't run, it should not go through.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shifted to an error instead of warning.

Comment on lines +292 to +295
# Quantization the experts apply to their own unquantized (``_a1``) input,
# e.g. MXFP8 for AITER's MXFP4 w4a16 kernels. An all-to-all may apply it
# before dispatch instead, which the experts then consume as is.
deferred_activation_key: QuantKey | None = None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to have @mgoin's review here.

@mergify

mergify Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @heachary.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 16, 2026
ningding01 pushed a commit to ningding01/vllm that referenced this pull request Nov 17, 2026
Cherry-picked from vllm-project#61962.

MXFP4 W4A16 experts declare BF16 activations, so MoRI dispatches raw BF16 and
AITER quantizes to group-32 mxfp8 on arrival. Quantizing in prepare() moves
that to once per token on the rank that owns it and roughly halves the
dispatch payload. AITER's stage-1 path then sorts the E8M0 scales instead of
running its own quant.

Signed-off-by: sabreshao <sabre.shao@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit 75e330e)
…mori-mxfp8-dispatch

Signed-off-by: Hemanth Acharya <heachary@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	vllm/models/deepseek_v41/quant_config.py
@mergify mergify Bot removed the needs-rebase label Nov 17, 2026

@fxmarty-amd fxmarty-amd left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if you can address the above

Please see #68160 for context about my pushback, that can also clean things up afterwards!

@Fangzhou-Ai

Copy link
Copy Markdown
Collaborator

Hi @heachary thanks for the PR. A quick question as you PR mentioend "MXFP4 W4A16 experts declare BF16 activations" why don't we just use native a8w4 MoE in this case?

heachary and others added 2 commits November 17, 2026 17:32
Which activation quantization AITER_MXFP4_BF16 experts run depends on the
backend, the MoE config and the requested activation, not on the checkpoint
format, so it now lives in oracle/mxfp4.py instead of Mxfp4MoEMethod.
make_mxfp4_moe_quant_config sets deferred_activation_key when given the
MoE config and the activation key passed to backend selection, applying the
user override as select_mxfp4_moe_backend does.

Every MXFP4 MoE method passes them: Mxfp4MoEMethod, GptOssMxfp4MoEMethod,
Quark OCP MX, online MXFP4 (and FP8 re-quant through it), compressed-tensors
and INC W4A4. Only DeepSeek-V4/V4.1 default to MXFP8; the others dispatch
MXFP8 when the user requests moe.activation=mxfp8 on those experts.

Requested-by: fxmarty-amd <fxmarty-amd@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Hemanth Acharya <heachary@amd.com>
An explicit quantization_config.moe.activation that the experts do not
compute with now raises instead of being ignored with a warning: MXFP8 is
what the interleaved w4a16 experts compute with, MXFP4 the DeepSeek-V4.1
a4w4 opt-in, and SiTU whatever VLLM_ROCM_USE_AITER_MOE_SITUV2 selects. A
model default that does not apply still falls back to BF16 dispatch.

Requested-by: fxmarty-amd <fxmarty-amd@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Hemanth Acharya <heachary@amd.com>
@fxmarty-amd

Copy link
Copy Markdown
Contributor

afaik #66564 adds AITER_MXFP4_MXFP8 MOE backend, but so far this was hacked into AITER_MXFP4_BF16 through

os.environ["AITER_BF16_FP8_MOE_BOUND"] = "0"

@heachary

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #103744 for commit a33e71f704f3.

ℹ️ This PR is 43 commits behind upstream main. Merged upstream main into this branch as a33e71f704f3 before starting CI. Pull the branch before pushing more changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build deepseek Related to DeepSeek models quantization ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

7 participants