Repository navigation
[Online quantization] Deprecate VLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT/VLLM_ROCM_MUSE_MOE_ONLINE_QUANT in favor of standard online quantization API - #67529
Draft
fxmarty-amd wants to merge 18 commits into
Conversation
…t env vars with online quantization Remove VLLM_FP8_MOE_REQUANT, VLLM_FP8_MOE_REQUANT_ACT and VLLM_ROCM_MUSE_MOE_ONLINE_QUANT. Block-FP8 experts are requantized by Mxfp4OnlineMoEMethod with Fp8MoEMethod as the requantization source (new Fp8MoEMethod.dequantize_weight), and Muse Spark online MXFP4 experts subclass Mxfp4OnlineMoEMethod; both are selected with --quantization-config. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
fxmarty-amd
temporarily deployed
to
recipes-preview
November 13, 2026 17:43 — with
GitHub Actions
Inactive
Contributor
VLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT/VLLM_ROCM_MUSE_MOE_ONLINE_QUANT with standard online quantization API
…v vars, share block-FP8 dequantization Keep VLLM_FP8_MOE_REQUANT[_ACT] and VLLM_ROCM_MUSE_MOE_ONLINE_QUANT as deprecated aliases redirected to the equivalent --quantization-config with a warning (removal in v0.35.0). Move the block-FP8 MoE dequantization to fp8_utils and plug it into the fp8, compressed-tensors and Quark MoE methods. Document block-FP8 MoE requantization in the online quantization docs. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
fxmarty-amd
temporarily deployed
to
recipes-preview
November 13, 2026 18:01 — with
GitHub Actions
Inactive
1 task done
Contributor
|
Documentation preview: https://vllm--67529.org.readthedocs.build/en/67529/ |
github-actions Bot
added a commit
to vllm-project/recipes
that referenced
this pull request
Nov 13, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…tions through AITER_BF16_FP8_MOE_BOUND activation: mxfp8 selects AITER_MXFP4_BF16 with bound 0 (MXFP8 activations for any token count) and activation: null sets a very high bound (bf16 activations); other requests keep the previous bound with a TODO. Map VLLM_FP8_MOE_REQUANT_ACT a8w4/a16w4 to mxfp8/null, make the legacy redirect set model_config.quantization_config, and use mxfp8 in the Mistral Large 4 recipe. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
fxmarty-amd
temporarily deployed
to
recipes-preview
November 13, 2026 18:21 — with
GitHub Actions
Inactive
github-actions Bot
added a commit
to vllm-project/recipes
that referenced
this pull request
Nov 13, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
fxmarty-amd
commented
Nov 13, 2026
| @@ -0,0 +1,62 @@ | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
Contributor
Author
There was a problem hiding this comment.
This is to keep backward compatibility with VLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT/VLLM_ROCM_MUSE_MOE_ONLINE_QUANT, but I could just remove them from envs.py if this is prefered.
…_STRICT_QUANT_SCHEME and address self-review Serve activation: mxfp8 with AITER_MXFP4_BF16 on gfx950 only, forbid adding activation quantization to weight-only requests in strict mode, derive the AITER bf16/fp8 bound from the layer's resolved activation, record the FP8 scale block shape at weight creation, restrict the ML4 specialized-decode check to moe, and document strict mode for the AITER MXFP4 kernels. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
fxmarty-amd
temporarily deployed
to
recipes-preview
November 13, 2026 18:55 — with
GitHub Actions
Inactive
Add a ParallelGroup protocol and an override_parallel_groups() context manager (usable as a decorator) to parallel_state, so tests can swap in a single-rank fake without monkeypatching private module globals. Use it in test_models_fse_init instead of the single_rank_parallel_state fixture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
Fuse the dist/GPU test cases into single tests that use override_parallel_groups() instead of dist_init, vectorize the block-FP8 dequantization reference, and skip the global cleanup for CPU-only tests. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
fxmarty-amd
temporarily deployed
to
recipes-preview
November 13, 2026 19:38 — with
GitHub Actions
Inactive
fxmarty-amd
commented
Nov 13, 2026
| return _PCP | ||
|
|
||
|
|
||
| _GROUP_NAMES = ("_WORLD", "_TP", "_ETP", "_DCP", "_PP", "_DP", "_EP", "_PCP") |
Contributor
Author
There was a problem hiding this comment.
Courtesy of tomasruizt@dbc36f0 for faster tests.
May be submitted standalone if you prefer
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
VLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT/VLLM_ROCM_MUSE_MOE_ONLINE_QUANT with standard online quantization APIVLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT/VLLM_ROCM_MUSE_MOE_ONLINE_QUANT in favor of standard online quantization API
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…layer Create the weights of a requantization source on the meta device and let the layerwise loader materialize and requantize one layer at a time, so the peak load memory is the quantized model plus one layer of serialized weights instead of the whole serialized model. Extend test_online_quant_peak_mem to a requantization case. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
6 tasks done
fxmarty-amd
temporarily deployed
to
recipes-preview
November 16, 2026 19:55 — with
GitHub Actions
Inactive
6 tasks done
github-actions Bot
added a commit
to vllm-project/recipes
that referenced
this pull request
Nov 16, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 task done
These tests only build layers with a single-rank parallel-group fake, so the per-test distributed and memory cleanup only added about 0.35 s each (about 100 s in total). Signed-off-by: Felix Marty <Felix.Marty@amd.com>
fxmarty-amd
temporarily deployed
to
recipes-preview
November 16, 2026 20:34 — with
GitHub Actions
Inactive
…ad code, env conflict check Set AITER_BF16_FP8_MOE_BOUND from the DeepSeek-V4 selector too (it no longer got the value from the weight conversion), remove unreachable code after a return in get_quant_config, only reject VLLM_FP8_MOE_REQUANT=mxfp4 together with --quantization-config, warn that VLLM_FP8_MOE_REQUANT_ACT=a16w4 no longer selects the Triton backend, and fix the nested JSON in a recipe comment. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…ect the Muse env var with --quantization-config AITER reads AITER_BF16_FP8_MOE_BOUND from the environment, so once a layer asks for bf16 activations (activation: null) later layers no longer take it back. VLLM_ROCM_MUSE_MOE_ONLINE_QUANT combined with --quantization-config is now an error, like VLLM_FP8_MOE_REQUANT. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…zation Signed-off-by: Felix Marty <Felix.Marty@amd.com> # Conflicts: # tests/quantization/test_online.py
…ethod's resolved activation The online MXFP4 method now passes its resolved activation (targets spec, deprecated env vars) to the bound setter instead of the setter reading the global quantization_config.moe, which is unset for both. The specialized-decode error message names the backend only. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…olved spec scope Replace the activation_explicit argument of select_mxfp4_moe_backend: OnlineQuantizationConfig.get_quant_method builds the online method inside layer_target_spec_scope(quant_spec), so the oracles read the layer's resolved activation (targets, moe, or the deprecated env vars) through user_activation_override like for a global quantization_config.moe. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
The oracle sets this process-wide variable at backend selection, and an explicit bf16 request is kept for the process: start every test of the two modules from AITER's default and restore the variable afterwards with an autouse fixture instead of per-test setenv calls. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
fxmarty-amd
temporarily deployed
to
recipes-preview
November 16, 2026 21:18 — with
GitHub Actions
Inactive
github-actions Bot
added a commit
to vllm-project/recipes
that referenced
this pull request
Nov 16, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Keep the single-rank fake and override_parallel_groups() in tests/single_rank_group.py instead of vllm/distributed/parallel_state.py, which is no longer touched by this PR. Reset the AITER state of test_models_fse_init in a finalizer so it also runs when a case fails, and say in the specialized-decode presets that the experts are checked once the weights are loaded. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…TP monkeypatches Replace the dist_init fixture of the online quantization tests and the hand-written get_tensor_model_parallel_rank / world_size patches (rank 0, world size 1) with override_parallel_groups(SingleRankGroup()). The TP2 loading test keeps its patches. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
github-actions Bot
added a commit
to vllm-project/recipes
that referenced
this pull request
Nov 16, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
fxmarty-amd
marked this pull request as draft
November 17, 2026 12:19
xaguilar-amd
added a commit
to xaguilar-amd/vllm
that referenced
this pull request
Nov 17, 2026
…on-config Select the block-FP8 to int4 expert requant with --quantization-config.moe.weight int4_per_group_32, the flag Kimi-K3 already uses for the same int4 format and kernel, instead of a new VLLM_FP8_MOE_REQUANT value: that env var is being deprecated in favor of --quantization-config (vllm-project#67529). The method refuses FP8 experts without block scales and non-ROCm platforms, and the flag conflicts with VLLM_FP8_MOE_REQUANT=mxfp4. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Xavier Aguilar <xavier.aguilarfruto@amd.com>
6 tasks done
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR will be broken in sub-PRs:
tests/single_rank_group.py),VLLM_STRICT_QUANT_SCHEMEsupport forAITER_MXFP4_BF16backend,VLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT/VLLM_ROCM_MUSE_MOE_ONLINE_QUANTdeprecation in favor of standard online quantization APIDisclosure
AI assistance was used. The changes were reviewed and tested manually.
Overview
This PR deprecates
VLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT/VLLM_ROCM_MUSE_MOE_ONLINE_QUANTin favor of standard online quantization API through--quantization-config, using https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization/online.This PR is inline with the online quantization extension RFC: #52167
Replacements:
VLLM_FP8_MOE_REQUANT="mxfp4" VLLM_FP8_MOE_REQUANT_ACT="a4w4"=>--quantization-config.moe mxfp4VLLM_FP8_MOE_REQUANT="mxfp4" VLLM_FP8_MOE_REQUANT_ACT="a8w4"=>--quantization-config.moe='{"weight":"mxfp4","activation":"mxfp8"}'This PR also makes it so that
--quantization-config.moe.activation nulltruly abides by BF16 requested activation (no silent override), integrating withVLLM_STRICT_QUANT_SCHEMEfrom #59460.Two ROCm-only env vars (
VLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT, added with Mistral Large 4 in #64534) andVLLM_ROCM_MUSE_MOE_ONLINE_QUANT(added with Muse Spark in #61879) implemented routed-expert MXFP4, now supported through--quantization-configafter #64378. This PR deletes the bespoke code behind them and routes both models through the online quantization infra. The three env vars are kept, redirect to the equivalent--quantization-config, and log awarning_onceannouncing their removal in v0.35.0.Block-FP8 checkpoints are requantized by the generic
Mxfp4OnlineMoEMethodwith the checkpoint MoE method as the requantization source. For that,Fp8MoEMethod,CompressedTensorsW8A8Fp8MoEMethodandQuarkW8A8Fp8MoEMethodget adequantize_weightbuilt on a shareddequantize_block_fp8_moe_weightshelper. Muse Spark's online MXFP4 experts subclassMxfp4OnlineMoEMethodand keep the route-preserving AITER experts.Not a duplicate: it builds on the requantization-source mechanism and removes the parallel env-var paths; no open PR removes them.
Claims
online/fp8_requant.py(198 lines,Fp8BlockExpertsAsMxfp4FusedMoE) is deleted.fp8, compressed-tensors and Quark methods can be requantized by any online MoE method, not only MXFP4 on ROCm (partial trailing blocks and w1/w3 shards handled); per-tensor / per-channel FP8 raisesNotImplementedError.VLLM_FP8_MOE_REQUANT=mxfp4witha4w4is--quantization-config.moe mxfp4, witha8w4(default) it is--quantization-config.moe='{"weight":"mxfp4","activation":"mxfp8"}', witha16w4--quantization-config.moe='{"weight":"mxfp4","activation":null}';VLLM_ROCM_MUSE_MOE_ONLINE_QUANT=mxfp4adds"ignore":["*layers.0.*","*layers.<N-1>.*"]. As before, the FP8 vars only act on block-FP8 checkpoints on ROCm.AITER_MXFP4_BF16serveactivation: mxfp8(gfx950 only, integrated withVLLM_STRICT_QUANT_SCHEME, documented indocs/features/quantization/README.md) and setsAITER_BF16_FP8_MOE_BOUNDat backend selection from the layer's request (targetsincluded):mxfp8->0(MXFP8 activations for any token count),null->2**31 - 1(bf16 activations for any token count).--quantization-configwith the first and last layers excluded throughignore.Behavior changes vs
main:VLLM_FP8_MOE_REQUANT=mxfp4together with--quantization-configis an error (the FP8 env vars apply only when no--quantization-configis given).AITER_BF16_FP8_MOE_BOUNDis set at backend selection by bothselect_mxfp4_moe_backendandselect_deepseek_v4_mxfp4_moe_backend, for every selection ofAITER_MXFP4_BF16(the DeepSeek-V4 checkpoint path included: it used to get the0from the weight conversion). An explicitactivation: nullon any layer is kept for the whole process.VLLM_ROCM_MUSE_MOE_ONLINE_QUANT=mxfp4together with--quantization-configis an error.AITER_BF16_FP8_MOE_BOUNDwas unconditionally set to0forAITER_MXFP4_BF16, i.e. MXFP8 activations even when the backend was selected for bf16 activations. An explicitactivation: nullnow gets bf16 activations (bound2**31 - 1); any other request keeps0(bf16 underVLLM_STRICT_QUANT_SCHEME=1, which forbids adding activation quantization to a weight-only request), with a TODO in the oracle (this silently overrides the model-requested activation key). AITER itself (resolve_activation_dtypeinaiter/fused_moe.py) only honors the bound for the interleaved kernels on gfx950 and uses bf16 activations elsewhere, soAITER_MXFP4_BF16is not selected foractivation: mxfp8off gfx950.VLLM_FP8_MOE_REQUANT_ACT=a16w4no longer selectsaiter_triton_mxfp4_bf16implicitly: it maps toactivation: null, i.e. bf16 activations on the interleavedAITER_MXFP4_BF16kernels.Fp8BlockExpertsAsMxfp4FusedMoEdid: the peak load memory is the quantized model plus one layer of serialized weights and the dequantization staging (Mistral-Large-4 at TP4, see the end-to-end table: 147.9 GiB peak versus 143.7 GiB for the removed class, with 135.7 GiB of final model memory).--quantization-configat startup; experts that are not on the AITER a4w4 MXFP4 backend are refused after weight loading (MonoUnsupported).Left to other PRs / TODO:
activation: null, a16w4) of the interleaved AITER kernels was only checked at the selection/weight-conversion level, not numerically;mxfp8runs the same kernel configuration as the removeda8w4. The two tests that requestactivation: nullon a tiny Granite MoE (mxfp4_null_activation_override,targets_activation_override) fail withUnsupported kernel config for moe heuristic dispatchon that backend, so they selectaiter_triton_mxfp4_bf16on ROCm; whether larger models (Mistral-Large-4) have kernels for it was not checked.AITER_BF16_FP8_MOE_BOUNDis a process-wide environment variable read by AITER on every call, so layers with differentactivationrequests (for example throughtargets) onAITER_MXFP4_BF16share one value (onlynullis kept, once requested). A per-call argument needs an AITER change.activation: null) wins over any other layer'smxfp8request,VLLM_STRICT_QUANT_SCHEME=1included, and it carries over to later engines built in the same process; nothing warns or raises when the requests differ.AITER_BF16_FP8_MOE_BOUNDis overwritten at backend selection without a log message.weight_scale_inv) is no longer reported as unloaded bydefault_loader.track_weights_loadingand the layer is processed with zeros. Online quantization of bf16 checkpoints has the same exemption; fixing it touches the layerwise loader.extra_argsand the specialized-decode modes both pass--quantization-config; the order the recipe builder passes them in was not verified.Validation
Unit tests
New / modified tests:
pytest -s -vvvvv \ tests/quantization/test_online.py \ tests/kernels/moe/test_moe_kernel_oracle.py \ tests/kernels/moe/test_muse_spark_rocm_bf16_moe.py \ tests/models/mistral/mistral_large4/amd_specialized_decode/test_runtime.py \ tests/model_executor/layers/test_fused_shared_expert.py \ -k "dequantize_weight or configures_prequantized_moe or requantizes_block_fp8 or deprecated or activation_bound or serves_mxfp8 or muse_online or quantization_config or refusal or test_models_fse_init or online_mxfp4 or test_online_quantization or test_online_quant_peak_mem"Related existing tests (unmodified):
Ran on MI355X (gfx950), ROCm (torch HIP 7.15.26333), vLLM commit db0a337, with
HF_HUB_CACHEpointing to a writable cache. The base of this PR has a bug with Mistral indexer cache, so the fix from a separate branch (#67984) was applied locally.End-to-end tests
Mistral-Large-4-1T-A52B (FP8 checkpoint) with routed experts requantized to MXFP4 at load, TP4, on this PR versus
main(the deprecated env vars). Peak memory is the per-GPU VRAM sampled at 1 Hz until the weights are loaded and requantized.This PR:
main:Online quantization logs shown only with this PR (rank 0 of the server log;
mainlogsFP8 block experts re-quantized to MXFP4 at load (a4w4, AITER_MXFP4_MXFP4)from the removed class instead):main(VLLM_FP8_MOE_REQUANT=mxfp4,VLLM_FP8_MOE_REQUANT_ACT=a4w4)--quantization-config.moe mxfp4)GSM8K results below are not accuracy claims, they are just used to test that there is nothing obviously wrong.
GSM8K Platinum against each server (single repeat), with
lm-evaluation-harnessat https://github.com/fxmarty-amd/lm-evaluation-harness/tree/503ad16e088dc9dea08e29f6df76dbb26129941b. The model thinks by default (reasoning_efforthigh), and the server runs--reasoning-parser mistral, which moves the thinking out of the message content, sothink_end_tokenis empty. Sampling is the model card's recommendation (temperature=1.0,top_p=0.95), no seed:LMEVAL_LOG_LEVEL=DEBUG lm_eval --model local-chat-completions \ --model_args model=mistralai/Mistral-Large-4-1T-A52B,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=128,timeout=99999,tokenized_requests=False,think_end_token='',server_sampling_defaults=True \ --tasks gsm8k_platinum \ --gen_kwargs max_gen_toks=8000,reasoning_effort=high,temperature=1.0,top_p=0.95 \ --apply_chat_template --repeats 1 \ --log_samples --output_path results/mistral_large_4_gsm8k_platinumGSM8K Platinum
exact_match, single repeat:main,VLLM_FP8_MOE_REQUANT=mxfp4,VLLM_FP8_MOE_REQUANT_ACT=a4w4--quantization-config.moe mxfp4Details
_pad_requantized_weights, so that restriction is gone.dequantize_block_fp8_moe_weightstakes the checkpointweight_block_size(as the compressed-tensors and Quark methods) and rejects a scale grid that does not match it, so FP8 scales refined under TP (weight_scale_refine) are not supported as a requantization source (they were not onmaineither); this is the part to scrutinise. Only thefp8method is covered by a real load (test_online_mxfp4_moe_requantizes_block_fp8_checkpoint); compressed-tensors and Quark are covered by the numerics test with synthetic state.experts_clsafterMxfp4OnlineMoEMethod.__init__, so oracle selection runs first and must land onAITER_MXFP4_MXFP4, otherwise it raises.docs/features/quantization/online.mdgets the block-FP8 MoE source row.tests/single_rank_group.py(SingleRankGroupandoverride_parallel_groups(), adapted from tomasruizt@dbc36f0 and kept out ofvllm/distributed) lets the new dist/GPU tests use a single-rank fake instead ofdist_init(these tests went from 71s to 8s).