Skip to content

[Online quantization] Deprecate VLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT/VLLM_ROCM_MUSE_MOE_ONLINE_QUANT in favor of standard online quantization API - #67529

Draft
fxmarty-amd wants to merge 18 commits into
vllm-project:mainfrom
fxmarty-amd:cleanup-online-quantization

Conversation

@fxmarty-amd

@fxmarty-amd fxmarty-amd commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

This PR will be broken in sub-PRs:

Disclosure

AI assistance was used. The changes were reviewed and tested manually.

Overview

This PR deprecates VLLM_FP8_MOE_REQUANT / VLLM_FP8_MOE_REQUANT_ACT / VLLM_ROCM_MUSE_MOE_ONLINE_QUANT in favor of standard online quantization API through --quantization-config, using https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization/online.

This PR is inline with the online quantization extension RFC: #52167

Replacements:

  • VLLM_FP8_MOE_REQUANT="mxfp4" VLLM_FP8_MOE_REQUANT_ACT="a4w4" => --quantization-config.moe mxfp4
  • VLLM_FP8_MOE_REQUANT="mxfp4" VLLM_FP8_MOE_REQUANT_ACT="a8w4" => --quantization-config.moe='{"weight":"mxfp4","activation":"mxfp8"}'

This PR also makes it so that --quantization-config.moe.activation null truly abides by BF16 requested activation (no silent override), integrating with VLLM_STRICT_QUANT_SCHEME from #59460.

Two ROCm-only env vars (VLLM_FP8_MOE_REQUANT / VLLM_FP8_MOE_REQUANT_ACT, added with Mistral Large 4 in #64534) and VLLM_ROCM_MUSE_MOE_ONLINE_QUANT (added with Muse Spark in #61879) implemented routed-expert MXFP4, now supported through --quantization-config after #64378. This PR deletes the bespoke code behind them and routes both models through the online quantization infra. The three env vars are kept, redirect to the equivalent --quantization-config, and log a warning_once announcing their removal in v0.35.0.

Block-FP8 checkpoints are requantized by the generic Mxfp4OnlineMoEMethod with the checkpoint MoE method as the requantization source. For that, Fp8MoEMethod, CompressedTensorsW8A8Fp8MoEMethod and QuarkW8A8Fp8MoEMethod get a dequantize_weight built on a shared dequantize_block_fp8_moe_weights helper. Muse Spark's online MXFP4 experts subclass Mxfp4OnlineMoEMethod and keep the route-preserving AITER experts.

Not a duplicate: it builds on the requantization-source mechanism and removes the parallel env-var paths; no open PR removes them.

Claims

  • online/fp8_requant.py (198 lines, Fp8BlockExpertsAsMxfp4FusedMoE) is deleted.
  • Block-FP8 experts of the fp8, compressed-tensors and Quark methods can be requantized by any online MoE method, not only MXFP4 on ROCm (partial trailing blocks and w1/w3 shards handled); per-tensor / per-channel FP8 raises NotImplementedError.
  • Block-FP8 experts requantized to MXFP4 are byte-identical to online-quantizing the same dequantized bf16 experts (AITER backend).
  • The deprecated env vars map to config: VLLM_FP8_MOE_REQUANT=mxfp4 with a4w4 is --quantization-config.moe mxfp4, with a8w4 (default) it is --quantization-config.moe='{"weight":"mxfp4","activation":"mxfp8"}', with a16w4 --quantization-config.moe='{"weight":"mxfp4","activation":null}'; VLLM_ROCM_MUSE_MOE_ONLINE_QUANT=mxfp4 adds "ignore":["*layers.0.*","*layers.<N-1>.*"]. As before, the FP8 vars only act on block-FP8 checkpoints on ROCm.
  • The MXFP4 oracle lets AITER_MXFP4_BF16 serve activation: mxfp8 (gfx950 only, integrated with VLLM_STRICT_QUANT_SCHEME, documented in docs/features/quantization/README.md) and sets AITER_BF16_FP8_MOE_BOUND at backend selection from the layer's request (targets included): mxfp8 -> 0 (MXFP8 activations for any token count), null -> 2**31 - 1 (bf16 activations for any token count).
  • Muse Spark online MXFP4 is selected by --quantization-config with the first and last layers excluded through ignore.

Behavior changes vs main:

  • Setting VLLM_FP8_MOE_REQUANT=mxfp4 together with --quantization-config is an error (the FP8 env vars apply only when no --quantization-config is given).
  • AITER_BF16_FP8_MOE_BOUND is set at backend selection by both select_mxfp4_moe_backend and select_deepseek_v4_mxfp4_moe_backend, for every selection of AITER_MXFP4_BF16 (the DeepSeek-V4 checkpoint path included: it used to get the 0 from the weight conversion). An explicit activation: null on any layer is kept for the whole process.
  • VLLM_ROCM_MUSE_MOE_ONLINE_QUANT=mxfp4 together with --quantization-config is an error.
  • AITER_BF16_FP8_MOE_BOUND was unconditionally set to 0 for AITER_MXFP4_BF16, i.e. MXFP8 activations even when the backend was selected for bf16 activations. An explicit activation: null now gets bf16 activations (bound 2**31 - 1); any other request keeps 0 (bf16 under VLLM_STRICT_QUANT_SCHEME=1, which forbids adding activation quantization to a weight-only request), with a TODO in the oracle (this silently overrides the model-requested activation key). AITER itself (resolve_activation_dtype in aiter/fused_moe.py) only honors the bound for the interleaved kernels on gfx950 and uses bf16 activations elsewhere, so AITER_MXFP4_BF16 is not selected for activation: mxfp8 off gfx950.
  • VLLM_FP8_MOE_REQUANT_ACT=a16w4 no longer selects aiter_triton_mxfp4_bf16 implicitly: it maps to activation: null, i.e. bf16 activations on the interleaved AITER_MXFP4_BF16 kernels.
  • Requantization sources (block-FP8 and the other serialized checkpoints, MoE and linear) are created on the meta device and materialized one layer at a time by the layerwise loader, as Fp8BlockExpertsAsMxfp4FusedMoE did: the peak load memory is the quantized model plus one layer of serialized weights and the dequantization staging (Mistral-Large-4 at TP4, see the end-to-end table: 147.9 GiB peak versus 143.7 GiB for the removed class, with 135.7 GiB of final model memory).
  • Mistral Large 4 specialized decode no longer reads the env vars nor inspects --quantization-config at startup; experts that are not on the AITER a4w4 MXFP4 backend are refused after weight loading (MonoUnsupported).

Left to other PRs / TODO:

  • The bf16-activation path (activation: null, a16w4) of the interleaved AITER kernels was only checked at the selection/weight-conversion level, not numerically; mxfp8 runs the same kernel configuration as the removed a8w4. The two tests that request activation: null on a tiny Granite MoE (mxfp4_null_activation_override, targets_activation_override) fail with Unsupported kernel config for moe heuristic dispatch on that backend, so they select aiter_triton_mxfp4_bf16 on ROCm; whether larger models (Mistral-Large-4) have kernels for it was not checked.
  • The oracle still silently overrides the model-requested activation key in the remaining cases (TODO in the code).
  • AITER_BF16_FP8_MOE_BOUND is a process-wide environment variable read by AITER on every call, so layers with different activation requests (for example through targets) on AITER_MXFP4_BF16 share one value (only null is kept, once requested). A per-call argument needs an AITER change.
  • Because of that sharing, a bf16 request (activation: null) wins over any other layer's mxfp8 request, VLLM_STRICT_QUANT_SCHEME=1 included, and it carries over to later engines built in the same process; nothing warns or raises when the requests differ.
  • A user-exported AITER_BF16_FP8_MOE_BOUND is overwritten at backend selection without a log message.
  • Requantization sources are on the meta device, so a missing checkpoint weight (for example one weight_scale_inv) is no longer reported as unloaded by default_loader.track_weights_loading and the layer is processed with zeros. Online quantization of bf16 checkpoints has the same exemption; fixing it touches the layerwise loader.
  • Recipe: the AMD FP8 variant's extra_args and the specialized-decode modes both pass --quantization-config; the order the recipe builder passes them in was not verified.

Validation

Unit tests

New / modified tests:

pytest -s -vvvvv \
  tests/quantization/test_online.py \
  tests/kernels/moe/test_moe_kernel_oracle.py \
  tests/kernels/moe/test_muse_spark_rocm_bf16_moe.py \
  tests/models/mistral/mistral_large4/amd_specialized_decode/test_runtime.py \
  tests/model_executor/layers/test_fused_shared_expert.py \
  -k "dequantize_weight or configures_prequantized_moe or requantizes_block_fp8 or deprecated or activation_bound or serves_mxfp8 or muse_online or quantization_config or refusal or test_models_fse_init or online_mxfp4 or test_online_quantization or test_online_quant_peak_mem"
214 passed, 9 skipped, 386 deselected, 16 warnings in 261.85s (0:04:21)

Related existing tests (unmodified):

pytest -s -vvvvv \
  tests/quantization/test_online_mxfp4.py \
  tests/quantization/test_online_shared_expert_fusion.py \
  tests/quantization/test_keep_param_objects.py \
  tests/quantization/test_gfx950_moe.py
68 passed, 46 skipped, 16 warnings in 73.16s (0:01:13)

Ran on MI355X (gfx950), ROCm (torch HIP 7.15.26333), vLLM commit db0a337, with HF_HUB_CACHE pointing to a writable cache. The base of this PR has a bug with Mistral indexer cache, so the fix from a separate branch (#67984) was applied locally.

End-to-end tests

Mistral-Large-4-1T-A52B (FP8 checkpoint) with routed experts requantized to MXFP4 at load, TP4, on this PR versus main (the deprecated env vars). Peak memory is the per-GPU VRAM sampled at 1 Hz until the weights are loaded and requantized.

This PR:

VLLM_ROCM_USE_AITER=1 \
vllm serve mistralai/Mistral-Large-4-1T-A52B \
  --tensor-parallel-size 4 \
  --tokenizer-mode mistral --config-format mistral --load-format mistral \
  --max-model-len 262144 --max-num-batched-tokens 8192 --max-num-seqs 128 \
  --quantization-config.moe mxfp4 \
  --reasoning-parser mistral

main:

VLLM_ROCM_USE_AITER=1 \
VLLM_FP8_MOE_REQUANT=mxfp4 VLLM_FP8_MOE_REQUANT_ACT=a4w4 \
vllm serve mistralai/Mistral-Large-4-1T-A52B \
  --tensor-parallel-size 4 \
  --tokenizer-mode mistral --config-format mistral --load-format mistral \
  --max-model-len 262144 --max-num-batched-tokens 8192 --max-num-seqs 128 \
  --reasoning-parser mistral

Online quantization logs shown only with this PR (rank 0 of the server log; main logs FP8 block experts re-quantized to MXFP4 at load (a4w4, AITER_MXFP4_MXFP4) from the removed class instead):

(Worker_TP0 pid=81933) INFO 11-16 19:26:35 [base_loader.py:164] Quantizing 71 layers of types: mlp.experts: 71 (from moe: mxfp4)
(Worker_TP0 pid=81933) INFO 11-16 19:27:35 [base_loader.py:177] Online quantization as part of model loading took 1.22 seconds
Path Peak VRAM per GPU during load Model memory after loading Loading time
main (VLLM_FP8_MOE_REQUANT=mxfp4, VLLM_FP8_MOE_REQUANT_ACT=a4w4) 143.7 GiB 135.73 GiB 59.1 s
This PR (--quantization-config.moe mxfp4) 147.9 GiB 135.75 GiB 63.2 s

GSM8K results below are not accuracy claims, they are just used to test that there is nothing obviously wrong.

GSM8K Platinum against each server (single repeat), with lm-evaluation-harness at https://github.com/fxmarty-amd/lm-evaluation-harness/tree/503ad16e088dc9dea08e29f6df76dbb26129941b. The model thinks by default (reasoning_effort high), and the server runs --reasoning-parser mistral, which moves the thinking out of the message content, so think_end_token is empty. Sampling is the model card's recommendation (temperature=1.0, top_p=0.95), no seed:

LMEVAL_LOG_LEVEL=DEBUG lm_eval --model local-chat-completions \
  --model_args model=mistralai/Mistral-Large-4-1T-A52B,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=128,timeout=99999,tokenized_requests=False,think_end_token='',server_sampling_defaults=True \
  --tasks gsm8k_platinum \
  --gen_kwargs max_gen_toks=8000,reasoning_effort=high,temperature=1.0,top_p=0.95 \
  --apply_chat_template --repeats 1 \
  --log_samples --output_path results/mistral_large_4_gsm8k_platinum

GSM8K Platinum exact_match, single repeat:

Case Result
main, VLLM_FP8_MOE_REQUANT=mxfp4, VLLM_FP8_MOE_REQUANT_ACT=a4w4 0.9934 ± 0.0023
this PR with --quantization-config.moe mxfp4 0.9934 ± 0.0023

Details

  • The old class required the FP8 checkpoint's expert dims to be unpadded and raised otherwise; the infrastructure pads in _pad_requantized_weights, so that restriction is gone.
  • dequantize_block_fp8_moe_weights takes the checkpoint weight_block_size (as the compressed-tensors and Quark methods) and rejects a scale grid that does not match it, so FP8 scales refined under TP (weight_scale_refine) are not supported as a requantization source (they were not on main either); this is the part to scrutinise. Only the fp8 method is covered by a real load (test_online_mxfp4_moe_requantizes_block_fp8_checkpoint); compressed-tensors and Quark are covered by the numerics test with synthetic state.
  • Muse: the Muse subclass overrides experts_cls after Mxfp4OnlineMoEMethod.__init__, so oracle selection runs first and must land on AITER_MXFP4_MXFP4, otherwise it raises.
  • Docs: docs/features/quantization/online.md gets the block-FP8 MoE source row.
  • tests/single_rank_group.py (SingleRankGroup and override_parallel_groups(), adapted from tomasruizt@dbc36f0 and kept out of vllm/distributed) lets the new dist/GPU tests use a single-rank fake instead of dist_init (these tests went from 71s to 8s).

…t env vars with online quantization

Remove VLLM_FP8_MOE_REQUANT, VLLM_FP8_MOE_REQUANT_ACT and VLLM_ROCM_MUSE_MOE_ONLINE_QUANT. Block-FP8 experts are requantized by Mxfp4OnlineMoEMethod with Fp8MoEMethod as the requantization source (new Fp8MoEMethod.dequantize_weight), and Muse Spark online MXFP4 experts subclass Mxfp4OnlineMoEMethod; both are selected with --quantization-config.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
@mergify

mergify Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

@mergify mergify Bot added recipe mistral Related to Mistral models quantization rocm Related to AMD ROCm labels Nov 13, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Nov 13, 2026
@fxmarty-amd fxmarty-amd changed the title [Quantization][ROCm] Replace the FP8 MoE requant and Muse online-quant env vars with online quantization [Online quantization] Replace VLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT/VLLM_ROCM_MUSE_MOE_ONLINE_QUANT with standard online quantization API Nov 13, 2026
…v vars, share block-FP8 dequantization

Keep VLLM_FP8_MOE_REQUANT[_ACT] and VLLM_ROCM_MUSE_MOE_ONLINE_QUANT as deprecated aliases redirected to the equivalent --quantization-config with a warning (removal in v0.35.0). Move the block-FP8 MoE dequantization to fp8_utils and plug it into the fp8, compressed-tensors and Quark MoE methods. Document block-FP8 MoE requantization in the online quantization docs.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
@mergify

mergify Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--67529.org.readthedocs.build/en/67529/

@mergify mergify Bot added the documentation Improvements or additions to documentation label Nov 13, 2026
github-actions Bot added a commit to vllm-project/recipes that referenced this pull request Nov 13, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…tions through AITER_BF16_FP8_MOE_BOUND

activation: mxfp8 selects AITER_MXFP4_BF16 with bound 0 (MXFP8 activations for any token count) and activation: null sets a very high bound (bf16 activations); other requests keep the previous bound with a TODO. Map VLLM_FP8_MOE_REQUANT_ACT a8w4/a16w4 to mxfp8/null, make the legacy redirect set model_config.quantization_config, and use mxfp8 in the Mistral Large 4 recipe.
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
github-actions Bot added a commit to vllm-project/recipes that referenced this pull request Nov 13, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@@ -0,0 +1,62 @@
# SPDX-License-Identifier: Apache-2.0

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to keep backward compatibility with VLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT/VLLM_ROCM_MUSE_MOE_ONLINE_QUANT, but I could just remove them from envs.py if this is prefered.

…_STRICT_QUANT_SCHEME and address self-review

Serve activation: mxfp8 with AITER_MXFP4_BF16 on gfx950 only, forbid adding activation quantization to weight-only requests in strict mode, derive the AITER bf16/fp8 bound from the layer's resolved activation, record the FP8 scale block shape at weight creation, restrict the ML4 specialized-decode check to moe, and document strict mode for the AITER MXFP4 kernels.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
tomasruizt and others added 2 commits November 13, 2026 19:34
Add a ParallelGroup protocol and an override_parallel_groups() context
manager (usable as a decorator) to parallel_state, so tests can swap in a
single-rank fake without monkeypatching private module globals.
Use it in test_models_fse_init instead of the single_rank_parallel_state
fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
Fuse the dist/GPU test cases into single tests that use override_parallel_groups() instead of dist_init, vectorize the block-FP8 dequantization reference, and skip the global cleanup for CPU-only tests.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Comment thread vllm/distributed/parallel_state.py Outdated
return _PCP


_GROUP_NAMES = ("_WORLD", "_TP", "_ETP", "_DCP", "_PP", "_DP", "_EP", "_PCP")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mergify

mergify Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fxmarty-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 16, 2026
@fxmarty-amd fxmarty-amd changed the title [Online quantization] Replace VLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT/VLLM_ROCM_MUSE_MOE_ONLINE_QUANT with standard online quantization API [Online quantization] Deprecate VLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT/VLLM_ROCM_MUSE_MOE_ONLINE_QUANT in favor of standard online quantization API Nov 16, 2026
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…layer

Create the weights of a requantization source on the meta device and let the layerwise loader materialize and requantize one layer at a time, so the peak load memory is the quantized model plus one layer of serialized weights instead of the whole serialized model. Extend test_online_quant_peak_mem to a requantization case.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
github-actions Bot added a commit to vllm-project/recipes that referenced this pull request Nov 16, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
These tests only build layers with a single-rank parallel-group fake, so the per-test distributed and memory cleanup only added about 0.35 s each (about 100 s in total).

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…ad code, env conflict check

Set AITER_BF16_FP8_MOE_BOUND from the DeepSeek-V4 selector too (it no longer got the value from the weight conversion), remove unreachable code after a return in get_quant_config, only reject VLLM_FP8_MOE_REQUANT=mxfp4 together with --quantization-config, warn that VLLM_FP8_MOE_REQUANT_ACT=a16w4 no longer selects the Triton backend, and fix the nested JSON in a recipe comment.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…ect the Muse env var with --quantization-config

AITER reads AITER_BF16_FP8_MOE_BOUND from the environment, so once a layer asks for bf16 activations (activation: null) later layers no longer take it back. VLLM_ROCM_MUSE_MOE_ONLINE_QUANT combined with --quantization-config is now an error, like VLLM_FP8_MOE_REQUANT.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…zation

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

# Conflicts:
#	tests/quantization/test_online.py
…ethod's resolved activation

The online MXFP4 method now passes its resolved activation (targets spec, deprecated env vars) to the bound setter instead of the setter reading the global quantization_config.moe, which is unset for both. The specialized-decode error message names the backend only.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…olved spec scope

Replace the activation_explicit argument of select_mxfp4_moe_backend: OnlineQuantizationConfig.get_quant_method builds the online method inside layer_target_spec_scope(quant_spec), so the oracles read the layer's resolved activation (targets, moe, or the deprecated env vars) through user_activation_override like for a global quantization_config.moe.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
The oracle sets this process-wide variable at backend selection, and an explicit bf16 request is kept for the process: start every test of the two modules from AITER's default and restore the variable afterwards with an autouse fixture instead of per-test setenv calls.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
@mergify mergify Bot removed the needs-rebase label Nov 16, 2026
github-actions Bot added a commit to vllm-project/recipes that referenced this pull request Nov 16, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Keep the single-rank fake and override_parallel_groups() in tests/single_rank_group.py instead of vllm/distributed/parallel_state.py, which is no longer touched by this PR. Reset the AITER state of test_models_fse_init in a finalizer so it also runs when a case fails, and say in the specialized-decode presets that the experts are checked once the weights are loaded.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…TP monkeypatches

Replace the dist_init fixture of the online quantization tests and the hand-written get_tensor_model_parallel_rank / world_size patches (rank 0, world size 1) with override_parallel_groups(SingleRankGroup()). The TP2 loading test keeps its patches.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
github-actions Bot added a commit to vllm-project/recipes that referenced this pull request Nov 16, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@mergify

mergify Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fxmarty-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 17, 2026
@fxmarty-amd
fxmarty-amd marked this pull request as draft November 17, 2026 12:19
xaguilar-amd added a commit to xaguilar-amd/vllm that referenced this pull request Nov 17, 2026
…on-config

Select the block-FP8 to int4 expert requant with
--quantization-config.moe.weight int4_per_group_32, the flag Kimi-K3
already uses for the same int4 format and kernel, instead of a new
VLLM_FP8_MOE_REQUANT value: that env var is being deprecated in favor of
--quantization-config (vllm-project#67529).

The method refuses FP8 experts without block scales and non-ROCm
platforms, and the flag conflicts with VLLM_FP8_MOE_REQUANT=mxfp4.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Xavier Aguilar <xavier.aguilarfruto@amd.com>

This branch was successfully deployed

1 active deployment
recipes-preview — 7449cd22 Deployed Nov 16, 2026 by fxmarty-amd via dispatch #651
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation mistral Related to Mistral models needs-rebase quantization recipe rocm Related to AMD ROCm

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

2 participants