Skip to content

[Quantization] Lower peak memory of online requantization, support block-FP8 to MXFP4 - #68171

Open
fxmarty-amd wants to merge 5 commits into
vllm-project:mainfrom
fxmarty-amd:lower-peak-mem-requantization
Open

fxmarty-amd wants to merge 5 commits into
vllm-project:mainfrom
fxmarty-amd:lower-peak-mem-requantization

Conversation

@fxmarty-amd

@fxmarty-amd fxmarty-amd commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Disclosure

AI assistance was used. The changes were reviewed and tested manually.

Overview

Online requantization (a serialized checkpoint, e.g. NVFP4 / MXFP8 / block-FP8, converted to another format at load through --quantization-config) allocated the serialized weights of the whole model on the device before converting them, so the peak load memory was the serialized model plus the quantized model.

This PR now creates requantization source on the meta device and materializes them the layerwise loader one layer at a time, a layer is requantization as soon as it is loaded. This is the same mechanism bf16 online quantization already uses.

Mistral-Large-4-1T-A52B (FP8 block checkpoint, experts requantized to MXFP4, TP4):

Configuration Peak VRAM per GPU during load (1 Hz) Model memory after loading
Base commit fa6c77f, VLLM_FP8_MOE_REQUANT=mxfp4, VLLM_FP8_MOE_REQUANT_ACT=a4w4 144.2 GiB 135.73 GiB
Base commit fa6c77f + dequantize_weight only (without the meta device changes in requantization.py), --quantization-config.moe mxfp4 287.9 GiB (device capacity, no OOM) 136.98 GiB
This PR, --quantization-config.moe mxfp4 147.9 GiB 135.75 GiB

This PR also adds dequantize_weight to the block-FP8 MoE methods:

  • Fp8MoEMethod,
  • CompressedTensorsW8A8Fp8MoEMethod,
  • QuarkW8A8Fp8MoEMethod,

built on a shared dequantize_block_fp8_moe_weights helper (chunked to bound the fp32 staging memory).

Split from #67529.

Related: #67136 (CPU offloading with online quantization, reviewers asked for progressive quantization during loading) and #67290 (quantize MoE experts one at a time during loading for every online scheme). This PR bounds the peak of requantization sources to one layer; per-expert granularity is left to #67290.

Claims

  • Requantization sources (MoE and linear) are on the meta device until their layer is loaded: test_online_quant_peak_mem[requantization] (Qwen3-0.6B-MXFP8 requantized to FP8 per-channel) keeps the peak load memory below 1.4x the final model memory; it fails on the base commit (peak 1.09 GiB > 1.064 GiB).
  • Block-FP8 experts of the fp8, compressed-tensors and Quark methods can be dequantized (partial trailing blocks and w1/w3 shards handled, bit-exact against a naive reference); per-tensor / per-channel FP8 raises NotImplementedError when dequantize_weight is called, in all three methods.
  • Block-FP8 experts requantized to MXFP4 (AITER backend, a4w4) are byte-identical to online-quantizing the same dequantized bf16 experts.
  • Mistral-Large-4-1T-A52B (FP8 block checkpoint) TP4 with --quantization-config.moe mxfp4: 135.75 GiB of model memory per GPU, 147.9 GiB peak during load, MXFP4 experts quantized in 2 s on top of the load.

Behavior changes vs base:

  • A requantization source layer that the checkpoint only partially provides (for example one missing weight_scale_inv) now raises a ValueError at the end of loading, instead of being requantized from zeros. This is checked in finalize_layerwise_processing; a layer with no tensor at all in the checkpoint is not detected.
  • --quantization-config.moe mxfp4 is now accepted for block-FP8 checkpoints. VLLM_FP8_MOE_REQUANT and Fp8BlockExpertsAsMxfp4FusedMoE are untouched.

Left to other PRs:

Validation

Unit tests

New / modified tests:

pytest -s -vvvvv tests/quantization/test_online.py \
  -k "dequantize_weight or configures_prequantized_moe or requantizes_block_fp8 or test_online_quant_peak_mem or online_prequantized or weight_loading_requantization or requantized_moe or test_online_quantization or removes_source_scales or requantization_source"
124 passed, 6 skipped, 131 deselected, 16 warnings in 233.28s (0:03:53)

The full tests/quantization/test_online.py:

232 passed, 29 skipped, 16 warnings in 334.09s (0:05:34)

Related existing tests (unmodified):

pytest -s -vvvvv \
  tests/quantization/test_online_mxfp4.py \
  tests/quantization/test_online_shared_expert_fusion.py \
  tests/quantization/test_keep_param_objects.py \
  tests/quantization/test_gfx950_moe.py
68 passed, 46 skipped, 28 warnings in 181.87s (0:03:01)

Without the vllm/ changes, the new peak memory test fails:

E       AssertionError: peak_memory_gib=1.09 higher than 1.0639999999999998
1 failed, 260 deselected, 16 warnings in 33.80s

Ran on MI355X (gfx950), ROCm (torch 2.13.0+rocm10.0.0), vLLM commit 147b12f (base fa6c77f).

End-to-end tests

Mistral-Large-4-1T-A52B (FP8 block checkpoint), routed experts requantized to MXFP4 at load, TP4:

VLLM_ROCM_USE_AITER=1 \
vllm serve mistralai/Mistral-Large-4-1T-A52B \
  --tensor-parallel-size 4 \
  --tokenizer-mode mistral --config-format mistral --load-format mistral \
  --max-model-len 262144 --max-num-batched-tokens 8192 --max-num-seqs 128 \
  --quantization-config.moe mxfp4 \
  --reasoning-parser mistral
(Worker_TP0) INFO [base_loader.py:164] Quantizing 71 layers of types: mlp.experts: 71 (from moe: mxfp4)
(Worker_TP0) INFO [base_loader.py:177] Online quantization as part of model loading took 1.98 seconds
(Worker_TP0) INFO [model_runner.py:449] Model loading took 135.75 GiB memory and 397.34 seconds
Configuration Peak VRAM per GPU during load (1 Hz) Model memory after loading
Base commit fa6c77f, VLLM_FP8_MOE_REQUANT=mxfp4, VLLM_FP8_MOE_REQUANT_ACT=a4w4 144.2 GiB 135.73 GiB
Base commit fa6c77f + dequantize_weight only (without the meta device changes in requantization.py), --quantization-config.moe mxfp4 287.9 GiB (device capacity, no OOM) 136.98 GiB
This PR, --quantization-config.moe mxfp4 147.9 GiB 135.75 GiB

The load time is dominated by a cold page cache (396 s weight read).

Without the requantization.py changes, the whole FP8 model is loaded on the device before requantization: the sampled VRAM sits at ~253 GiB for the load, then reaches the device capacity (~288 GiB) while experts are requantized (8 s, instead of 2 s), and loading only just succeeds. This is the memory overhead this PR removes.

GSM8K is not an accuracy claim, only a check that nothing is obviously broken: full GSM8K Platinum against each server (single repeat), with lm-evaluation-harness at https://github.com/fxmarty-amd/lm-evaluation-harness/tree/503ad16e088dc9dea08e29f6df76dbb26129941b. The model thinks by default (reasoning_effort high), and the server runs --reasoning-parser mistral, which moves the thinking out of the message content, so think_end_token is empty. Sampling is the model card's recommendation (temperature=1.0, top_p=0.95), no seed:

LMEVAL_LOG_LEVEL=DEBUG lm_eval --model local-chat-completions \
  --model_args model=mistralai/Mistral-Large-4-1T-A52B,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=128,timeout=99999,tokenized_requests=False,think_end_token='',server_sampling_defaults=True \
  --tasks gsm8k_platinum \
  --gen_kwargs max_gen_toks=8000,reasoning_effort=high,temperature=1.0,top_p=0.95 \
  --apply_chat_template --repeats 1 \
  --log_samples --output_path results/mistral_large_4_gsm8k_platinum

Base commit (online requantization through the env vars):

VLLM_ROCM_USE_AITER=1 VLLM_FP8_MOE_REQUANT=mxfp4 VLLM_FP8_MOE_REQUANT_ACT=a4w4 \
vllm serve mistralai/Mistral-Large-4-1T-A52B \
  --tensor-parallel-size 4 \
  --tokenizer-mode mistral --config-format mistral --load-format mistral \
  --max-model-len 262144 --max-num-batched-tokens 8192 --max-num-seqs 128 \
  --reasoning-parser mistral

exact_match (flexible-extract and strict-match are identical), 4 GPUs each:

Case gsm8k_platinum
Base commit fa6c77f, VLLM_FP8_MOE_REQUANT=mxfp4, VLLM_FP8_MOE_REQUANT_ACT=a4w4 0.9950 ± 0.0020
This PR, --quantization-config.moe mxfp4 0.9950 ± 0.0020

Details

  • Reviewers should scrutinise create_requantization_source_weights / release_requantization_source_weights in https://github.com/vllm-project/vllm/blob/fa6c77f5954ef22e705cd82456cc731ba67d0ae9/vllm/model_executor/layers/quantization/online/requantization.py: the source parameters are re-created by the layerwise loader on materialization, so release keys on the requantization_source attribute instead of parameter identity.
  • Tests that build a requantization source by hand now call _materialize_requantization_source since the weights start on the meta device.
  • The dequantization staging is bounded by _DEQUANT_CHUNK_BYTES (1 GiB) per chunk of experts.
  • dequant_block_fp8 (fp8_utils) is shared with requant_weight_ue8m0_inplace, Fp8BlockExpertsAsMxfp4FusedMoE and the Mistral specialized decode weights, which had their own copies.
  • The online MoE target resets layer.weight_block_size after the source creates its weights, so the source's block size does not reach the kernel selection of non-block targets.

…block-FP8 to MXFP4

Create the weights of a requantization source on the meta device and let the layerwise loader materialize and requantize one layer at a time, so the peak load memory is the quantized model plus one layer of serialized weights instead of the whole serialized model.

Add `dequantize_weight` to the block-FP8 MoE methods (fp8, compressed-tensors, Quark) so that block-FP8 experts can be requantized by online quantization, e.g. to MXFP4 with `--quantization-config.moe mxfp4`.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…sed-tensors and Quark dequantize_weight

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…cate block-FP8 dequantization

Raise when a requantization source layer is finalized partially loaded instead of requantizing zeros. Reject non-block FP8 MoE sources when dequantize_weight is called in all three methods. Do not leak the source's weight_block_size on the layer. Share dequant_block_fp8 with requant_weight_ue8m0_inplace, the removable FP8 requant class and the Mistral specialized decode.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…source creates its weights

The block-FP8 source sets weight_block_size on the layer for its serialized weights. The online MoE target owns the layer's quantization attributes: reset it after the source's create_weights instead of snapshotting it in the mixin.

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Comment on lines +265 to +269
raise NotImplementedError(
"MoE requantization from compressed-tensors FP8 is only supported "
"for block-scaled weights, got weight strategy "
f"{self.weight_quant.strategy}."
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the only other strategies are per-tensor and per-channel scaling. If it's not too much of an ask, can we support them too so we don't need to raise NotImplemented here?

My agent also flagged that docs/features/quantization/online.md needs an update

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we trim the extra tests?

Also, my agent reported that we can extend test_online_quantization(which has real weight loader) to cover block-FP8?

_DEQUANT_CHUNK_BYTES = 1 << 30


def dequantize_block_fp8_moe_weights(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, dequant goes through BF16 first then requantize. Probably not a big issue here but I think using FP32 should be better overall.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants