Repository navigation
[Quantization] Lower peak memory of online requantization, support block-FP8 to MXFP4 - #68171
Open
fxmarty-amd wants to merge 5 commits into
Open
fxmarty-amd wants to merge 5 commits into
fxmarty-amd wants to merge 5 commits into
Conversation
…block-FP8 to MXFP4 Create the weights of a requantization source on the meta device and let the layerwise loader materialize and requantize one layer at a time, so the peak load memory is the quantized model plus one layer of serialized weights instead of the whole serialized model. Add `dequantize_weight` to the block-FP8 MoE methods (fp8, compressed-tensors, Quark) so that block-FP8 experts can be requantized by online quantization, e.g. to MXFP4 with `--quantization-config.moe mxfp4`. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
fxmarty-amd
marked this pull request as ready for review
November 17, 2026 13:41
fxmarty-amd
requested review from
AndreasKaratzas,
mgoin,
pavanimajety,
robertgshaw2-redhat,
tjtanaa,
tlrmchlsmth,
yewentao256 and
zyongye
as code owners
November 17, 2026 13:41
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…sed-tensors and Quark dequantize_weight Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…cate block-FP8 dequantization Raise when a requantization source layer is finalized partially loaded instead of requantizing zeros. Reject non-block FP8 MoE sources when dequantize_weight is called in all three methods. Do not leak the source's weight_block_size on the layer. Share dequant_block_fp8 with requant_weight_ue8m0_inplace, the removable FP8 requant class and the Mistral specialized decode. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
…source creates its weights The block-FP8 source sets weight_block_size on the layer for its serialized weights. The online MoE target owns the layer's quantization attributes: reset it after the source's create_weights instead of snapshotting it in the mixin. Signed-off-by: Felix Marty <Felix.Marty@amd.com>
6 tasks done
gau-nernst
reviewed
Nov 18, 2026
Comment on lines
+265
to
+269
| raise NotImplementedError( | ||
| "MoE requantization from compressed-tensors FP8 is only supported " | ||
| "for block-scaled weights, got weight strategy " | ||
| f"{self.weight_quant.strategy}." | ||
| ) |
Contributor
There was a problem hiding this comment.
I believe the only other strategies are per-tensor and per-channel scaling. If it's not too much of an ask, can we support them too so we don't need to raise NotImplemented here?
My agent also flagged that docs/features/quantization/online.md needs an update
Contributor
There was a problem hiding this comment.
Can we trim the extra tests?
Also, my agent reported that we can extend test_online_quantization(which has real weight loader) to cover block-FP8?
| _DEQUANT_CHUNK_BYTES = 1 << 30 | ||
|
|
||
|
|
||
| def dequantize_block_fp8_moe_weights( |
Contributor
There was a problem hiding this comment.
IIUC, dequant goes through BF16 first then requantize. Probably not a big issue here but I think using FP32 should be better overall.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disclosure
AI assistance was used. The changes were reviewed and tested manually.
Overview
Online requantization (a serialized checkpoint, e.g. NVFP4 / MXFP8 / block-FP8, converted to another format at load through
--quantization-config) allocated the serialized weights of the whole model on the device before converting them, so the peak load memory was the serialized model plus the quantized model.This PR now creates requantization source on the meta device and materializes them the layerwise loader one layer at a time, a layer is requantization as soon as it is loaded. This is the same mechanism bf16 online quantization already uses.
Mistral-Large-4-1T-A52B(FP8 block checkpoint, experts requantized to MXFP4, TP4):VLLM_FP8_MOE_REQUANT=mxfp4,VLLM_FP8_MOE_REQUANT_ACT=a4w4dequantize_weightonly (without the meta device changes inrequantization.py),--quantization-config.moe mxfp4--quantization-config.moe mxfp4This PR also adds
dequantize_weightto the block-FP8 MoE methods:Fp8MoEMethod,CompressedTensorsW8A8Fp8MoEMethod,QuarkW8A8Fp8MoEMethod,built on a shared
dequantize_block_fp8_moe_weightshelper (chunked to bound the fp32 staging memory).Split from #67529.
Related: #67136 (CPU offloading with online quantization, reviewers asked for progressive quantization during loading) and #67290 (quantize MoE experts one at a time during loading for every online scheme). This PR bounds the peak of requantization sources to one layer; per-expert granularity is left to #67290.
Claims
test_online_quant_peak_mem[requantization](Qwen3-0.6B-MXFP8 requantized to FP8 per-channel) keeps the peak load memory below 1.4x the final model memory; it fails on the base commit (peak 1.09 GiB > 1.064 GiB).fp8, compressed-tensors and Quark methods can be dequantized (partial trailing blocks and w1/w3 shards handled, bit-exact against a naive reference); per-tensor / per-channel FP8 raisesNotImplementedErrorwhendequantize_weightis called, in all three methods.--quantization-config.moe mxfp4: 135.75 GiB of model memory per GPU, 147.9 GiB peak during load, MXFP4 experts quantized in 2 s on top of the load.Behavior changes vs base:
weight_scale_inv) now raises aValueErrorat the end of loading, instead of being requantized from zeros. This is checked infinalize_layerwise_processing; a layer with no tensor at all in the checkpoint is not detected.--quantization-config.moe mxfp4is now accepted for block-FP8 checkpoints.VLLM_FP8_MOE_REQUANTandFp8BlockExpertsAsMxfp4FusedMoEare untouched.Left to other PRs:
VLLM_FP8_MOE_REQUANT*/Fp8BlockExpertsAsMxfp4FusedMoEin favor of the above: [Online quantization] DeprecateVLLM_FP8_MOE_REQUANT/VLLM_FP8_MOE_REQUANT_ACT/VLLM_ROCM_MUSE_MOE_ONLINE_QUANTin favor of standard online quantization API #67529.activation: mxfp8/null) onAITER_MXFP4_BF16, only a4w4 is covered here.get_weights_for_quantization, which concatenates the dequantized chunks into a full bf16 layer before quantizing it (needed for an amax over all experts); their peak is one serialized layer plus a full bf16 layer plus the quantized layer. Making their_quantize_*helpers chunk-wise is part of [Feature]: Quantize MoE experts one at a time during loading for every online quantization scheme #67290.Validation
Unit tests
New / modified tests:
pytest -s -vvvvv tests/quantization/test_online.py \ -k "dequantize_weight or configures_prequantized_moe or requantizes_block_fp8 or test_online_quant_peak_mem or online_prequantized or weight_loading_requantization or requantized_moe or test_online_quantization or removes_source_scales or requantization_source"The full
tests/quantization/test_online.py:Related existing tests (unmodified):
Without the
vllm/changes, the new peak memory test fails:Ran on MI355X (gfx950), ROCm (torch 2.13.0+rocm10.0.0), vLLM commit 147b12f (base fa6c77f).
End-to-end tests
Mistral-Large-4-1T-A52B (FP8 block checkpoint), routed experts requantized to MXFP4 at load, TP4:
VLLM_FP8_MOE_REQUANT=mxfp4,VLLM_FP8_MOE_REQUANT_ACT=a4w4dequantize_weightonly (without the meta device changes inrequantization.py),--quantization-config.moe mxfp4--quantization-config.moe mxfp4The load time is dominated by a cold page cache (396 s weight read).
Without the
requantization.pychanges, the whole FP8 model is loaded on the device before requantization: the sampled VRAM sits at ~253 GiB for the load, then reaches the device capacity (~288 GiB) while experts are requantized (8 s, instead of 2 s), and loading only just succeeds. This is the memory overhead this PR removes.GSM8K is not an accuracy claim, only a check that nothing is obviously broken: full GSM8K Platinum against each server (single repeat), with
lm-evaluation-harnessat https://github.com/fxmarty-amd/lm-evaluation-harness/tree/503ad16e088dc9dea08e29f6df76dbb26129941b. The model thinks by default (reasoning_efforthigh), and the server runs--reasoning-parser mistral, which moves the thinking out of the message content, sothink_end_tokenis empty. Sampling is the model card's recommendation (temperature=1.0,top_p=0.95), no seed:LMEVAL_LOG_LEVEL=DEBUG lm_eval --model local-chat-completions \ --model_args model=mistralai/Mistral-Large-4-1T-A52B,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=128,timeout=99999,tokenized_requests=False,think_end_token='',server_sampling_defaults=True \ --tasks gsm8k_platinum \ --gen_kwargs max_gen_toks=8000,reasoning_effort=high,temperature=1.0,top_p=0.95 \ --apply_chat_template --repeats 1 \ --log_samples --output_path results/mistral_large_4_gsm8k_platinumBase commit (online requantization through the env vars):
exact_match(flexible-extract and strict-match are identical), 4 GPUs each:VLLM_FP8_MOE_REQUANT=mxfp4,VLLM_FP8_MOE_REQUANT_ACT=a4w4--quantization-config.moe mxfp4Details
create_requantization_source_weights/release_requantization_source_weightsin https://github.com/vllm-project/vllm/blob/fa6c77f5954ef22e705cd82456cc731ba67d0ae9/vllm/model_executor/layers/quantization/online/requantization.py: the source parameters are re-created by the layerwise loader on materialization, so release keys on therequantization_sourceattribute instead of parameter identity._materialize_requantization_sourcesince the weights start on the meta device._DEQUANT_CHUNK_BYTES(1 GiB) per chunk of experts.dequant_block_fp8(fp8_utils) is shared withrequant_weight_ue8m0_inplace,Fp8BlockExpertsAsMxfp4FusedMoEand the Mistral specialized decode weights, which had their own copies.layer.weight_block_sizeafter the source creates its weights, so the source's block size does not reach the kernel selection of non-block targets.