Skip to content

[Quantization] Support CPU offloading with online NVFP4 MoE quantization - #67136

Merged
gau-nernst merged 3 commits into
vllm-project:mainfrom
Kh4L:online-quant-cpu-offload
Nov 17, 2026
Merged

gau-nernst merged 3 commits into
vllm-project:mainfrom
Kh4L:online-quant-cpu-offload

Conversation

@Kh4L

@Kh4L Kh4L commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Overview

Two fixes for online quantization combined with --cpu-offload-gb, split out of #64552 at @gau-nernst's suggestion. #64552 (Muse Spark online NVFP4) now builds on this PR.

  1. Crash fix. With --cpu-offload-gb, any model quantized online failed to load with Cannot copy out of meta tensor: UVAOffloader.wrap_modules runs while the online-quantized weights are still on the meta device. The offloader now skips meta parameters and logs once that they stay in GPU memory. online.md says so too.
  2. CPU staging for nvfp4_per_token MoE. When the UVA offloader targets a layer's experts, device memory is short. Nvfp4OnlineMoEMethod then sets the new LayerReloadingInfo.stage_on_cpu, and materialize_layer materializes that layer's full-precision experts on the CPU. _quantize_moe_weight_to_nvfp4 moves one expert at a time to the GPU and writes into GPU outputs allocated up front, so the whole BF16 layer is never on the GPU.

Claims

  • --cpu-offload-gb works with online quantization. Online-quantized weights are not offloaded; a one-time warning says so. Offloading them after quantization is a TODO.
  • UVAOffloader.targets() exposes the existing --cpu-offload-params segment match, used by the offloader and the staging gate.
  • Staging applies only with a UVAOffloader whose --cpu-offload-params select the experts. Neither the prefetch offloader nor a UVA offloader that skips the experts stages.
  • stage_on_cpu is honored on the first load and on a layerwise reload, and reset() keeps it. record_metadata_for_reloading() is unchanged.
  • Without TP sharding, each expert's amax is computed on the GPU from the copy the quantizer reads anyway. TP > 1 keeps the whole-layer amax that its all-reduce needs. The scales are bit-identical to before.

Validation

Unit tests on one GB300 DGX Station (job 3187363 on this code, main bc4ca78fd1, image nightly-cf5cea2da37d, torch 2.14.0+cu130, FlashInfer 0.7.3):

Test Result
tests/basic_correctness/cpu_offload/test_cpu_offload.py -k "online_quantization or skips_meta" 2 passed. On main the serving test fails with Cannot copy out of meta tensor
tests/quantization/test_online.py -k test_online_nvfp4_quantizes_original_expert_weights (CUDA and CPU-staged input) 2 passed
tests/model_executor/model_loader/test_reload.py + test_direct_reload.py 122 passed, 3 failed. Main fails the same 3

The 3 test_reload.py failures (test_kv_scale_reload, test_online_quantize_reload[DeepSeek-V3-debug…-fp8_per_tensor-1], test_reload_weights_with_lora_moe) are out-of-memory on the 250 GiB GPU, where the extensible KV cache takes ~245 GiB.

End to end: Muse Spark Open NVFP4 with its two BF16 MoE layers quantized online (#64552), --cpu-offload-gb 264 --cpu-offload-params experts, one GB300 Station, nvidia-smi every 0.5 s:

Weight-loading GPU plateau
Staging through restore_device, undone by record_metadata_for_reloading() 225 GB (peak 232.8)
stage_on_cpu 201 GB
No online quantization, --cpu-offload-gb 300 203 GB

Staging added 18 s (+4.6%) of model loading for the two layers. That was measured before the per-expert amax on the GPU.

CI: the offload tests run in the existing CPU Offload step. test_online.py is not run by any active CI step: the general quantization step is commented out on main.

Details

Why an explicit flag. record_metadata_for_reloading() runs after the model is built and resets every layer's restore_device to the default device. Encoding staging in restore_device was undone by that reset, which is the 225 GB row above.

Weight reload. A staged layer is also re-materialized on the CPU during a layerwise reload, when the quantized weights and the KV cache already take GPU memory. Weights sent as GPU tensors are then copied to the CPU and back one expert at a time. max_layer_reload_bytes still sizes the headroom as if the layer lands on the GPU, which overestimates and is safe.

Other online MoE methods. Only nvfp4_per_token stages for now. Moving the decision to OnlineMoEMethodBase.create_weights is a TODO, once the other _quantize_* helpers accept CPU input.

Duplicate check: no open PR covers the UVA meta-parameter crash or CPU staging for online quantization. I searched for UVAOffloader meta online quantization, cpu-offload-gb online quantization and nvfp4_per_token cpu offload.

AI assistance (Claude) was used to write the code, the tests and this description. I reviewed every changed line and ran the GPU validation above.


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

`--cpu-offload-gb` crashed with online quantization: the UVA offloader
tried to copy the meta weights that online quantization creates. It now
skips them, warns once that they stay in GPU memory, and exposes the
`--cpu-offload-params` match as `UVAOffloader.targets()`.

When the offloader targets a layer's experts, `nvfp4_per_token` sets the
new `LayerReloadingInfo.stage_on_cpu`: `materialize_layer` then stages
the full-precision experts on the CPU, and each expert moves to the GPU
to be quantized. Each expert's amax is computed on the device from that
copy (TP > 1 keeps the whole-layer amax its all-reduce needs).

Signed-off-by: Serge Panev <spanev@nvidia.com>
Assisted-by: Claude

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

@mergify

mergify Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--67136.org.readthedocs.build/en/67136/

@mergify mergify Bot added documentation Improvements or additions to documentation quantization labels Nov 12, 2026

@gau-nernst gau-nernst left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stamp to unblock. but @fxmarty-amd probably should take another look

```

!!! note
`--cpu-offload-gb` does not offload the weights that online quantization

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-blocking but personally I feel this is a strange interaction to have

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same.. If this is a support scope limit, at least there should be a clear NotImplementedError raised with a design that is extendible later to all schemes, not only nvfp4_per_token

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed it's odd that staging depends on offloading. The general fix is what @gau-nernst describes: quantize one routed expert at a time during loading, for every scheme (like #64315 / #64378). Then neither the CPU staging nor this note is needed. I scoped this PR to the crash and to the one scheme that needed memory headroom. I'll open an issue for per-expert online MoE quantization across schemes and link it from the TODO.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed #67290 for this.

Comment on lines +146 to +150
# UVA offloading of the experts means device memory is short: stage the
# full-precision experts on the CPU and quantize one at a time on the
# device.
# TODO: move this decision to OnlineMoEMethodBase.create_weights, for
# the other online MoE methods, once their quantizers take CPU input.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in general online quant should try to keep peak memory low as a good rule of thumb, but maybe that require a more careful follow up. So maybe online quant should always load and quantize per routed expert.

I'm not too familiar with the code here so a lot of analysis is done by my agent. For reference, I do per-expert requantization unconditionally here #64315

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was chunking implemented in #64378 for NVFP4->MXFP4 re-quantization but I am not certain it was implemented for every x->y quantization.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, see the docs thread. Per-expert quantization during loading would replace the CPU staging, and I'd rather do that for all schemes in one follow-up than only for nvfp4 here.

@gau-nernst

Copy link
Copy Markdown
Contributor

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #102063 for commit f8ad3cb4ae45.

ℹ️ This PR is 131 commits behind upstream main. Merged upstream main into this branch as f8ad3cb4ae45 before starting CI. Pull the branch before pushing more changes.

@arpera arpera added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 12, 2026
@github-actions

Copy link
Copy Markdown

✅ @Kh4L, CI is now available for this PR.

  • /ci run starts upstream CI; /amd-ci run starts AMD CI only.
  • Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
  • /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

@stecasta stecasta left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Kh4L, thanks for splitting this out.

LGTM. Minor: does the prefetch offloader (--offload-group-size) hit the same Cannot copy out of meta tensor? _CpuParamOffloader._offload_to_cpu_internal copies param.data when it wraps the module, and PrefetchOffloader.wrap_modules has no meta check. If so, moving targets() and the meta skip to BaseOffloader would cover both, since PrefetchOffloader.wrap_modules repeats the same segment match inline. Fine as a follow-up.

@fxmarty-amd fxmarty-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see anything blocking here, should just be generalized at some point.

I am not sure to see the benefit of CPU offloading if we support (not certain where we are right now):

  • Chunked quantization of MOE experts
  • process_weights_after_loading called during loading (i.e. e.g. you never have full FP8 -> dequantized BF16/FP32 staged, this is done progressively. Same for BF16->NVF4, FP8->NVFP4)

```

!!! note
`--cpu-offload-gb` does not offload the weights that online quantization

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same.. If this is a support scope limit, at least there should be a clear NotImplementedError raised with a design that is extendible later to all schemes, not only nvfp4_per_token

Comment on lines +146 to +150
# UVA offloading of the experts means device memory is short: stage the
# full-precision experts on the CPU and quantize one at a time on the
# device.
# TODO: move this decision to OnlineMoEMethodBase.create_weights, for
# the other online MoE methods, once their quantizers take CPU input.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was chunking implemented in #64378 for NVFP4->MXFP4 re-quantization but I am not certain it was implemented for every x->y quantization.

@Kh4L

Kh4L commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

@stecasta Good catch: PrefetchOffloader.wrap_modules has no meta check, so it likely crashes the same way. I'll move targets() and the meta skip to BaseOffloader in a follow-up, so both offloaders share them.

@Kh4L

Kh4L commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

❌ CI already ran on this commit: https://buildkite.com/vllm/ci/builds/102063. No new CI build was started. Push new changes to run CI again, use /ci retry to rerun failed jobs, or ask a reviewer with write access to rerun it.

@fxmarty-amd

Copy link
Copy Markdown
Contributor

process_weights_after_loading called during loading (i.e. e.g. you never have full FP8 -> dequantized BF16/FP32 staged, this is done progressively. Same for BF16->NVF4, FP8->NVFP4)

fyi I am adding it part of #67529, see the changes in vllm/model_executor/layers/quantization/online/requantization.py - happy if this lands elsewhere standalone.

@Kh4L

Kh4L commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

/ci retry

@Kh4L

Kh4L commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @fxmarty-amd, that progressive processing in online/requantization.py is what #67290 asks for, and landing it as part of #67529 is fine by me. I'll link #67290 to it. Once #67529 is in, the NVFP4 path can move onto it and the CPU staging this PR adds for nvfp4_per_token goes away.

@github-actions

Copy link
Copy Markdown

✅ Queued 2 failed job(s) for retry in Buildkite CI #102063.

@gau-nernst

Copy link
Copy Markdown
Contributor

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #103327 for commit 098097b6862b, running 1 failed step(s) from Buildkite CI #102063.

@gau-nernst
gau-nernst merged commit ca04fba into vllm-project:main Nov 17, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation quantization ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants