Skip to content

[Quantization] Online weight-only NVFP4 for MoE layers - #68018

Open
stecasta wants to merge 2 commits into
vllm-project:mainfrom
stecasta:nvfp4-weight-only-moe
Open

stecasta wants to merge 2 commits into
vllm-project:mainfrom
stecasta:nvfp4-weight-only-moe

Conversation

@stecasta

Copy link
Copy Markdown
Contributor

Purpose

Follow-up to #67006: nvfp4_weight_only now also quantizes routed experts, as suggested in review there. The online NVFP4 MoE method accepts activation: null and runs the experts on the existing W4A16 NVFP4 MoE backends (Marlin, Humming). No new kernels.

  • --quantization-config.linear nvfp4_weight_only or --quantization-config.moe nvfp4_weight_only quantize only one layer kind.
  • GPUs without the native NVFP4 quant op quantize the experts with the existing PyTorch reference quantizer.
  • nvfp4_per_token is unchanged.

Note: #67136 also edits _quantize_moe_weight_to_nvfp4; whichever lands second rebases.

Test Plan

  • pytest tests/quantization/test_online.py tests/quantization/test_quantization_config_args.py, stock vs this PR, on GB10, GB300 and RTX PRO 6000.
  • e2e on Qwen/Qwen3.6-35B-A3B (BF16): aiperf, SPEED-Bench coding, OSL 8K. GSM8K via tests/evals/gsm8k/gsm8k_eval.py (1319 questions).

Test Result

Tests: no regressions on any of the three GPUs; the new tests pass. Marlin is auto-selected for the W4A16 experts on all three.

Decode tok/s at c=1, --quantization <arm>:

DGX Spark RTX PRO 6000 Max-Q weights GSM8K
BF16 n/a 170.3 65.5 GiB 0.955
fp8_per_channel 50.9 211.3 34.1 GiB 0.958
nvfp4_weight_only, main (experts BF16) 47.7 186.2 63.0 GiB 0.958
nvfp4_weight_only, this PR 65.9 215.2 19.9 GiB 0.961

+38% (Spark) and +16% (RTX PRO 6000) decode over the current shorthand, 3.2x less weight memory, accuracy unchanged.

This PR was written with AI assistance; I reviewed every line and ran the tests and benchmarks above. Searched open PRs for online weight-only NVFP4 MoE: none besides the overlap noted above.

nvfp4_weight_only now also quantizes routed experts: the online NVFP4
MoE method accepts activation=null and runs the experts on the W4A16
NVFP4 MoE backends (Marlin, Humming). One layer kind alone stays
available via --quantization-config.linear or --quantization-config.moe
nvfp4_weight_only.

Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify

mergify Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--68018.org.readthedocs.build/en/68018/

@mergify mergify Bot added documentation Improvements or additions to documentation quantization labels Nov 16, 2026
@mergify

mergify Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @stecasta.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 17, 2026
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>

# Conflicts:
#	vllm/model_executor/layers/quantization/online/nvfp4.py
@mergify mergify Bot removed the needs-rebase label Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation quantization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant