Skip to content

[Model] Quantize Muse Spark BF16 routed experts with online nvfp4_per_token - #64552

Open
Kh4L wants to merge 6 commits into
vllm-project:mainfrom
Kh4L:muse-spark-nvfp4-bf16-experts
Open

Kh4L wants to merge 6 commits into
vllm-project:mainfrom
Kh4L:muse-spark-nvfp4-bf16-experts

Conversation

@Kh4L

@Kh4L Kh4L commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Independent of #67136, which holds the generic --cpu-offload-gb fixes for online quantization. Either can merge first. Serving on one Station needs --cpu-offload-gb, so it needs both; the end-to-end runs below used both.

Overview

The ModelOpt NVFP4 checkpoint of Muse Spark Open keeps the routed experts of layers 0 and 61 in BF16, 24 GiB each. With online quantization targets, those two layers are now quantized to NVFP4 at load time, with per-token activation scales. All 62 MoE layers then run the FlashInfer TRT-LLM NVFP4 MoE, and the weights shrink by ~35 GiB.

vllm serve meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
  --quantization-config '{"targets":{"*layers.0.feed_forward.experts*":"nvfp4_per_token","*layers.61.feed_forward.experts*":"nvfp4_per_token"}}'

This PR was reworked after @fxmarty-amd's review. The muse_spark_quantize_bf16_experts hf-override field and the static input scales borrowed from neighbouring layers are gone. Muse now goes through the generic online quantization API, using the stock Nvfp4OnlineMoEMethod and the NVFP4 oracle.

Claims

  • Online targets now reach Muse's routed experts. The layers the checkpoint leaves in BF16 go through resolve_quant_method. Before this PR, MuseSparkRoutedExperts._get_quant_method bypassed it, so the targets were silently ignored.
  • Muse's post-expert RMSNorm stays applied. MuseMoERunner attaches its finalize hook as the TRT-LLM experts' post_gemm2_hook.
  • Unsupported targets raise an error. These are:
    • any other online MoE method or backend, whose experts would skip the norm;
    • a target on the experts that are already NVFP4 in the checkpoint.
  • CPU offloading (--cpu-offload-gb, needed on one Station) is [Quantization] Support CPU offloading with online NVFP4 MoE quantization #67136. Its Muse test case, test_muse_spark_online_nvfp4_moe_numeric with the BF16 experts staged on the CPU, goes into whichever of the two PRs merges second.
  • Results on one GB300 DGX Station:
    • on/264 uses about the same GPU memory as off/300, with 33 GiB less offloaded;
    • decode +9.9%;
    • load time +18 s;
    • logits closer to BF16 than the original static-scale version (top-1 agreement and KL).

Validation

All runs on one GB300 DGX Station: image vllm/vllm-openai:nightly-76c6f3f2, precompiled wheel from main a11d3a15f6, torch 2.14.0+cu130, FlashInfer 0.7.2.post1. Common flags:

VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1 vllm serve meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
  --max-model-len 32768 --max-num-seqs 4 --cpu-offload-gb <G> --cpu-offload-params experts \
  --kv-cache-memory-bytes 17179869184 --hf-overrides '{"text_config": {"max_position_embeddings": 32768}}' \
  --kernel-config '{"enable_flashinfer_autotune": false}' --reasoning-parser muse

Memory. Switch off vs on, both with a 300 GiB offload budget: "Model loading took" 196.96 → 161.46 GiB (−35.50 GiB). With the switch on, a 264 GiB budget fits the same GPU memory (195.21 GiB), so 33 GiB fewer expert weights are offloaded to the CPU.

GPU memory by phase (nvidia-smi every 0.5 s, GB):

Phase off/300 on/264, without the staging fix on/264, this PR
Weight-loading plateau 203 225 (peak 232.8) 201
Post-load step 210 208 208
Warm-up peak 227.7 ~226 226.8
"Model loading took" 196.96 GiB, 491.4 s 195.21 GiB, 390.8 s 195.21 GiB, 408.7 s
  • Before the staging fix, three on/264 loads took 270–272 s to load the weights (off/300: 265.75 s), and none stalled. The run with the fix was measured as total model-loading time only.
  • Staging costs +18 s (+4.6%) of model loading, to quantize the two layers from the CPU.
  • At the current head (explicit flag, UVA-only gate, per-expert amax; job 3173708 on main f9fe2f1a47): weight-loading plateau 201 GB again and "Model loading took" 194.86 GiB. That run's load time (450.5 s) isn't comparable to the table: newer main and FlashInfer, freshly downloaded checkpoint.
  • Logs: Quantizing 2 layers of types: feed_forward.experts ..., plus one MuseSpark model.layers.{0,61}.feed_forward.experts: experts quantized online to NVFP4 with per-token activation scales (FLASHINFER_TRTLLM). line per layer.

Speed. SPEED-Bench 8K, 6-prompt screen, same node and build:

Run Decode tok/s TTFT
off/300 26.20 2.25 s
on/264 28.80 (+9.9%) 1.86 s

Logit A/B.

  • Teacher-forced on 408 sequences from GPQA, GSM8K and SPEED-Bench (249,103 scored positions), top-20 prompt logprobs.
  • ΔNLL is NLL(on) − NLL(off).
  • c1/c4 = --max-num-seqs 1 and 4.
Comparison Top-1 agreement Top-20 KL ΔNLL
Noise floor: off vs off (c1 vs c4) 98.04% 0.0050 +0.0006
Noise floor: on vs on (c1 vs c4) 98.47% 0.0049 +0.0013
off vs on, c1 95.54% 0.0204 +0.0054 (se 0.0006)
off vs on, c4 95.24% 0.0207 +0.0061 (se 0.0006)
Original static-scale version, c1 94.98% 0.0224 −0.0093 (se 0.0005)
Original static-scale version, c4 95.06% 0.0233 −0.0047
  • Closer to BF16 than the static-scale version: per-token activation scales give higher top-1 agreement and lower KL.
  • But ΔNLL flipped sign.
    • On the reference text this version is slightly worse than BF16: ppl ×1.0054.
    • The static-scale version was slightly better: ppl ×0.991.
    • The static-scale rows come from a different node, nightly and base.

Task accuracy (original static-scale version, stock checkpoint):

  • GPQA Diamond, 198 questions, temperature 1.0, TP=2 on two Stations: off 86.36% vs on 86.87%, paired diff +0.51 pts, 95% CI [−2.53, +3.54].
  • GSM8K, 200 questions, one Station: off 98.0% vs on 97.0%.

Smoke: correct answers on off/300 and on/264.

Unit tests on GB300 (job 3187363, Muse and #67136 changes together, main bc4ca78fd1, FlashInfer 0.7.3; the generic tests are listed in #67136):

Test Result
tests/kernels/moe/test_muse_spark_trtllm_moe.py 15 passed, 2 skipped (MegaMoE), including the CPU-offload case now deferred (see Claims)
tests/kernels/moe/test_muse_spark_moe_deployment.py 22 passed
  • Measured numeric errors: rel(out, ref) = 0.163 (limit 0.2) and rel(out, unnormed) = 7.1 (floor 0.5).

Pending: TP=2 has not been run for this version, because no two Stations were free. The original static-scale version ran at TP=2 on two Stations (the GPQA runs above). Under TP, the per-expert amax is max-reduced across ranks, so the scales match quantizing the whole expert.

CI:

  • test_muse_spark_trtllm_moe.py is added to the B200 Kernels step.

Details

Why there is no MuseSparkNvfp4OnlineMoEMethod. The NVFP4 oracle already picks TrtLlmNvFp4ExpertsMonolithic for Muse: MiniMax2 routing, per-token activation and post_gemm2_hook. Muse's hidden size (4096) needs no padding.

Experts that are NVFP4 in the checkpoint keep Muse's ModelOpt method. That method keeps split w1/w3 global scales, so re-quantizing them is not wired up. A target on them raises an error instead of being silently ignored.

Duplicate check: no open PR covers online quantization for Muse. I searched for muse spark nvfp4 and nvfp4_per_token.

AI assistance (Claude) was used to write the code, the tests and this description. I reviewed every changed line and ran the GPU validation above.


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

@Kh4L
Kh4L requested a review from a team October 29, 2026 17:24

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify

mergify Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--64552.org.readthedocs.build/en/64552/

@mergify mergify Bot added documentation Improvements or additions to documentation ci/build quantization nvidia labels Oct 29, 2026
@mergify

mergify Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Kh4L.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Oct 29, 2026
@github-actions

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

@jeejeelee jeejeelee mentioned this pull request Oct 30, 2026
9 of 35 tasks

@fxmarty-amd fxmarty-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see comment

Comment thread docs/features/quantization/modelopt.md Outdated
```bash
vllm serve meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
--hf-overrides '{"text_config": {"muse_spark_quantize_bf16_experts": true}}'
```

@fxmarty-amd fxmarty-amd Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use existing online quantization infrastructure here?

Namely:

  • --quantization-config.targets:
     --quantization-config '{"targets":{"*layers.0.feed_forward.experts*":"nvfp4_per_token", "*layers.61.feed_forward.experts*":"nvfp4_per_token"}}'

In #32412 / #32029 and followup #52167, the goal is to have vLLM frontend not be dependent on the model producer library (modelopt, compressed-tensors, etc.), and if possible not on the model arch.

As muse spark MOE is a normal RoutedExperts, the API at https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/online is the one that fits naturally.

A welcome extension would be to have Nvfpv4OnlineMoEMethod & NVFP4 oracle compatible with Muse, if any change is needed (or have MuseSparkNvfp4OnlineMoEMethod(Nvfpv4OnlineMoEMethod), alternatively, but using oracle would be cleaner)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, agreed. I've reworked this PR on top of the online quantization API. Your command is now the switch, and the muse_spark_quantize_bf16_experts field is gone:

--quantization-config '{"targets":{"*layers.0.feed_forward.experts*":"nvfp4_per_token","*layers.61.feed_forward.experts*":"nvfp4_per_token"}}'

Why the command did nothing for Muse before: MuseSparkRoutedExperts._get_quant_method overrides RoutedExperts._get_quant_method and never called resolve_quant_method, so the targets never reached layers 0 and 61.

What changed

  • Routing. Layers the checkpoint leaves in BF16 now go through resolve_quant_method. Layers that are NVFP4 in the checkpoint keep Muse's ModelOpt method, which needs the split w1/w3 global scales.
  • No MuseSparkNvfp4OnlineMoEMethod. The stock Nvfp4OnlineMoEMethod and the NVFP4 oracle already select the TRT-LLM NVFP4 experts for Muse: MiniMax2 routing, per-token activation, and the existing post_gemm2_hook, which Muse's post-expert RMSNorm uses.
  • The only Muse-specific piece is in MuseMoERunner.process_weights_after_loading. It attaches the finalize hook to the online experts, which only exist after loading, and logs each converted layer.
  • Other online MoE methods are rejected. An expert class without the hook would silently skip the norm.
  • Per-token activation scales replace our neighbour-borrowed static input scales, so that code is gone.

Three generic fixes, needed under --cpu-offload-gb, which is how we run Muse on one GB300

  1. UVAOffloader skips meta parameters. Online quantization creates its weights on meta, and the offloader crashed with "Cannot copy out of meta tensor". This affects any online-quantized model with --cpu-offload-gb; there is a regression test in test_cpu_offload.py.
  2. Nvfp4OnlineMoEMethod stages the BF16 experts on the CPU when an offloader is active.
    • _quantize_moe_weight_to_nvfp4 then quantizes one expert at a time on the GPU, into GPU tensors allocated up front.
    • This replaces the per-expert list plus torch.stack, which helps the non-offload path too.
  3. record_metadata_for_reloading() (shared reload code) keeps a CPU restore_device.
    • It runs after the model is built and reset every layer's restore device to the default device. That silently undid the staging in (2), so the BF16 layer was still materialized whole on the GPU: a 225 GB plateau during weight loading vs 201 GB with the fix.
    • Only a CPU device is kept, so everything else behaves as before, including layers built under torch.device("meta").
    • A staged layer is also re-materialized on the CPU during a layerwise weight reload. A reload has even less GPU headroom than the first load.
    • The upstream reload tests give the same results with and without this change.

Limitation: re-quantizing Muse's checkpoint-NVFP4 experts isn't wired up, because Muse's NVFP4 method keeps split w1/w3 global scales. A target that matches those layers now raises an error instead of being silently ignored.

Results on one GB300

  • Memory: −35.5 GiB of weights.
  • Speed: +9.9% decode on a SPEED-Bench 8K screen (26.20 → 28.80 tok/s), from the 33 GiB less CPU offload that frees up.
  • Load time: +18 s.
  • Logits:
    • Per-token scales are closer to BF16 than our static scales were: top-1 agreement 95.5% vs 95.0%, KL 0.020 vs 0.022.
    • ΔNLL is now +0.005 rather than −0.009.

The PR description has the full tables. TP=2 for this version is still pending: no two Stations have been free.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

appreciated!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Resolving.

@jeejeelee jeejeelee self-assigned this Nov 5, 2026
@jeejeelee

Copy link
Copy Markdown
Member

QQ: have you validated the accuracy

@Kh4L

Kh4L commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

QQ: have you validated the accuracy
@jeejeelee
yes, on a DGX Station (GB300), comparing the two BF16 routed-expert layers kept in BF16 (off) vs quantized to NVFP4 at load time (on), stock nvidia checkpoint:

  • GPQA Diamond (198 q, temp 1.0, TP=2 on 2 Stations): off 86.36% vs on 86.87%, paired diff +0.51 pts, 95% CI [-2.53, +3.54]
  • GSM8K (200 q, 1 Station): off 98.0% vs on 97.0%
  • logit A/B (408 prompts from GPQA / GSM8K / SpeedBench, 249K scored positions, top-20 prompt logprobs):
    • this PR's original static-scale version: top-1 agreement 94.98%, KL 0.0224, ΔNLL -0.0093
    • reworked onto the online-quant API (per @fxmarty-amd's review, pushing shortly): top-1 95.54%, KL 0.0204, ΔNLL +0.0054
    • run-to-run noise floor for the same arm: top-1 ~98.2%, KL ~0.005

so it's measurably different from BF16 (KL ~4x the noise floor) but task accuracy is unchanged within noise. the GPQA and GSM8K runs were on the original version; the rework is at least as close to BF16 at the logit level

@Kh4L
Kh4L force-pushed the muse-spark-nvfp4-bf16-experts branch from fb3f05a to d505949 Compare November 5, 2026 20:55
@Kh4L
Kh4L requested a review from AndreasKaratzas as a code owner November 5, 2026 20:55
@mergify mergify Bot removed the needs-rebase label Nov 5, 2026
@Kh4L Kh4L changed the title [Model] Quantize Muse Spark BF16 routed experts to NVFP4 at load time [Model] Quantize Muse Spark BF16 routed experts with online nvfp4_per_token Nov 5, 2026
@mergify

mergify Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Kh4L.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 8, 2026

@fxmarty-amd fxmarty-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on online quantization side, thanks for adjusting!

Some agent review items nonetheless, feel free to close irrelevant threads

Comment thread vllm/model_executor/offloader/uva.py Outdated
Comment on lines +101 to +104
# Online quantization creates meta weights and replaces them after
# loading; they have no data to offload.
if p.is_meta:
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skipping meta params fixes the crash, but it also means online-quantized weights are never offloaded: wrap_modules runs once at init (meta), and the quantized tensors created later by process_weights_after_loading are plain device tensors that the offloader never sees. For a BF16 model served with e.g. --quantization fp8_per_tensor --cpu-offload-gb N, every Linear/MoE weight is meta at wrap time, so --cpu-offload-gb becomes a silent no-op and the user may OOM without any hint.

Could we at least log a warning (once) when params are skipped because they are meta, stating that online-quantized layers stay on device? Longer term, offloading the post-quantization tensors (e.g. re-wrapping after layerwise processing) would make --cpu-offload-gb behave as documented. The docs line "The quantized experts stay on the GPU" covers Muse but not the generic case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, it was silent. UVAOffloader now logs once when it skips a weight that online quantization will create: "CPU offloading skips weights that online quantization creates at load time: they stay in GPU memory."

The meta check now runs after the --cpu-offload-params match, so the warning only fires for weights the user asked to offload. online.md also has a note that --cpu-offload-gb doesn't offload online-quantized weights.

Offloading the quantized tensors after processing is left for a follow-up; the code has a TODO.

Comment on lines +47 to +52
compare_two_settings(
model="hmellor/tiny-random-LlamaForCausalLM",
arg1=["--quantization", "fp8_per_tensor"],
arg2=["--quantization", "fp8_per_tensor", "--cpu-offload-gb", "1"],
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With fp8_per_tensor on this Llama, all Linear weights are meta when UVAOffloader.wrap_modules runs, so (with the new meta skip) essentially nothing gets offloaded and arg1/arg2 run the same placement. The test therefore only guards the crash, not that offloading + online quantization actually work together. Consider asserting the regression explicitly (no exception) and documenting in the docstring that online-quantized weights are not offloaded, or checking offloaded bytes so a future fix/regression in this behavior is visible.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right: it only guarded the crash. The docstring now says so, and that online-quantized weights aren't offloaded.

I also added a cheap test, test_uva_offloader_skips_meta_weights. It wraps a meta Linear and asserts that no bytes are offloaded and the weight stays meta. A future fix (or regression) in this behaviour will show up there.

Comment on lines +83 to +85
# Keep CPU staging chosen by online quantization under CPU offloading.
if info.restore_device.type != "cpu":
info.restore_device = torch.get_default_device()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using restore_device.type == "cpu" as the signal for "online quantization chose CPU staging" is implicit: get_layerwise_info initializes restore_device from torch.get_default_device() at first access (

def get_layerwise_info(layer: torch.nn.Module) -> LayerReloadingInfo:
"""Get information related to restoring and layerwise processing. If no previous
information existed, a new entry is constructed
"""
if layer not in LAYERWISE_INFO:
LAYERWISE_INFO[layer] = LayerReloadingInfo(
restore_metadata=({}, {}),
restore_device=torch.get_default_device(),
)
return LAYERWISE_INFO[layer]
), so any layer whose info happens to be created while the default device is CPU will now also keep CPU forever, whereas before it was always reset here. An explicit field on LayerReloadingInfo (e.g. stage_on_cpu: bool, set by Nvfp4OnlineMoEMethod.create_weights and honored here / in materialize_layer) would make the intent clear and avoid accidental CPU materialization.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, with an explicit LayerReloadingInfo.stage_on_cpu:

  • Nvfp4OnlineMoEMethod.create_weights sets it.
  • materialize_layer honours it, on both the first load and a layerwise reload.
  • reset() keeps it.

record_metadata_for_reloading is back to upstream: every layer's restore device is reset to the default device. So a layer whose info happens to be created under a CPU default device no longer stays on the CPU.

Comment on lines +137 to +143
# CPU weight offloading means device memory is short: stage the
# full-precision experts on the CPU and quantize one at a time on the
# device.
if self.requantization_source is None and not isinstance(
get_offloader(), NoopOffloader
):
get_layerwise_info(layer).restore_device = torch.device("cpu")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two questions on the CPU-staging gate:

  1. It triggers for any non-NoopOffloader, including PrefetchOffloader and a UVAOffloader whose --cpu-offload-params does not match the experts at all. Is staging (and its extra H2D copies on every reload) intended in those cases, or should it be tied to the UVA/cpu_offload_gb path?
  2. Nothing here is NVFP4-specific: the FP8/MXFP8/INT8 online MoE methods face the same "full BF16 layer materialized on device" peak under offloading. Would it be cleaner to put the staging decision in OnlineMoEMethodBase.create_weights (https://github.com/vllm-project/vllm/blob/7c3545509102bc8a3d20aed75a50bdc9ed739dad/vllm/model_executor/layers/quantization/online/moe_base.py) and let each _quantize_* helper handle a CPU input, instead of a NVFP4-only override? Fine to defer, but a TODO would help.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Tied to UVA now. Staging applies only when the offloader is a UVAOffloader whose --cpu-offload-params select the experts. I factored the existing segment match into UVAOffloader.targets() and used it in both places. The prefetch offloader, or a UVA offloader that doesn't target the experts, no longer stages.
  2. Agreed it isn't NVFP4-specific. There's a TODO to move the decision to OnlineMoEMethodBase.create_weights once the other _quantize_* helpers accept CPU input.

)

amax = weight_amax(weight.flatten(1), dim=-1).to(torch.float32)
amax = weight_amax(weight.flatten(1), dim=-1).to(device, torch.float32)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When weight is CPU-staged, weight_amax runs aminmax over the whole BF16 layer (~16 GiB for w13 on Muse) on the CPU, which likely accounts for a good part of the reported +18 s load time. Since the amax is per expert, the reduction could be done on the device inside the loop (one expert copy already happens there). Under moe_tp_size > 1 the all-reduce needs all amaxes first, so this would need either two passes or a per-expert amax computed on device before the loop (copying each expert twice). Probably not blocking, just noting it as an easy win for the moe_tp_size == 1 case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done for moe_tp_size == 1: each expert's amax is computed on the device, inside the existing per-expert loop, from the copy the quantizer already reads. TP > 1 keeps the whole-layer amax that the all-reduce needs.

The scales are bit-identical (amax is an exact max; same fp32 ops). I checked the global and decode scales against the old path on random BF16 experts, including an all-zero expert that hits the clamp. On a GB300 load the staging still holds (201 GB GPU plateau during weight loading, same as before). I haven't measured the load-time effect cleanly yet: the run was on a newer main and FlashInfer with a freshly downloaded checkpoint, so it isn't comparable to the earlier numbers.

Comment on lines +829 to +835
if isinstance(quant_method, OnlineMoEMethodBase):
if not (
isinstance(quant_method, Nvfp4OnlineMoEMethod)
and quant_method.nvfp4_backend == NvFp4MoeBackend.FLASHINFER_TRTLLM
):
raise NotImplementedError(
"MuseSpark online MoE quantization supports only "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

post_gemm2_hook is a generic FusedMoEExperts attribute (https://github.com/vllm-project/vllm/blob/7c3545509102bc8a3d20aed75a50bdc9ed739dad/vllm/model_executor/layers/fused_moe/modular_kernel.py) and is honored by several backends (Triton, Marlin, TRT-LLM BF16/NVFP4). Is the restriction to Nvfp4OnlineMoEMethod + FLASHINFER_TRTLLM a real kernel limitation, or just what was validated? If the latter, a check on whether the selected experts_cls supports the hook (e.g. a class attribute) would avoid hardcoding one method/backend pair here and in process_weights_after_loading, and would let fp8_per_tensor/mxfp8 targets work later. Otherwise a short comment explaining why other backends would skip the norm would help.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's partly a real requirement and partly what was validated.

The real requirement: Muse normalizes each expert's output before the top-k sum. The experts must stop after GEMM2 and hand the rows to post_gemm2_hook; a backend that finalizes inside the kernel would silently skip the norm.

What was validated: on an NVFP4 checkpoint's BF16 layers, only TRT-LLM NVFP4 has been validated with Muse's in-kernel MiniMax2 routing. Since #66139 (merged yesterday), online targets on a BF16 checkpoint go to Muse's Triton FP8 experts first; I rebased onto it and kept both paths.

I added a comment saying this, rather than a capability attribute. The attribute would touch the base class and five backends, and the routing constraint still needs a Muse-side check. I'm happy to add it when an fp8_per_tensor or mxfp8 target for Muse is wanted.

@stecasta stecasta left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Kh4L
Kh4L force-pushed the muse-spark-nvfp4-bf16-experts branch from d505949 to b310d34 Compare November 10, 2026 20:04
@mergify mergify Bot removed the needs-rebase label Nov 10, 2026
@gau-nernst

Copy link
Copy Markdown
Contributor

Would it be better to separate this into 2 PRs?

  1. Online quantizations with --cpu-offload-gb fixes
  2. Muse Spark online quantization

@Kh4L

Kh4L commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Good call, done: the generic --cpu-offload-gb fixes (UVA meta-parameter skip, stage_on_cpu CPU staging, per-expert amax) are now #67136. This PR builds on it: until #67136 merges, its commit shows up here first, and only the second commit is Muse-specific. The code is unchanged from the previous head b310d34868; only the online.md wording moved between the two PRs.

The ModelOpt NVFP4 checkpoint of Muse Spark Open keeps the routed experts
of layers 0 and 61 in BF16. `nvfp4_per_token` online targets now quantize
them to NVFP4, so every MoE layer runs Muse's TRT-LLM NVFP4 experts.
Online targets that match NVFP4 experts, or select another online MoE
method on an NVFP4 checkpoint, are rejected.

Signed-off-by: Serge Panev <spanev@nvidia.com>
Assisted-by: Claude
@Kh4L
Kh4L force-pushed the muse-spark-nvfp4-bf16-experts branch from 6e170f7 to 0dddf50 Compare November 12, 2026 02:19
@Kh4L

Kh4L commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Update: I unstacked it, so this PR is now independent of #67136 and only has the Muse commit. Either can merge first. The one Muse test case that needs #67136 (the numeric test with the BF16 experts staged on the CPU) goes into whichever PR merges second.

@arpera arpera added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 12, 2026
@github-actions

Copy link
Copy Markdown

✅ @Kh4L, CI is now available for this PR.

  • /ci run starts upstream CI; /amd-ci run starts AMD CI only.
  • Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
  • /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

@Kh4L

Kh4L commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #102215 for commit 03c7ca044dbb.

ℹ️ This PR is 171 commits behind upstream main. Merged upstream main into this branch as 03c7ca044dbb before starting CI. Pull the branch before pushing more changes.

@mergify

mergify Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Hi @Kh4L, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

@Kh4L

Kh4L commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #103212 for commit 13e7abbe5181.

ℹ️ This PR is 176 commits behind upstream main. Merged upstream main into this branch as 13e7abbe5181 before starting CI. Pull the branch before pushing more changes.

gau-nernst and others added 2 commits November 17, 2026 01:54
Trigger on muse_moe_runner.py, the code the test covers, instead of the
whole muse_spark/common and muse_spark/nvidia directories, so Muse model
and multimodal changes don't run the full B200 kernel step. Drop the
online/nvfp4.py trigger, which is generic quantization code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build documentation Improvements or additions to documentation nvidia quantization ready ONLY add when PR is ready to merge/full CI is needed

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

7 participants