Skip to content

[Quant] Route hybrid NVFP4 MXFP4 draft experts through the CUTLASS MoE method on SM90/SM120 - #40173

Open
sethforprivacy wants to merge 1 commit into
sgl-project:mainfrom
sethforprivacy:fix/hybrid-nvfp4-draft-moe-sm120
Open

sethforprivacy wants to merge 1 commit into
sgl-project:mainfrom
sethforprivacy:fix/hybrid-nvfp4-draft-moe-sm120

Conversation

@sethforprivacy

@sethforprivacy sethforprivacy commented Sep 18, 2026 •

Copy link
Copy Markdown

Motivation

HybridFp8NvFp4Config (the config loader.py builds for nvidia/DeepSeek-V4*-NVFP4 checkpoints: FP8 attention/dense + NVFP4 routed experts) sends the FusedMoE layers that remain MXFP4 — the bundled DSpark/MTP draft experts, excluded from the NVFP4 map — to Mxfp4FlashinferTrtllmMoEMethod unconditionally. That TRT-LLM MoE kernel exists only on SM100. Fp8Config.get_quant_method already handles the identical layers of the stock checkpoint by selecting Mxfp4FlashinferCutlassMoEMethod on SM90/SM120 ("SM100 uses TRT-LLM; SM90 uses W4A16 and SM120 uses MXFP8xMXFP4"), so the hybrid path is the only place the platform branch is missing.

Modifications

One platform branch in HybridFp8NvFp4Config.get_quant_method, mirroring fp8.py: on SM90/SM120 return Mxfp4FlashinferCutlassMoEMethod(Fp8MoEMethod(self), prefix=prefix); otherwise the existing TRT-LLM method. No change to the NVFP4 routed-expert method, to any non-MoE layer, or to SM100.

Accuracy / speed

Served on 4× NVIDIA DGX Spark (GB10, SM121), TP4/EP4, nvidia/DeepSeek-V4.1-Flash-NVFP4 @ 3431dde3, --moe-runner-backend flashinfer_cutlass, DSpark block 5, --enable-decoder-swa-bounded-replay, on top of e087e662 with this change bind-mounted. Without it the draft cannot load on this hardware; with it the draft loads (Initialized DSpark draft runner ... gamma=5) and serves:

  • 112/112 structured tool/JSON requests; needle retrieval exact at 131K / 262K / 655K prompt tokens; 8 concurrent 262K prompts 8/8.
  • Decode C1/C4/C8 36.9 / 80.8 / 107.4 tok/s with the draft vs 27.7 / 71.6 / 109.4 without.
  • Output is deterministic across boots (two boots gave byte-identical top-20 logprobs at 1,440 panel positions); first-token distributions identical to the no-speculation boot on the same checkpoint.

Full record with raw JSON: sparkring performance record (§5–6). Not tested on SM90 or SM100 hardware; SM100 code path is unchanged.

Checklist

  • ruff check, ruff format --check, isort --check-only --profile black clean on the file.
  • CI.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #35336643252
Latest PR Test (Extra): ❌ Run #35336642917
Latest PR Test (AMD ROCm 10): ❌ Run #35336643182

…E method on SM90/SM120

HybridFp8NvFp4Config sent FusedMoE layers that stay MXFP4 (the DeepSeek V4
DSpark/MTP draft experts of nvidia/DeepSeek-V4*-NVFP4 checkpoints) to
Mxfp4FlashinferTrtllmMoEMethod unconditionally. The TRT-LLM MoE kernel exists
only on SM100; Fp8Config.get_quant_method already selects the CUTLASS
MXFP8xMXFP4 method on SM90/SM120 for the same layers of the stock checkpoint.
Mirror that platform branch in the hybrid config so the NVFP4 checkpoints load
with speculation on those parts. SM100 behaviour is unchanged.

Served with this change on 4x DGX Spark (GB10, SM121, TP4/EP4) for
nvidia/DeepSeek-V4.1-Flash-NVFP4 with DSpark speculation; record and gates:
https://github.com/FujitsuPolycom/sparkring/blob/main/performance/records/deepseek-v41-flash/sglang-79f656a6-nvfp4-20260917.md

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions github-actions Bot added the quant LLM Quantization label Sep 18, 2026
detain added a commit to detain/sglang that referenced this pull request Oct 20, 2026
…/deferred/avoid map

- 3A [24-35]: NVFP4 MoE boot sgl-project#39807/sgl-project#40173, block-FP8 TP refine sgl-project#42566, k-pool indexer split sgl-project#43619,
  SM120 skinny GEMM sgl-project#42865/sgl-project#42867, DeepGEMM warmup guards sgl-project#32686/sgl-project#43712, autotune sgl-project#43363/sgl-project#40420,
  mHC overlap sgl-project#42985, FP8-einsum precompile sgl-project#42821.
- 3B [36-43]: PCIe CustomAllreduce VMM sgl-project#44448, EP local expert ids sgl-project#44480, shm TP broadcast sgl-project#44202,
  PCIe-IPC all-gather sgl-project#42863 + multicast guard sgl-project#41888, TP-sampling consistency sgl-project#41916/sgl-project#38565,
  draft embed/lm_head VRAM sgl-project#37155.
- sgl-project#39807 toolchain commit verified safe against our CUDA_HOME=12.9 pin (cu13 fallback needs CUDA_HOME unset AND glibc>=2.41; we are 2.39).
- stack_want.txt: execution-status banner. Branch tip 5dd6ec7.
@sethforprivacy

Copy link
Copy Markdown
Author

Posted by Claude (AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work.

@ch-wan @Edwardf0t1 gentle ping. Main still routes hybrid NVFP4 MXFP4 draft experts only to the TRT-LLM MoE method (modelopt_quant.py, MTP fallback), which has no SM90/SM120 kernel. This sends those two arches to Mxfp4FlashinferCutlassMoEMethod, the same way Fp8Config.get_quant_method does. It still merges cleanly. CI needs a maintainer's run-ci label to start.

@xiongyidxx

Copy link
Copy Markdown

Field validation for this PR from a 4-node DGX Spark (GB10, sm_121) fleet, running the NVIDIA DeepSeek-V4.1-Flash-1017-NVFP4 checkpoint (hybrid FP8 dense + NVFP4 routed experts) through SGLang nightly + TP4:

1. The broken bucket extends to sm_121, not just SM90/SM120.
On our GPUs the flashinfer_trtllm FP4 MoE path fails exactly as described — the trtllm-gen FP4 MoE cubins ship for sm100/103/107 only, so the unconditionally-selected Mxfp4FlashinferTrtllmMoEMethod has no kernel here. Passing --moe-runner-backend flashinfer_cutlass makes the main model path work on sm_121 (CUTLASS JIT compiles), which matches the routing this PR describes for SM120. So sm_121 should be grouped with SM90/SM120 in the selection logic.

2. The unconditional TRT-LLM selection also blocks speculative decoding entirely on these parts.
With the NVFP4 checkpoint we could not enable --speculative-algorithm DSPARK at all: the draft/MTP path reaches the same MXFP4 expert layers and the call sites in srt/layers/quantization/mxfp4_flashinfer_trtllm_moe.py (the forward path around :404 and :417) call the trtllm MoE op directly with no backend switch. CLI-level workarounds (get_quant_method overrides, AUTO dispatch guards) do not reach those hardcoded calls.

3. Why it matters numerically (field numbers, same decode runner, 4-node TP4).
Without speculative decoding the NVFP4 checkpoint ran at ~10 tok/s (c=1) / ~66 tok/s (c=8) aggregate; the same fleet's FP8 production line (with DSPARK) runs ~70/201. The delta is dominated by the missing spec-decode path, not by the FP4 experts themselves — so routing the draft experts through CUTLASS on sm_121 (this PR's change) is what would let an NVFP4 deployment stay in the same performance class.

4. Independent confirmation that the CUTLASS draft path is viable on sm_121.
The Mxfp4FlashinferCutlassMoEMethod selection already builds and runs on sm_121 for the main MoE layers (we ran it end-to-end: text + vision smoke, 1M-context request, no transport or shape errors on 4 ranks). We did not separately A/B the draft layers because the current unconditional selection never lets them reach CUTLASS — i.e. exactly the gap this PR closes.

If useful, I can re-run our NVFP4 speculative A/B on the branch this PR builds and post numbers here once a build with the change is available.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

quant LLM Quantization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants