Repository navigation
[Quant] Route hybrid NVFP4 MXFP4 draft experts through the CUTLASS MoE method on SM90/SM120 - #40173
sethforprivacy wants to merge 1 commit into
Conversation
…E method on SM90/SM120 HybridFp8NvFp4Config sent FusedMoE layers that stay MXFP4 (the DeepSeek V4 DSpark/MTP draft experts of nvidia/DeepSeek-V4*-NVFP4 checkpoints) to Mxfp4FlashinferTrtllmMoEMethod unconditionally. The TRT-LLM MoE kernel exists only on SM100; Fp8Config.get_quant_method already selects the CUTLASS MXFP8xMXFP4 method on SM90/SM120 for the same layers of the stock checkpoint. Mirror that platform branch in the hybrid config so the NVFP4 checkpoints load with speculation on those parts. SM100 behaviour is unchanged. Served with this change on 4x DGX Spark (GB10, SM121, TP4/EP4) for nvidia/DeepSeek-V4.1-Flash-NVFP4 with DSpark speculation; record and gates: https://github.com/FujitsuPolycom/sparkring/blob/main/performance/records/deepseek-v41-flash/sglang-79f656a6-nvfp4-20260917.md Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…/deferred/avoid map - 3A [24-35]: NVFP4 MoE boot sgl-project#39807/sgl-project#40173, block-FP8 TP refine sgl-project#42566, k-pool indexer split sgl-project#43619, SM120 skinny GEMM sgl-project#42865/sgl-project#42867, DeepGEMM warmup guards sgl-project#32686/sgl-project#43712, autotune sgl-project#43363/sgl-project#40420, mHC overlap sgl-project#42985, FP8-einsum precompile sgl-project#42821. - 3B [36-43]: PCIe CustomAllreduce VMM sgl-project#44448, EP local expert ids sgl-project#44480, shm TP broadcast sgl-project#44202, PCIe-IPC all-gather sgl-project#42863 + multicast guard sgl-project#41888, TP-sampling consistency sgl-project#41916/sgl-project#38565, draft embed/lm_head VRAM sgl-project#37155. - sgl-project#39807 toolchain commit verified safe against our CUDA_HOME=12.9 pin (cu13 fallback needs CUDA_HOME unset AND glibc>=2.41; we are 2.39). - stack_want.txt: execution-status banner. Branch tip 5dd6ec7.
|
Posted by Claude (AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work. @ch-wan @Edwardf0t1 gentle ping. Main still routes hybrid NVFP4 MXFP4 draft experts only to the TRT-LLM MoE method ( |
|
Field validation for this PR from a 4-node DGX Spark (GB10, sm_121) fleet, running the NVIDIA 1. The broken bucket extends to sm_121, not just SM90/SM120. 2. The unconditional TRT-LLM selection also blocks speculative decoding entirely on these parts. 3. Why it matters numerically (field numbers, same decode runner, 4-node TP4). 4. Independent confirmation that the CUTLASS draft path is viable on sm_121. If useful, I can re-run our NVFP4 speculative A/B on the branch this PR builds and post numbers here once a build with the change is available. |
Motivation
HybridFp8NvFp4Config(the configloader.pybuilds fornvidia/DeepSeek-V4*-NVFP4checkpoints: FP8 attention/dense + NVFP4 routed experts) sends the FusedMoE layers that remain MXFP4 — the bundled DSpark/MTP draft experts, excluded from the NVFP4 map — toMxfp4FlashinferTrtllmMoEMethodunconditionally. That TRT-LLM MoE kernel exists only on SM100.Fp8Config.get_quant_methodalready handles the identical layers of the stock checkpoint by selectingMxfp4FlashinferCutlassMoEMethodon SM90/SM120 ("SM100 uses TRT-LLM; SM90 uses W4A16 and SM120 uses MXFP8xMXFP4"), so the hybrid path is the only place the platform branch is missing.Modifications
One platform branch in
HybridFp8NvFp4Config.get_quant_method, mirroringfp8.py: on SM90/SM120 returnMxfp4FlashinferCutlassMoEMethod(Fp8MoEMethod(self), prefix=prefix); otherwise the existing TRT-LLM method. No change to the NVFP4 routed-expert method, to any non-MoE layer, or to SM100.Accuracy / speed
Served on 4× NVIDIA DGX Spark (GB10, SM121), TP4/EP4,
nvidia/DeepSeek-V4.1-Flash-NVFP4@3431dde3,--moe-runner-backend flashinfer_cutlass, DSpark block 5,--enable-decoder-swa-bounded-replay, on top ofe087e662with this change bind-mounted. Without it the draft cannot load on this hardware; with it the draft loads (Initialized DSpark draft runner ... gamma=5) and serves:Full record with raw JSON: sparkring performance record (§5–6). Not tested on SM90 or SM100 hardware; SM100 code path is unchanged.
Checklist
ruff check,ruff format --check,isort --check-only --profile blackclean on the file.🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #35336643252
Latest PR Test (Extra): ❌ Run #35336642917
Latest PR Test (AMD ROCm 10): ❌ Run #35336643182