Repository navigation
perf: optimize small-M NVFP4 MoE on SM120 (v0.5.15) - #35422
zhachar14h-cell wants to merge 1 commit into
Conversation
Route eligible BF16 ModelOpt W4A4 decode workloads through the FlashInfer CUTLASS fused MoE path. Preserve the existing grouped CUTLASS fallback and support both Up/Gate weight layouts without copying expert weights.
|
@Fridge003 Could you please add the Local pre-commit passes and all 27 supported NVFP4 tests pass. The current AMD gate failure is only from the missing This PR targets |
|
Superseded by #35617. The latest |
Motivation
On GB10 / SM121 with SGLang v0.5.15, decode-heavy ModelOpt NVFP4 MoE workloads use very small token counts (typically M=1/2, with M<=8 during CUDA Graph capture). The regular FlashInfer TRT-LLM route has non-trivial dispatch setup at these sizes.
This PR adds an explicit opt-in path for the Qwen3.6-35B-A3B NVFP4 shape. It reuses the FlashInfer SM120 CUTLASS fused MoE kernel and the already packed ModelOpt W4A4 weights. The default path is unchanged.
This PR targets
release/v0.5.15because the relevant NVFP4 JIT and runner interfaces have since been refactored onmain; a main-branch port needs separate validation against the new weight preparation path.Modifications
nvfp4_small_moefor BF16, M<=8, hidden size 2048, intermediate size 512, and top-k<=8.SGLANG_NVFP4_SMALL_MOE=1.[Up; Gate]and historical[Gate; Up]W13 layouts without copying expert weight tensors.Accuracy Tests
Tested on NVIDIA GB10 / SM121 with Torch 2.11.0+cu130 and FlashInfer 0.6.12.
Additional validation:
pytest: 27 supported NVFP4 tests passed.Speed Tests and Profiling
Direct fused MoE latency:
End-to-end validation, Qwen3.6-35B-A3B-NVFP4, 2048 input tokens to 200 output tokens, concurrency 1, n=20:
These end-to-end numbers validate stability and integration; they are not presented as an isolated speedup attributable only to this dispatch change.
Checklist