Skip to content

perf: optimize small-M NVFP4 MoE on SM120 (v0.5.15) - #35422

Closed
zhachar14h-cell wants to merge 1 commit into
sgl-project:release/v0.5.15from
zhachar14h-cell:perf/nvfp4-small-m-moe-sm120-v0515
Closed

zhachar14h-cell wants to merge 1 commit into
sgl-project:release/v0.5.15from
zhachar14h-cell:perf/nvfp4-small-m-moe-sm120-v0515

Conversation

@zhachar14h-cell

@zhachar14h-cell zhachar14h-cell commented Aug 19, 2026 •

Copy link
Copy Markdown

Motivation

On GB10 / SM121 with SGLang v0.5.15, decode-heavy ModelOpt NVFP4 MoE workloads use very small token counts (typically M=1/2, with M<=8 during CUDA Graph capture). The regular FlashInfer TRT-LLM route has non-trivial dispatch setup at these sizes.

This PR adds an explicit opt-in path for the Qwen3.6-35B-A3B NVFP4 shape. It reuses the FlashInfer SM120 CUTLASS fused MoE kernel and the already packed ModelOpt W4A4 weights. The default path is unchanged.

This PR targets release/v0.5.15 because the relevant NVFP4 JIT and runner interfaces have since been refactored on main; a main-branch port needs separate validation against the new weight preparation path.

Modifications

  • Add nvfp4_small_moe for BF16, M<=8, hidden size 2048, intermediate size 512, and top-k<=8.
  • Dispatch eligible ModelOpt NVFP4 decode calls when SGLANG_NVFP4_SMALL_MOE=1.
  • Preserve the existing TRT-LLM path for all unsupported shapes and configurations.
  • Support both [Up; Gate] and historical [Gate; Up] W13 layouts without copying expert weight tensors.
  • Reject GEMM1 clamp and per-token activation cases from the opt-in route.

Accuracy Tests

Tested on NVIDIA GB10 / SM121 with Torch 2.11.0+cu130 and FlashInfer 0.6.12.

Case Relative L2 error vs W4A4 reference Max abs diff
M=1 0.36% <=8.96e-4
M=2 0.64% <=8.96e-4
M=4 0.51% <=8.96e-4

Additional validation:

  • Five deterministic prompts produced identical outputs.
  • CUDA Graph capture succeeded for decode batch sizes 1/2/4.
  • pytest: 27 supported NVFP4 tests passed.
  • The full blockwise MoE parameterization has one pre-existing FP16 failure on SM120: the kernel explicitly supports only BF16 output. The BF16 case passes.

Speed Tests and Profiling

Direct fused MoE latency:

Tokens Latency
M=1 80.6 us
M=2 81.9 us
M=4 74.8 us

End-to-end validation, Qwen3.6-35B-A3B-NVFP4, 2048 input tokens to 200 output tokens, concurrency 1, n=20:

  • Output throughput: 61.07 tok/s
  • Mean TPOT: 15.11 ms
  • Mean TTFT: 267.80 ms
  • Success: 20/20
  • Concurrency 4 smoke test: 8/8 successful

These end-to-end numbers validate stability and integration; they are not presented as an isolated speedup attributable only to this dispatch change.

Checklist

  • Format code with pre-commit (all modified-file hooks pass).
  • Add new CI unit tests. Existing NVFP4 tests and model-level validation are included above.
  • Update documentation. The path remains experimental and opt-in.
  • Provide accuracy and speed benchmark results.
  • Follow the SGLang code style guidance.

Route eligible BF16 ModelOpt W4A4 decode workloads through the FlashInfer CUTLASS fused MoE path. Preserve the existing grouped CUTLASS fallback and support both Up/Gate weight layouts without copying expert weights.
@zhachar14h-cell

Copy link
Copy Markdown
Author

@Fridge003 Could you please add the run-ci label (or run /tag-run-ci-label) for this PR?

Local pre-commit passes and all 27 supported NVFP4 tests pass. The current AMD gate failure is only from the missing run-ci label, not a test failure.

This PR targets release/v0.5.15 because the relevant NVFP4 interfaces have been refactored on main. Please let me know if you would prefer a separately validated main-branch port. Thanks!

@zhachar14h-cell

Copy link
Copy Markdown
Author

Superseded by #35617.

The latest main already has a dedicated FlashInfer CUTLASS MoE runner. The replacement PR fixes SM120 backend resolution before weight preparation instead of switching from the TRT-LLM layout at runtime, and includes main-branch unit tests. Closing this release-branch version in favor of the main-compatible fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blackwell SM100/SM120

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant