Skip to content

fix: select FlashInfer CUTLASS for ModelOpt mixed MoE on SM120 - #35617

Open
zhachar14h-cell wants to merge 1 commit into
sgl-project:mainfrom
zhachar14h-cell:perf/nvfp4-small-m-moe-main
Open

zhachar14h-cell wants to merge 1 commit into
sgl-project:mainfrom
zhachar14h-cell:perf/nvfp4-small-m-moe-main

Conversation

@zhachar14h-cell

@zhachar14h-cell zhachar14h-cell commented Aug 20, 2026 •

Copy link
Copy Markdown

Motivation

On SM120, the quantization-driven MoE backend resolver selects flashinfer_cutlass for modelopt_fp4, but not for modelopt_mixed.

A mixed-precision checkpoint with NVFP4 MoE experts therefore leaves --moe-runner-backend auto unresolved. The NVFP4 MoE layer can then fall through to the TRT-LLM runner, although the trtllm-gen FP4 MoE kernels are SM100-only and use a different shuffled weight layout. Users currently have to pass --moe-runner-backend flashinfer_cutlass explicitly on SM120.

This supersedes #35422. Current main already has a dedicated FlashInfer CUTLASS runner, so selecting that runner before weight preparation is safer than switching kernels at runtime.

Modifications

  • Resolve modelopt_mixed + SM120 + auto to flashinfer_cutlass, matching the existing modelopt_fp4 behavior.
  • Preserve explicit backend selections.
  • Preserve existing behavior on non-SM120 devices.
  • Extend the model override unit test to cover both ModelOpt modes and the fallback boundaries.

Accuracy Tests

This change only selects an existing runner before weight preparation; it does not modify kernel math or quantization scales.

71 passed, 1 skipped, 4 subtests passed

The focused backend-resolution test covers:

  • modelopt_fp4 and modelopt_mixed on SM120 -> flashinfer_cutlass
  • explicit flashinfer_trtllm -> unchanged
  • modelopt_mixed on non-SM120 -> unchanged

Speed Tests and Profiling

No kernel implementation changes are included. After this patch, the automatic configuration selects the same FlashInfer CUTLASS path as an explicit --moe-runner-backend flashinfer_cutlass launch.

Checklist

  • Format code with pre-commit.
  • Add unit tests.
  • No documentation update is required because no public option is added or changed.
  • Accuracy impact reviewed; no numerical path is modified.
  • Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ❌ Run #32325693849
Latest PR Test (Extra): ❌ Run #32325693438

Resolve the auto MoE runner to flashinfer_cutlass for modelopt_mixed checkpoints on SM120, matching the existing modelopt_fp4 behavior. This selects a supported backend before weights are prepared while preserving explicit backend choices.
@zhachar14h-cell

Copy link
Copy Markdown
Author

@Fridge003 Could you please add the run-ci label (or run /tag-run-ci-label) for this main-branch fix?

This replaces #35422 with the architecture used by current main: SM120 modelopt_mixed + auto now selects the existing flashinfer_cutlass runner before weight preparation. The full model-override unit test passes (71 passed, 1 skipped), pre-commit passes, and the minimum-main-base maintenance check is green. Thanks!

detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
Comment-only; no behavior change (overrides.py battery 112P/1S/181sub
unchanged by this commit, trtllm-gen file still its pre-existing
8F no-CUDA class, dispatcher suite still 12P).

Verdict recorded in the block above _model_config_or_none:
  * For the crash sgl-project#45520 targets (flagless ModelOpt NVFP4 under `auto`
    falling through to SM100-only trtllm-gen -> g1_scale_c AttributeError
    at warmup): upstream's checkpoint-resolved rule makes entry 230's
    published-value fix REDUNDANT whenever the config is readable, and
    refines it (W4A16 experts now answer marlin, not cutlass).
  * For the SILENT EXPERT-ID MISROUTE entry 230 also fixed (published
    auto vs built GLOBAL-ids runner: rank 0 routes right, EP ranks >0
    route nothing): sgl-project#45520 closes only the readable-config half and
    intentionally leaves `auto` when it cannot read the expert-algo list
    -- exactly the shape the misroute lives in. Entry 230's pieces are
    therefore KEPT and load-bearing there: the fail-safe config reads,
    the algo-unknown mixed->cutlass publish, and the once-per-pair
    mismatch warning in token_dispatcher/standard.py (untouched by this
    family). The patch is downgraded from primary to fail-safe layer;
    nothing deleted, per the port brief.
  * 489c... (entry 247 b12x) territory: the W4A16->marlin refinement
    changes what `auto` publishes for W4A16-MIXED checkpoints on
    SM110/SM120 (entry 230 blanket-published flashinfer_cutlass for ALL
    mixed names, which for a pure-W4A16 mixed checkpoint diverged from
    its per-layer marlin build -- a latent instance of the very
    published-vs-built divergence entry 230 exists to prevent; sgl-project#45520
    removes it). b12x itself stays named-explicit-only: auto never
    resolves to flashinfer_b12x_w4a16 before or after sgl-project#45520, and
    test_auto_never_resolves_to_b12x remains green.
  * The bare "modelopt" name still does not refine to fp4/mixed at arg
    resolution (entry 230's KNOWN GAP, unchanged by sgl-project#45520): an explicit
    --quantization modelopt boot on SM120 must still name the runner.
    Upstream work that would close it via the modelopt_mixed mapping is
    sgl-project#35617 territory -- referenced here only, not ported.

Co-authored-by: port-moe-contracts subagent
(cherry picked from commit aa671ba8cf32c5e270d7219064506a39a20c9cca)
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…mily + S-EXT1 + riders + S-RACE + LLGUARD/ENVREG), S-EXT1 and S-RACE items CLOSED, entry-256's raw-env deviation and LL-vs-PCIe-IPC collision warning both marked CLOSED with the BOOT_DELTA (h') needle in landing terms, sgl-project#35617 left as the sole FAM-A7 survivor, sgl-project#44732 re-pointed to the operator's merge-sgl-project#5 answer, and the KV-ROUTE DECISION RECORD added (HOLD NVFP4-KV until armed PCIe-IPC is measured post-merge; preferred-if-ever sequence merge -> sgl-project#43356-delta -> sgl-project#46452 NOT sgl-project#43240 -> optional sgl-project#43471 -> fork BUILD-A QSA FP4 reads (template sgl-project#38724) -> BUILD-B L3 scale sidecar -> one GPU window; sgl-project#46141 template, sgl-project#44462 NEXT, MXFP4 declined, entry-247 re-sync watch-string modelopt_e4m3_k32)
detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
… C closure -- the post-merge port queue is now decided

ARRIVAL SCAN (git log 12dc623..2a6c204 --grep) + LIVE gh STATE +
head-sha comparison for EVERY deferred item. Verdicts:
  RELEASED-BY-MERGE: sgl-project#42689 (@ 7a632cf in-range -- the SECTION-B
    cherry is now redundant; behavior note rides the restart).
  CLOSED (C-items whose PRs arrived and whose integration this stack
    performed): C1 (the dedicated merge week WAS merge sgl-project#5; all eight
    precursor PRs in-range), C2 (sgl-project#44993/sgl-project#44994 -- B5), C3 (sgl-project#45232/
    sgl-project#45236/sgl-project#45334 -- B2), C4 (sgl-project#45238/sgl-project#45241/sgl-project#45242 -- B3), C5 (sgl-project#44912/
    sgl-project#44913 -- arrived; entry-230 canary re-planted as its guard),
    C9 (sgl-project#39727 -- verified OPT-IN, no purge; BOOT_DELTA anti-needle),
    C0a (arrival shas already recorded at entry 310 -- Section C now
    carries NO open code-integration items: C6/C10 retain only their
    explicitly-named docs/boot-measurement remainders, re-stated).
  STILL-OPEN: sgl-project#44342 (head unchanged; auto-merge comfort LOST to
    #46970x), sgl-project#46153/sgl-project#37106/sgl-project#32479/sgl-project#45391 (gh CONFLICTING -- stale
    bases, rebase before any port), sgl-project#41624/sgl-project#42525/sgl-project#42660/sgl-project#44462/sgl-project#35617
    (open, heads unchanged), sgl-project#46191/sgl-project#39845-class (watch-list, none in
    range).
  HOLD STANDS (operator decision, binding): sgl-project#43356/sgl-project#46452/sgl-project#43240/
    sgl-project#43471 KV route -- step-1 'MERGE sgl-project#5 lands first' is NOW MET; the
    sequence may be re-visited ONLY with the post-merge PCIe-IPC
    measurement at an operator sitting. sgl-project#44732: the 'one purge if done
    WITH the merge' premise LAPSED when merge sgl-project#5 completed without it
    -- later execution costs the SAME single purge once; no penalty,
    no haste. Recorded for the operator, not decided by B6.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant