Repository navigation
fix: select FlashInfer CUTLASS for ModelOpt mixed MoE on SM120 - #35617
Open
zhachar14h-cell wants to merge 1 commit into
Open
zhachar14h-cell wants to merge 1 commit into
zhachar14h-cell wants to merge 1 commit into
Conversation
Resolve the auto MoE runner to flashinfer_cutlass for modelopt_mixed checkpoints on SM120, matching the existing modelopt_fp4 behavior. This selects a supported backend before weights are prepared while preserving explicit backend choices.
3 of 5 tasks
Author
|
@Fridge003 Could you please add the This replaces #35422 with the architecture used by current |
4 of 5 tasks
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
Comment-only; no behavior change (overrides.py battery 112P/1S/181sub unchanged by this commit, trtllm-gen file still its pre-existing 8F no-CUDA class, dispatcher suite still 12P). Verdict recorded in the block above _model_config_or_none: * For the crash sgl-project#45520 targets (flagless ModelOpt NVFP4 under `auto` falling through to SM100-only trtllm-gen -> g1_scale_c AttributeError at warmup): upstream's checkpoint-resolved rule makes entry 230's published-value fix REDUNDANT whenever the config is readable, and refines it (W4A16 experts now answer marlin, not cutlass). * For the SILENT EXPERT-ID MISROUTE entry 230 also fixed (published auto vs built GLOBAL-ids runner: rank 0 routes right, EP ranks >0 route nothing): sgl-project#45520 closes only the readable-config half and intentionally leaves `auto` when it cannot read the expert-algo list -- exactly the shape the misroute lives in. Entry 230's pieces are therefore KEPT and load-bearing there: the fail-safe config reads, the algo-unknown mixed->cutlass publish, and the once-per-pair mismatch warning in token_dispatcher/standard.py (untouched by this family). The patch is downgraded from primary to fail-safe layer; nothing deleted, per the port brief. * 489c... (entry 247 b12x) territory: the W4A16->marlin refinement changes what `auto` publishes for W4A16-MIXED checkpoints on SM110/SM120 (entry 230 blanket-published flashinfer_cutlass for ALL mixed names, which for a pure-W4A16 mixed checkpoint diverged from its per-layer marlin build -- a latent instance of the very published-vs-built divergence entry 230 exists to prevent; sgl-project#45520 removes it). b12x itself stays named-explicit-only: auto never resolves to flashinfer_b12x_w4a16 before or after sgl-project#45520, and test_auto_never_resolves_to_b12x remains green. * The bare "modelopt" name still does not refine to fp4/mixed at arg resolution (entry 230's KNOWN GAP, unchanged by sgl-project#45520): an explicit --quantization modelopt boot on SM120 must still name the runner. Upstream work that would close it via the modelopt_mixed mapping is sgl-project#35617 territory -- referenced here only, not ported. Co-authored-by: port-moe-contracts subagent (cherry picked from commit aa671ba8cf32c5e270d7219064506a39a20c9cca)
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
…mily + S-EXT1 + riders + S-RACE + LLGUARD/ENVREG), S-EXT1 and S-RACE items CLOSED, entry-256's raw-env deviation and LL-vs-PCIe-IPC collision warning both marked CLOSED with the BOOT_DELTA (h') needle in landing terms, sgl-project#35617 left as the sole FAM-A7 survivor, sgl-project#44732 re-pointed to the operator's merge-sgl-project#5 answer, and the KV-ROUTE DECISION RECORD added (HOLD NVFP4-KV until armed PCIe-IPC is measured post-merge; preferred-if-ever sequence merge -> sgl-project#43356-delta -> sgl-project#46452 NOT sgl-project#43240 -> optional sgl-project#43471 -> fork BUILD-A QSA FP4 reads (template sgl-project#38724) -> BUILD-B L3 scale sidecar -> one GPU window; sgl-project#46141 template, sgl-project#44462 NEXT, MXFP4 declined, entry-247 re-sync watch-string modelopt_e4m3_k32)
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 30, 2026
… C closure -- the post-merge port queue is now decided ARRIVAL SCAN (git log 12dc623..2a6c204 --grep) + LIVE gh STATE + head-sha comparison for EVERY deferred item. Verdicts: RELEASED-BY-MERGE: sgl-project#42689 (@ 7a632cf in-range -- the SECTION-B cherry is now redundant; behavior note rides the restart). CLOSED (C-items whose PRs arrived and whose integration this stack performed): C1 (the dedicated merge week WAS merge sgl-project#5; all eight precursor PRs in-range), C2 (sgl-project#44993/sgl-project#44994 -- B5), C3 (sgl-project#45232/ sgl-project#45236/sgl-project#45334 -- B2), C4 (sgl-project#45238/sgl-project#45241/sgl-project#45242 -- B3), C5 (sgl-project#44912/ sgl-project#44913 -- arrived; entry-230 canary re-planted as its guard), C9 (sgl-project#39727 -- verified OPT-IN, no purge; BOOT_DELTA anti-needle), C0a (arrival shas already recorded at entry 310 -- Section C now carries NO open code-integration items: C6/C10 retain only their explicitly-named docs/boot-measurement remainders, re-stated). STILL-OPEN: sgl-project#44342 (head unchanged; auto-merge comfort LOST to #46970x), sgl-project#46153/sgl-project#37106/sgl-project#32479/sgl-project#45391 (gh CONFLICTING -- stale bases, rebase before any port), sgl-project#41624/sgl-project#42525/sgl-project#42660/sgl-project#44462/sgl-project#35617 (open, heads unchanged), sgl-project#46191/sgl-project#39845-class (watch-list, none in range). HOLD STANDS (operator decision, binding): sgl-project#43356/sgl-project#46452/sgl-project#43240/ sgl-project#43471 KV route -- step-1 'MERGE sgl-project#5 lands first' is NOW MET; the sequence may be re-visited ONLY with the post-merge PCIe-IPC measurement at an operator sitting. sgl-project#44732: the 'one purge if done WITH the merge' premise LAPSED when merge sgl-project#5 completed without it -- later execution costs the SAME single purge once; no penalty, no haste. Recorded for the operator, not decided by B6.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On SM120, the quantization-driven MoE backend resolver selects
flashinfer_cutlassformodelopt_fp4, but not formodelopt_mixed.A mixed-precision checkpoint with NVFP4 MoE experts therefore leaves
--moe-runner-backend autounresolved. The NVFP4 MoE layer can then fall through to the TRT-LLM runner, although the trtllm-gen FP4 MoE kernels are SM100-only and use a different shuffled weight layout. Users currently have to pass--moe-runner-backend flashinfer_cutlassexplicitly on SM120.This supersedes #35422. Current
mainalready has a dedicated FlashInfer CUTLASS runner, so selecting that runner before weight preparation is safer than switching kernels at runtime.Modifications
modelopt_mixed + SM120 + autotoflashinfer_cutlass, matching the existingmodelopt_fp4behavior.Accuracy Tests
This change only selects an existing runner before weight preparation; it does not modify kernel math or quantization scales.
The focused backend-resolution test covers:
modelopt_fp4andmodelopt_mixedon SM120 ->flashinfer_cutlassflashinfer_trtllm-> unchangedmodelopt_mixedon non-SM120 -> unchangedSpeed Tests and Profiling
No kernel implementation changes are included. After this patch, the automatic configuration selects the same FlashInfer CUTLASS path as an explicit
--moe-runner-backend flashinfer_cutlasslaunch.Checklist
CI States
Latest PR Test (Base): ❌ Run #32325693849
Latest PR Test (Extra): ❌ Run #32325693438