Repository navigation
[Fix] [NVIDIA] moe: Pick a supported runner for auto-detected ModelOpt NVFP4 MoE on SM110/SM120 - #45520
[Fix] [NVIDIA] moe: Pick a supported runner for auto-detected ModelOpt NVFP4 MoE on SM110/SM120#45520waynehacking8 wants to merge 2 commits into
Conversation
…t NVFP4 MoE on SM110/SM120 With --quantization unset, the SM110/SM120 rule never saw modelopt_fp4, so auto stayed on auto and the ModelOpt NVFP4 MoE method fell through to FLASHINFER_TRTLLM, whose kernels are SM100-only. nvidia/Qwen3.6-35B-A3B-NVFP4 then failed at warmup with "'FusedMoE' object has no attribute 'g1_scale_c'". When --quantization is unset, use the quantization ModelConfig resolved from the checkpoint, and choose by expert format: marlin for W4A16_NVFP4 experts and flashinfer_cutlass for NVFP4 (W4A4) experts of a modelopt_mixed checkpoint, flashinfer_cutlass for modelopt_fp4. The MIXED_PRECISION expert helper moves to utils so the Qwen3 MoE override and this rule share it. Signed-off-by: WEI CHENG CHIU <waynehacking8@gmail.com>
26cb238 to
7032238
Compare
|
@Fridge003 this fixes a gap in the SM110/SM120 The PR uses the checkpoint's quantization when the flag is unset, and it also covers |
…ted runner for auto-detected ModelOpt NVFP4 MoE on SM110/SM120 STATE: gh-verified OPEN at 7032238 (mergeable). Method per the ledger's open-port convention: git apply -3 clean for utils/common.py (get_mixed_precision_moe_quant_algos hoist) and model_overrides/qwen3_moe.py (helper now imported from utils); overrides.py + test_model_overrides.py hand-merged (the upstream hunk's preimage is the RAW view.quantization=="modelopt_fp4" block -- this tree carries the entry-230 fork patch at that exact spot). What upstream fixes: with --quantization unset the SM110/SM120 rule never saw the checkpoint's ModelOpt NVFP4, auto stayed auto, and the layer fell through to FLASHINFER_TRTLLM -> g1_scale_c AttributeError at warmup. Now the rule reads the checkpoint-resolved quantization and discriminates modelopt_mixed by expert format: W4A16_NVFP4 experts -> marlin, NVFP4 experts -> flashinfer_cutlass; modelopt_fp4 -> flashinfer_cutlass. This is the UPSTREAM HALF of our entry-230 fork patch (the published-value fix), living in the same _moe_runner_backend_quant_constraints hook the entries-230/244 retargeted. Fork union on top of sgl-project#45520's shape (all pinned by tests): * every model-config read goes through _model_config_or_none (fail-safe; upstream reads model_config_of(view) raw) -- entry 230's "unreadable config must leave the resolution exactly as it was" contract, kept by its RuntimeError test case; * modelopt_mixed with an UNREADABLE expert list publishes flashinfer_cutlass rather than staying auto (entry 230's reasoning: mixed derives only when some NVFP4/W4A16 layer exists; leaving auto is the silent-misroute shape). Upstream leaves auto here; the golden NVFP4-MOE-ARM case (readable config, NVFP4 experts) lands the SAME answer either way: flashinfer_cutlass. * _quantization_explicitly_unset read via getattr(default False) so the CPU resolver stubs keep constructing; production always sets it. * _checkpoint_quantization_or_none KEPT (delegating) -- the entry-230 patch is not deleted; see the reconcile note commit for the redundancy analysis. Tests: upstream's test_sm1xx_modelopt_nvfp4_moe_auto_avoids_trtllm_gen spliced byte-faithful at its upstream anchor (marlin / flashinfer_cutlass / flashinfer_cutlass across both platforms, 2 subtests); the entry-230 fixture pass (111P/1S/179sub at base) is unchanged in semantics and still green -> file now 112 passed, 1 skipped, 181 subtests. Co-authored-by: port-moe-contracts subagent (cherry picked from commit f9faef3da2644e1e723df8ffd2cc86c8fbbee1bc)
Comment-only; no behavior change (overrides.py battery 112P/1S/181sub unchanged by this commit, trtllm-gen file still its pre-existing 8F no-CUDA class, dispatcher suite still 12P). Verdict recorded in the block above _model_config_or_none: * For the crash sgl-project#45520 targets (flagless ModelOpt NVFP4 under `auto` falling through to SM100-only trtllm-gen -> g1_scale_c AttributeError at warmup): upstream's checkpoint-resolved rule makes entry 230's published-value fix REDUNDANT whenever the config is readable, and refines it (W4A16 experts now answer marlin, not cutlass). * For the SILENT EXPERT-ID MISROUTE entry 230 also fixed (published auto vs built GLOBAL-ids runner: rank 0 routes right, EP ranks >0 route nothing): sgl-project#45520 closes only the readable-config half and intentionally leaves `auto` when it cannot read the expert-algo list -- exactly the shape the misroute lives in. Entry 230's pieces are therefore KEPT and load-bearing there: the fail-safe config reads, the algo-unknown mixed->cutlass publish, and the once-per-pair mismatch warning in token_dispatcher/standard.py (untouched by this family). The patch is downgraded from primary to fail-safe layer; nothing deleted, per the port brief. * 489c... (entry 247 b12x) territory: the W4A16->marlin refinement changes what `auto` publishes for W4A16-MIXED checkpoints on SM110/SM120 (entry 230 blanket-published flashinfer_cutlass for ALL mixed names, which for a pure-W4A16 mixed checkpoint diverged from its per-layer marlin build -- a latent instance of the very published-vs-built divergence entry 230 exists to prevent; sgl-project#45520 removes it). b12x itself stays named-explicit-only: auto never resolves to flashinfer_b12x_w4a16 before or after sgl-project#45520, and test_auto_never_resolves_to_b12x remains green. * The bare "modelopt" name still does not refine to fp4/mixed at arg resolution (entry 230's KNOWN GAP, unchanged by sgl-project#45520): an explicit --quantization modelopt boot on SM120 must still name the runner. Upstream work that would close it via the modelopt_mixed mapping is sgl-project#35617 territory -- referenced here only, not ported. Co-authored-by: port-moe-contracts subagent (cherry picked from commit aa671ba8cf32c5e270d7219064506a39a20c9cca)
…20 commits since 12dc623 @ merge-sgl-project#4) Conflicts: 167 files (149 content / 14 add/add / 4 modify/delete) at the settled 309 tip d28e7cd -- dossier predicted 146 +/-25; the +21 shift is entries 275-309 landing qsa-ring/hicache-ctrl/gdn/moe/llguard deltas into the conflict zone. By class: A 13 (auto-takes-upstream, dedupe-verified), B 80, C 56, D 18. 102 TODO(merge5) re-plant/fold markers planted: B1 x15, B2 x29, B3 x12, B4 x23, B5 x23. Re-plant stock: 117 files under /tmp/opencode/merge5-dossier/replant/ (76 fork-hunks patches, 19 upstream-side patches, 14 AA fork copies, 8 preremove snapshots) -- sessions 2/3 re-plant from real code, not memory. D-list outcomes (must-not-auto-resolve, handled personally): - sgl-project#45289-vs-sgl-project#44858 (entry 301): VERIFIED, not re-resolved -- the gate/up-order final form is in origin/main (5b1dbc2 in range); scales test taken as upstream oracle, fork 191-line version saved to replant/ for B5 diffing. sgl-project#44858's call-site shape CANNOT survive (our base never had it). - entry-230 dispatcher vs sgl-project#44993/sgl-project#44994: took upstream structure in token_dispatcher/standard.py; published-value CONTRACT lives upstream via get_moe_runner_backend() in the LoRA-EP guard; fork's warn-once mismatch diagnostic re-plants (B5 marker). UNION GATE: standard_dispatcher 16P (was 12P, suite grew), model_overrides 121P/1S/210sub (was 112P/1S/181sub), fold probe 6P/2F (upstream added a runner_backend arg to the predicate -- B5 re-pins the probe). - entry-219 gauge names vs sgl-project#43727 return + sgl-project#44968: PER-POOL NAMES KEPT as the serving schema -- hicache_host_pool_{used,total}_tokens{pool} dataclass fields, Gauge registrations and labels(pool=) emission re-restored over the #4697x structure; hicache_host_pool_evicted_tokens_total already survived auto-merge; producer dict restored by union in metrics_reporter. ANTI-NEEDLE SATISFIED: old mamba-only gauges are NOT the serving schema. - entry-221 eager-direct add/add: DEDUPED to upstream (bodies identical except est_time 15->9); MEASUREMENT DEBT RECORDED: prefill t/s must be re-measured post-B4 before trusting the est_time. - entry-168 D2H fence vs sgl-project#44800: the fence is ALREADY UPSTREAM (both load_fence_stream + write_fence_stream with the load-thread-corruption comment in origin/main's cache_controller.py) -- nothing re-planted; the two real regions resolved as import-union + comment wording (merge sgl-project#4 stale-tail pattern avoided). Fork hunks still saved to replant/ (belt). - entry-209 clamp + 216/218 ordering + 288-295 hicache-ctrl-era vs #46970x KV rewrite: ADOPTED-UPSTREAM across unified_radix_cache/hicache_storage/ hybrid_cache_controller/memory_pool/kv_cache_builder; EVERY fork hunk extracted to replant/*.fork-hunks.patch; B3 re-plant markers planted. sgl-project#45703's short-full-match seam arrives with upstream. - sgl-project#45090 SGLANG_ENABLE_UNIFIED_RADIX_TREE: REMOVAL TAKEN (fork had already moved it to _DeprecatedEnv; now gone). grep: ZERO .py readers repo-wide; 2 upstream-owned docs jsx snippets still export it (deepseek-v4, glm-5.2 -- upstream's own debt). OPERATOR-OWED CHECK: confirm no /root harness exports it (unreadable from this seat). - sgl-project#39727 unified L3 key: VERIFIED OPT-IN -- merged hicache_storage.py keeps the DEFAULT legacy suffix composition _<model>_<tp_rank>_<tp_size> (content-anchor: HiCacheFile._init_config_suffix, raise-guarded when _unified_chunk_spans is set). ANTI-NEEDLE: no silent key-scheme switch; warm /root/hicache_store contents stay addressable. Modify/delete (4, all victims of upstream sgl-project#45672 test prune + URC split): - test_unified_radix_cache_unittest.py: REMOVED (adopted split) -- its bare unified_tree_core_inspector imports are unreachable post-sgl-project#45236 re-home; fork source + hunks saved; B2 marker planted in the new core test. - test_fused_moe_triton_config.py: REMOVED -- imports common_utils, absent from both sides post-sgl-project#45672; the 2 live entry-3/235 tests re-plant in B2. - test_scheduler_batch_is_full_latch.py: REMOVED -- fork delta was 3 stub-attr lines documenting sgl-project#46643's absorbed arms; upstream deletion correct. - test_qsa_kv_pool.py: KEPT -- fork's 147 added lines are entry-187/200 sharding + 285-287 QSA-ring tests with NO upstream replacement; qsa_kv_pool.py sides converged byte-identical; collects clean. B2 marker. Cascading silent-auto-merge repairs (classes not in the conflict set but broken by auto-merge + union splices -- all found by gates, now fixed): kernels/ops/activation/__init__.py (dead _CUDA registration dup), kernels/ops/ diffusion/__init__.py (5-tuple _SPECS entry), kernels ops embeddings/moe/ attention/mamba __init__ (KernelSpec fields removed upstream: description/ format_signature stripped, CapabilityRequirement->CUDA_ONLY), base_runner/ eager_runner cp_extend + shared_read_event imports retargeted per upstream's "no imports in runner_utils/__init__" contract, and 4 fork helpers grafted back onto their take-upstream homes because merged code/tests still import them (memory_pool.spec_intermediate_bytes_per_req, kv_cache_builder. _adjust_hicache_size_for_draft + _device_pool_size_per_token, hybrid_cache_controller.local_restorable_mask, validation_hook. validate_replayssm_spec_algorithm, arg_groups/moe_hook b12x validator). flashinfer_cutlass.py flipped to FORK-OWNED (b12x route is fork-only, 760 vs 95 lines; upstream sgl-project#44993 hunks = B5 fold patch saved). Silent-semantics notes: sgl-project#46127/sgl-project#44912 riders arrived via per-PR picks (union); sgl-project#45520 (entry 302) is STILL OPEN upstream -- entry-230 fail-safe layer stays as entry 303 records; if a future sync brings it, re-apply the fail-safe reads. qwen4_exp holder-contract fused (ctor takes optional batch). Gates at commit: import sglang OK (CVD=9, prod venv); check_registered_tests/ check_no_bare_pytest_main/check_no_registered_tests_in_package rc=0; ruff F821 +E9+format clean on the resolution set; collect-only over unit/{mem_cache, model_executor,distributed,layers/moe} = 4337 nodes, 3 ERRORS: 2x cargo/torch- sys libtorch probe (REPRODUCED IDENTICALLY AT THE PRE-MERGE TIP -- env issue, not merge damage) + test_hicache_async_ready_sync.py (entry-253-era attribute gone with the URC rewrite -- EXPECTED pre-B3). sessions 2/3 owed: B2 test re-home, B3 mem_cache+rust re-plant (+hicache-ctrl re-ports), B4 scheduler/admission, B5 moe/quant/dispatcher, B6 environ+C0a+ CAVEATS re-pin. PROD BOOT FORBIDDEN until B3 completes (209 clamp + 218 ordering re-plant).
…/ hicliveness), header qwennext-stack-318-2026-10-29, tip b7c656f, CAVEATS re-pin at the 318 tip + b12x-inert/gate-split + storage-liveness + sgl-project#45520-merge#6 + S-B12XDENSE bullets 6 commits from three branches all born at exactly 78783f0: ZERO conflicts. Records: entry 316 proved INERT ON BOTH ARMS from the predicate and all 10 battery rows unmoved, with the gate-split trap MEASURED LIVE (torch.version. cuda 13.0 -> gate True vs default nvcc symlink 12.9); entry 317's subsume-vs-land table + the fraction placed AFTER entry-219's override + the 106.9->100 GB test-scaffold-not-prod caveat; entry 318 (ONE entry, four commits) the silent-total-L3-loss premise quoted from the base, the always-enter and flag-then-publish invariants (the latter correcting my own dispatch sketch), the width-8 interim, non-consuming extra-config validation, and the new hicache_storage_healthy gauge. Battery: every predicted number reproduced EXCEPT the worker's own final mem_cache read (1887P/155F) -- my tip is 1892P/153F with a failure SET identical to base (diff 0 lines); their 2 extra reds are entry-290 dir-context flake, recorded not propagated. Nothing pushed.
…gl-project#44462(317)/sgl-project#44342-trunk(318) in both the FAM-A3b block and the B6 re-triage verdicts; 10 dated corrections (mergeable flips: sgl-project#46191 REAL -> WATCH-HIGH with the re-derive-vs-QSA_RING_GROUPS=2 instruction, sgl-project#39845 DISPUTED as UNKNOWN-not-true plus the general 'UNKNOWN is no signal' rule; sgl-project#44344 -> SECTION R DOA with the entry-268 unconditional-clear subsumption citation and the mergeable-is-a-trap note; sgl-project#42525 target path moved fla->linear; sgl-project#45520 STILL OPEN -> re-apply rule ARMED FOR MERGE sgl-project#6; b12x-package archived claim DISPUTED with the frozen cutlass-dsl==4.8.0 pin risk standing; sgl-project#46153+sgl-project#41624 gated as ONE QAD bundle; sgl-project#37106 PARKED on eroded motivation with named revive conditions; sgl-project#32479 gated behind the PCIe-IPC measurement and DE-COUPLED from sgl-project#45391; sgl-project#44462's NEXT note spent); NEW S-item S-B12XDENSE filed
… commits since 2a6c204 @ merge-sgl-project#5) Conflicts: 110 files (106 content / 4 add-add / 0 modify/delete) at the settled 321 tip 2d87757 -- vs merge sgl-project#5's 167; the smaller count despite 950 commits (merge sgl-project#5: 1,120) is the merge-era's re-plant landing INSIDE upstream's new window (many fork families -- qwen4 varlen, fast_file, hicache-liveness, the 209-218 controller semantics -- now collide as DUAL IMPLEMENTATIONS: upstream grew the same families independently in the window). By class: A 7 (auto-takes-upstream, dedupe-verified), B 36 (mechanical union/custom splices), C 46 (upstream structure taken, fork hunks extracted for re-plant), D 21 (fork-owned flips + dual-impl adjudications, incl. the whole fast_file family). 228 conflict regions total: 69 union (all def/class-collision-checked), 118 take-upstream (104 marked, 14 dedupe-verified), 25 keep-fork (17 fold-marked, 8 strict supersets), 16 custom import/region splices. 121 TODO(merge6) markers planted (tree census): b1 12 / b2 23 / b3 18 / b4 39 / b5 17 / b6 12. (Resolver census at region-resolution was 117; the cascade-repair grafts below added the rest.) REPLANT INVENTORY: /tmp/opencode/merge6-dossier/ replant/ -- per conflicted file: <name>.fork-hunks.patch (full 2a6c..321 delta), <name>.upstream-side.patch (full 2a6c..eb73 delta), <name>.ours-dropped.txt / <name>.theirs-dropped.txt region snapshots, MANIFEST.tsv (file, zone, regions, kinds, fork-hunk counts, per-region notes). Work records (conflict census JSON, decision tables, class table, logs) under .../work/. D-list outcomes (must-not-auto-resolve, handled deliberately): - sgl-project#45520 (entry 302): STILL OPEN upstream at eb73a50 (gh-verified + ZERO "sgl-project#45520" strings in the +950 window). Entry-230's fail-safe layer is UNTOUCHED and stays (standard.py warn-once + published-auto guards intact; the re-apply rule armed in CAVEATS was NOT needed this merge). - sgl-project#44590 (entry 228): the window did NOT move the floor further -- upstream pyproject stays sglang-kernel==0.4.10 (absorbed at merge sgl-project#5 via sgl-project#43922) and the engine assert_pkg_version stays 0.4.9 in BOTH trees; the _load_inline_fragment_probes_available() probe never existed on either side (docs-mention only). Nothing to re-assert; both pins kept byte-identical. - flashinfer_cutlass.py (fork-owned per merge-sgl-project#5 flip): did NOT conflict; 56 b12x refs intact, dispatcher reads untouched. - modelopt_quant sgl-project#44993/sgl-project#44994-ERA B12X W4A16 arms: the fork's defer-swizzle/b12x conditions at L2941-2996 are CONTESTED against upstream's skip_swizzled_blockscales rewrite -> took upstream, b5 markers + region snapshots (boot-integrity fields live in the fork-owned cutlass file, which survived; the b12x selection ARMS need the B5-style re-plant). - DUAL-IMPLEMENTATION FAMILIES (the new pattern of this merge; upstream grew the same features the fork replanted -- take-upstream + fold-diff replant, NEVER a silent pick): kernels/ops/mamba/qwen4_short_conv.py + its test suite, srt/models/qwen4_exp.py varlen trio, mem_cache/storage/fast_file/fast_file_store.py + test_hicache_fast_file_unit.py, test_hicache_sidecar_restore.py, fused_moe_triton/layer.py nvfp4-deferred clause, tokenizer_manager _validate_top_logprobs_num, hicache_best_practices.mdx. All fork-side versions snapshotted; replant batches diff ours-dropped.txt against the taken upstream. - environ.py: took upstream's SGLANG_EXACT_CHUNK_FILL removal; kept ONLY the fork-added SGLANG_SPLIT_PREFILL_AT_MAMBA_FORK line (custom splice). - entry-219 gauge schema: per-pool hicache_host_pool_{used,total}_tokens names SURVIVED the auto-merge (metrics_collector/hicache_pool_stats/metrics_reporter/ urc all intact); the marked b3 controller regions must RE-VERIFY the producer dict end-to-end at re-plant, not assume it. Cascading silent-auto-merge repairs (files NOT in the conflict set, broken by auto-merge splices -- all found by the ruff-F821/check_* gates, all fixed here): utils/common.py upstream prune RE-HOMED the fork's get_mixed_precision_moe_quant_algos into configs/quant_utils.py (next to the moved siblings) + the make_layers empty_cache into layers/utils/layer_list.py, importers rewired; mem_cache/pool_host/base.py silently LOST the entry-sgl-project#44462-era host_memory_budget_bytes() helper whose caller survived -- grafted back; model_executor/runner/eager_runner.py CP helpers retargeted to upstream's NEW layers/cp/utils (cp_shard_model_inputs/cp_gather_after_forward) replacing the retired runner_utils.cp_extend pair; runner/flashinfer_autotune.py + qsa/mqa.py + modelopt_quant.py import retargets for the utils/device + layers/dp_attention move; mem_cache/memory_pool.py lossless-verify tuple buffers re-grafted around upstream's replayssm skip-chain; unified_radix_cache.py host-capacity deadline preamble re-grafted (marker kept); qwen4_exp.py _use_qwen4_direct_decode re-grafted (its kernels survived); qwen4_exp_ple_table.py _HOST_PAGE_TABLES memo dict re-grafted; rotary_embedding/factory.py get_rope param restored; multimodal_gen forward_context teacache_plan param restored + server_args.py Literal import; parallel_executor.py rebuilt as fork-owned + upstream's raise_if_cancelled guards folded into the fork loop structure; hybrid_cache_controller.py _reduce_prefetch_ack if/zero_ splice repaired; python_bindings.rs pyo3 signature REORDERED to match the unioned fn new() argument order (mamba_value_seqlen position -- a positional-bind footgun, fixed pre-commit); test_io_struct.py get_context/MAX_PARALLEL_SAMPLES imports; stat_loggers_di + test_flashinfer_autotune_sync + fused_lora_compose + tp_worker + compressed_tensors + minimax_h3 + hc_mix + scheduler_mixin + pool_configurator + scheduler/tokenizer_manager import prunes/grafts. Gates at commit (CVD=9, prod venv, worktree PYTHONPATH): import sglang OK; check_registered_tests / check_no_bare_pytest_main / check_no_registered_tests_in_package ALL rc=0; ruff (F401,F821,F823,UP037 @ v0.17.1 vs pinned 0.15.1) ZERO new errors on the 3,269 merge-touched .py -- the residual 161 are BYTE-IDENTICAL INHERITED rows present at BOTH parents (kernels/aot, api_types, mm entrypoints/utils, etc.; 0.17-only detections, not merge damage); ruff format clean on the same set; py_compile clean; CARGO sglang-radix-tree check+test GREEN: 966 passed / 0 failed / 1 ignored (vs merge-sgl-project#5-era 943P -- upstream grew the suite +23; NOTE the crate is now `exclude`d from the rust/ workspace with extension-manifests metadata -- `cargo -p sglang-radix-tree` from rust/ no longer works; run inside the crate). NON-ADOPTIONS honored: sglang-kernel floor 0.4.9 (engine assert) untouched; sgl-project#45520 not brought (fail-safe layer intact by rule). Re-plant batches owed (per marker zones): B2-2026 test re-home (23), B3-2026 mem_cache/controllers/rust (18), B4-2026 scheduler/managers/runner (39), B5-2026 moe/quant/kernels/qwen4 (17), B6-2026 environ/arg_groups/docs re-pins (12), B1-2026 diffusion folds (12). PROD BOOT FORBIDDEN until the B3/B4 zones re-plant the entry-209/218 controller semantics (the trailing_endpoint gate will read RED at this commit, exactly as at merge-sgl-project#5's damage state).
…+ re-plant era CLOSED at b86b7c9 (125/125 markers; grep census ZERO verified at this tree): header STACK epoch 321-2026-10-30 -> 329-2026-11-10 (supersession note kept) + +merge6-era-closeout-2026-11-10 trailer line + entries-count 1-329; tip/HEAD re-pinned (era CODE TIP b86b7c9 NAMED with the full wave chain, superseded values dea2641/e9f0ae1838/065fdcdf33/b7c656f69f/5cde7a5ede kept, docs tip recorded as this wave) + a prominent ** LIVE/BOOT STATE ** line (main checkout detached @ 2d87757 = the ONLY boot-validated state, 9b 16:58:52 warm all-needles-hit per prod-health-2026-11-10.txt; era branch CPU-green but NOT BOOTED -- switching prod = operator-chosen checkout + the entry-326 boot-watch). ENTRY 324: imports+bootfix trunk wave -- sgl-project#48781 require_attn_tp_gather->dp_attention relocation invisible to markers, base_runner deletion-only fix unmasking the 145E/32E/31E collection cascade (1dcc2d6); THEN the operator-live crash: maybe_init_router_registration read tokenizer_ipc_name before init_ipc_channels at OUR fork's entry-201 init order, call moved back to upstream's position, RED(2F)/GREEN(2P) test_tokenizer_manager_init_order.py, family scan clean + report-only carry (10a1c08). ENTRY 325: RE-PLANT ERA TABLE b1 12 / b3 18 / b4 39 / b5 17 / b2 23 / b6 16 = 125/125 per-wave dispositions (compressed from the commit bodies, all shas cited) + the NINE ERA CATCHES with FULL CREDIT: _rope_cache_key 10/11-arg universal get_rope crash (15 spec F were its fingerprints), PLE file-backend env registration gap (AttributeError every boot), COW-catcher doubled-staging-arm cascade bug, stat_loggers DI test restored from its corrupted merge hybrid, w13 refined-grid mis-key, the budget-contract decision (fork accounting RESTORED, evidence 106.9GB/100GB = the entry-317 leak shape; INDEXER collective order verified), verify-gate SUBSUME pick (sgl-project#44047 eats sgl-project#42779, both intents preserved), mm_process_config ambient-widths root-cause (validators now state the full 8-way layout), TLFIX twin re-pinned 282/285. ENTRY 326: BATTERY RE-PIN TABLE -- era pins managers 1307P/0F, spec 406P/0F, server_args 496P/0F, sampling 252P/0F, distributed 213P/0F, mem_cache 2965P/24F (the ACCEPTED device-only 24-node list over 8 files), cargo 966P/0F/1-ignored, five registration/wheel gates rc=0 + the PATH-TO-BOOTABLE statement: CPU-proven vs GPU-window debt -- first-boot needles incl. the sampling.cuh-vs-0.7.2 JIT arbitration (if it JIT-fails, wheel-trio-then-pin per S-FI073, never a lone re-pin), autotune caches SHOULD hit warm (no retune owed on 0.7.2), entry-322 verify-at-boot quartet, and the still-riding 321-era watches (sgl-project#45472/sgl-project#46643 TTFT+starvation, first-delta tool-stream latency). ENTRY 327: WHEELS & VENV RECORD -- pairing matrix (FI 0.7.2 HELD, sgl-kernel 0.4.10/0.4.10 era-consistent with the RETIRED-rule cross-ref, torch/cutlass-dsl MATCH), blake3 1.0.11 INSTALLED with ZERO-DRIFT PROOF (0 battery lines vs ~380; matrix re-read unchanged; negative-shim prefix-sim causality) + no-pip/uv-note (operator lane), the two tree-side wheel reverts with reachability + residual-symbol evidence, check_dockerfile_wheel_pins re-green, FI-0.7.3 adoption contract restated with post-era facts (upstream code now expects 0.7.3 in one reported JIT path; tree HELD 0.7.2; first boot arbitrates), dist-name/module-name note sglang_kernel/sgl_kernel. ENTRY 328: INCIDENT RECORD 2026-11-10 ~16:4x-16:58, honest narrative -- re-plant waves landed into the EDITABLE-IMPORT working tree while :30000/:30001 ran; 9b died ~pre-16:50 (cause UNVERIFIABLE from the user acct, /root logs unreadable; worker-respawn-importing-changed-code is the LEADING MECHANISM, stated AS SUCH); the operator restart hit the init-order crash = the era's flagship find; recovery = merge6-era-backup @ b069e6a + detached checkout 2d87757, 9b re-booted WARM 16:58:52 all needles hit, 9d NEVER dropped (Nov 3, HTTP 200); NEW ERA LAW: working tree = prod import path, integration by branch ref-move ONLY; b2's emergency-freeze WIP 7ccf76a (author 16:55:02) kept in range and completed as C1 -- provenance note. ENTRY 329: BACKFILL AUDIT RECORD -- window 2026-08-10..2026-10-27, universe 2714 = 1830 in-tree / 255 already-documented / 628 NEW REJECTS / 1 NEW-INTERESTING; method one-liners; the qwen4-main-squashed side-branch class discovery (REJECT-MERGED-TO-SIDE-BRANCH x44 -- future sweeps MUST filter by base; orchestrator's "206" NOT FOUND in the artifacts, 44 recorded as verified); sgl-project#36043 NVFP4 skinny GEMM SM120 full paragraph (closed-unmerged, absent-tree-verified, W4A4 M<=16 = our exact decode cell, BENCHMARK-BEFORE-PORT -> S-36043-BENCH filed); artifact paths. EVALUATED: the 671-line DOCS-BLOCK.txt inserted VERBATIM as its own dated block (165 dropped list-commas tidied at insertion, noted once at the block head; nothing else reflowed). DATED CORRECTIONS where the waves superseded 322/323-era operational text: the header PROD-BOOT-FORBIDDEN bullet and entry 323's WHEEL DRIFT + STATE lines get [11-10 ERA-CLOSE CORRECTION] annotations (classes closed, wheel landed, ban now purely the operator gate); marker-zone math re-verified against the wave ledgers (121-at-merge vs 125 recount stands correct; planted == consumed per zone). WATCH LIST: era-close block appended (sgl-project#36043 bench-gated; sgl-project#42790/sgl-project#40916 arrived-via-merge6 noted; sgl-project#43703 revert-trap cross-ref from entry 323; sgl-project#45520 STILL-OPEN re-confirmed at era close -- zero in-window refs, b5 verified the fail-safe layer intact; bootfix's report-only get_tokenizer_worker_class/entrypoints wiring carry). Docs-only: NO .py touched; check_registered_tests rc=0 re-run against this tree. NOTHING PUSHED; the live tree was NOT touched by this wave (ref-move integration follows with HEAD/status equality proofs).
Motivation
ModelOpt NVFP4 MoE checkpoints crash at warmup on SM120 with the default
--moe-runner-backend auto. On one RTX PRO 6000 Blackwell (SM120), main4eb606b2afails this way fornvidia/Qwen3-30B-A3B-NVFP4(NVFP4 experts) and fornvidia/Qwen3.6-35B-A3B-NVFP4(MIXED_PRECISION with W4A16_NVFP4 experts):The SM110/SM120 rule never sees the checkpoint's quantization. #41765 extended an existing SM120 rule to SM110; it resolves
autotoflashinfer_cutlassformodelopt_fp4, but it compares--quantization, which stays unset when the quantization comes from the checkpoint, as in the usual launch. SoModelOptNvFp4FusedMoEMethod.create_moe_runnerresolvesautoto FLASHINFER_TRTLLM for every capability 10.0 and up, while the weight preparation follows the global backend (stillauto) and never creates the TRT-LLM scales.Modifications
--moe-runner-backend autoand--quantizationnot given,_moe_runner_backend_quant_constraintsnow uses the quantizationModelConfigresolved from the checkpoint.modelopt_mixedcheckpoint whose experts areW4A16_NVFP4getsmarlin, which the Nemotron-H override and the SM100 Qwen3 MoE override already pick for W4A16 MoE layers underauto.modelopt_fp4keepsflashinfer_cutlass, and amodelopt_mixedcheckpoint whose experts areNVFP4now gets it too. The log line names the backend and quantization it picked.get_mixed_precision_moe_quant_algosmoves frommodel_overrides/qwen3_moe.pytoconfigs/quant_utils.py, so the Qwen3 MoE override and this rule use the same function.test_sm1xx_modelopt_nvfp4_moe_auto_avoids_trtllm_genintest_model_overrides.pycovers mixed W4A16 experts, mixed NVFP4 experts and plain NVFP4, with the platform flag set to SM120 and to SM110.An explicit
--quantization modelopt_mixedis handled by the same branch, and an explicit--quantization modelopt_fp4keepsflashinfer_cutlass. The platform flagis_sm120matches every capability 12.x (with CUDA 12.8 or newer), so SM121 takes the same path. I have no SM110 device; the rule is shared with SM120 and the test runs both flags.Known limits:
--quantization modeloptstays onauto, as on main. The FlashInfer Cutlass check inhandle_moe_kernel_configaccepts only the specific ModelOpt names, so resolving the generic flag needs that check changed too.--quantization modelopt_fp4keepsflashinfer_cutlass, as on main.modelopt_mixedcheckpoint that declares its expert algorithms only inhf_quant_config.jsonstays onauto; the shared helper readsconfig.json'squantization_config, as the Qwen3 MoE and Nemotron-H overrides do.W4A16_NVFP4checkpoints resolve tomodelopt_fp4and getflashinfer_cutlass; I have no such MoE checkpoint to test.Related PRs:
modelopt_mixedto the same check. It helps only an explicit--quantization modelopt_mixedand sends W4A16 experts toflashinfer_cutlass.create_moe_runnerto FLASHINFER_CUTLASS on SM12x and makesenable_flashinfer_cutlass_moefollow its resolved backend, but leaves the global backend atauto.Accuracy Tests
Both checkpoints now start on
auto, and GSM8K is unchanged within noise against main with the same backend passed explicitly (one RTX PRO 6000 Blackwell Max-Q, TP1, sgl-eval 0.1.3):autoautoThe last column counts the questions only one side answered correctly. Server logs, per-run backend checks, sgl-eval outputs and the scripts are in https://gist.github.com/waynehacking8/a3fb1c44b09958b10998fe5b7c210e40.
The MoE layers ran on the backend in the table: a probe recorded
ModelOptNvFp4FusedMoEMethodwith that backend at runner creation and in the forward for every run, and only this PR'sautoruns log the rule'sUse ... as MoE runner backend on SM110/SM120line.Settings that differ from the cookbook (the Qwen3.6 RTX Spark recipe already pins
--moe-runner-backend flashinfer_cutlass; these runs test the backend choice itself):--mem-fraction-static 0.8.--tool-call-parser qwen3_coderand--mamba-radix-cache-strategy extra_buffer; the runs use--disable-radix-cache.SGLANG_FLASHINFER_AUTOTUNE_CACHE=0, and the eval does not apply Qwen3.6's recommendedpresence_penalty1.5.The GPU runs used this library change as one commit on main
4eb606b2a. The branch now merges main3e8d7c9e09; the only library difference from that commit is where the helper lives and the imports that follow from it, and the branch adds the mixed NVFP4 test case. With main merged,test_model_overrides.pyhas no new failures: 128 passed, and its 3 failures also occur on main without this PR.956b9a24ee, the new test fails withAssertionError: 'auto' != 'marlin'.--quantizationunset, the two checkpoints resolve tomarlinandflashinfer_cutlass, againstautoon main. The explicit-flag cases in the known limits were checked the same way.Speed Tests and Profiling
No kernel changes:
autonow resolves to backends that already exist. Decode TPOT at c=1, accuracy launch above:autoTPOT on this PR's
autoruns is 3.1% higher than main's explicit runs in both rows, one server boot per cell. The diff only changes server-arg resolution, which runs once at startup, and both arms resolve to the same server args apart from the random seed, so the MoE runner path is the same. For Qwen3.6 the gap is within what I saw between main boots: a separate main boot with the crash command's flags (table below) gave 3.33 ms. For Qwen3-30B it isn't: all five of this PR's prompts (4.777 to 4.787 ms) are slower than all ten from main's two flashinfer_cutlass boots, this table's and the crash-flags boot below (4.257 to 4.669 ms). This PR's boot came first in both pairs, so boot order is one candidate; I haven't found the cause. I can run three boots per arm in alternating order if a reviewer wants it settled.Backend comparison on main
4eb606b2awith the crash command's flags and the backend passed explicitly, one boot each:flashinfer_cutlass also runs Qwen3.6, which ships expert input scales, but marlin decodes faster, so
autopicks marlin. For NVFP4 experts this PR keeps the SM120 rule's existing flashinfer_cutlass choice. Marlin also runs Qwen3-30B on main and decodes faster here; I left that choice alone because this PR is about the crash, and it may be worth a separate look.TPOT is the median over 5 random 512-token prompts, measured through
/generate. Each prompt's value is:Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #42571317653
Latest PR Test (Extra): ❌ Run #42571317533
Latest PR Test (AMD ROCm 10): ❌ Run #42571317521
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: