Repository navigation
[Refactor] moe: Derive layer contracts from the selected quant method - #44993
Merged
Merged
Conversation
ch-wan
requested review from
1am9trash,
AgainstEntropy,
Alisehen,
AniZpZ,
BBuf,
BowenBao,
Edwardf0t1,
FlamingoPg,
Fridge003,
HaiShaw,
Jiminator,
OrangeRedeng,
Ying1123,
b8zhong,
fzyzcjy,
hubertlu-tw,
ispobock,
kevin-mii,
kkHuang-amd,
kpham-sgl,
merrymercy,
mickqian,
mmangkad,
niehen6174,
ping1jing2,
pyc96 and
yichiche
as code owners
October 18, 2026 12:06
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
…20 commits since 12dc623 @ merge-sgl-project#4) Conflicts: 167 files (149 content / 14 add/add / 4 modify/delete) at the settled 309 tip d28e7cd -- dossier predicted 146 +/-25; the +21 shift is entries 275-309 landing qsa-ring/hicache-ctrl/gdn/moe/llguard deltas into the conflict zone. By class: A 13 (auto-takes-upstream, dedupe-verified), B 80, C 56, D 18. 102 TODO(merge5) re-plant/fold markers planted: B1 x15, B2 x29, B3 x12, B4 x23, B5 x23. Re-plant stock: 117 files under /tmp/opencode/merge5-dossier/replant/ (76 fork-hunks patches, 19 upstream-side patches, 14 AA fork copies, 8 preremove snapshots) -- sessions 2/3 re-plant from real code, not memory. D-list outcomes (must-not-auto-resolve, handled personally): - sgl-project#45289-vs-sgl-project#44858 (entry 301): VERIFIED, not re-resolved -- the gate/up-order final form is in origin/main (5b1dbc2 in range); scales test taken as upstream oracle, fork 191-line version saved to replant/ for B5 diffing. sgl-project#44858's call-site shape CANNOT survive (our base never had it). - entry-230 dispatcher vs sgl-project#44993/sgl-project#44994: took upstream structure in token_dispatcher/standard.py; published-value CONTRACT lives upstream via get_moe_runner_backend() in the LoRA-EP guard; fork's warn-once mismatch diagnostic re-plants (B5 marker). UNION GATE: standard_dispatcher 16P (was 12P, suite grew), model_overrides 121P/1S/210sub (was 112P/1S/181sub), fold probe 6P/2F (upstream added a runner_backend arg to the predicate -- B5 re-pins the probe). - entry-219 gauge names vs sgl-project#43727 return + sgl-project#44968: PER-POOL NAMES KEPT as the serving schema -- hicache_host_pool_{used,total}_tokens{pool} dataclass fields, Gauge registrations and labels(pool=) emission re-restored over the #4697x structure; hicache_host_pool_evicted_tokens_total already survived auto-merge; producer dict restored by union in metrics_reporter. ANTI-NEEDLE SATISFIED: old mamba-only gauges are NOT the serving schema. - entry-221 eager-direct add/add: DEDUPED to upstream (bodies identical except est_time 15->9); MEASUREMENT DEBT RECORDED: prefill t/s must be re-measured post-B4 before trusting the est_time. - entry-168 D2H fence vs sgl-project#44800: the fence is ALREADY UPSTREAM (both load_fence_stream + write_fence_stream with the load-thread-corruption comment in origin/main's cache_controller.py) -- nothing re-planted; the two real regions resolved as import-union + comment wording (merge sgl-project#4 stale-tail pattern avoided). Fork hunks still saved to replant/ (belt). - entry-209 clamp + 216/218 ordering + 288-295 hicache-ctrl-era vs #46970x KV rewrite: ADOPTED-UPSTREAM across unified_radix_cache/hicache_storage/ hybrid_cache_controller/memory_pool/kv_cache_builder; EVERY fork hunk extracted to replant/*.fork-hunks.patch; B3 re-plant markers planted. sgl-project#45703's short-full-match seam arrives with upstream. - sgl-project#45090 SGLANG_ENABLE_UNIFIED_RADIX_TREE: REMOVAL TAKEN (fork had already moved it to _DeprecatedEnv; now gone). grep: ZERO .py readers repo-wide; 2 upstream-owned docs jsx snippets still export it (deepseek-v4, glm-5.2 -- upstream's own debt). OPERATOR-OWED CHECK: confirm no /root harness exports it (unreadable from this seat). - sgl-project#39727 unified L3 key: VERIFIED OPT-IN -- merged hicache_storage.py keeps the DEFAULT legacy suffix composition _<model>_<tp_rank>_<tp_size> (content-anchor: HiCacheFile._init_config_suffix, raise-guarded when _unified_chunk_spans is set). ANTI-NEEDLE: no silent key-scheme switch; warm /root/hicache_store contents stay addressable. Modify/delete (4, all victims of upstream sgl-project#45672 test prune + URC split): - test_unified_radix_cache_unittest.py: REMOVED (adopted split) -- its bare unified_tree_core_inspector imports are unreachable post-sgl-project#45236 re-home; fork source + hunks saved; B2 marker planted in the new core test. - test_fused_moe_triton_config.py: REMOVED -- imports common_utils, absent from both sides post-sgl-project#45672; the 2 live entry-3/235 tests re-plant in B2. - test_scheduler_batch_is_full_latch.py: REMOVED -- fork delta was 3 stub-attr lines documenting sgl-project#46643's absorbed arms; upstream deletion correct. - test_qsa_kv_pool.py: KEPT -- fork's 147 added lines are entry-187/200 sharding + 285-287 QSA-ring tests with NO upstream replacement; qsa_kv_pool.py sides converged byte-identical; collects clean. B2 marker. Cascading silent-auto-merge repairs (classes not in the conflict set but broken by auto-merge + union splices -- all found by gates, now fixed): kernels/ops/activation/__init__.py (dead _CUDA registration dup), kernels/ops/ diffusion/__init__.py (5-tuple _SPECS entry), kernels ops embeddings/moe/ attention/mamba __init__ (KernelSpec fields removed upstream: description/ format_signature stripped, CapabilityRequirement->CUDA_ONLY), base_runner/ eager_runner cp_extend + shared_read_event imports retargeted per upstream's "no imports in runner_utils/__init__" contract, and 4 fork helpers grafted back onto their take-upstream homes because merged code/tests still import them (memory_pool.spec_intermediate_bytes_per_req, kv_cache_builder. _adjust_hicache_size_for_draft + _device_pool_size_per_token, hybrid_cache_controller.local_restorable_mask, validation_hook. validate_replayssm_spec_algorithm, arg_groups/moe_hook b12x validator). flashinfer_cutlass.py flipped to FORK-OWNED (b12x route is fork-only, 760 vs 95 lines; upstream sgl-project#44993 hunks = B5 fold patch saved). Silent-semantics notes: sgl-project#46127/sgl-project#44912 riders arrived via per-PR picks (union); sgl-project#45520 (entry 302) is STILL OPEN upstream -- entry-230 fail-safe layer stays as entry 303 records; if a future sync brings it, re-apply the fail-safe reads. qwen4_exp holder-contract fused (ctor takes optional batch). Gates at commit: import sglang OK (CVD=9, prod venv); check_registered_tests/ check_no_bare_pytest_main/check_no_registered_tests_in_package rc=0; ruff F821 +E9+format clean on the resolution set; collect-only over unit/{mem_cache, model_executor,distributed,layers/moe} = 4337 nodes, 3 ERRORS: 2x cargo/torch- sys libtorch probe (REPRODUCED IDENTICALLY AT THE PRE-MERGE TIP -- env issue, not merge damage) + test_hicache_async_ready_sync.py (entry-253-era attribute gone with the URC rewrite -- EXPECTED pre-B3). sessions 2/3 owed: B2 test re-home, B3 mem_cache+rust re-plant (+hicache-ctrl re-ports), B4 scheduler/admission, B5 moe/quant/dispatcher, B6 environ+C0a+ CAVEATS re-pin. PROD BOOT FORBIDDEN until B3 completes (209 clamp + 218 ordering re-plant).
05yuki
pushed a commit
to 05yuki/sglang-upstream
that referenced
this pull request
Oct 30, 2026
Since sgl-project#44993 the top-k folds routed_scaling_factor into its weights whenever the quant method says so, and KTEPWrapperMethod forwards that question to its GPU method (the MXFP4 ones answer yes). DeepseekV2MoE still multiplied the routed output by the factor again for every KT layer, so the routed experts came out 1.5x too strong on DeepSeek V4.1. Paired NLL on 11,496 tokens against the checkpoint's own inference code: 2.878 before, 2.596 after (reference 2.594).
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 30, 2026
…-arg predicate -- entries 230/239-era fold contract Upstream sgl-project#44993 (4f8f06f, arrived in-range) changed _fuses_routed_scaling_factor_in_topk(quant_method) to (quant_method, runner_backend): the runner now arrives explicitly from FusedMoE.__init__ (layer.py:652, quant_method.get_moe_runner_backend), and the Marlin exclusion widened to is_marlin() or is_lora_marlin(). The probe's 2F were its stub calling the OLD shape (TypeError: missing runner_backend). Re-adapted WITHOUT weakening beliefs: _layer_fold_verdict() mirrors the layer's call shape; stub keeps _moe_runner_backend feeding through the real getter. Docstring updated with the sgl-project#44993 delta. NEW pin added: lora_marlin does not fold (the widened exclusion arm is now under contract). Battery ACTUALS (CVD=9, prod venv, PYTHONPATH=python): test_modelopt_nvfp4_fold_contract: 6P/2F -> **9P/0F** (damage 2F closed, +1 new lora_marlin case).
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 30, 2026
…d flashinfer_cutlass + entry-230 dispatcher contract re-plant + qwen4_exp PLE router-API re-home -- entries 9,174,225,230,238,244,247,167 THREE BOOT-INTEGRITY/HONESTY ITEMS, one moe-contract cluster: (1) flashinfer_cutlass.py (marker consumed): upstream's in-range delta folded INTO the fork-owned file (merge resolution had already woven the tune-range helper): FlashInferCutlassMoeQuantInfo.pair_w2_weight FIELD (modelopt_quant :3826 PASSED it -- unexpected-kwarg TypeError on the CUDA FP4 cutlass route as merged, B3-class), the sm120 native MTP-MMA route (_sm120_mtp_mma_supported + _flashinfer_autotune_active + try_nvfp4_moe call, upstream's position after _prepare_input / before the fork smallm decision; conditions are disjoint from our gated SwiGLU arm), its fail-loud paired-weight guard, and the all-gather-prefill Standard delegation on the flashinfer-a2a fused func. is_sm120 import added. b12x route + D-3 constants (source_format/w13_layout, flashinfer 0.7.2) untouched; the flip-provenance note replaces the marker. (2) token_dispatcher/standard.py (marker consumed): entry-225's rescue is now IN THE CORE by sgl-project#44993 (mapping keys off the BUILT runner) -- absorbed, not re-planted; the experimental_sgl_trtllm arm's loss is UPSTREAM'S OWN churn (sgl-project#27329 added in-premerge, sgl-project#44993 deleted -- topk.py still special-cases it). RE-PLANTED: the fork-only b12x arm of _keeps_global_expert_ids (without it a named flashinfer_b12x_w4a16 runner gets LOCAL ids from the dispatcher and its own expert_map re-remaps -- EP>1 misroute) and the entry-230 warn-once published-vs-built convention canary, re-anchored on sgl-project#44993's shape (warn once per (published, built) pair at EP>1; the fp4-allgather launch decision still keys off the published value). (3) qwen4_exp.py (marker consumed) + its packed_short_conv test: the two pre-#46970x reads (req_to_token_pool.get_ngram_context, deleted by sgl-project#46970..sgl-project#46991; pool.ngram_pool.context) re-homed onto the layer_router NGRAM_PART view (single read feeds both consumers -- they were always the same data; .context is the same-named buffer the view exposes). Test seam swapped to the router stub (the 1F was PRE-EXISTING at HEAD, verified by stash). Battery ACTUALS (CVD=9, prod venv, PYTHONPATH=python, import sglang OK): fold probe: 6P/2F -> 9P (B5-1, committed separately) standard_dispatcher + runner_backend + b12x_route: 104P/91sub GREEN +swiglu_params +smallm (cluster of 5 files): 137P GREEN pre/post qwen4 PLE family (8 files): was 1F (pre-existing) -> 85P/12S/4sub GREEN ruff F821+E9 + format: clean on all 5 files
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 30, 2026
…ies 28,165,166,186,194,206,296,297,299,300
PER-MARKER TRUTH (census: B5 markers in this set now ZERO):
kernels/aot topk.cu (28): VERIFIED-ABSORBED -- upstream's in-range
legacy_radix_topk::select rewrite gates the threshold-bin population
against the stash BEFORE staging (coarse_population >= StashEntries ->
full-row byte-per-pass refinement), structurally fixing the overflow
the fork patched after-the-fact; jit twin keeps its s_prefix_bins.
kernels/ops/attention/utils.py (194): VERIFIED-ABSORBED -- the function
re-sited to ops/attention/kv_indices.py (spec_kv_index_token_blocks with
the 32768 gate) and every consumer imports it at that module; the saved
utils re-export would be a dead hunk (zero importers).
elementwise/elementwise.py (166): VERIFIED-PRESENT -- the batch-invariant
num_warps clamp rides upstream's guard_or_false refactor ('or
is_batch_invariant_mode_enabled()' intact).
kvcache/__init__.py (186): VERIFIED-PRESENT -- the store_cache_fp8
registry entry is the merge commit's own graft (entry 310), imports +
capabilities in place.
moe/hash_topk.py + moe/topk.py (296/297/299/300): the merged upstream
canonicals of sgl-project#44901/sgl-project#44904/sgl-project#44912/sgl-project#44913 are IN RANGE (all four
MERGED commits proven ancestors of 2a6c204); every fork symbol
(fused_shared_expert_weight, is_routed_scaling_factor_fused_in_topk,
num_shared_topk_slots x54, has_per_rank_fused_shared_slots,
SharedExpertsConfig wiring) verified present. Markers stale.
moe/flashinfer_trtllm.py (165): SUPERSEDED-BY-EQUIVALENT -- upstream's
own in-range rewrite of the deferred block covers the routed triple
(do_finalize=not defer + _make_deferred_finalize_output) fail-LOUDER
(valid_deferred_input raise); the fork's sgl-project#6077 ignored-autotune-buffer
targeted the OLD block shape and is deliberately NOT restored (GPU
perf-only, defer+routed is not our production arm).
moe/mega_moe.py (296): deep_gemm transform_sf redirect RE-PLANTED to the
guarded wrapper (import-safety contract; sibling fix in dsv4 below).
fused_moe_triton/layer.py (165/206/239): _built_runner_backend hunk
ABSORBED by sgl-project#44993 (config carries the built runner); the entry-206
supports_routed_deferred_finalize property RE-PLANTED (upstream's
packed_nvfp4_routed_deferred covers TRTLLM only, NOT trtllm_routed --
our runner arm), with the sgl-project#44993 2-arg fold predicate; the debug-log OR
re-plant paired.
models/deepseek_v2.py (296/297/300/236): fold chain verified; the
entry-206 format-ternary consumer arm RE-PLANTED (BYPASSED ->
supports_deferred_finalize, else supports_routed_deferred_finalize --
restoring what sgl-project#44993's rewrite dropped).
Battery ACTUALS (CVD=9, prod venv, PYTHONPATH=python, import sglang OK):
fused_shared_expert_scaling + mega_moe_deepgemm_api + moe_runner_
extensions + topk_routing_inputs + model_overrides + kernels_namespace:
GREEN (see B5-4 commit for full 43-file battery actuals)
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 30, 2026
… C closure -- the post-merge port queue is now decided ARRIVAL SCAN (git log 12dc623..2a6c204 --grep) + LIVE gh STATE + head-sha comparison for EVERY deferred item. Verdicts: RELEASED-BY-MERGE: sgl-project#42689 (@ 7a632cf in-range -- the SECTION-B cherry is now redundant; behavior note rides the restart). CLOSED (C-items whose PRs arrived and whose integration this stack performed): C1 (the dedicated merge week WAS merge sgl-project#5; all eight precursor PRs in-range), C2 (sgl-project#44993/sgl-project#44994 -- B5), C3 (sgl-project#45232/ sgl-project#45236/sgl-project#45334 -- B2), C4 (sgl-project#45238/sgl-project#45241/sgl-project#45242 -- B3), C5 (sgl-project#44912/ sgl-project#44913 -- arrived; entry-230 canary re-planted as its guard), C9 (sgl-project#39727 -- verified OPT-IN, no purge; BOOT_DELTA anti-needle), C0a (arrival shas already recorded at entry 310 -- Section C now carries NO open code-integration items: C6/C10 retain only their explicitly-named docs/boot-measurement remainders, re-stated). STILL-OPEN: sgl-project#44342 (head unchanged; auto-merge comfort LOST to #46970x), sgl-project#46153/sgl-project#37106/sgl-project#32479/sgl-project#45391 (gh CONFLICTING -- stale bases, rebase before any port), sgl-project#41624/sgl-project#42525/sgl-project#42660/sgl-project#44462/sgl-project#35617 (open, heads unchanged), sgl-project#46191/sgl-project#39845-class (watch-list, none in range). HOLD STANDS (operator decision, binding): sgl-project#43356/sgl-project#46452/sgl-project#43240/ sgl-project#43471 KV route -- step-1 'MERGE sgl-project#5 lands first' is NOW MET; the sequence may be re-visited ONLY with the post-merge PCIe-IPC measurement at an operator sitting. sgl-project#44732: the 'one purge if done WITH the merge' premise LAPSED when merge sgl-project#5 completed without it -- later execution costs the SAME single purge once; no penalty, no haste. Recorded for the operator, not decided by B6.
This was referenced Oct 31, 2026
05yuki
pushed a commit
to 05yuki/sglang-upstream
that referenced
this pull request
Nov 1, 2026
Since sgl-project#44993 the top-k folds routed_scaling_factor into its weights whenever the quant method says so, and KTEPWrapperMethod forwards that question to its GPU method (the MXFP4 ones answer yes). DeepseekV2MoE still multiplied the routed output by the factor again for every KT layer, so the routed experts came out 1.5x too strong on DeepSeek V4.1. Paired NLL on 11,496 tokens against the checkpoint's own inference code: 2.878 before, 2.596 after (reference 2.594).
This was referenced Nov 2, 2026
05yuki
pushed a commit
to 05yuki/sglang-upstream
that referenced
this pull request
Nov 4, 2026
Since sgl-project#44993 the top-k folds routed_scaling_factor into its weights whenever the quant method says so, and KTEPWrapperMethod forwards that question to its GPU method (the MXFP4 ones answer yes). DeepseekV2MoE still multiplied the routed output by the factor again for every KT layer, so the routed experts came out 1.5x too strong on DeepSeek V4.1. Paired NLL on 11,496 tokens against the checkpoint's own inference code: 2.878 before, 2.596 after (reference 2.594).
This was referenced Nov 5, 2026
detain
added a commit
to detain/sglang
that referenced
this pull request
Nov 11, 2026
… commits since 2a6c204 @ merge-sgl-project#5) Conflicts: 110 files (106 content / 4 add-add / 0 modify/delete) at the settled 321 tip 2d87757 -- vs merge sgl-project#5's 167; the smaller count despite 950 commits (merge sgl-project#5: 1,120) is the merge-era's re-plant landing INSIDE upstream's new window (many fork families -- qwen4 varlen, fast_file, hicache-liveness, the 209-218 controller semantics -- now collide as DUAL IMPLEMENTATIONS: upstream grew the same families independently in the window). By class: A 7 (auto-takes-upstream, dedupe-verified), B 36 (mechanical union/custom splices), C 46 (upstream structure taken, fork hunks extracted for re-plant), D 21 (fork-owned flips + dual-impl adjudications, incl. the whole fast_file family). 228 conflict regions total: 69 union (all def/class-collision-checked), 118 take-upstream (104 marked, 14 dedupe-verified), 25 keep-fork (17 fold-marked, 8 strict supersets), 16 custom import/region splices. 121 TODO(merge6) markers planted (tree census): b1 12 / b2 23 / b3 18 / b4 39 / b5 17 / b6 12. (Resolver census at region-resolution was 117; the cascade-repair grafts below added the rest.) REPLANT INVENTORY: /tmp/opencode/merge6-dossier/ replant/ -- per conflicted file: <name>.fork-hunks.patch (full 2a6c..321 delta), <name>.upstream-side.patch (full 2a6c..eb73 delta), <name>.ours-dropped.txt / <name>.theirs-dropped.txt region snapshots, MANIFEST.tsv (file, zone, regions, kinds, fork-hunk counts, per-region notes). Work records (conflict census JSON, decision tables, class table, logs) under .../work/. D-list outcomes (must-not-auto-resolve, handled deliberately): - sgl-project#45520 (entry 302): STILL OPEN upstream at eb73a50 (gh-verified + ZERO "sgl-project#45520" strings in the +950 window). Entry-230's fail-safe layer is UNTOUCHED and stays (standard.py warn-once + published-auto guards intact; the re-apply rule armed in CAVEATS was NOT needed this merge). - sgl-project#44590 (entry 228): the window did NOT move the floor further -- upstream pyproject stays sglang-kernel==0.4.10 (absorbed at merge sgl-project#5 via sgl-project#43922) and the engine assert_pkg_version stays 0.4.9 in BOTH trees; the _load_inline_fragment_probes_available() probe never existed on either side (docs-mention only). Nothing to re-assert; both pins kept byte-identical. - flashinfer_cutlass.py (fork-owned per merge-sgl-project#5 flip): did NOT conflict; 56 b12x refs intact, dispatcher reads untouched. - modelopt_quant sgl-project#44993/sgl-project#44994-ERA B12X W4A16 arms: the fork's defer-swizzle/b12x conditions at L2941-2996 are CONTESTED against upstream's skip_swizzled_blockscales rewrite -> took upstream, b5 markers + region snapshots (boot-integrity fields live in the fork-owned cutlass file, which survived; the b12x selection ARMS need the B5-style re-plant). - DUAL-IMPLEMENTATION FAMILIES (the new pattern of this merge; upstream grew the same features the fork replanted -- take-upstream + fold-diff replant, NEVER a silent pick): kernels/ops/mamba/qwen4_short_conv.py + its test suite, srt/models/qwen4_exp.py varlen trio, mem_cache/storage/fast_file/fast_file_store.py + test_hicache_fast_file_unit.py, test_hicache_sidecar_restore.py, fused_moe_triton/layer.py nvfp4-deferred clause, tokenizer_manager _validate_top_logprobs_num, hicache_best_practices.mdx. All fork-side versions snapshotted; replant batches diff ours-dropped.txt against the taken upstream. - environ.py: took upstream's SGLANG_EXACT_CHUNK_FILL removal; kept ONLY the fork-added SGLANG_SPLIT_PREFILL_AT_MAMBA_FORK line (custom splice). - entry-219 gauge schema: per-pool hicache_host_pool_{used,total}_tokens names SURVIVED the auto-merge (metrics_collector/hicache_pool_stats/metrics_reporter/ urc all intact); the marked b3 controller regions must RE-VERIFY the producer dict end-to-end at re-plant, not assume it. Cascading silent-auto-merge repairs (files NOT in the conflict set, broken by auto-merge splices -- all found by the ruff-F821/check_* gates, all fixed here): utils/common.py upstream prune RE-HOMED the fork's get_mixed_precision_moe_quant_algos into configs/quant_utils.py (next to the moved siblings) + the make_layers empty_cache into layers/utils/layer_list.py, importers rewired; mem_cache/pool_host/base.py silently LOST the entry-sgl-project#44462-era host_memory_budget_bytes() helper whose caller survived -- grafted back; model_executor/runner/eager_runner.py CP helpers retargeted to upstream's NEW layers/cp/utils (cp_shard_model_inputs/cp_gather_after_forward) replacing the retired runner_utils.cp_extend pair; runner/flashinfer_autotune.py + qsa/mqa.py + modelopt_quant.py import retargets for the utils/device + layers/dp_attention move; mem_cache/memory_pool.py lossless-verify tuple buffers re-grafted around upstream's replayssm skip-chain; unified_radix_cache.py host-capacity deadline preamble re-grafted (marker kept); qwen4_exp.py _use_qwen4_direct_decode re-grafted (its kernels survived); qwen4_exp_ple_table.py _HOST_PAGE_TABLES memo dict re-grafted; rotary_embedding/factory.py get_rope param restored; multimodal_gen forward_context teacache_plan param restored + server_args.py Literal import; parallel_executor.py rebuilt as fork-owned + upstream's raise_if_cancelled guards folded into the fork loop structure; hybrid_cache_controller.py _reduce_prefetch_ack if/zero_ splice repaired; python_bindings.rs pyo3 signature REORDERED to match the unioned fn new() argument order (mamba_value_seqlen position -- a positional-bind footgun, fixed pre-commit); test_io_struct.py get_context/MAX_PARALLEL_SAMPLES imports; stat_loggers_di + test_flashinfer_autotune_sync + fused_lora_compose + tp_worker + compressed_tensors + minimax_h3 + hc_mix + scheduler_mixin + pool_configurator + scheduler/tokenizer_manager import prunes/grafts. Gates at commit (CVD=9, prod venv, worktree PYTHONPATH): import sglang OK; check_registered_tests / check_no_bare_pytest_main / check_no_registered_tests_in_package ALL rc=0; ruff (F401,F821,F823,UP037 @ v0.17.1 vs pinned 0.15.1) ZERO new errors on the 3,269 merge-touched .py -- the residual 161 are BYTE-IDENTICAL INHERITED rows present at BOTH parents (kernels/aot, api_types, mm entrypoints/utils, etc.; 0.17-only detections, not merge damage); ruff format clean on the same set; py_compile clean; CARGO sglang-radix-tree check+test GREEN: 966 passed / 0 failed / 1 ignored (vs merge-sgl-project#5-era 943P -- upstream grew the suite +23; NOTE the crate is now `exclude`d from the rust/ workspace with extension-manifests metadata -- `cargo -p sglang-radix-tree` from rust/ no longer works; run inside the crate). NON-ADOPTIONS honored: sglang-kernel floor 0.4.9 (engine assert) untouched; sgl-project#45520 not brought (fail-safe layer intact by rule). Re-plant batches owed (per marker zones): B2-2026 test re-home (23), B3-2026 mem_cache/controllers/rust (18), B4-2026 scheduler/managers/runner (39), B5-2026 moe/quant/kernels/qwen4 (17), B6-2026 environ/arg_groups/docs re-pins (12), B1-2026 diffusion folds (12). PROD BOOT FORBIDDEN until the B3/B4 zones re-plant the entry-209/218 controller semantics (the trailing_endpoint gate will read RED at this commit, exactly as at merge-sgl-project#5's damage state).
YaoleWu
added a commit
to YaoleWu/sglang
that referenced
this pull request
Nov 12, 2026
… factor Since sgl-project#44993, TopK folds `routed_scaling_factor` into the top-k weights when the quant method reports `fuse_routed_scaling_factor_in_topk`. The KT wrapper forwards that report from its MXFP4 methods, yet the DeepseekV2 MoE epilogue still scaled the routed output whenever the quant method is a KTEPWrapperMethod — DSv4.1 (factor 1.5) came out at 2.25x per layer. Make the decision single-sourced: `KTEPWrapperMethod` declares a duck-typed `applies_routing_weights_as_given = True` (same style as the fold flag), and one `_post_moe_needs_routed_scaling()` helper replaces the two hand-copied conditionals in `forward_normal_dual_stream` and `_post_combine_hook`. The old `or isinstance(...)` clause made the CUDA short-circuit dead for KT; the helper makes that impossible by construction. KT is scaled exactly once on every device; CUDA/MUSA/XPU/AITER and plain-CPU paths keep their runner- vs epilogue-owned semantics unchanged. Supersedes sgl-project#45810 (absorbing its teacher-forced accuracy evidence: mean NLL 2.5960 vs reference 2.5940, was 2.8780 before). Signed-off-by: wuyaole <112309453+LeslieWylie@users.noreply.github.com>
This was referenced Nov 13, 2026
05yuki
pushed a commit
to 05yuki/sglang-upstream
that referenced
this pull request
Nov 14, 2026
Since sgl-project#44993 the top-k folds routed_scaling_factor into its weights whenever the quant method says so, and KTEPWrapperMethod forwards that question to its GPU method (the MXFP4 ones answer yes). DeepseekV2MoE still multiplied the routed output by the factor again for every KT layer, so the routed experts came out 1.5x too strong on DeepSeek V4.1. Paired NLL on 11,496 tokens against the checkpoint's own inference code: 2.878 before, 2.596 after (reference 2.594).
This was referenced Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR was part of a stack (oldest at bottom):
Motivation
A MoE quantization method can execute on Triton while the requested backend remains FlashInfer or AITER. Dispatch, weight preparation and routing must follow the layer that actually executes. Backend identity also does not prove that a native router implements the model's routing policy: AUTO-resolved NVFP4 must preserve custom routing and renormalization.
Modifications
Defaultenum value 0 remains a real policy, distinct from an unspecified selector.Equivalent native policies are declared for normalized MiMoV2 grouped sigmoid/bias, BailingMoE grouped sigmoid/bias or plain softmax, and Cohere2 sigmoid. Other configurations remain materialized. DeepSeek's logits-only dual-stream graph requires a BYPASSED router even when this PR is used independently.
Llama4 FP8 explicitly selects the model's top-1 sigmoid native policy. Gemma4 NVFP4 materializes its model router so selected-expert softmax weights retain the learned per-expert multiplier. These change behavior relative to the earlier native defaults; CPU contract tests cover the boundaries, but no full-model accuracy A/B is claimed for either model.
BailingLinear resolves its optional AWQ provider only when AWQ weights are loaded, so ordinary model construction does not require
vllm. The Kimi constructor fixture declares the routing selector explicitly.Requested-backend reads remain where they select a method, allocate compatible shapes or establish model-wide policy.
CI fixture repair
Current head:
4275a5026050615139c3658a72b6a7d05d65b17f.The DeepSeek dual-stream test fixture now declares its native runner, BYPASSED TopK format and static expert dispatch policy. The production routing guard remains intact. This repairs the missing
topkerror in the registered CUDA graph test.Validation: the original fixture reproduced the reported error at both the lower and upper PR production revisions. The corrected full test passed on one H200 GPU at both revisions, including opcheck, fullgraph compilation and CUDA graph capture/replay for replicated and non-replicated shared outputs. The tested fixture bytes match this commit. Applicable pre-commit checks passed.
GPU environment: Python 3.12.3, Torch 2.13.0+cu130, Triton 3.7.1, sgl-kernel 0.4.7 and FlashInfer 0.6.18. The test substitutes small CUDA operations for expert kernels; model loading, MoE numerical accuracy and full CI were not rerun for this fixture repair. Earlier validation below remains tied to its stated revisions.
Earlier validation after rebase
Head:
fc2dd3d90bf86fe1b61741c1bb8d28a4e121dc8a. Rebased onto main553836c1647b3bfbeb266b49d76ec2c9df012425; basecheng/hot-fix/server-args-positional-apiis77300dcce02c0b20d68cbc687bb56b54c4a97ace.GPU, full-model accuracy/throughput and full CI were not rerun for this rebase. Prior hardware measurements below remain evidence for their named earlier revisions; checked CUTLASS/Marlin/GGUF consumer sources are unchanged, which is source evidence rather than a new GPU run.
Previous validation (before rebase)
Previous head:
b37b73a1ae1bdf3d73b6108deb76a14679a079b4, based on #44991 and main693d33330647c57f82e09ac39a34122fa46b5763.Earlier validation remains historical: the previous stack passed 856 CPU tests across 43 modules with 7 skips. At
97c954fdf334055dee221483622a5b8ae02c6e05, a mixed block-FP8/BF16 Qwen3MoE smoke on 4 NVIDIA B300 GPUs (TP4/EP4) matched all 24 greedy IDs against Triton, with maximum generated-token logprob difference 0.0039258003. Neither result is full-model accuracy validation of this head.Native TRT-LLM selector parity and full-model accuracy/performance have not been run for these policy changes. Unsupported ModelOpt per-tensor FP8 and compressed MXINT4 materialized-routing capabilities remain limited; rejection still occurs at apply rather than binding, and no compatibility claim is made for unverified conditional checkpoints. ROCm/NPU/XPU/gfx1250 execution and full CI remain unverified. Refs #44291, #40623; related #41636.
CI States
Latest PR Test (Base): ❌ Run #39514001815
Latest PR Test (Extra): ❌ Run #39514001794
Latest PR Test (AMD ROCm 10): ❌ Run #39514001830