Skip to content

[Refactor] moe: Derive layer contracts from the selected quant method - #44993

Merged
ch-wan merged 33 commits into
mainfrom
cheng/refactor/moe-layer-backend-semantics
Oct 20, 2026
Merged

ch-wan merged 33 commits into
mainfrom
cheng/refactor/moe-layer-backend-semantics

Conversation

@ch-wan

@ch-wan ch-wan commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

This PR was part of a stack (oldest at bottom):

Motivation

A MoE quantization method can execute on Triton while the requested backend remains FlashInfer or AITER. Dispatch, weight preparation and routing must follow the layer that actually executes. Backend identity also does not prove that a native router implements the model's routing policy: AUTO-resolved NVFP4 must preserve custom routing and renormalization.

Modifications

  • Quantization methods expose their effective execution backend, including wrapped methods, secondary runners and direct kernels. FusedMoE binds dispatch, expert-ID layout, transport payloads, post-load transforms and each model router to that contract.
  • TRT-LLM in-kernel routing requires a declared policy matching the model's scoring, grouping, bias, renormalization, custom-router and shared-slot behavior. Other routers retain materialized STANDARD output. Explicit Default enum value 0 remains a real policy, distinct from an unspecified selector.
  • TRT-LLM routed adapters consume materialized weights without renormalizing them again. Unsupported materialized per-tensor FP8 input fails before the kernel; an undeclared per-tensor router retains the Triton fallback and its compatible weight layout.
  • Materialized TRT-LLM routes include the routed factor for every effective quantization method, including compressed-tensors FP8/NVFP4 schemes. Grouped/biased fallback producers apply the factor independently of renormalization while preserving shared contributions. Native routers still apply their own factor once. Ordinary loaders zero untouched padded weight tails, including repeated loads.
  • Preserve model-wide DP layout for fallback payloads, gate AITER-specific transforms on the effective backend, and keep weight-only helpers with no runner valid. Binding is idempotent for the same contract and rejects conflicting reuse. The gfx1250 W4A8 internal router rejects unsupported expert sharding.

Equivalent native policies are declared for normalized MiMoV2 grouped sigmoid/bias, BailingMoE grouped sigmoid/bias or plain softmax, and Cohere2 sigmoid. Other configurations remain materialized. DeepSeek's logits-only dual-stream graph requires a BYPASSED router even when this PR is used independently.

Llama4 FP8 explicitly selects the model's top-1 sigmoid native policy. Gemma4 NVFP4 materializes its model router so selected-expert softmax weights retain the learned per-expert multiplier. These change behavior relative to the earlier native defaults; CPU contract tests cover the boundaries, but no full-model accuracy A/B is claimed for either model.

BailingLinear resolves its optional AWQ provider only when AWQ weights are loaded, so ordinary model construction does not require vllm. The Kimi constructor fixture declares the routing selector explicitly.

Requested-backend reads remain where they select a method, allocate compatible shapes or establish model-wide policy.

CI fixture repair

Current head: 4275a5026050615139c3658a72b6a7d05d65b17f.

The DeepSeek dual-stream test fixture now declares its native runner, BYPASSED TopK format and static expert dispatch policy. The production routing guard remains intact. This repairs the missing topk error in the registered CUDA graph test.

Validation: the original fixture reproduced the reported error at both the lower and upper PR production revisions. The corrected full test passed on one H200 GPU at both revisions, including opcheck, fullgraph compilation and CUDA graph capture/replay for replicated and non-replicated shared outputs. The tested fixture bytes match this commit. Applicable pre-commit checks passed.

GPU environment: Python 3.12.3, Torch 2.13.0+cu130, Triton 3.7.1, sgl-kernel 0.4.7 and FlashInfer 0.6.18. The test substitutes small CUDA operations for expert kernels; model loading, MoE numerical accuracy and full CI were not rerun for this fixture repair. Earlier validation below remains tied to its stated revisions.

Earlier validation after rebase

Head: fc2dd3d90bf86fe1b61741c1bb8d28a4e121dc8a. Rebased onto main 553836c1647b3bfbeb266b49d76ec2c9df012425; base cheng/hot-fix/server-args-positional-api is 77300dcce02c0b20d68cbc687bb56b54c4a97ace.

GPU, full-model accuracy/throughput and full CI were not rerun for this rebase. Prior hardware measurements below remain evidence for their named earlier revisions; checked CUTLASS/Marlin/GGUF consumer sources are unchanged, which is source evidence rather than a new GPU run.

Previous validation (before rebase)

Previous head: b37b73a1ae1bdf3d73b6108deb76a14679a079b4, based on #44991 and main 693d33330647c57f82e09ac39a34122fa46b5763.

  • At this exact head, 4 focused CPU modules passed: 28 tests plus 48 reported subtests, covering model policy constructors, shared scaling, ModelOpt dispatch and FP8 ownership. The full StandardDispatcher module also passed 21 tests plus 45 subtests on source bytes verified against this commit.
  • The same new tests fail against the previous production sources for the intended missing scale, non-normalized shared compensation, model admission and dual-stream conditions. Compressed FP8/NVFP4 cases use real scheme/FusedMoE construction, TopK and dispatch with unit/nonunit factors and both renormalization choices. Model policy tests control FP8 quantization/native-provider boundaries; these CPU checks do not execute native quantized GEMMs.
  • The integrated [Refactor] moe: Derive routing inputs from the layer and active forward #44994 head separately passed 119 CPU tests across 12 modules, plus 200 reported subtests. Configured hooks and registration/entrypoint lint pass; strict parsing collects 2,472 files / 3,006 registrations. CPU execution: Python 3.12.15, Torch 2.14.1+cu130, GPUs hidden.

Earlier validation remains historical: the previous stack passed 856 CPU tests across 43 modules with 7 skips. At 97c954fdf334055dee221483622a5b8ae02c6e05, a mixed block-FP8/BF16 Qwen3MoE smoke on 4 NVIDIA B300 GPUs (TP4/EP4) matched all 24 greedy IDs against Triton, with maximum generated-token logprob difference 0.0039258003. Neither result is full-model accuracy validation of this head.

Native TRT-LLM selector parity and full-model accuracy/performance have not been run for these policy changes. Unsupported ModelOpt per-tensor FP8 and compressed MXINT4 materialized-routing capabilities remain limited; rejection still occurs at apply rather than binding, and no compatibility claim is made for unverified conditional checkpoints. ROCm/NPU/XPU/gfx1250 execution and full CI remain unverified. Refs #44291, #40623; related #41636.


CI States

Latest PR Test (Base): ❌ Run #39514001815
Latest PR Test (Extra): ❌ Run #39514001794
Latest PR Test (AMD ROCm 10): ❌ Run #39514001830

detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…20 commits since 12dc623 @ merge-sgl-project#4)

Conflicts: 167 files (149 content / 14 add/add / 4 modify/delete) at the settled
309 tip d28e7cd -- dossier predicted 146 +/-25; the +21 shift is entries
275-309 landing qsa-ring/hicache-ctrl/gdn/moe/llguard deltas into the conflict
zone. By class: A 13 (auto-takes-upstream, dedupe-verified), B 80, C 56, D 18.
102 TODO(merge5) re-plant/fold markers planted: B1 x15, B2 x29, B3 x12, B4 x23,
B5 x23. Re-plant stock: 117 files under /tmp/opencode/merge5-dossier/replant/
(76 fork-hunks patches, 19 upstream-side patches, 14 AA fork copies, 8
preremove snapshots) -- sessions 2/3 re-plant from real code, not memory.

D-list outcomes (must-not-auto-resolve, handled personally):
- sgl-project#45289-vs-sgl-project#44858 (entry 301): VERIFIED, not re-resolved -- the gate/up-order
  final form is in origin/main (5b1dbc2 in range); scales test taken as
  upstream oracle, fork 191-line version saved to replant/ for B5 diffing.
  sgl-project#44858's call-site shape CANNOT survive (our base never had it).
- entry-230 dispatcher vs sgl-project#44993/sgl-project#44994: took upstream structure in
  token_dispatcher/standard.py; published-value CONTRACT lives upstream via
  get_moe_runner_backend() in the LoRA-EP guard; fork's warn-once mismatch
  diagnostic re-plants (B5 marker). UNION GATE: standard_dispatcher 16P (was
  12P, suite grew), model_overrides 121P/1S/210sub (was 112P/1S/181sub),
  fold probe 6P/2F (upstream added a runner_backend arg to the predicate --
  B5 re-pins the probe).
- entry-219 gauge names vs sgl-project#43727 return + sgl-project#44968: PER-POOL NAMES KEPT as the
  serving schema -- hicache_host_pool_{used,total}_tokens{pool} dataclass
  fields, Gauge registrations and labels(pool=) emission re-restored over the
  #4697x structure; hicache_host_pool_evicted_tokens_total already survived
  auto-merge; producer dict restored by union in metrics_reporter. ANTI-NEEDLE
  SATISFIED: old mamba-only gauges are NOT the serving schema.
- entry-221 eager-direct add/add: DEDUPED to upstream (bodies identical except
  est_time 15->9); MEASUREMENT DEBT RECORDED: prefill t/s must be re-measured
  post-B4 before trusting the est_time.
- entry-168 D2H fence vs sgl-project#44800: the fence is ALREADY UPSTREAM (both
  load_fence_stream + write_fence_stream with the load-thread-corruption
  comment in origin/main's cache_controller.py) -- nothing re-planted; the two
  real regions resolved as import-union + comment wording (merge sgl-project#4 stale-tail
  pattern avoided). Fork hunks still saved to replant/ (belt).
- entry-209 clamp + 216/218 ordering + 288-295 hicache-ctrl-era vs #46970x KV
  rewrite: ADOPTED-UPSTREAM across unified_radix_cache/hicache_storage/
  hybrid_cache_controller/memory_pool/kv_cache_builder; EVERY fork hunk
  extracted to replant/*.fork-hunks.patch; B3 re-plant markers planted.
  sgl-project#45703's short-full-match seam arrives with upstream.
- sgl-project#45090 SGLANG_ENABLE_UNIFIED_RADIX_TREE: REMOVAL TAKEN (fork had already
  moved it to _DeprecatedEnv; now gone). grep: ZERO .py readers repo-wide;
  2 upstream-owned docs jsx snippets still export it (deepseek-v4, glm-5.2 --
  upstream's own debt). OPERATOR-OWED CHECK: confirm no /root harness exports
  it (unreadable from this seat).
- sgl-project#39727 unified L3 key: VERIFIED OPT-IN -- merged hicache_storage.py keeps
  the DEFAULT legacy suffix composition _<model>_<tp_rank>_<tp_size>
  (content-anchor: HiCacheFile._init_config_suffix, raise-guarded when
  _unified_chunk_spans is set). ANTI-NEEDLE: no silent key-scheme switch;
  warm /root/hicache_store contents stay addressable.

Modify/delete (4, all victims of upstream sgl-project#45672 test prune + URC split):
- test_unified_radix_cache_unittest.py: REMOVED (adopted split) -- its bare
  unified_tree_core_inspector imports are unreachable post-sgl-project#45236 re-home;
  fork source + hunks saved; B2 marker planted in the new core test.
- test_fused_moe_triton_config.py: REMOVED -- imports common_utils, absent
  from both sides post-sgl-project#45672; the 2 live entry-3/235 tests re-plant in B2.
- test_scheduler_batch_is_full_latch.py: REMOVED -- fork delta was 3 stub-attr
  lines documenting sgl-project#46643's absorbed arms; upstream deletion correct.
- test_qsa_kv_pool.py: KEPT -- fork's 147 added lines are entry-187/200
  sharding + 285-287 QSA-ring tests with NO upstream replacement;
  qsa_kv_pool.py sides converged byte-identical; collects clean. B2 marker.

Cascading silent-auto-merge repairs (classes not in the conflict set but
broken by auto-merge + union splices -- all found by gates, now fixed):
kernels/ops/activation/__init__.py (dead _CUDA registration dup), kernels/ops/
diffusion/__init__.py (5-tuple _SPECS entry), kernels ops embeddings/moe/
attention/mamba __init__ (KernelSpec fields removed upstream: description/
format_signature stripped, CapabilityRequirement->CUDA_ONLY), base_runner/
eager_runner cp_extend + shared_read_event imports retargeted per upstream's
"no imports in runner_utils/__init__" contract, and 4 fork helpers grafted
back onto their take-upstream homes because merged code/tests still import
them (memory_pool.spec_intermediate_bytes_per_req, kv_cache_builder.
_adjust_hicache_size_for_draft + _device_pool_size_per_token,
hybrid_cache_controller.local_restorable_mask, validation_hook.
validate_replayssm_spec_algorithm, arg_groups/moe_hook b12x validator).
flashinfer_cutlass.py flipped to FORK-OWNED (b12x route is fork-only, 760 vs
95 lines; upstream sgl-project#44993 hunks = B5 fold patch saved).

Silent-semantics notes: sgl-project#46127/sgl-project#44912 riders arrived via per-PR picks (union);
sgl-project#45520 (entry 302) is STILL OPEN upstream -- entry-230 fail-safe layer stays
as entry 303 records; if a future sync brings it, re-apply the fail-safe reads.
qwen4_exp holder-contract fused (ctor takes optional batch).

Gates at commit: import sglang OK (CVD=9, prod venv); check_registered_tests/
check_no_bare_pytest_main/check_no_registered_tests_in_package rc=0; ruff F821
+E9+format clean on the resolution set; collect-only over unit/{mem_cache,
model_executor,distributed,layers/moe} = 4337 nodes, 3 ERRORS: 2x cargo/torch-
sys libtorch probe (REPRODUCED IDENTICALLY AT THE PRE-MERGE TIP -- env issue,
not merge damage) + test_hicache_async_ready_sync.py (entry-253-era attribute
gone with the URC rewrite -- EXPECTED pre-B3).

sessions 2/3 owed: B2 test re-home, B3 mem_cache+rust re-plant (+hicache-ctrl
re-ports), B4 scheduler/admission, B5 moe/quant/dispatcher, B6 environ+C0a+
CAVEATS re-pin. PROD BOOT FORBIDDEN until B3 completes (209 clamp + 218
ordering re-plant).
05yuki pushed a commit to 05yuki/sglang-upstream that referenced this pull request Oct 30, 2026
Since sgl-project#44993 the top-k folds routed_scaling_factor into its weights whenever the
quant method says so, and KTEPWrapperMethod forwards that question to its GPU
method (the MXFP4 ones answer yes). DeepseekV2MoE still multiplied the routed
output by the factor again for every KT layer, so the routed experts came out
1.5x too strong on DeepSeek V4.1. Paired NLL on 11,496 tokens against the
checkpoint's own inference code: 2.878 before, 2.596 after (reference 2.594).
detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
…-arg predicate -- entries 230/239-era fold contract

Upstream sgl-project#44993 (4f8f06f, arrived in-range) changed
_fuses_routed_scaling_factor_in_topk(quant_method) to
(quant_method, runner_backend): the runner now arrives explicitly from
FusedMoE.__init__ (layer.py:652, quant_method.get_moe_runner_backend), and the
Marlin exclusion widened to is_marlin() or is_lora_marlin(). The probe's 2F
were its stub calling the OLD shape (TypeError: missing runner_backend).

Re-adapted WITHOUT weakening beliefs: _layer_fold_verdict() mirrors the layer's
call shape; stub keeps _moe_runner_backend feeding through the real getter.
Docstring updated with the sgl-project#44993 delta. NEW pin added: lora_marlin does not
fold (the widened exclusion arm is now under contract).

Battery ACTUALS (CVD=9, prod venv, PYTHONPATH=python):
  test_modelopt_nvfp4_fold_contract: 6P/2F -> **9P/0F** (damage 2F closed,
  +1 new lora_marlin case).
detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
…d flashinfer_cutlass + entry-230 dispatcher contract re-plant + qwen4_exp PLE router-API re-home -- entries 9,174,225,230,238,244,247,167

THREE BOOT-INTEGRITY/HONESTY ITEMS, one moe-contract cluster:

(1) flashinfer_cutlass.py (marker consumed): upstream's in-range delta folded
  INTO the fork-owned file (merge resolution had already woven the tune-range
  helper): FlashInferCutlassMoeQuantInfo.pair_w2_weight FIELD (modelopt_quant
  :3826 PASSED it -- unexpected-kwarg TypeError on the CUDA FP4 cutlass route
  as merged, B3-class), the sm120 native MTP-MMA route (_sm120_mtp_mma_supported
  + _flashinfer_autotune_active + try_nvfp4_moe call, upstream's position after
  _prepare_input / before the fork smallm decision; conditions are
  disjoint from our gated SwiGLU arm), its fail-loud paired-weight guard, and
  the all-gather-prefill Standard delegation on the flashinfer-a2a fused func.
  is_sm120 import added. b12x route + D-3 constants (source_format/w13_layout,
  flashinfer 0.7.2) untouched; the flip-provenance note replaces the marker.

(2) token_dispatcher/standard.py (marker consumed): entry-225's rescue is now
  IN THE CORE by sgl-project#44993 (mapping keys off the BUILT runner) -- absorbed, not
  re-planted; the experimental_sgl_trtllm arm's loss is UPSTREAM'S OWN churn
  (sgl-project#27329 added in-premerge, sgl-project#44993 deleted -- topk.py still special-cases it).
  RE-PLANTED: the fork-only b12x arm of _keeps_global_expert_ids (without it a
  named flashinfer_b12x_w4a16 runner gets LOCAL ids from the dispatcher and its
  own expert_map re-remaps -- EP>1 misroute) and the entry-230 warn-once
  published-vs-built convention canary, re-anchored on sgl-project#44993's shape (warn
  once per (published, built) pair at EP>1; the fp4-allgather launch decision
  still keys off the published value).

(3) qwen4_exp.py (marker consumed) + its packed_short_conv test: the two
  pre-#46970x reads (req_to_token_pool.get_ngram_context, deleted by
  sgl-project#46970..sgl-project#46991; pool.ngram_pool.context) re-homed onto the
  layer_router NGRAM_PART view (single read feeds both consumers -- they were
  always the same data; .context is the same-named buffer the view exposes).
  Test seam swapped to the router stub (the 1F was PRE-EXISTING at HEAD,
  verified by stash).

Battery ACTUALS (CVD=9, prod venv, PYTHONPATH=python, import sglang OK):
  fold probe: 6P/2F -> 9P (B5-1, committed separately)
  standard_dispatcher + runner_backend + b12x_route: 104P/91sub GREEN
  +swiglu_params +smallm (cluster of 5 files): 137P GREEN pre/post
  qwen4 PLE family (8 files): was 1F (pre-existing) -> 85P/12S/4sub GREEN
  ruff F821+E9 + format: clean on all 5 files
detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
…ies 28,165,166,186,194,206,296,297,299,300

PER-MARKER TRUTH (census: B5 markers in this set now ZERO):
  kernels/aot topk.cu (28): VERIFIED-ABSORBED -- upstream's in-range
  legacy_radix_topk::select rewrite gates the threshold-bin population
  against the stash BEFORE staging (coarse_population >= StashEntries ->
  full-row byte-per-pass refinement), structurally fixing the overflow
  the fork patched after-the-fact; jit twin keeps its s_prefix_bins.
  kernels/ops/attention/utils.py (194): VERIFIED-ABSORBED -- the function
  re-sited to ops/attention/kv_indices.py (spec_kv_index_token_blocks with
  the 32768 gate) and every consumer imports it at that module; the saved
  utils re-export would be a dead hunk (zero importers).
  elementwise/elementwise.py (166): VERIFIED-PRESENT -- the batch-invariant
  num_warps clamp rides upstream's guard_or_false refactor ('or
  is_batch_invariant_mode_enabled()' intact).
  kvcache/__init__.py (186): VERIFIED-PRESENT -- the store_cache_fp8
  registry entry is the merge commit's own graft (entry 310), imports +
  capabilities in place.
  moe/hash_topk.py + moe/topk.py (296/297/299/300): the merged upstream
  canonicals of sgl-project#44901/sgl-project#44904/sgl-project#44912/sgl-project#44913 are IN RANGE (all four
  MERGED commits proven ancestors of 2a6c204); every fork symbol
  (fused_shared_expert_weight, is_routed_scaling_factor_fused_in_topk,
  num_shared_topk_slots x54, has_per_rank_fused_shared_slots,
  SharedExpertsConfig wiring) verified present. Markers stale.
  moe/flashinfer_trtllm.py (165): SUPERSEDED-BY-EQUIVALENT -- upstream's
  own in-range rewrite of the deferred block covers the routed triple
  (do_finalize=not defer + _make_deferred_finalize_output) fail-LOUDER
  (valid_deferred_input raise); the fork's sgl-project#6077 ignored-autotune-buffer
  targeted the OLD block shape and is deliberately NOT restored (GPU
  perf-only, defer+routed is not our production arm).
  moe/mega_moe.py (296): deep_gemm transform_sf redirect RE-PLANTED to the
  guarded wrapper (import-safety contract; sibling fix in dsv4 below).
  fused_moe_triton/layer.py (165/206/239): _built_runner_backend hunk
  ABSORBED by sgl-project#44993 (config carries the built runner); the entry-206
  supports_routed_deferred_finalize property RE-PLANTED (upstream's
  packed_nvfp4_routed_deferred covers TRTLLM only, NOT trtllm_routed --
  our runner arm), with the sgl-project#44993 2-arg fold predicate; the debug-log OR
  re-plant paired.
  models/deepseek_v2.py (296/297/300/236): fold chain verified; the
  entry-206 format-ternary consumer arm RE-PLANTED (BYPASSED ->
  supports_deferred_finalize, else supports_routed_deferred_finalize --
  restoring what sgl-project#44993's rewrite dropped).

Battery ACTUALS (CVD=9, prod venv, PYTHONPATH=python, import sglang OK):
  fused_shared_expert_scaling + mega_moe_deepgemm_api + moe_runner_
  extensions + topk_routing_inputs + model_overrides + kernels_namespace:
  GREEN (see B5-4 commit for full 43-file battery actuals)
detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
… C closure -- the post-merge port queue is now decided

ARRIVAL SCAN (git log 12dc623..2a6c204 --grep) + LIVE gh STATE +
head-sha comparison for EVERY deferred item. Verdicts:
  RELEASED-BY-MERGE: sgl-project#42689 (@ 7a632cf in-range -- the SECTION-B
    cherry is now redundant; behavior note rides the restart).
  CLOSED (C-items whose PRs arrived and whose integration this stack
    performed): C1 (the dedicated merge week WAS merge sgl-project#5; all eight
    precursor PRs in-range), C2 (sgl-project#44993/sgl-project#44994 -- B5), C3 (sgl-project#45232/
    sgl-project#45236/sgl-project#45334 -- B2), C4 (sgl-project#45238/sgl-project#45241/sgl-project#45242 -- B3), C5 (sgl-project#44912/
    sgl-project#44913 -- arrived; entry-230 canary re-planted as its guard),
    C9 (sgl-project#39727 -- verified OPT-IN, no purge; BOOT_DELTA anti-needle),
    C0a (arrival shas already recorded at entry 310 -- Section C now
    carries NO open code-integration items: C6/C10 retain only their
    explicitly-named docs/boot-measurement remainders, re-stated).
  STILL-OPEN: sgl-project#44342 (head unchanged; auto-merge comfort LOST to
    #46970x), sgl-project#46153/sgl-project#37106/sgl-project#32479/sgl-project#45391 (gh CONFLICTING -- stale
    bases, rebase before any port), sgl-project#41624/sgl-project#42525/sgl-project#42660/sgl-project#44462/sgl-project#35617
    (open, heads unchanged), sgl-project#46191/sgl-project#39845-class (watch-list, none in
    range).
  HOLD STANDS (operator decision, binding): sgl-project#43356/sgl-project#46452/sgl-project#43240/
    sgl-project#43471 KV route -- step-1 'MERGE sgl-project#5 lands first' is NOW MET; the
    sequence may be re-visited ONLY with the post-merge PCIe-IPC
    measurement at an operator sitting. sgl-project#44732: the 'one purge if done
    WITH the merge' premise LAPSED when merge sgl-project#5 completed without it
    -- later execution costs the SAME single purge once; no penalty,
    no haste. Recorded for the operator, not decided by B6.
05yuki pushed a commit to 05yuki/sglang-upstream that referenced this pull request Nov 1, 2026
Since sgl-project#44993 the top-k folds routed_scaling_factor into its weights whenever the
quant method says so, and KTEPWrapperMethod forwards that question to its GPU
method (the MXFP4 ones answer yes). DeepseekV2MoE still multiplied the routed
output by the factor again for every KT layer, so the routed experts came out
1.5x too strong on DeepSeek V4.1. Paired NLL on 11,496 tokens against the
checkpoint's own inference code: 2.878 before, 2.596 after (reference 2.594).
05yuki pushed a commit to 05yuki/sglang-upstream that referenced this pull request Nov 4, 2026
Since sgl-project#44993 the top-k folds routed_scaling_factor into its weights whenever the
quant method says so, and KTEPWrapperMethod forwards that question to its GPU
method (the MXFP4 ones answer yes). DeepseekV2MoE still multiplied the routed
output by the factor again for every KT layer, so the routed experts came out
1.5x too strong on DeepSeek V4.1. Paired NLL on 11,496 tokens against the
checkpoint's own inference code: 2.878 before, 2.596 after (reference 2.594).
detain added a commit to detain/sglang that referenced this pull request Nov 11, 2026
… commits since 2a6c204 @ merge-sgl-project#5)

Conflicts: 110 files (106 content / 4 add-add / 0 modify/delete) at the settled
321 tip 2d87757 -- vs merge sgl-project#5's 167; the smaller count despite 950 commits
(merge sgl-project#5: 1,120) is the merge-era's re-plant landing INSIDE upstream's new
window (many fork families -- qwen4 varlen, fast_file, hicache-liveness, the
209-218 controller semantics -- now collide as DUAL IMPLEMENTATIONS: upstream
grew the same families independently in the window). By class: A 7
(auto-takes-upstream, dedupe-verified), B 36 (mechanical union/custom splices),
C 46 (upstream structure taken, fork hunks extracted for re-plant), D 21
(fork-owned flips + dual-impl adjudications, incl. the whole fast_file family).
228 conflict regions total: 69 union (all def/class-collision-checked), 118
take-upstream (104 marked, 14 dedupe-verified), 25 keep-fork (17 fold-marked,
8 strict supersets), 16 custom import/region splices.

121 TODO(merge6) markers planted (tree census): b1 12 / b2 23 / b3 18 / b4 39 /
b5 17 / b6 12. (Resolver census at region-resolution was 117; the cascade-repair
grafts below added the rest.) REPLANT INVENTORY: /tmp/opencode/merge6-dossier/
replant/ -- per conflicted file: <name>.fork-hunks.patch (full 2a6c..321 delta),
<name>.upstream-side.patch (full 2a6c..eb73 delta), <name>.ours-dropped.txt /
<name>.theirs-dropped.txt region snapshots, MANIFEST.tsv (file, zone, regions,
kinds, fork-hunk counts, per-region notes). Work records (conflict census JSON,
decision tables, class table, logs) under .../work/.

D-list outcomes (must-not-auto-resolve, handled deliberately):
- sgl-project#45520 (entry 302): STILL OPEN upstream at eb73a50 (gh-verified + ZERO
  "sgl-project#45520" strings in the +950 window). Entry-230's fail-safe layer is UNTOUCHED
  and stays (standard.py warn-once + published-auto guards intact; the re-apply
  rule armed in CAVEATS was NOT needed this merge).
- sgl-project#44590 (entry 228): the window did NOT move the floor further -- upstream
  pyproject stays sglang-kernel==0.4.10 (absorbed at merge sgl-project#5 via sgl-project#43922) and
  the engine assert_pkg_version stays 0.4.9 in BOTH trees; the
  _load_inline_fragment_probes_available() probe never existed on either side
  (docs-mention only). Nothing to re-assert; both pins kept byte-identical.
- flashinfer_cutlass.py (fork-owned per merge-sgl-project#5 flip): did NOT conflict; 56
  b12x refs intact, dispatcher reads untouched.
- modelopt_quant sgl-project#44993/sgl-project#44994-ERA B12X W4A16 arms: the fork's
  defer-swizzle/b12x conditions at L2941-2996 are CONTESTED against upstream's
  skip_swizzled_blockscales rewrite -> took upstream, b5 markers + region
  snapshots (boot-integrity fields live in the fork-owned cutlass file, which
  survived; the b12x selection ARMS need the B5-style re-plant).
- DUAL-IMPLEMENTATION FAMILIES (the new pattern of this merge; upstream grew
  the same features the fork replanted -- take-upstream + fold-diff replant,
  NEVER a silent pick): kernels/ops/mamba/qwen4_short_conv.py + its test suite,
  srt/models/qwen4_exp.py varlen trio, mem_cache/storage/fast_file/fast_file_store.py +
  test_hicache_fast_file_unit.py, test_hicache_sidecar_restore.py,
  fused_moe_triton/layer.py nvfp4-deferred clause,
  tokenizer_manager _validate_top_logprobs_num, hicache_best_practices.mdx.
  All fork-side versions snapshotted; replant batches diff
  ours-dropped.txt against the taken upstream.
- environ.py: took upstream's SGLANG_EXACT_CHUNK_FILL removal; kept ONLY the
  fork-added SGLANG_SPLIT_PREFILL_AT_MAMBA_FORK line (custom splice).
- entry-219 gauge schema: per-pool hicache_host_pool_{used,total}_tokens names
  SURVIVED the auto-merge (metrics_collector/hicache_pool_stats/metrics_reporter/
  urc all intact); the marked b3 controller regions must RE-VERIFY the producer
  dict end-to-end at re-plant, not assume it.

Cascading silent-auto-merge repairs (files NOT in the conflict set, broken by
auto-merge splices -- all found by the ruff-F821/check_* gates, all fixed here):
utils/common.py upstream prune RE-HOMED the fork's get_mixed_precision_moe_quant_algos
into configs/quant_utils.py (next to the moved siblings) + the make_layers
empty_cache into layers/utils/layer_list.py, importers rewired;
mem_cache/pool_host/base.py silently LOST the entry-sgl-project#44462-era
host_memory_budget_bytes() helper whose caller survived -- grafted back;
model_executor/runner/eager_runner.py CP helpers retargeted to upstream's NEW
layers/cp/utils (cp_shard_model_inputs/cp_gather_after_forward) replacing the
retired runner_utils.cp_extend pair; runner/flashinfer_autotune.py + qsa/mqa.py +
modelopt_quant.py import retargets for the utils/device + layers/dp_attention
move; mem_cache/memory_pool.py lossless-verify tuple buffers re-grafted around
upstream's replayssm skip-chain; unified_radix_cache.py host-capacity deadline
preamble re-grafted (marker kept); qwen4_exp.py _use_qwen4_direct_decode
re-grafted (its kernels survived); qwen4_exp_ple_table.py _HOST_PAGE_TABLES
memo dict re-grafted; rotary_embedding/factory.py get_rope param restored;
multimodal_gen forward_context teacache_plan param restored + server_args.py
Literal import; parallel_executor.py rebuilt as fork-owned + upstream's
raise_if_cancelled guards folded into the fork loop structure;
hybrid_cache_controller.py _reduce_prefetch_ack if/zero_ splice repaired;
python_bindings.rs pyo3 signature REORDERED to match the unioned fn new()
argument order (mamba_value_seqlen position -- a positional-bind footgun, fixed
pre-commit); test_io_struct.py get_context/MAX_PARALLEL_SAMPLES imports;
stat_loggers_di + test_flashinfer_autotune_sync + fused_lora_compose +
tp_worker + compressed_tensors + minimax_h3 + hc_mix + scheduler_mixin +
pool_configurator + scheduler/tokenizer_manager import prunes/grafts.

Gates at commit (CVD=9, prod venv, worktree PYTHONPATH): import sglang OK;
check_registered_tests / check_no_bare_pytest_main / check_no_registered_tests_in_package
ALL rc=0; ruff (F401,F821,F823,UP037 @ v0.17.1 vs pinned 0.15.1) ZERO new errors
on the 3,269 merge-touched .py -- the residual 161 are BYTE-IDENTICAL INHERITED
rows present at BOTH parents (kernels/aot, api_types, mm entrypoints/utils, etc.;
0.17-only detections, not merge damage); ruff format clean on the same set;
py_compile clean; CARGO sglang-radix-tree check+test GREEN: 966 passed / 0 failed
/ 1 ignored (vs merge-sgl-project#5-era 943P -- upstream grew the suite +23; NOTE the crate
is now `exclude`d from the rust/ workspace with extension-manifests metadata --
`cargo -p sglang-radix-tree` from rust/ no longer works; run inside the crate).

NON-ADOPTIONS honored: sglang-kernel floor 0.4.9 (engine assert) untouched;
sgl-project#45520 not brought (fail-safe layer intact by rule).

Re-plant batches owed (per marker zones): B2-2026 test re-home (23), B3-2026
mem_cache/controllers/rust (18), B4-2026 scheduler/managers/runner (39),
B5-2026 moe/quant/kernels/qwen4 (17), B6-2026 environ/arg_groups/docs re-pins
(12), B1-2026 diffusion folds (12). PROD BOOT FORBIDDEN until the B3/B4 zones
re-plant the entry-209/218 controller semantics (the trailing_endpoint gate will
read RED at this commit, exactly as at merge-sgl-project#5's damage state).
YaoleWu added a commit to YaoleWu/sglang that referenced this pull request Nov 12, 2026
… factor

Since sgl-project#44993, TopK folds `routed_scaling_factor` into the top-k weights when
the quant method reports `fuse_routed_scaling_factor_in_topk`. The KT wrapper
forwards that report from its MXFP4 methods, yet the DeepseekV2 MoE epilogue
still scaled the routed output whenever the quant method is a KTEPWrapperMethod
— DSv4.1 (factor 1.5) came out at 2.25x per layer.

Make the decision single-sourced: `KTEPWrapperMethod` declares a duck-typed
`applies_routing_weights_as_given = True` (same style as the fold flag), and
one `_post_moe_needs_routed_scaling()` helper replaces the two hand-copied
conditionals in `forward_normal_dual_stream` and `_post_combine_hook`. The
old `or isinstance(...)` clause made the CUDA short-circuit dead for KT; the
helper makes that impossible by construction. KT is scaled exactly once on
every device; CUDA/MUSA/XPU/AITER and plain-CPU paths keep their runner- vs
epilogue-owned semantics unchanged.

Supersedes sgl-project#45810 (absorbing its teacher-forced accuracy evidence: mean NLL
2.5960 vs reference 2.5940, was 2.8780 before).

Signed-off-by: wuyaole <112309453+LeslieWylie@users.noreply.github.com>
05yuki pushed a commit to 05yuki/sglang-upstream that referenced this pull request Nov 14, 2026
Since sgl-project#44993 the top-k folds routed_scaling_factor into its weights whenever the
quant method says so, and KTEPWrapperMethod forwards that question to its GPU
method (the MXFP4 ones answer yes). DeepseekV2MoE still multiplied the routed
output by the factor again for every KT layer, so the routed experts came out
1.5x too strong on DeepSeek V4.1. Paired NLL on 11,496 tokens against the
checkpoint's own inference code: 2.878 before, 2.596 after (reference 2.594).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blackwell SM100/SM120 deepseek diffusion SGLang Diffusion quant LLM Quantization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants