Skip to content

[Fix] [NVIDIA] moe: Pick a supported runner for auto-detected ModelOpt NVFP4 MoE on SM110/SM120 - #45520

Open
waynehacking8 wants to merge 2 commits into
sgl-project:mainfrom
waynehacking8:r107-sm1xx-modelopt-moe-auto
Open

waynehacking8 wants to merge 2 commits into
sgl-project:mainfrom
waynehacking8:r107-sm1xx-modelopt-moe-auto

Conversation

@waynehacking8

@waynehacking8 waynehacking8 commented Oct 21, 2026 •

Copy link
Copy Markdown

Motivation

ModelOpt NVFP4 MoE checkpoints crash at warmup on SM120 with the default --moe-runner-backend auto. On one RTX PRO 6000 Blackwell (SM120), main 4eb606b2a fails this way for nvidia/Qwen3-30B-A3B-NVFP4 (NVFP4 experts) and for nvidia/Qwen3.6-35B-A3B-NVFP4 (MIXED_PRECISION with W4A16_NVFP4 experts):

python -m sglang.launch_server --model-path nvidia/Qwen3.6-35B-A3B-NVFP4 --tp 1 --context-length 32768 --mem-fraction-static 0.85 --disable-radix-cache
...
AttributeError: 'FusedMoE' object has no attribute 'g1_scale_c'

The SM110/SM120 rule never sees the checkpoint's quantization. #41765 extended an existing SM120 rule to SM110; it resolves auto to flashinfer_cutlass for modelopt_fp4, but it compares --quantization, which stays unset when the quantization comes from the checkpoint, as in the usual launch. So ModelOptNvFp4FusedMoEMethod.create_moe_runner resolves auto to FLASHINFER_TRTLLM for every capability 10.0 and up, while the weight preparation follows the global backend (still auto) and never creates the TRT-LLM scales.

Modifications

  • On SM110/SM120, with --moe-runner-backend auto and --quantization not given, _moe_runner_backend_quant_constraints now uses the quantization ModelConfig resolved from the checkpoint.
  • A modelopt_mixed checkpoint whose experts are W4A16_NVFP4 gets marlin, which the Nemotron-H override and the SM100 Qwen3 MoE override already pick for W4A16 MoE layers under auto. modelopt_fp4 keeps flashinfer_cutlass, and a modelopt_mixed checkpoint whose experts are NVFP4 now gets it too. The log line names the backend and quantization it picked.
  • get_mixed_precision_moe_quant_algos moves from model_overrides/qwen3_moe.py to configs/quant_utils.py, so the Qwen3 MoE override and this rule use the same function.
  • test_sm1xx_modelopt_nvfp4_moe_auto_avoids_trtllm_gen in test_model_overrides.py covers mixed W4A16 experts, mixed NVFP4 experts and plain NVFP4, with the platform flag set to SM120 and to SM110.

An explicit --quantization modelopt_mixed is handled by the same branch, and an explicit --quantization modelopt_fp4 keeps flashinfer_cutlass. The platform flag is_sm120 matches every capability 12.x (with CUDA 12.8 or newer), so SM121 takes the same path. I have no SM110 device; the rule is shared with SM120 and the test runs both flags.

Known limits:

  • Generic --quantization modelopt stays on auto, as on main. The FlashInfer Cutlass check in handle_moe_kernel_config accepts only the specific ModelOpt names, so resolving the generic flag needs that check changed too.
  • Explicit method flags are taken as given: a mixed W4A16 checkpoint launched with --quantization modelopt_fp4 keeps flashinfer_cutlass, as on main.
  • A modelopt_mixed checkpoint that declares its expert algorithms only in hf_quant_config.json stays on auto; the shared helper reads config.json's quantization_config, as the Qwen3 MoE and Nemotron-H overrides do.
  • Plain W4A16_NVFP4 checkpoints resolve to modelopt_fp4 and get flashinfer_cutlass; I have no such MoE checkpoint to test.
  • Neither checkpoint I ran has mixed NVFP4 experts, so that branch is covered by the unit test only.

Related PRs:

Accuracy Tests

Both checkpoints now start on auto, and GSM8K is unchanged within noise against main with the same backend passed explicitly (one RTX PRO 6000 Blackwell Max-Q, TP1, sgl-eval 0.1.3):

checkpoint main, auto this PR, auto main, explicit backend right on one side only (main, PR)
nvidia/Qwen3.6-35B-A3B-NVFP4 fails at warmup (crash command above) marlin, 0.930 (186/200) marlin, 0.935 (187/200) 8, 7
nvidia/Qwen3-30B-A3B-NVFP4 fails at warmup (crash command above with this model) flashinfer_cutlass, 0.955 (191/200) flashinfer_cutlass, 0.945 (189/200) 3, 5

The last column counts the questions only one side answered correctly. Server logs, per-run backend checks, sgl-eval outputs and the scripts are in https://gist.github.com/waynehacking8/a3fb1c44b09958b10998fe5b7c210e40.

python -m sglang.launch_server --model-path nvidia/Qwen3.6-35B-A3B-NVFP4 --tp 1 --disable-radix-cache --reasoning-parser qwen3 --mem-fraction-static 0.8 --moe-runner-backend {auto,marlin}
python -m sglang.launch_server --model-path nvidia/Qwen3-30B-A3B-NVFP4 --tp 1 --disable-radix-cache --reasoning-parser qwen3 --moe-runner-backend {auto,flashinfer_cutlass}
sgl-eval run gsm8k --base-url http://127.0.0.1:<port>/v1 --num-examples 200 --temperature 0.7 --top-p 0.8 --max-tokens 32768 --seed 0 --chat-template-kwarg enable_thinking=false

The MoE layers ran on the backend in the table: a probe recorded ModelOptNvFp4FusedMoEMethod with that backend at runner creation and in the forward for every run, and only this PR's auto runs log the rule's Use ... as MoE runner backend on SM110/SM120 line.

Settings that differ from the cookbook (the Qwen3.6 RTX Spark recipe already pins --moe-runner-backend flashinfer_cutlass; these runs test the backend choice itself):

  • Kept: the reasoning parser and --mem-fraction-static 0.8.
  • Dropped: --tool-call-parser qwen3_coder and --mamba-radix-cache-strategy extra_buffer; the runs use --disable-radix-cache.
  • Qwen3-30B: the Qwen3 page has no NVFP4 recipe.
  • All runs set SGLANG_FLASHINFER_AUTOTUNE_CACHE=0, and the eval does not apply Qwen3.6's recommended presence_penalty 1.5.

The GPU runs used this library change as one commit on main 4eb606b2a. The branch now merges main 3e8d7c9e09; the only library difference from that commit is where the helper lives and the imports that follow from it, and the branch adds the mixed NVFP4 test case. With main merged, test_model_overrides.py has no new failures: 128 passed, and its 3 failures also occur on main without this PR.

PYTHONPATH=python python -m unittest test/registered/unit/test_model_overrides.py
  • With only the test file added to 956b9a24ee, the new test fails with AssertionError: 'auto' != 'marlin'.
  • With the platform reported as SM120 and --quantization unset, the two checkpoints resolve to marlin and flashinfer_cutlass, against auto on main. The explicit-flag cases in the known limits were checked the same way.

Speed Tests and Profiling

No kernel changes: auto now resolves to backends that already exist. Decode TPOT at c=1, accuracy launch above:

checkpoint, backend main, explicit backend this PR, auto change
Qwen3.6, marlin 3.23 ms 3.33 ms +3.1%
Qwen3-30B, flashinfer_cutlass 4.63 ms 4.78 ms +3.1%

TPOT on this PR's auto runs is 3.1% higher than main's explicit runs in both rows, one server boot per cell. The diff only changes server-arg resolution, which runs once at startup, and both arms resolve to the same server args apart from the random seed, so the MoE runner path is the same. For Qwen3.6 the gap is within what I saw between main boots: a separate main boot with the crash command's flags (table below) gave 3.33 ms. For Qwen3-30B it isn't: all five of this PR's prompts (4.777 to 4.787 ms) are slower than all ten from main's two flashinfer_cutlass boots, this table's and the crash-flags boot below (4.257 to 4.669 ms). This PR's boot came first in both pairs, so boot order is one candidate; I haven't found the cause. I can run three boots per arm in alternating order if a reviewer wants it settled.

Backend comparison on main 4eb606b2a with the crash command's flags and the backend passed explicitly, one boot each:

checkpoint marlin flashinfer_cutlass
Qwen3.6 (W4A16 experts) 3.33 ms 3.93 ms
Qwen3-30B (NVFP4 experts) 4.19 ms 4.66 ms

flashinfer_cutlass also runs Qwen3.6, which ships expert input scales, but marlin decodes faster, so auto picks marlin. For NVFP4 experts this PR keeps the SM120 rule's existing flashinfer_cutlass choice. Marlin also runs Qwen3-30B on main and decodes faster here; I left that choice alone because this PR is about the crash, and it may be worth a separate look.

TPOT is the median over 5 random 512-token prompts, measured through /generate. Each prompt's value is:

(latency at 257 tokens - latency at 1 token) / 256

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #42571317653
Latest PR Test (Extra): ❌ Run #42571317533
Latest PR Test (AMD ROCm 10): ❌ Run #42571317521

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

…t NVFP4 MoE on SM110/SM120

With --quantization unset, the SM110/SM120 rule never saw modelopt_fp4, so auto stayed on auto and the ModelOpt NVFP4 MoE method fell through to FLASHINFER_TRTLLM, whose kernels are SM100-only. nvidia/Qwen3.6-35B-A3B-NVFP4 then failed at warmup with "'FusedMoE' object has no attribute 'g1_scale_c'".

When --quantization is unset, use the quantization ModelConfig resolved from the checkpoint, and choose by expert format: marlin for W4A16_NVFP4 experts and flashinfer_cutlass for NVFP4 (W4A4) experts of a modelopt_mixed checkpoint, flashinfer_cutlass for modelopt_fp4. The MIXED_PRECISION expert helper moves to utils so the Qwen3 MoE override and this rule share it.

Signed-off-by: WEI CHENG CHIU <waynehacking8@gmail.com>
@waynehacking8
waynehacking8 force-pushed the r107-sm1xx-modelopt-moe-auto branch from 26cb238 to 7032238 Compare October 21, 2026 05:45
@waynehacking8

Copy link
Copy Markdown
Author

@Fridge003 this fixes a gap in the SM110/SM120 auto MoE rule in _moe_runner_backend_quant_constraints, the one #41765 extended to SM110. The rule compares --quantization, which stays unset for ModelOpt NVFP4 checkpoints such as nvidia/Qwen3-30B-A3B-NVFP4 that declare their quantization only in their own config, so on the default launch it never fires and SM120 warmup fails with 'FusedMoE' object has no attribute 'g1_scale_c'.

The PR uses the checkpoint's quantization when the flag is unset, and it also covers modelopt_mixed: a mixed checkpoint whose experts are W4A16_NVFP4 goes to marlin, which the SM100 Qwen3 MoE and Nemotron-H overrides already pick for those layers. The RTX PRO 6000 runs and the unit test are in the description. Could you review it, or add run-ci if it looks right?

detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…ted runner for auto-detected ModelOpt NVFP4 MoE on SM110/SM120

STATE: gh-verified OPEN at 7032238 (mergeable).
Method per the ledger's open-port convention: git apply -3 clean for
utils/common.py (get_mixed_precision_moe_quant_algos hoist) and
model_overrides/qwen3_moe.py (helper now imported from utils);
overrides.py + test_model_overrides.py hand-merged (the upstream hunk's
preimage is the RAW view.quantization=="modelopt_fp4" block -- this tree
carries the entry-230 fork patch at that exact spot).

What upstream fixes: with --quantization unset the SM110/SM120 rule never
saw the checkpoint's ModelOpt NVFP4, auto stayed auto, and the layer fell
through to FLASHINFER_TRTLLM -> g1_scale_c AttributeError at warmup. Now
the rule reads the checkpoint-resolved quantization and discriminates
modelopt_mixed by expert format: W4A16_NVFP4 experts -> marlin,
NVFP4 experts -> flashinfer_cutlass; modelopt_fp4 -> flashinfer_cutlass.
This is the UPSTREAM HALF of our entry-230 fork patch (the published-value
fix), living in the same _moe_runner_backend_quant_constraints hook the
entries-230/244 retargeted.

Fork union on top of sgl-project#45520's shape (all pinned by tests):
  * every model-config read goes through _model_config_or_none (fail-safe;
    upstream reads model_config_of(view) raw) -- entry 230's "unreadable
    config must leave the resolution exactly as it was" contract, kept by
    its RuntimeError test case;
  * modelopt_mixed with an UNREADABLE expert list publishes
    flashinfer_cutlass rather than staying auto (entry 230's reasoning:
    mixed derives only when some NVFP4/W4A16 layer exists; leaving auto
    is the silent-misroute shape). Upstream leaves auto here; the golden
    NVFP4-MOE-ARM case (readable config, NVFP4 experts) lands the SAME
    answer either way: flashinfer_cutlass.
  * _quantization_explicitly_unset read via getattr(default False) so the
    CPU resolver stubs keep constructing; production always sets it.
  * _checkpoint_quantization_or_none KEPT (delegating) -- the entry-230
    patch is not deleted; see the reconcile note commit for the
    redundancy analysis.

Tests: upstream's test_sm1xx_modelopt_nvfp4_moe_auto_avoids_trtllm_gen
spliced byte-faithful at its upstream anchor (marlin / flashinfer_cutlass /
flashinfer_cutlass across both platforms, 2 subtests); the entry-230
fixture pass (111P/1S/179sub at base) is unchanged in semantics and still
green -> file now 112 passed, 1 skipped, 181 subtests.

Co-authored-by: port-moe-contracts subagent
(cherry picked from commit f9faef3da2644e1e723df8ffd2cc86c8fbbee1bc)
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
Comment-only; no behavior change (overrides.py battery 112P/1S/181sub
unchanged by this commit, trtllm-gen file still its pre-existing
8F no-CUDA class, dispatcher suite still 12P).

Verdict recorded in the block above _model_config_or_none:
  * For the crash sgl-project#45520 targets (flagless ModelOpt NVFP4 under `auto`
    falling through to SM100-only trtllm-gen -> g1_scale_c AttributeError
    at warmup): upstream's checkpoint-resolved rule makes entry 230's
    published-value fix REDUNDANT whenever the config is readable, and
    refines it (W4A16 experts now answer marlin, not cutlass).
  * For the SILENT EXPERT-ID MISROUTE entry 230 also fixed (published
    auto vs built GLOBAL-ids runner: rank 0 routes right, EP ranks >0
    route nothing): sgl-project#45520 closes only the readable-config half and
    intentionally leaves `auto` when it cannot read the expert-algo list
    -- exactly the shape the misroute lives in. Entry 230's pieces are
    therefore KEPT and load-bearing there: the fail-safe config reads,
    the algo-unknown mixed->cutlass publish, and the once-per-pair
    mismatch warning in token_dispatcher/standard.py (untouched by this
    family). The patch is downgraded from primary to fail-safe layer;
    nothing deleted, per the port brief.
  * 489c... (entry 247 b12x) territory: the W4A16->marlin refinement
    changes what `auto` publishes for W4A16-MIXED checkpoints on
    SM110/SM120 (entry 230 blanket-published flashinfer_cutlass for ALL
    mixed names, which for a pure-W4A16 mixed checkpoint diverged from
    its per-layer marlin build -- a latent instance of the very
    published-vs-built divergence entry 230 exists to prevent; sgl-project#45520
    removes it). b12x itself stays named-explicit-only: auto never
    resolves to flashinfer_b12x_w4a16 before or after sgl-project#45520, and
    test_auto_never_resolves_to_b12x remains green.
  * The bare "modelopt" name still does not refine to fp4/mixed at arg
    resolution (entry 230's KNOWN GAP, unchanged by sgl-project#45520): an explicit
    --quantization modelopt boot on SM120 must still name the runner.
    Upstream work that would close it via the modelopt_mixed mapping is
    sgl-project#35617 territory -- referenced here only, not ported.

Co-authored-by: port-moe-contracts subagent
(cherry picked from commit aa671ba8cf32c5e270d7219064506a39a20c9cca)
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…20 commits since 12dc623 @ merge-sgl-project#4)

Conflicts: 167 files (149 content / 14 add/add / 4 modify/delete) at the settled
309 tip d28e7cd -- dossier predicted 146 +/-25; the +21 shift is entries
275-309 landing qsa-ring/hicache-ctrl/gdn/moe/llguard deltas into the conflict
zone. By class: A 13 (auto-takes-upstream, dedupe-verified), B 80, C 56, D 18.
102 TODO(merge5) re-plant/fold markers planted: B1 x15, B2 x29, B3 x12, B4 x23,
B5 x23. Re-plant stock: 117 files under /tmp/opencode/merge5-dossier/replant/
(76 fork-hunks patches, 19 upstream-side patches, 14 AA fork copies, 8
preremove snapshots) -- sessions 2/3 re-plant from real code, not memory.

D-list outcomes (must-not-auto-resolve, handled personally):
- sgl-project#45289-vs-sgl-project#44858 (entry 301): VERIFIED, not re-resolved -- the gate/up-order
  final form is in origin/main (5b1dbc2 in range); scales test taken as
  upstream oracle, fork 191-line version saved to replant/ for B5 diffing.
  sgl-project#44858's call-site shape CANNOT survive (our base never had it).
- entry-230 dispatcher vs sgl-project#44993/sgl-project#44994: took upstream structure in
  token_dispatcher/standard.py; published-value CONTRACT lives upstream via
  get_moe_runner_backend() in the LoRA-EP guard; fork's warn-once mismatch
  diagnostic re-plants (B5 marker). UNION GATE: standard_dispatcher 16P (was
  12P, suite grew), model_overrides 121P/1S/210sub (was 112P/1S/181sub),
  fold probe 6P/2F (upstream added a runner_backend arg to the predicate --
  B5 re-pins the probe).
- entry-219 gauge names vs sgl-project#43727 return + sgl-project#44968: PER-POOL NAMES KEPT as the
  serving schema -- hicache_host_pool_{used,total}_tokens{pool} dataclass
  fields, Gauge registrations and labels(pool=) emission re-restored over the
  #4697x structure; hicache_host_pool_evicted_tokens_total already survived
  auto-merge; producer dict restored by union in metrics_reporter. ANTI-NEEDLE
  SATISFIED: old mamba-only gauges are NOT the serving schema.
- entry-221 eager-direct add/add: DEDUPED to upstream (bodies identical except
  est_time 15->9); MEASUREMENT DEBT RECORDED: prefill t/s must be re-measured
  post-B4 before trusting the est_time.
- entry-168 D2H fence vs sgl-project#44800: the fence is ALREADY UPSTREAM (both
  load_fence_stream + write_fence_stream with the load-thread-corruption
  comment in origin/main's cache_controller.py) -- nothing re-planted; the two
  real regions resolved as import-union + comment wording (merge sgl-project#4 stale-tail
  pattern avoided). Fork hunks still saved to replant/ (belt).
- entry-209 clamp + 216/218 ordering + 288-295 hicache-ctrl-era vs #46970x KV
  rewrite: ADOPTED-UPSTREAM across unified_radix_cache/hicache_storage/
  hybrid_cache_controller/memory_pool/kv_cache_builder; EVERY fork hunk
  extracted to replant/*.fork-hunks.patch; B3 re-plant markers planted.
  sgl-project#45703's short-full-match seam arrives with upstream.
- sgl-project#45090 SGLANG_ENABLE_UNIFIED_RADIX_TREE: REMOVAL TAKEN (fork had already
  moved it to _DeprecatedEnv; now gone). grep: ZERO .py readers repo-wide;
  2 upstream-owned docs jsx snippets still export it (deepseek-v4, glm-5.2 --
  upstream's own debt). OPERATOR-OWED CHECK: confirm no /root harness exports
  it (unreadable from this seat).
- sgl-project#39727 unified L3 key: VERIFIED OPT-IN -- merged hicache_storage.py keeps
  the DEFAULT legacy suffix composition _<model>_<tp_rank>_<tp_size>
  (content-anchor: HiCacheFile._init_config_suffix, raise-guarded when
  _unified_chunk_spans is set). ANTI-NEEDLE: no silent key-scheme switch;
  warm /root/hicache_store contents stay addressable.

Modify/delete (4, all victims of upstream sgl-project#45672 test prune + URC split):
- test_unified_radix_cache_unittest.py: REMOVED (adopted split) -- its bare
  unified_tree_core_inspector imports are unreachable post-sgl-project#45236 re-home;
  fork source + hunks saved; B2 marker planted in the new core test.
- test_fused_moe_triton_config.py: REMOVED -- imports common_utils, absent
  from both sides post-sgl-project#45672; the 2 live entry-3/235 tests re-plant in B2.
- test_scheduler_batch_is_full_latch.py: REMOVED -- fork delta was 3 stub-attr
  lines documenting sgl-project#46643's absorbed arms; upstream deletion correct.
- test_qsa_kv_pool.py: KEPT -- fork's 147 added lines are entry-187/200
  sharding + 285-287 QSA-ring tests with NO upstream replacement;
  qsa_kv_pool.py sides converged byte-identical; collects clean. B2 marker.

Cascading silent-auto-merge repairs (classes not in the conflict set but
broken by auto-merge + union splices -- all found by gates, now fixed):
kernels/ops/activation/__init__.py (dead _CUDA registration dup), kernels/ops/
diffusion/__init__.py (5-tuple _SPECS entry), kernels ops embeddings/moe/
attention/mamba __init__ (KernelSpec fields removed upstream: description/
format_signature stripped, CapabilityRequirement->CUDA_ONLY), base_runner/
eager_runner cp_extend + shared_read_event imports retargeted per upstream's
"no imports in runner_utils/__init__" contract, and 4 fork helpers grafted
back onto their take-upstream homes because merged code/tests still import
them (memory_pool.spec_intermediate_bytes_per_req, kv_cache_builder.
_adjust_hicache_size_for_draft + _device_pool_size_per_token,
hybrid_cache_controller.local_restorable_mask, validation_hook.
validate_replayssm_spec_algorithm, arg_groups/moe_hook b12x validator).
flashinfer_cutlass.py flipped to FORK-OWNED (b12x route is fork-only, 760 vs
95 lines; upstream sgl-project#44993 hunks = B5 fold patch saved).

Silent-semantics notes: sgl-project#46127/sgl-project#44912 riders arrived via per-PR picks (union);
sgl-project#45520 (entry 302) is STILL OPEN upstream -- entry-230 fail-safe layer stays
as entry 303 records; if a future sync brings it, re-apply the fail-safe reads.
qwen4_exp holder-contract fused (ctor takes optional batch).

Gates at commit: import sglang OK (CVD=9, prod venv); check_registered_tests/
check_no_bare_pytest_main/check_no_registered_tests_in_package rc=0; ruff F821
+E9+format clean on the resolution set; collect-only over unit/{mem_cache,
model_executor,distributed,layers/moe} = 4337 nodes, 3 ERRORS: 2x cargo/torch-
sys libtorch probe (REPRODUCED IDENTICALLY AT THE PRE-MERGE TIP -- env issue,
not merge damage) + test_hicache_async_ready_sync.py (entry-253-era attribute
gone with the URC rewrite -- EXPECTED pre-B3).

sessions 2/3 owed: B2 test re-home, B3 mem_cache+rust re-plant (+hicache-ctrl
re-ports), B4 scheduler/admission, B5 moe/quant/dispatcher, B6 environ+C0a+
CAVEATS re-pin. PROD BOOT FORBIDDEN until B3 completes (209 clamp + 218
ordering re-plant).
detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
…/ hicliveness), header qwennext-stack-318-2026-10-29, tip b7c656f, CAVEATS re-pin at the 318 tip + b12x-inert/gate-split + storage-liveness + sgl-project#45520-merge#6 + S-B12XDENSE bullets

6 commits from three branches all born at exactly 78783f0: ZERO conflicts.
Records: entry 316 proved INERT ON BOTH ARMS from the predicate and all 10
battery rows unmoved, with the gate-split trap MEASURED LIVE (torch.version.
cuda 13.0 -> gate True vs default nvcc symlink 12.9); entry 317's
subsume-vs-land table + the fraction placed AFTER entry-219's override + the
106.9->100 GB test-scaffold-not-prod caveat; entry 318 (ONE entry, four
commits) the silent-total-L3-loss premise quoted from the base, the
always-enter and flag-then-publish invariants (the latter correcting my own
dispatch sketch), the width-8 interim, non-consuming extra-config validation,
and the new hicache_storage_healthy gauge. Battery: every predicted number
reproduced EXCEPT the worker's own final mem_cache read (1887P/155F) -- my
tip is 1892P/153F with a failure SET identical to base (diff 0 lines); their
2 extra reds are entry-290 dir-context flake, recorded not propagated.
Nothing pushed.
detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
…gl-project#44462(317)/sgl-project#44342-trunk(318) in both the FAM-A3b block and the B6 re-triage verdicts; 10 dated corrections (mergeable flips: sgl-project#46191 REAL -> WATCH-HIGH with the re-derive-vs-QSA_RING_GROUPS=2 instruction, sgl-project#39845 DISPUTED as UNKNOWN-not-true plus the general 'UNKNOWN is no signal' rule; sgl-project#44344 -> SECTION R DOA with the entry-268 unconditional-clear subsumption citation and the mergeable-is-a-trap note; sgl-project#42525 target path moved fla->linear; sgl-project#45520 STILL OPEN -> re-apply rule ARMED FOR MERGE sgl-project#6; b12x-package archived claim DISPUTED with the frozen cutlass-dsl==4.8.0 pin risk standing; sgl-project#46153+sgl-project#41624 gated as ONE QAD bundle; sgl-project#37106 PARKED on eroded motivation with named revive conditions; sgl-project#32479 gated behind the PCIe-IPC measurement and DE-COUPLED from sgl-project#45391; sgl-project#44462's NEXT note spent); NEW S-item S-B12XDENSE filed
@github-actions github-actions Bot added the quant LLM Quantization label Nov 9, 2026
detain added a commit to detain/sglang that referenced this pull request Nov 11, 2026
… commits since 2a6c204 @ merge-sgl-project#5)

Conflicts: 110 files (106 content / 4 add-add / 0 modify/delete) at the settled
321 tip 2d87757 -- vs merge sgl-project#5's 167; the smaller count despite 950 commits
(merge sgl-project#5: 1,120) is the merge-era's re-plant landing INSIDE upstream's new
window (many fork families -- qwen4 varlen, fast_file, hicache-liveness, the
209-218 controller semantics -- now collide as DUAL IMPLEMENTATIONS: upstream
grew the same families independently in the window). By class: A 7
(auto-takes-upstream, dedupe-verified), B 36 (mechanical union/custom splices),
C 46 (upstream structure taken, fork hunks extracted for re-plant), D 21
(fork-owned flips + dual-impl adjudications, incl. the whole fast_file family).
228 conflict regions total: 69 union (all def/class-collision-checked), 118
take-upstream (104 marked, 14 dedupe-verified), 25 keep-fork (17 fold-marked,
8 strict supersets), 16 custom import/region splices.

121 TODO(merge6) markers planted (tree census): b1 12 / b2 23 / b3 18 / b4 39 /
b5 17 / b6 12. (Resolver census at region-resolution was 117; the cascade-repair
grafts below added the rest.) REPLANT INVENTORY: /tmp/opencode/merge6-dossier/
replant/ -- per conflicted file: <name>.fork-hunks.patch (full 2a6c..321 delta),
<name>.upstream-side.patch (full 2a6c..eb73 delta), <name>.ours-dropped.txt /
<name>.theirs-dropped.txt region snapshots, MANIFEST.tsv (file, zone, regions,
kinds, fork-hunk counts, per-region notes). Work records (conflict census JSON,
decision tables, class table, logs) under .../work/.

D-list outcomes (must-not-auto-resolve, handled deliberately):
- sgl-project#45520 (entry 302): STILL OPEN upstream at eb73a50 (gh-verified + ZERO
  "sgl-project#45520" strings in the +950 window). Entry-230's fail-safe layer is UNTOUCHED
  and stays (standard.py warn-once + published-auto guards intact; the re-apply
  rule armed in CAVEATS was NOT needed this merge).
- sgl-project#44590 (entry 228): the window did NOT move the floor further -- upstream
  pyproject stays sglang-kernel==0.4.10 (absorbed at merge sgl-project#5 via sgl-project#43922) and
  the engine assert_pkg_version stays 0.4.9 in BOTH trees; the
  _load_inline_fragment_probes_available() probe never existed on either side
  (docs-mention only). Nothing to re-assert; both pins kept byte-identical.
- flashinfer_cutlass.py (fork-owned per merge-sgl-project#5 flip): did NOT conflict; 56
  b12x refs intact, dispatcher reads untouched.
- modelopt_quant sgl-project#44993/sgl-project#44994-ERA B12X W4A16 arms: the fork's
  defer-swizzle/b12x conditions at L2941-2996 are CONTESTED against upstream's
  skip_swizzled_blockscales rewrite -> took upstream, b5 markers + region
  snapshots (boot-integrity fields live in the fork-owned cutlass file, which
  survived; the b12x selection ARMS need the B5-style re-plant).
- DUAL-IMPLEMENTATION FAMILIES (the new pattern of this merge; upstream grew
  the same features the fork replanted -- take-upstream + fold-diff replant,
  NEVER a silent pick): kernels/ops/mamba/qwen4_short_conv.py + its test suite,
  srt/models/qwen4_exp.py varlen trio, mem_cache/storage/fast_file/fast_file_store.py +
  test_hicache_fast_file_unit.py, test_hicache_sidecar_restore.py,
  fused_moe_triton/layer.py nvfp4-deferred clause,
  tokenizer_manager _validate_top_logprobs_num, hicache_best_practices.mdx.
  All fork-side versions snapshotted; replant batches diff
  ours-dropped.txt against the taken upstream.
- environ.py: took upstream's SGLANG_EXACT_CHUNK_FILL removal; kept ONLY the
  fork-added SGLANG_SPLIT_PREFILL_AT_MAMBA_FORK line (custom splice).
- entry-219 gauge schema: per-pool hicache_host_pool_{used,total}_tokens names
  SURVIVED the auto-merge (metrics_collector/hicache_pool_stats/metrics_reporter/
  urc all intact); the marked b3 controller regions must RE-VERIFY the producer
  dict end-to-end at re-plant, not assume it.

Cascading silent-auto-merge repairs (files NOT in the conflict set, broken by
auto-merge splices -- all found by the ruff-F821/check_* gates, all fixed here):
utils/common.py upstream prune RE-HOMED the fork's get_mixed_precision_moe_quant_algos
into configs/quant_utils.py (next to the moved siblings) + the make_layers
empty_cache into layers/utils/layer_list.py, importers rewired;
mem_cache/pool_host/base.py silently LOST the entry-sgl-project#44462-era
host_memory_budget_bytes() helper whose caller survived -- grafted back;
model_executor/runner/eager_runner.py CP helpers retargeted to upstream's NEW
layers/cp/utils (cp_shard_model_inputs/cp_gather_after_forward) replacing the
retired runner_utils.cp_extend pair; runner/flashinfer_autotune.py + qsa/mqa.py +
modelopt_quant.py import retargets for the utils/device + layers/dp_attention
move; mem_cache/memory_pool.py lossless-verify tuple buffers re-grafted around
upstream's replayssm skip-chain; unified_radix_cache.py host-capacity deadline
preamble re-grafted (marker kept); qwen4_exp.py _use_qwen4_direct_decode
re-grafted (its kernels survived); qwen4_exp_ple_table.py _HOST_PAGE_TABLES
memo dict re-grafted; rotary_embedding/factory.py get_rope param restored;
multimodal_gen forward_context teacache_plan param restored + server_args.py
Literal import; parallel_executor.py rebuilt as fork-owned + upstream's
raise_if_cancelled guards folded into the fork loop structure;
hybrid_cache_controller.py _reduce_prefetch_ack if/zero_ splice repaired;
python_bindings.rs pyo3 signature REORDERED to match the unioned fn new()
argument order (mamba_value_seqlen position -- a positional-bind footgun, fixed
pre-commit); test_io_struct.py get_context/MAX_PARALLEL_SAMPLES imports;
stat_loggers_di + test_flashinfer_autotune_sync + fused_lora_compose +
tp_worker + compressed_tensors + minimax_h3 + hc_mix + scheduler_mixin +
pool_configurator + scheduler/tokenizer_manager import prunes/grafts.

Gates at commit (CVD=9, prod venv, worktree PYTHONPATH): import sglang OK;
check_registered_tests / check_no_bare_pytest_main / check_no_registered_tests_in_package
ALL rc=0; ruff (F401,F821,F823,UP037 @ v0.17.1 vs pinned 0.15.1) ZERO new errors
on the 3,269 merge-touched .py -- the residual 161 are BYTE-IDENTICAL INHERITED
rows present at BOTH parents (kernels/aot, api_types, mm entrypoints/utils, etc.;
0.17-only detections, not merge damage); ruff format clean on the same set;
py_compile clean; CARGO sglang-radix-tree check+test GREEN: 966 passed / 0 failed
/ 1 ignored (vs merge-sgl-project#5-era 943P -- upstream grew the suite +23; NOTE the crate
is now `exclude`d from the rust/ workspace with extension-manifests metadata --
`cargo -p sglang-radix-tree` from rust/ no longer works; run inside the crate).

NON-ADOPTIONS honored: sglang-kernel floor 0.4.9 (engine assert) untouched;
sgl-project#45520 not brought (fail-safe layer intact by rule).

Re-plant batches owed (per marker zones): B2-2026 test re-home (23), B3-2026
mem_cache/controllers/rust (18), B4-2026 scheduler/managers/runner (39),
B5-2026 moe/quant/kernels/qwen4 (17), B6-2026 environ/arg_groups/docs re-pins
(12), B1-2026 diffusion folds (12). PROD BOOT FORBIDDEN until the B3/B4 zones
re-plant the entry-209/218 controller semantics (the trailing_endpoint gate will
read RED at this commit, exactly as at merge-sgl-project#5's damage state).
detain added a commit to detain/sglang that referenced this pull request Nov 11, 2026
…+ re-plant era CLOSED at b86b7c9 (125/125 markers; grep census ZERO verified at this tree): header STACK epoch 321-2026-10-30 -> 329-2026-11-10 (supersession note kept) + +merge6-era-closeout-2026-11-10 trailer line + entries-count 1-329; tip/HEAD re-pinned (era CODE TIP b86b7c9 NAMED with the full wave chain, superseded values dea2641/e9f0ae1838/065fdcdf33/b7c656f69f/5cde7a5ede kept, docs tip recorded as this wave) + a prominent ** LIVE/BOOT STATE ** line (main checkout detached @ 2d87757 = the ONLY boot-validated state, 9b 16:58:52 warm all-needles-hit per prod-health-2026-11-10.txt; era branch CPU-green but NOT BOOTED -- switching prod = operator-chosen checkout + the entry-326 boot-watch). ENTRY 324: imports+bootfix trunk wave -- sgl-project#48781 require_attn_tp_gather->dp_attention relocation invisible to markers, base_runner deletion-only fix unmasking the 145E/32E/31E collection cascade (1dcc2d6); THEN the operator-live crash: maybe_init_router_registration read tokenizer_ipc_name before init_ipc_channels at OUR fork's entry-201 init order, call moved back to upstream's position, RED(2F)/GREEN(2P) test_tokenizer_manager_init_order.py, family scan clean + report-only carry (10a1c08). ENTRY 325: RE-PLANT ERA TABLE b1 12 / b3 18 / b4 39 / b5 17 / b2 23 / b6 16 = 125/125 per-wave dispositions (compressed from the commit bodies, all shas cited) + the NINE ERA CATCHES with FULL CREDIT: _rope_cache_key 10/11-arg universal get_rope crash (15 spec F were its fingerprints), PLE file-backend env registration gap (AttributeError every boot), COW-catcher doubled-staging-arm cascade bug, stat_loggers DI test restored from its corrupted merge hybrid, w13 refined-grid mis-key, the budget-contract decision (fork accounting RESTORED, evidence 106.9GB/100GB = the entry-317 leak shape; INDEXER collective order verified), verify-gate SUBSUME pick (sgl-project#44047 eats sgl-project#42779, both intents preserved), mm_process_config ambient-widths root-cause (validators now state the full 8-way layout), TLFIX twin re-pinned 282/285. ENTRY 326: BATTERY RE-PIN TABLE -- era pins managers 1307P/0F, spec 406P/0F, server_args 496P/0F, sampling 252P/0F, distributed 213P/0F, mem_cache 2965P/24F (the ACCEPTED device-only 24-node list over 8 files), cargo 966P/0F/1-ignored, five registration/wheel gates rc=0 + the PATH-TO-BOOTABLE statement: CPU-proven vs GPU-window debt -- first-boot needles incl. the sampling.cuh-vs-0.7.2 JIT arbitration (if it JIT-fails, wheel-trio-then-pin per S-FI073, never a lone re-pin), autotune caches SHOULD hit warm (no retune owed on 0.7.2), entry-322 verify-at-boot quartet, and the still-riding 321-era watches (sgl-project#45472/sgl-project#46643 TTFT+starvation, first-delta tool-stream latency). ENTRY 327: WHEELS & VENV RECORD -- pairing matrix (FI 0.7.2 HELD, sgl-kernel 0.4.10/0.4.10 era-consistent with the RETIRED-rule cross-ref, torch/cutlass-dsl MATCH), blake3 1.0.11 INSTALLED with ZERO-DRIFT PROOF (0 battery lines vs ~380; matrix re-read unchanged; negative-shim prefix-sim causality) + no-pip/uv-note (operator lane), the two tree-side wheel reverts with reachability + residual-symbol evidence, check_dockerfile_wheel_pins re-green, FI-0.7.3 adoption contract restated with post-era facts (upstream code now expects 0.7.3 in one reported JIT path; tree HELD 0.7.2; first boot arbitrates), dist-name/module-name note sglang_kernel/sgl_kernel. ENTRY 328: INCIDENT RECORD 2026-11-10 ~16:4x-16:58, honest narrative -- re-plant waves landed into the EDITABLE-IMPORT working tree while :30000/:30001 ran; 9b died ~pre-16:50 (cause UNVERIFIABLE from the user acct, /root logs unreadable; worker-respawn-importing-changed-code is the LEADING MECHANISM, stated AS SUCH); the operator restart hit the init-order crash = the era's flagship find; recovery = merge6-era-backup @ b069e6a + detached checkout 2d87757, 9b re-booted WARM 16:58:52 all needles hit, 9d NEVER dropped (Nov 3, HTTP 200); NEW ERA LAW: working tree = prod import path, integration by branch ref-move ONLY; b2's emergency-freeze WIP 7ccf76a (author 16:55:02) kept in range and completed as C1 -- provenance note. ENTRY 329: BACKFILL AUDIT RECORD -- window 2026-08-10..2026-10-27, universe 2714 = 1830 in-tree / 255 already-documented / 628 NEW REJECTS / 1 NEW-INTERESTING; method one-liners; the qwen4-main-squashed side-branch class discovery (REJECT-MERGED-TO-SIDE-BRANCH x44 -- future sweeps MUST filter by base; orchestrator's "206" NOT FOUND in the artifacts, 44 recorded as verified); sgl-project#36043 NVFP4 skinny GEMM SM120 full paragraph (closed-unmerged, absent-tree-verified, W4A4 M<=16 = our exact decode cell, BENCHMARK-BEFORE-PORT -> S-36043-BENCH filed); artifact paths. EVALUATED: the 671-line DOCS-BLOCK.txt inserted VERBATIM as its own dated block (165 dropped list-commas tidied at insertion, noted once at the block head; nothing else reflowed). DATED CORRECTIONS where the waves superseded 322/323-era operational text: the header PROD-BOOT-FORBIDDEN bullet and entry 323's WHEEL DRIFT + STATE lines get [11-10 ERA-CLOSE CORRECTION] annotations (classes closed, wheel landed, ban now purely the operator gate); marker-zone math re-verified against the wave ledgers (121-at-merge vs 125 recount stands correct; planted == consumed per zone). WATCH LIST: era-close block appended (sgl-project#36043 bench-gated; sgl-project#42790/sgl-project#40916 arrived-via-merge6 noted; sgl-project#43703 revert-trap cross-ref from entry 323; sgl-project#45520 STILL-OPEN re-confirmed at era close -- zero in-window refs, b5 verified the fail-safe layer intact; bootfix's report-only get_tokenizer_worker_class/entrypoints wiring carry). Docs-only: NO .py touched; check_registered_tests rc=0 re-run against this tree. NOTHING PUSHED; the live tree was NOT touched by this wave (ref-move integration follows with HEAD/status equality proofs).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

quant LLM Quantization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant