Skip to content

[Perf] [NVIDIA] unquant: Autotune low-M BF16 GEMMs via FlashInfer mm_bf16 - #43703

Merged
zcnrex merged 9 commits into
sgl-project:mainfrom
zcnrex:codex/bf16-m1-direct
Nov 7, 2026
Merged

zcnrex merged 9 commits into
sgl-project:mainfrom
zcnrex:codex/bf16-m1-direct

Conversation

@zcnrex

@zcnrex zcnrex commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Depends on #48468 (bump to FlashInfer 0.7.3). Merge after it lands.

This PR needs flashinfer-ai/flashinfer#6360, which first shipped in FlashInfer 0.7.3. On 0.7.2 the BF16 tuner times each candidate kernel in isolation, so every small GEMM measures at about 10 µs and its low-M picks are close to a tie-break. #6360 makes the tuner rank tactics on a cold launch chain instead, which is what lets the autotuned picks replace the hand tables. On 0.7.2 this PR would delete the hand rows without a reliable replacement.

Motivation

Low-M BF16 decode GEMMs on Blackwell leave measurable performance on the table versus cuBLAS. So far that gap has been closed by hand-measured per-shape tables (_BF16_SPLITK_TUNED_TACTICS, _BF16_WARP_SPLITK_TUNED_TACTICS), which need new rows for every model, weight shape and M. This PR lets FlashInfer pick the kernel per shape and M bucket instead, and removes the tables.

FlashInfer's mm_bf16(backend="auto") autotunes across its BF16 runners; with pdl=True those are the CuTe DSL low-M kernels (direct, warp split-K, cluster split-K), cuDNN, TGV and TinyGEMM. sglang's existing FlashInfer autotune pass drives the tuning, so no per-model or per-shape code is needed.

Modifications

  • On by default when the BF16 GEMM backend is CuTe DSL (the default on Blackwell). Contiguous BF16 linears with M <= 32 (with or without a BF16 bias) go to mm_bf16(pdl=True, backend="auto"), in both _bf16_gemm_dispatch_impl and apply_into. --disable-flashinfer-autotune turns it off.
  • Tuning: inside the FlashInfer autotune context, a larger M (the autotune dummy forward) profiles the M <= 32 buckets on a 32-row slice of each linear. should_run_flashinfer_autotune now also fires when this path is active.
  • First boot tunes each unique BF16 weight shape once; later boots load the cache.
  • Removes the split-K path the autotuned one replaces: both hand tables (all 133 rows: GB300 TP16, MuseSpark TP4/TP8 and its DFLASH draft, Qwen4-Exp TP4, the hidden-8192 TP4/TP8 rows), the direct and cluster split-K dispatch branches, and the Qwen4-Exp precompile hook. The FlashInfer imports stay lazy in initialize_bf16_gemm_config.
  • SM107 (Vera Rubin) keeps main's GLM-5 warp split-K table from [Perf] [NVIDIA] glm5: Route small-M BF16 GEMMs to FlashInfer warp split-K with PDL on SM107 #47354 unchanged, and the autotuned path stays off there: it was only measured on SM100/SM103.
  • The TGV allowlist (_MUSE_SPARK_TGV_M_RANGES) and the Kimi-K3 tables are untouched; above M = 32, and for K3's direct use_cutedsl_bf16_gemm call, they still apply.

Accuracy Tests

Qwen3-VL-4B-Instruct, TP1, 1x B300, FlashInfer 0.7.3. GSM8K via sgl-eval run gsm8k (0.1.3), all 1319 questions, 16 threads, temperature 0, top_p 0.95, --max-tokens 4096:

score truncated
--disable-flashinfer-autotune 93.25% 1.67%
this PR 93.56% 1.14%

Outputs are not bitwise identical to cuBLAS (different accumulation order). Truncations are temperature-0 repetition loops in both arms.

Real dispatch on B300: M in {1, 3, 8, 16, 24, 32} takes the autotuned kernel for the plain, biased, apply_into and addend paths, while M in {33, 64} does not; all outputs are within 1e-2 relative error of an FP32 reference.

Speed Tests and Profiling

Same model, GPU and FlashInfer 0.7.3. bench_one_batch_server, input 1024 / output 512, steady-state output tok/s, mean of two runs per arm in ON/OFF/OFF/ON order (the two runs of each arm agree within 0.6%, 2.0% for ON at bs 16):

bs autotune off this PR change ITL ms
1 425 458 +7.7% 2.35 → 2.18
2 839 901 +7.3% 2.38 → 2.22
4 1678 1798 +7.2% 2.38 → 2.22
8 3163 3379 +6.8% 2.53 → 2.37
16 5794 5971 +3.1% 2.76 → 2.68
24 7523 7854 +4.4% 3.19 → 3.06
32 9713 9953 +2.5% 3.29 → 3.22

Startup: the first boot tunes the 4 decoder shapes; scheduler startup is 195 s, vs 16 s with autotune off. Cached boots run no tuning: 22 s vs 16 s.

Kernel level (B300 sm_103, same SM as GB300; FlashInfer 0.7.3; cold L2: each timed CUDA graph rotates through ≥1 GiB of distinct weights; PDL on). 32 weight shapes (every hand-table shape, the MuseSpark/DFLASH TGV shapes, Qwen3-VL-4B) at M = 1, 4, 8, 12, 16, 24, 32: 224 cells, each run through main's real dispatch (hand row, TGV or cuBLAS), against this PR's autotuned pick (tuned once per shape at M = 32, as the server does) and cuBLAS:

main's kernel cells this PR vs main (geomean) PR >5% slower
hand split-K / warp / direct 104 1.05x 2
TGV 37 1.06x 2
cuBLAS 83 1.29x 0
all 224 1.14x 4

This PR vs cuBLAS: 1.39x geomean, never more than 3% slower. It beats main by more than 5% in 119 cells. The 4 cells where it is more than 5% slower:

  • (32, 8192, 2048): 8.01 vs 7.20 µs (0.90x). MuseSpark TP4 o_proj; main picks TGV.
  • (24, 8192, 2048): 0.91x. Same shape.
  • (1, 320, 2560): 0.93x. A hand direct row.
  • (24, 512, 8192): 0.95x. A GB300 TP16 hand row.

Per-shape tuning takes 6–64 s (median 31 s).

Known limitations

  • M > 32 is not routed: the CuTe DSL low-M kernels stop at M = 32, and auto above that was not measured against the existing TGV/cuBLAS dispatch.
  • With --disable-flashinfer-autotune, the removed hand rows no longer apply, and those shapes use TGV/cuBLAS.
  • Every Blackwell BF16 server pays the first-boot tuning unless its SGLANG_CACHE_DIR persists.

Checklist

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ✅ Run #42311285608
Latest PR Test (Extra): ❌ Run #42311285675
Latest PR Test (AMD ROCm 10): ⏳ Run #42311285570

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@github-actions github-actions Bot added the quant LLM Quantization label Oct 11, 2026
Opt-in SGLANG_OPT_BF16_LOW_M_AUTOTUNE routes bias-free BF16 linears with
M <= 16 through flashinfer mm_bf16(backend="cute-dsl"), which picks among the
direct, warp split-K and cluster split-K kernels per M bucket and weight shape.
The existing FlashInfer autotune pass tunes it with no per-shape table: its
dummy forward profiles the M <= 16 buckets on a row slice of each BF16 linear.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zcnrex
zcnrex force-pushed the codex/bf16-m1-direct branch from b606295 to e609ece Compare October 12, 2026 05:17
@zcnrex zcnrex changed the title [Perf] Use direct BF16 GEMM for single-row gate projections on SM103 [Perf] Autotune low-M BF16 GEMMs via FlashInfer CuTe DSL on Blackwell Oct 12, 2026
zcnrex and others added 4 commits October 11, 2026 22:23
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FlashInfer 0.7.1 persists the profiling policy in saved autotune configs, so
cached bf16_gemm tactics are reused while tuning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Autotuned low-M GEMMs now take bias too, so the M <= 16 Split-K/direct table
rows, the direct-kernel branch and the Qwen4-Exp precompile hook are
unreachable whenever autotune runs. Keep only the M = 24/32 GB300 rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YangXu1990uiuc pushed a commit to flashinfer-ai/flashinfer that referenced this pull request Oct 23, 2026
## 📌 Description

`mm_bf16(backend="cute-dsl")` autotuning ranked low-M tactics by
**isolated single-launch latency** rather than the cold-L2 chained cost
it was configured for, and picked measurably slower tactics than ones in
its own candidate set. `backend="auto"`, which has included the cute-dsl
runners since #6392, had the same problem and also ranked them against
cuDNN, TGV and TinyGEMM on isolated launches.

**Root cause.** #4908 added `TuningConfig.use_cold_l2_graph_replay` for
the BF16 cute-dsl config: rank tactics on a graph of launches, each with
its own input copy, timed after an L2 flush. #4361 then routed every
`use_cold_l2_cache=True` config to `cold_l2_profile`, which times one
graph launch per flushed sample, so the flag stopped having any effect.
A single isolated launch carries a ~6.3 µs floor on B300 (a trivial
`add_` measures 6.38 µs) plus a first-launch cost that differs by kernel
family (cluster split-K > warp split-K > direct). The real kernels take
2.8–6.4 µs, so candidates tied at ~10.24 µs and the pick came down to
noise. Neither the event timer (32 ns resolution) nor PDL (same with
`pdl=False`) causes it.

**Changes**
- `TuningConfig.cold_chain_launches` (with `uses_cold_chain`):
`cold_l2_profile` captures that many back-to-back launches in one graph,
each on its own profile-arena lane, and reports the median flushed
sample divided by the launch count. Every tactic of a cold-chain config
is measured this way. Cost is bounded by the sample count: the existing
untimed warm replay of the chain graph is timed, and chained samples are
capped at about 10 ms of chain time per tactic (at least 1, at most
`repeat`). Configs without a profile arena (e.g. the MXFP8 cute-dsl
config) are unchanged.
- `ValueProfileArena` accepts any non-overlapping layout: slots are
sized by the tensor's memory span and lanes keep its strides, so a
transposed (`W.T`) or padded weight is staged as-is. An operand whose
touched bytes exceed 2×L2 is chained over the caller's tensors without a
copy; a small gapped tensor whose storage span exceeds 2×L2 (copying it
into distinct lanes would be too large) is timed with single cold
launches.
- `_BF16_GEMM_SM100_TUNING_CONFIG` itself gets `cold_chain_launches=8`
and the weight in the profile arena, so every SM100 `mm_bf16` backend
(`auto`, cute-dsl, cuDNN, CUTLASS, TGV, TinyGEMM, cuBLASLt) ranks on the
cold chain with one policy per cache key.
- `TuningConfig.l2_cache_policy` (`hot` / `cold` / `cold-chain:<n>`) is
recorded by `_profiling_policy`, so persisted picks from isolated timing
or a different chain length are re-ranked.
- Cache replacement after re-tuning (an earlier runner's stale winner
must not shadow the new one once the policy changes) is handled by
#6894, now merged; this PR is rebased on it.
- `_apply_tuning_overrides` copies the config with
`dataclasses.replace`; it rebuilt `TuningConfig` field by field and
would drop new fields.
- cuBLASLt and CUTLASS BF16 requirements now reject a non-column-major B
(both kernels require contiguous `A` and `B.T`). Before,
`backend="auto"` offered them for a padded weight and autotuning crashed
in runner preparation (pre-existing; reproduced at `e2df159`).
- `pure_profile` keeps its untimed first graph replay for
`use_cold_l2_graph_replay`; the dead eviction buffer is removed.

## 📊 Performance

**Current revision (`96f9b994`): chain length and tuning cost.** B300
SXM6 AC (sm_103), CUDA 13.0, PyTorch 2.13.0, nvidia-cutlass-dsl 4.6.2,
`pdl=True`, `backend="auto"`. Harness: ≥2 GiB of distinct weights
rotated inside one CUDA graph, median per-GEMM µs over 15 replays; every
tactic the tuner profiles is timed on it, and "×best" is the tuned pick
/ fastest candidate. Each row re-tunes `auto` 3–8 times per chain length
L (L=1 = single-launch samples, as on `main`); values are the worst
trial.

| (M, N, K) | best µs | L=1 (`main`) | L=4 | L=5 | L=6 | L=7 | **L=8** |
L=16 | L=32 |
|---|---|---|---|---|---|---|---|---|---|
| (1, 512, 8192) | 2.76 | 1.040 | 1.040 | 1.038 | 1.061 | 1.036 |
**1.013** | 1.012 | 1.013 |
| (8, 512, 8192) | 3.60 | 1.311 | 1.006 | | | | **1.006** | 1.006 |
1.006 |
| (16, 512, 8192) | 3.64 | 1.015 | 1.015 | | | | **1.015** | 1.015 |
1.015 |
| (16, 256, 8192) | 3.59 | 1.225 | 1.095 | 1.005 | 1.002 | 1.005 |
**1.002** | 1.005 | 1.005 |
| (4, 3584, 2560) | 4.16 | 1.052 | 1.025 | 1.002 | 1.049 | 1.002 |
**1.003** | 1.003 | 1.000 |
| (2, 2560, 1536) | 2.62 | 1.124 | 1.110 | 1.033 | 1.014 | 1.014 |
**1.016** | 1.016 | 1.012 |
| (32, 3584, 2560) | 4.31 | 1.196 | 1.000 | | | | **1.000** | 1.000 |
1.000 |
| (256, 512, 8192) | 6.91 | 1.126 | 1.000 | | | | **1.000** | 1.000 |
1.000 |
| (512, 512, 8192) | 7.58 | 1.200 | | | | | **1.000** | | |
| (512, 256, 8192) | 6.68 | 1.132 | | | | | **1.001** | | |
| (1024, 256, 8192) | 7.69 | 1.115 | | | | | **1.000** | | |
| (1024, 1024, 2560) | 5.68 | 1.201 | | | | | **1.001** | | |
| (256, 3584, 2560) | 6.16 | 1.048 | | | | | **1.003** | | |
| (24, 19456, 2560) | 17.64 | 1.006 | | | | | **1.000** | | 1.000 |
| (512, 3584, 2560), (1024 / 2048, 512, 8192), (128–1024, 19456, 2560) |
8.2–62 | ≤ 1.004 | | | | | ≤ 1.004 | | |

L=8 is the shortest chain within 1.6% of the best candidate on every
cell; L ≤ 7 misses by up to 6%, and 8 ranks as well as 16 or 32.
Chaining matters for kernels below ~8 µs regardless of M (e.g. M=1024
with a small weight), so it is applied to every bucket rather than cut
off by M.

Warm `auto` tuning time (s), same shapes on both sides:

| | `main` (`a9dae250`) | L=32 | **L=8 (this PR)** |
|---|---|---|---|
| (4096, 19456, 2560), one bucket, 54 tactics | 1.36 | 6.33 | **2.09** |
| all 5 buckets, (19456, 2560), real M=16 | 0.80 | 5.20 | **1.92** |
| all 5 buckets, (512, 8192), real M=16 | 0.81 | 3.69 | **1.61** |

Cold (first-run) tuning is dominated by CuTe-DSL compilation: 44.7 →
49.1 s and 70.6 → 73.6 s for the two sweeps at L=32 on `main` vs this
PR, so the L=8 overhead there is a few percent.

Correction: the earlier (24, 19456, 2560) row tuned with
`tuning_buckets=[24]` but served without the override, so it measured
the cache-miss fallback on both revisions; the row above serves under
the same override.

The tables below were measured on the original revision against `main`.

B300 SXM (sm_103), CUDA 13.0, nvidia-cutlass-dsl 4.8.0, `pdl=True`.
Harness: rotate ≥2 GiB of distinct weight buffers inside one CUDA graph
(cold L2), time with CUDA events over replays, arms in ABAB order;
values are median µs per GEMM. Baseline is unpatched `main` (f63b2e7).

60 low-M cells (GB300 TP16 shapes at K=8192 plus K=2560/1536 shapes,
M=1–16), compared against hand-picked split-K/direct tactics:

| | main | this PR |
|---|---|---|
| tuned vs hand, geomean | 1.017× | **1.124×** |
| cells where tuned loses >5% to hand | 17 (worst 0.719×) | **0** (worst
0.983×) |
| tuned vs cuBLAS, geomean | 1.741× | 1.915× |
| tuned this PR vs tuned main, geomean | — | 1.108× |

Largest changes:

| (M, N, K) | cuBLAS | hand | main tuned | PR tuned |
|---|---|---|---|---|
| (1, 512, 8192) | 5.37 | 2.79 | 3.88 | **2.75** |
| (8, 512, 8192) | 5.84 | 3.68 | 5.02 | **3.68** |
| (16, 256, 8192) | 5.58 | 3.69 | 5.06 | **3.67** |
| (16, 512, 8192) | 6.05 | 3.75 | 5.16 | **3.75** |
| (4, 3584, 2560) | 7.72 | 4.58 | 4.93 | **3.72** |
| (2, 2560, 1536) | 5.33 | 2.77 | 3.11 | **2.43** |

Tuner-measured vs harness time per candidate, N=512 K=8192 M=16:

| tactic | harness | main tuner | PR tuner |
|---|---|---|---|
| SplitK (64,8,4,12) | 3.74 | 12.29 | 4.03 ← PR pick |
| WarpSplitK (16,8,256,13,2) | 4.43 | 10.38 | 4.48 |
| WarpSplitK (16,8,256,16,1) | 5.17 | 10.34 ← main pick | — |
| Direct (128,4,8) | 6.42 | 14.30 | 6.66 |

Tuning time per weight shape is unchanged (9–200 s, dominated by kernel
compilation).

**`backend="auto"`.** GB300 (sm_103), CUDA 13.0, nvidia-cutlass-dsl
4.8.0, same cold-L2 harness. Five shapes (N, K) ∈ {(256, 8192), (512,
8192), (3584, 2560), (2560, 1536), (19456, 2560)} × M ∈ {1, 4, 8, 16,
24, 32}, one tuning run per arm. Each cell is compared with the fastest
of five tuning arms (cute-dsl and auto, each under isolated, cold-chain
or CUPTI-span timing; geomean):

| | pdl=1 geomean | pdl=1 cells >5% slower | pdl=1 worst | pdl=0 geomean
|
|---|---|---|---|---|
| `auto`, isolated timing (before) | 1.171× | 21/30 | 1.71× | 1.110× |
| `auto`, cold chain (this PR) | **1.006×** | **1/30** | 1.06× | 1.047×
|
| `cute-dsl`, cold chain (this PR) | 1.013× | 3/30 | 1.10× | 1.012× |

Selected pdl=1 cells, µs:

| (M, N, K) | cuBLAS | `auto` before | `auto` this PR | `cute-dsl` this
PR |
|---|---|---|---|---|
| (16, 256, 8192) | 5.27 | 6.24 (warp split-K) | **3.65** (split-K) |
3.67 |
| (32, 256, 8192) | 5.41 | 5.92 (cuDNN) | **3.65** (split-K) | 3.68 |
| (8, 512, 8192) | 5.46 | 4.40 (TinyGEMM) | **3.67** (split-K) | 3.69 |
| (32, 3584, 2560) | 6.50 | 5.67 (warp split-K) | **4.58** (TGV) | 5.03
|
| (24, 19456, 2560) | **17.33** | 18.65 (warp split-K) | 17.43 (TGV) |
17.45 |

Under the cold chain, `auto` also picks other backends where they beat
cute-dsl: TGV at pdl=1, and cuBLASLt/cuDNN at pdl=0 for N=19456 K=2560 M
≥ 16 (16.8–17.0 µs vs 17.1–18.3 µs for cute-dsl). Ranking by CUPTI
kernel spans removes the launch floor but still prefers warp split-K
where split-K is 1.2–1.4× faster (pdl=1 geomean 1.072×), so it is not a
substitute for the chain.

## 🔍 Related Issues

- Regression from #4361 relative to the profiling behavior #4908
introduced.
- Found while routing sglang low-M BF16 linears through
`mm_bf16(backend="cute-dsl")` (sgl-project/sglang#43703).
- #6392 added the cute-dsl runners to `backend="auto"`; this PR makes
`auto` rank them on the cold chain.
- Supersedes #6361 (closed): instead of letting cuBLASLt compete inside
`backend="cute-dsl"` at M ≤ 32, `auto` ranks cute-dsl against the other
backends.

## 🚀 Pull Request Checklist

### ✅ Pre-commit Checks

- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [x] I have installed the hooks with `pre-commit install`.
- [x] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues. (Run on the changed files.)

## 🧪 Tests

- [x] On B300 SXM6 AC at `534622c5` (rebased on `main` after #6894 and
#6833) (PyTorch 2.13.0+cu130, cuDNN frontend 1.29.0, CuTe DSL 4.6.2):
`python -m pytest tests/gemm/test_mm_bf16.py
tests/autotuner/test_autotuner_core.py
tests/autotuner/test_autotuner_configs.py
tests/autotuner/test_autotune_cache_v2.py -q` — **332 passed, 1
skipped**.
- [x] CPU-only run of autotuner core / cache_v2 / configs (torch CPU) at
`534622c5`: the same 2 failures as `main` (GPU-only), plus 4 new passes.
- [x] New / updated tests:
- `test_cold_chain_times_arena_lanes_per_launch[dense|gapped]`:
transposed and padded arena inputs keep strides and contents; the graph
captures one launch per scheduled batch across distinct lanes; latency
is divided per launch.
- `test_cold_chain_caps_samples_by_chain_time`: 10 / 2 / 1 timed samples
for short / 4 ms / 50 ms chains.
- `test_value_profile_arena_layout_helpers`,
`test_cold_chain_reuses_inputs_for_payload_above_working_set`,
`test_cold_chain_profiling_policy_labels`,
`test_tuning_overrides_keep_cold_chain`.
`test_cold_chain_reuses_inputs_for_payload_above_working_set` also
checks that a gapped view whose span exceeds 2×L2 while its elements do
not is timed with single launches (verified end-to-end on B300 with the
review's `(512, 8192)` stride-`(1, 16384)` cuDNN case).
- `test_mm_bf16_autotune_weight_layouts[auto|cudnn × dense|padded]`:
autotuning stages dense and padded weights in the cold-chain arena and
matches PyTorch. The padded `auto` case crashed before the
cuBLASLt/CUTLASS requirement fix.
- [x] Original revision, GB300 (sm_103), PyTorch 2.14.1+cu130, CuTe DSL
4.8.0: `python -m pytest tests/gemm/test_mm_bf16.py
tests/autotuner/test_autotuner_core.py -q` — 173 passed, 4 skipped.
- [x] `pre-commit run` on all changed files passed, including mypy and
Ruff.

## 🔬 Experimental Track

- [ ] This PR is **experimental**: it adds or changes code under
`flashinfer/experimental/` and/or an `@flashinfer_experimental_api`.
Tracking issue: #
- [ ] The tracking issue names an owner, the reason for the experimental
path, and a graduation plan with a target release.
- [ ] Core changes are limited to a thin entry point (signature, shared
validation, feature-gate check, backend selection, handoff).
- [ ] Tests live in `tests/experimental/` and were validated on the
intended hardware; a runnable example is included.
- [ ] Nothing is registered in `flashinfer/aot.py`, and no experimental
backend is reachable from `backend="auto"` without
`FLASHINFER_ALLOW_EXPERIMENTAL_AUTO_BACKENDS=1`. (Calling an
`@flashinfer_experimental_api` or naming a backend explicitly is itself
the opt-in and needs no environment variable.)
- [ ] **Test scope declared below.** The experimental CI lane runs
exactly these targets, so keep them as narrow as the change allows.

```experimental-tests
# One target per line: a directory or a file. (A pytest ::selector is not
# supported -- the sharding runner cannot consume one.) Must be under
# tests/experimental/ and must exist. Delete these comment lines and add yours, e.g.
#
#   tests/experimental/test_my_backend.py
#   tests/experimental/my_backend/
#
# Declaring the whole tree (tests/experimental/) is allowed but means every
# experimental PR pays for every other feature's tests, in every matrix cell.
```

## Reviewer Notes

- The cold chain now applies to every SM100 `mm_bf16` backend, which
changes the persisted cache identity (`l2_cache_policy` becomes
`cold-chain:8`), so previously tuned BF16 entries are re-tuned.
- Tuning cost: one untimed 8-launch warm replay plus up to 10 timed
chained samples per tactic (timed samples capped at about 10 ms of chain
time). Warm `auto` tuning is about 1.5–2.4× `main` (table above); cold
tuning is compile-dominated. Arena memory is about 4×L2; weights above
2×L2 are chained in place without a copy.
- Not run on SM100/B200; current-revision numbers are B300 only.

AI-assisted (Claude Code and Codex).

🤖 Generated with [Claude Code](https://claude.com/claude-code)



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Performance**
* Improved cold-cache profiling for CUDA Graph workloads with
configurable multi-launch chains and per-launch latency reporting.
  * Added optimized profiling for dense BF16 GEMM weights.

* **Bug Fixes**
* Improved support for non-contiguous tensor inputs while preserving
their layouts.
* Improved BF16 GEMM support for padded weights, with autotuning enabled
or disabled.
* Fixed stale tuning results being retained after retuning selects a
different runner.
* Improved profiling behavior for oversized arena payloads and
differentiated hot, cold, and cold-chain measurements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
zcnrex and others added 2 commits November 4, 2026 20:00
Resolve conflicts with the new MuseSpark, Qwen4-Exp, DFLASH and warp split-K
BF16 rows; the FlashInfer imports for the autotuned path are now lazy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pile hook

The autotuned FlashInfer path serves M<=16 first, so these rows only applied
with --disable-flashinfer-autotune.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zcnrex zcnrex changed the title [Perf] Autotune low-M BF16 GEMMs via FlashInfer CuTe DSL on Blackwell [Perf] [NVIDIA] unquant: Autotune low-M BF16 GEMMs via FlashInfer CuTe DSL Nov 5, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zcnrex zcnrex changed the title [Perf] [NVIDIA] unquant: Autotune low-M BF16 GEMMs via FlashInfer CuTe DSL [Perf] [NVIDIA] unquant: Autotune low-M BF16 GEMMs via FlashInfer mm_bf16 Nov 5, 2026
@zcnrex
zcnrex marked this pull request as ready for review November 6, 2026 23:15

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@zcnrex
zcnrex requested a review from Alisehen as a code owner November 6, 2026 23:15
Keep main's SM107 GLM-5 warp split-K path (sgl-project#47354) and leave the
autotuned low-M path off SM107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zcnrex

zcnrex commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

@BBuf review

@BBuf

BBuf commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@BBuf

BBuf commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/e2e/models/test_qwen4_exp_models.py test/registered/e2e/models_large/test_nvidia_nemotron_3_super_nightly.py::TestNvidiaNemotron3SuperNightly::test_nemotron_3_super_bf16

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_qwen4_exp_models.py  # extra-b

🚀 8-gpu-h200 (1 test): ❌ View workflow run

cd test/ && python3 registered/e2e/models_large/test_nvidia_nemotron_3_super_nightly.py TestNvidiaNemotron3SuperNightly::test_nemotron_3_super_bf16  # nightly

🚀 8-gpu-b200 (1 test): ❌ View workflow run

cd test/ && python3 registered/e2e/models_large/test_nvidia_nemotron_3_super_nightly.py TestNvidiaNemotron3SuperNightly::test_nemotron_3_super_bf16  # nightly

@zcnrex

zcnrex commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/e2e/models_large/test_nvidia_nemotron_3_super_nightly.py::TestNvidiaNemotron3SuperNightly.test_nemotron_3_super_bf16

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models_large/test_nvidia_nemotron_3_super_nightly.py TestNvidiaNemotron3SuperNightly.test_nemotron_3_super_bf16  # nightly

🚀 8-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models_large/test_nvidia_nemotron_3_super_nightly.py TestNvidiaNemotron3SuperNightly.test_nemotron_3_super_bf16  # nightly

@zcnrex
zcnrex merged commit 19cee50 into sgl-project:main Nov 7, 2026
246 of 273 checks passed
@zcnrex
zcnrex deleted the codex/bf16-m1-direct branch November 7, 2026 07:33
hnyls2002 added a commit that referenced this pull request Nov 8, 2026
detain added a commit to detain/sglang that referenced this pull request Nov 11, 2026
…+ re-plant era CLOSED at b86b7c9 (125/125 markers; grep census ZERO verified at this tree): header STACK epoch 321-2026-10-30 -> 329-2026-11-10 (supersession note kept) + +merge6-era-closeout-2026-11-10 trailer line + entries-count 1-329; tip/HEAD re-pinned (era CODE TIP b86b7c9 NAMED with the full wave chain, superseded values dea2641/e9f0ae1838/065fdcdf33/b7c656f69f/5cde7a5ede kept, docs tip recorded as this wave) + a prominent ** LIVE/BOOT STATE ** line (main checkout detached @ 2d87757 = the ONLY boot-validated state, 9b 16:58:52 warm all-needles-hit per prod-health-2026-11-10.txt; era branch CPU-green but NOT BOOTED -- switching prod = operator-chosen checkout + the entry-326 boot-watch). ENTRY 324: imports+bootfix trunk wave -- sgl-project#48781 require_attn_tp_gather->dp_attention relocation invisible to markers, base_runner deletion-only fix unmasking the 145E/32E/31E collection cascade (1dcc2d6); THEN the operator-live crash: maybe_init_router_registration read tokenizer_ipc_name before init_ipc_channels at OUR fork's entry-201 init order, call moved back to upstream's position, RED(2F)/GREEN(2P) test_tokenizer_manager_init_order.py, family scan clean + report-only carry (10a1c08). ENTRY 325: RE-PLANT ERA TABLE b1 12 / b3 18 / b4 39 / b5 17 / b2 23 / b6 16 = 125/125 per-wave dispositions (compressed from the commit bodies, all shas cited) + the NINE ERA CATCHES with FULL CREDIT: _rope_cache_key 10/11-arg universal get_rope crash (15 spec F were its fingerprints), PLE file-backend env registration gap (AttributeError every boot), COW-catcher doubled-staging-arm cascade bug, stat_loggers DI test restored from its corrupted merge hybrid, w13 refined-grid mis-key, the budget-contract decision (fork accounting RESTORED, evidence 106.9GB/100GB = the entry-317 leak shape; INDEXER collective order verified), verify-gate SUBSUME pick (sgl-project#44047 eats sgl-project#42779, both intents preserved), mm_process_config ambient-widths root-cause (validators now state the full 8-way layout), TLFIX twin re-pinned 282/285. ENTRY 326: BATTERY RE-PIN TABLE -- era pins managers 1307P/0F, spec 406P/0F, server_args 496P/0F, sampling 252P/0F, distributed 213P/0F, mem_cache 2965P/24F (the ACCEPTED device-only 24-node list over 8 files), cargo 966P/0F/1-ignored, five registration/wheel gates rc=0 + the PATH-TO-BOOTABLE statement: CPU-proven vs GPU-window debt -- first-boot needles incl. the sampling.cuh-vs-0.7.2 JIT arbitration (if it JIT-fails, wheel-trio-then-pin per S-FI073, never a lone re-pin), autotune caches SHOULD hit warm (no retune owed on 0.7.2), entry-322 verify-at-boot quartet, and the still-riding 321-era watches (sgl-project#45472/sgl-project#46643 TTFT+starvation, first-delta tool-stream latency). ENTRY 327: WHEELS & VENV RECORD -- pairing matrix (FI 0.7.2 HELD, sgl-kernel 0.4.10/0.4.10 era-consistent with the RETIRED-rule cross-ref, torch/cutlass-dsl MATCH), blake3 1.0.11 INSTALLED with ZERO-DRIFT PROOF (0 battery lines vs ~380; matrix re-read unchanged; negative-shim prefix-sim causality) + no-pip/uv-note (operator lane), the two tree-side wheel reverts with reachability + residual-symbol evidence, check_dockerfile_wheel_pins re-green, FI-0.7.3 adoption contract restated with post-era facts (upstream code now expects 0.7.3 in one reported JIT path; tree HELD 0.7.2; first boot arbitrates), dist-name/module-name note sglang_kernel/sgl_kernel. ENTRY 328: INCIDENT RECORD 2026-11-10 ~16:4x-16:58, honest narrative -- re-plant waves landed into the EDITABLE-IMPORT working tree while :30000/:30001 ran; 9b died ~pre-16:50 (cause UNVERIFIABLE from the user acct, /root logs unreadable; worker-respawn-importing-changed-code is the LEADING MECHANISM, stated AS SUCH); the operator restart hit the init-order crash = the era's flagship find; recovery = merge6-era-backup @ b069e6a + detached checkout 2d87757, 9b re-booted WARM 16:58:52 all needles hit, 9d NEVER dropped (Nov 3, HTTP 200); NEW ERA LAW: working tree = prod import path, integration by branch ref-move ONLY; b2's emergency-freeze WIP 7ccf76a (author 16:55:02) kept in range and completed as C1 -- provenance note. ENTRY 329: BACKFILL AUDIT RECORD -- window 2026-08-10..2026-10-27, universe 2714 = 1830 in-tree / 255 already-documented / 628 NEW REJECTS / 1 NEW-INTERESTING; method one-liners; the qwen4-main-squashed side-branch class discovery (REJECT-MERGED-TO-SIDE-BRANCH x44 -- future sweeps MUST filter by base; orchestrator's "206" NOT FOUND in the artifacts, 44 recorded as verified); sgl-project#36043 NVFP4 skinny GEMM SM120 full paragraph (closed-unmerged, absent-tree-verified, W4A4 M<=16 = our exact decode cell, BENCHMARK-BEFORE-PORT -> S-36043-BENCH filed); artifact paths. EVALUATED: the 671-line DOCS-BLOCK.txt inserted VERBATIM as its own dated block (165 dropped list-commas tidied at insertion, noted once at the block head; nothing else reflowed). DATED CORRECTIONS where the waves superseded 322/323-era operational text: the header PROD-BOOT-FORBIDDEN bullet and entry 323's WHEEL DRIFT + STATE lines get [11-10 ERA-CLOSE CORRECTION] annotations (classes closed, wheel landed, ban now purely the operator gate); marker-zone math re-verified against the wave ledgers (121-at-merge vs 125 recount stands correct; planted == consumed per zone). WATCH LIST: era-close block appended (sgl-project#36043 bench-gated; sgl-project#42790/sgl-project#40916 arrived-via-merge6 noted; sgl-project#43703 revert-trap cross-ref from entry 323; sgl-project#45520 STILL-OPEN re-confirmed at era close -- zero in-window refs, b5 verified the fail-safe layer intact; bootfix's report-only get_tokenizer_worker_class/entrypoints wiring carry). Docs-only: NO .py touched; check_registered_tests rc=0 re-run against this tree. NOTHING PUSHED; the live tree was NOT touched by this wave (ref-move integration follows with HEAD/status equality proofs).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

quant LLM Quantization run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants