Repository navigation
[Perf] [NVIDIA] unquant: Autotune low-M BF16 GEMMs via FlashInfer mm_bf16 - #43703
Merged
Merged
Conversation
Opt-in SGLANG_OPT_BF16_LOW_M_AUTOTUNE routes bias-free BF16 linears with M <= 16 through flashinfer mm_bf16(backend="cute-dsl"), which picks among the direct, warp split-K and cluster split-K kernels per M bucket and weight shape. The existing FlashInfer autotune pass tunes it with no per-shape table: its dummy forward profiles the M <= 16 buckets on a row slice of each BF16 linear. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zcnrex
force-pushed
the
codex/bf16-m1-direct
branch
from
October 12, 2026 05:17
b606295 to
e609ece
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FlashInfer 0.7.1 persists the profiling policy in saved autotune configs, so cached bf16_gemm tactics are reused while tuning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Autotuned low-M GEMMs now take bias too, so the M <= 16 Split-K/direct table rows, the direct-kernel branch and the Qwen4-Exp precompile hook are unreachable whenever autotune runs. Keep only the M = 24/32 GB300 rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 12, 2026
YangXu1990uiuc
pushed a commit
to flashinfer-ai/flashinfer
that referenced
this pull request
Oct 23, 2026
## 📌 Description `mm_bf16(backend="cute-dsl")` autotuning ranked low-M tactics by **isolated single-launch latency** rather than the cold-L2 chained cost it was configured for, and picked measurably slower tactics than ones in its own candidate set. `backend="auto"`, which has included the cute-dsl runners since #6392, had the same problem and also ranked them against cuDNN, TGV and TinyGEMM on isolated launches. **Root cause.** #4908 added `TuningConfig.use_cold_l2_graph_replay` for the BF16 cute-dsl config: rank tactics on a graph of launches, each with its own input copy, timed after an L2 flush. #4361 then routed every `use_cold_l2_cache=True` config to `cold_l2_profile`, which times one graph launch per flushed sample, so the flag stopped having any effect. A single isolated launch carries a ~6.3 µs floor on B300 (a trivial `add_` measures 6.38 µs) plus a first-launch cost that differs by kernel family (cluster split-K > warp split-K > direct). The real kernels take 2.8–6.4 µs, so candidates tied at ~10.24 µs and the pick came down to noise. Neither the event timer (32 ns resolution) nor PDL (same with `pdl=False`) causes it. **Changes** - `TuningConfig.cold_chain_launches` (with `uses_cold_chain`): `cold_l2_profile` captures that many back-to-back launches in one graph, each on its own profile-arena lane, and reports the median flushed sample divided by the launch count. Every tactic of a cold-chain config is measured this way. Cost is bounded by the sample count: the existing untimed warm replay of the chain graph is timed, and chained samples are capped at about 10 ms of chain time per tactic (at least 1, at most `repeat`). Configs without a profile arena (e.g. the MXFP8 cute-dsl config) are unchanged. - `ValueProfileArena` accepts any non-overlapping layout: slots are sized by the tensor's memory span and lanes keep its strides, so a transposed (`W.T`) or padded weight is staged as-is. An operand whose touched bytes exceed 2×L2 is chained over the caller's tensors without a copy; a small gapped tensor whose storage span exceeds 2×L2 (copying it into distinct lanes would be too large) is timed with single cold launches. - `_BF16_GEMM_SM100_TUNING_CONFIG` itself gets `cold_chain_launches=8` and the weight in the profile arena, so every SM100 `mm_bf16` backend (`auto`, cute-dsl, cuDNN, CUTLASS, TGV, TinyGEMM, cuBLASLt) ranks on the cold chain with one policy per cache key. - `TuningConfig.l2_cache_policy` (`hot` / `cold` / `cold-chain:<n>`) is recorded by `_profiling_policy`, so persisted picks from isolated timing or a different chain length are re-ranked. - Cache replacement after re-tuning (an earlier runner's stale winner must not shadow the new one once the policy changes) is handled by #6894, now merged; this PR is rebased on it. - `_apply_tuning_overrides` copies the config with `dataclasses.replace`; it rebuilt `TuningConfig` field by field and would drop new fields. - cuBLASLt and CUTLASS BF16 requirements now reject a non-column-major B (both kernels require contiguous `A` and `B.T`). Before, `backend="auto"` offered them for a padded weight and autotuning crashed in runner preparation (pre-existing; reproduced at `e2df159`). - `pure_profile` keeps its untimed first graph replay for `use_cold_l2_graph_replay`; the dead eviction buffer is removed. ## 📊 Performance **Current revision (`96f9b994`): chain length and tuning cost.** B300 SXM6 AC (sm_103), CUDA 13.0, PyTorch 2.13.0, nvidia-cutlass-dsl 4.6.2, `pdl=True`, `backend="auto"`. Harness: ≥2 GiB of distinct weights rotated inside one CUDA graph, median per-GEMM µs over 15 replays; every tactic the tuner profiles is timed on it, and "×best" is the tuned pick / fastest candidate. Each row re-tunes `auto` 3–8 times per chain length L (L=1 = single-launch samples, as on `main`); values are the worst trial. | (M, N, K) | best µs | L=1 (`main`) | L=4 | L=5 | L=6 | L=7 | **L=8** | L=16 | L=32 | |---|---|---|---|---|---|---|---|---|---| | (1, 512, 8192) | 2.76 | 1.040 | 1.040 | 1.038 | 1.061 | 1.036 | **1.013** | 1.012 | 1.013 | | (8, 512, 8192) | 3.60 | 1.311 | 1.006 | | | | **1.006** | 1.006 | 1.006 | | (16, 512, 8192) | 3.64 | 1.015 | 1.015 | | | | **1.015** | 1.015 | 1.015 | | (16, 256, 8192) | 3.59 | 1.225 | 1.095 | 1.005 | 1.002 | 1.005 | **1.002** | 1.005 | 1.005 | | (4, 3584, 2560) | 4.16 | 1.052 | 1.025 | 1.002 | 1.049 | 1.002 | **1.003** | 1.003 | 1.000 | | (2, 2560, 1536) | 2.62 | 1.124 | 1.110 | 1.033 | 1.014 | 1.014 | **1.016** | 1.016 | 1.012 | | (32, 3584, 2560) | 4.31 | 1.196 | 1.000 | | | | **1.000** | 1.000 | 1.000 | | (256, 512, 8192) | 6.91 | 1.126 | 1.000 | | | | **1.000** | 1.000 | 1.000 | | (512, 512, 8192) | 7.58 | 1.200 | | | | | **1.000** | | | | (512, 256, 8192) | 6.68 | 1.132 | | | | | **1.001** | | | | (1024, 256, 8192) | 7.69 | 1.115 | | | | | **1.000** | | | | (1024, 1024, 2560) | 5.68 | 1.201 | | | | | **1.001** | | | | (256, 3584, 2560) | 6.16 | 1.048 | | | | | **1.003** | | | | (24, 19456, 2560) | 17.64 | 1.006 | | | | | **1.000** | | 1.000 | | (512, 3584, 2560), (1024 / 2048, 512, 8192), (128–1024, 19456, 2560) | 8.2–62 | ≤ 1.004 | | | | | ≤ 1.004 | | | L=8 is the shortest chain within 1.6% of the best candidate on every cell; L ≤ 7 misses by up to 6%, and 8 ranks as well as 16 or 32. Chaining matters for kernels below ~8 µs regardless of M (e.g. M=1024 with a small weight), so it is applied to every bucket rather than cut off by M. Warm `auto` tuning time (s), same shapes on both sides: | | `main` (`a9dae250`) | L=32 | **L=8 (this PR)** | |---|---|---|---| | (4096, 19456, 2560), one bucket, 54 tactics | 1.36 | 6.33 | **2.09** | | all 5 buckets, (19456, 2560), real M=16 | 0.80 | 5.20 | **1.92** | | all 5 buckets, (512, 8192), real M=16 | 0.81 | 3.69 | **1.61** | Cold (first-run) tuning is dominated by CuTe-DSL compilation: 44.7 → 49.1 s and 70.6 → 73.6 s for the two sweeps at L=32 on `main` vs this PR, so the L=8 overhead there is a few percent. Correction: the earlier (24, 19456, 2560) row tuned with `tuning_buckets=[24]` but served without the override, so it measured the cache-miss fallback on both revisions; the row above serves under the same override. The tables below were measured on the original revision against `main`. B300 SXM (sm_103), CUDA 13.0, nvidia-cutlass-dsl 4.8.0, `pdl=True`. Harness: rotate ≥2 GiB of distinct weight buffers inside one CUDA graph (cold L2), time with CUDA events over replays, arms in ABAB order; values are median µs per GEMM. Baseline is unpatched `main` (f63b2e7). 60 low-M cells (GB300 TP16 shapes at K=8192 plus K=2560/1536 shapes, M=1–16), compared against hand-picked split-K/direct tactics: | | main | this PR | |---|---|---| | tuned vs hand, geomean | 1.017× | **1.124×** | | cells where tuned loses >5% to hand | 17 (worst 0.719×) | **0** (worst 0.983×) | | tuned vs cuBLAS, geomean | 1.741× | 1.915× | | tuned this PR vs tuned main, geomean | — | 1.108× | Largest changes: | (M, N, K) | cuBLAS | hand | main tuned | PR tuned | |---|---|---|---|---| | (1, 512, 8192) | 5.37 | 2.79 | 3.88 | **2.75** | | (8, 512, 8192) | 5.84 | 3.68 | 5.02 | **3.68** | | (16, 256, 8192) | 5.58 | 3.69 | 5.06 | **3.67** | | (16, 512, 8192) | 6.05 | 3.75 | 5.16 | **3.75** | | (4, 3584, 2560) | 7.72 | 4.58 | 4.93 | **3.72** | | (2, 2560, 1536) | 5.33 | 2.77 | 3.11 | **2.43** | Tuner-measured vs harness time per candidate, N=512 K=8192 M=16: | tactic | harness | main tuner | PR tuner | |---|---|---|---| | SplitK (64,8,4,12) | 3.74 | 12.29 | 4.03 ← PR pick | | WarpSplitK (16,8,256,13,2) | 4.43 | 10.38 | 4.48 | | WarpSplitK (16,8,256,16,1) | 5.17 | 10.34 ← main pick | — | | Direct (128,4,8) | 6.42 | 14.30 | 6.66 | Tuning time per weight shape is unchanged (9–200 s, dominated by kernel compilation). **`backend="auto"`.** GB300 (sm_103), CUDA 13.0, nvidia-cutlass-dsl 4.8.0, same cold-L2 harness. Five shapes (N, K) ∈ {(256, 8192), (512, 8192), (3584, 2560), (2560, 1536), (19456, 2560)} × M ∈ {1, 4, 8, 16, 24, 32}, one tuning run per arm. Each cell is compared with the fastest of five tuning arms (cute-dsl and auto, each under isolated, cold-chain or CUPTI-span timing; geomean): | | pdl=1 geomean | pdl=1 cells >5% slower | pdl=1 worst | pdl=0 geomean | |---|---|---|---|---| | `auto`, isolated timing (before) | 1.171× | 21/30 | 1.71× | 1.110× | | `auto`, cold chain (this PR) | **1.006×** | **1/30** | 1.06× | 1.047× | | `cute-dsl`, cold chain (this PR) | 1.013× | 3/30 | 1.10× | 1.012× | Selected pdl=1 cells, µs: | (M, N, K) | cuBLAS | `auto` before | `auto` this PR | `cute-dsl` this PR | |---|---|---|---|---| | (16, 256, 8192) | 5.27 | 6.24 (warp split-K) | **3.65** (split-K) | 3.67 | | (32, 256, 8192) | 5.41 | 5.92 (cuDNN) | **3.65** (split-K) | 3.68 | | (8, 512, 8192) | 5.46 | 4.40 (TinyGEMM) | **3.67** (split-K) | 3.69 | | (32, 3584, 2560) | 6.50 | 5.67 (warp split-K) | **4.58** (TGV) | 5.03 | | (24, 19456, 2560) | **17.33** | 18.65 (warp split-K) | 17.43 (TGV) | 17.45 | Under the cold chain, `auto` also picks other backends where they beat cute-dsl: TGV at pdl=1, and cuBLASLt/cuDNN at pdl=0 for N=19456 K=2560 M ≥ 16 (16.8–17.0 µs vs 17.1–18.3 µs for cute-dsl). Ranking by CUPTI kernel spans removes the launch floor but still prefers warp split-K where split-K is 1.2–1.4× faster (pdl=1 geomean 1.072×), so it is not a substitute for the chain. ## 🔍 Related Issues - Regression from #4361 relative to the profiling behavior #4908 introduced. - Found while routing sglang low-M BF16 linears through `mm_bf16(backend="cute-dsl")` (sgl-project/sglang#43703). - #6392 added the cute-dsl runners to `backend="auto"`; this PR makes `auto` rank them on the cold chain. - Supersedes #6361 (closed): instead of letting cuBLASLt compete inside `backend="cute-dsl"` at M ≤ 32, `auto` ranks cute-dsl against the other backends. ## 🚀 Pull Request Checklist ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. (Run on the changed files.) ## 🧪 Tests - [x] On B300 SXM6 AC at `534622c5` (rebased on `main` after #6894 and #6833) (PyTorch 2.13.0+cu130, cuDNN frontend 1.29.0, CuTe DSL 4.6.2): `python -m pytest tests/gemm/test_mm_bf16.py tests/autotuner/test_autotuner_core.py tests/autotuner/test_autotuner_configs.py tests/autotuner/test_autotune_cache_v2.py -q` — **332 passed, 1 skipped**. - [x] CPU-only run of autotuner core / cache_v2 / configs (torch CPU) at `534622c5`: the same 2 failures as `main` (GPU-only), plus 4 new passes. - [x] New / updated tests: - `test_cold_chain_times_arena_lanes_per_launch[dense|gapped]`: transposed and padded arena inputs keep strides and contents; the graph captures one launch per scheduled batch across distinct lanes; latency is divided per launch. - `test_cold_chain_caps_samples_by_chain_time`: 10 / 2 / 1 timed samples for short / 4 ms / 50 ms chains. - `test_value_profile_arena_layout_helpers`, `test_cold_chain_reuses_inputs_for_payload_above_working_set`, `test_cold_chain_profiling_policy_labels`, `test_tuning_overrides_keep_cold_chain`. `test_cold_chain_reuses_inputs_for_payload_above_working_set` also checks that a gapped view whose span exceeds 2×L2 while its elements do not is timed with single launches (verified end-to-end on B300 with the review's `(512, 8192)` stride-`(1, 16384)` cuDNN case). - `test_mm_bf16_autotune_weight_layouts[auto|cudnn × dense|padded]`: autotuning stages dense and padded weights in the cold-chain arena and matches PyTorch. The padded `auto` case crashed before the cuBLASLt/CUTLASS requirement fix. - [x] Original revision, GB300 (sm_103), PyTorch 2.14.1+cu130, CuTe DSL 4.8.0: `python -m pytest tests/gemm/test_mm_bf16.py tests/autotuner/test_autotuner_core.py -q` — 173 passed, 4 skipped. - [x] `pre-commit run` on all changed files passed, including mypy and Ruff. ## 🔬 Experimental Track - [ ] This PR is **experimental**: it adds or changes code under `flashinfer/experimental/` and/or an `@flashinfer_experimental_api`. Tracking issue: # - [ ] The tracking issue names an owner, the reason for the experimental path, and a graduation plan with a target release. - [ ] Core changes are limited to a thin entry point (signature, shared validation, feature-gate check, backend selection, handoff). - [ ] Tests live in `tests/experimental/` and were validated on the intended hardware; a runnable example is included. - [ ] Nothing is registered in `flashinfer/aot.py`, and no experimental backend is reachable from `backend="auto"` without `FLASHINFER_ALLOW_EXPERIMENTAL_AUTO_BACKENDS=1`. (Calling an `@flashinfer_experimental_api` or naming a backend explicitly is itself the opt-in and needs no environment variable.) - [ ] **Test scope declared below.** The experimental CI lane runs exactly these targets, so keep them as narrow as the change allows. ```experimental-tests # One target per line: a directory or a file. (A pytest ::selector is not # supported -- the sharding runner cannot consume one.) Must be under # tests/experimental/ and must exist. Delete these comment lines and add yours, e.g. # # tests/experimental/test_my_backend.py # tests/experimental/my_backend/ # # Declaring the whole tree (tests/experimental/) is allowed but means every # experimental PR pays for every other feature's tests, in every matrix cell. ``` ## Reviewer Notes - The cold chain now applies to every SM100 `mm_bf16` backend, which changes the persisted cache identity (`l2_cache_policy` becomes `cold-chain:8`), so previously tuned BF16 entries are re-tuned. - Tuning cost: one untimed 8-launch warm replay plus up to 10 timed chained samples per tactic (timed samples capped at about 10 ms of chain time). Warm `auto` tuning is about 1.5–2.4× `main` (table above); cold tuning is compile-dominated. Arena memory is about 4×L2; weights above 2×L2 are chained in place without a copy. - Not run on SM100/B200; current-revision numbers are B300 only. AI-assisted (Claude Code and Codex). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Performance** * Improved cold-cache profiling for CUDA Graph workloads with configurable multi-launch chains and per-launch latency reporting. * Added optimized profiling for dense BF16 GEMM weights. * **Bug Fixes** * Improved support for non-contiguous tensor inputs while preserving their layouts. * Improved BF16 GEMM support for padded weights, with autotuning enabled or disabled. * Fixed stale tuning results being retained after retuning selects a different runner. * Improved profiling behavior for oversized arena payloads and differentiated hot, cold, and cold-chain measurements. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Resolve conflicts with the new MuseSpark, Qwen4-Exp, DFLASH and warp split-K BF16 rows; the FlashInfer imports for the autotuned path are now lazy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pile hook The autotuned FlashInfer path serves M<=16 first, so these rows only applied with --disable-flashinfer-autotune. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zcnrex
marked this pull request as ready for review
November 6, 2026 23:15
zcnrex
requested review from
AniZpZ,
BBuf,
Edwardf0t1,
FlamingoPg,
Fridge003,
HaiShaw,
OrangeRedeng,
Ying1123,
b8zhong,
ch-wan,
hnyls2002,
ispobock and
merrymercy
as code owners
November 6, 2026 23:15
Keep main's SM107 GLM-5 warp split-K path (sgl-project#47354) and leave the autotuned low-M path off SM107. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
@BBuf review |
Collaborator
|
/tag-and-rerun-ci |
Collaborator
|
/rerun-test test/registered/e2e/models/test_qwen4_exp_models.py test/registered/e2e/models_large/test_nvidia_nemotron_3_super_nightly.py::TestNvidiaNemotron3SuperNightly::test_nemotron_3_super_bf16 |
Contributor
|
🚀 🚀 🚀 |
Collaborator
Author
|
/rerun-test test/registered/e2e/models_large/test_nvidia_nemotron_3_super_nightly.py::TestNvidiaNemotron3SuperNightly.test_nemotron_3_super_bf16 |
Contributor
|
🚀 🚀 |
This was referenced Nov 7, 2026
3 tasks done
hnyls2002
added a commit
that referenced
this pull request
Nov 8, 2026
3 of 4 tasks
5 tasks done
detain
added a commit
to detain/sglang
that referenced
this pull request
Nov 11, 2026
…+ re-plant era CLOSED at b86b7c9 (125/125 markers; grep census ZERO verified at this tree): header STACK epoch 321-2026-10-30 -> 329-2026-11-10 (supersession note kept) + +merge6-era-closeout-2026-11-10 trailer line + entries-count 1-329; tip/HEAD re-pinned (era CODE TIP b86b7c9 NAMED with the full wave chain, superseded values dea2641/e9f0ae1838/065fdcdf33/b7c656f69f/5cde7a5ede kept, docs tip recorded as this wave) + a prominent ** LIVE/BOOT STATE ** line (main checkout detached @ 2d87757 = the ONLY boot-validated state, 9b 16:58:52 warm all-needles-hit per prod-health-2026-11-10.txt; era branch CPU-green but NOT BOOTED -- switching prod = operator-chosen checkout + the entry-326 boot-watch). ENTRY 324: imports+bootfix trunk wave -- sgl-project#48781 require_attn_tp_gather->dp_attention relocation invisible to markers, base_runner deletion-only fix unmasking the 145E/32E/31E collection cascade (1dcc2d6); THEN the operator-live crash: maybe_init_router_registration read tokenizer_ipc_name before init_ipc_channels at OUR fork's entry-201 init order, call moved back to upstream's position, RED(2F)/GREEN(2P) test_tokenizer_manager_init_order.py, family scan clean + report-only carry (10a1c08). ENTRY 325: RE-PLANT ERA TABLE b1 12 / b3 18 / b4 39 / b5 17 / b2 23 / b6 16 = 125/125 per-wave dispositions (compressed from the commit bodies, all shas cited) + the NINE ERA CATCHES with FULL CREDIT: _rope_cache_key 10/11-arg universal get_rope crash (15 spec F were its fingerprints), PLE file-backend env registration gap (AttributeError every boot), COW-catcher doubled-staging-arm cascade bug, stat_loggers DI test restored from its corrupted merge hybrid, w13 refined-grid mis-key, the budget-contract decision (fork accounting RESTORED, evidence 106.9GB/100GB = the entry-317 leak shape; INDEXER collective order verified), verify-gate SUBSUME pick (sgl-project#44047 eats sgl-project#42779, both intents preserved), mm_process_config ambient-widths root-cause (validators now state the full 8-way layout), TLFIX twin re-pinned 282/285. ENTRY 326: BATTERY RE-PIN TABLE -- era pins managers 1307P/0F, spec 406P/0F, server_args 496P/0F, sampling 252P/0F, distributed 213P/0F, mem_cache 2965P/24F (the ACCEPTED device-only 24-node list over 8 files), cargo 966P/0F/1-ignored, five registration/wheel gates rc=0 + the PATH-TO-BOOTABLE statement: CPU-proven vs GPU-window debt -- first-boot needles incl. the sampling.cuh-vs-0.7.2 JIT arbitration (if it JIT-fails, wheel-trio-then-pin per S-FI073, never a lone re-pin), autotune caches SHOULD hit warm (no retune owed on 0.7.2), entry-322 verify-at-boot quartet, and the still-riding 321-era watches (sgl-project#45472/sgl-project#46643 TTFT+starvation, first-delta tool-stream latency). ENTRY 327: WHEELS & VENV RECORD -- pairing matrix (FI 0.7.2 HELD, sgl-kernel 0.4.10/0.4.10 era-consistent with the RETIRED-rule cross-ref, torch/cutlass-dsl MATCH), blake3 1.0.11 INSTALLED with ZERO-DRIFT PROOF (0 battery lines vs ~380; matrix re-read unchanged; negative-shim prefix-sim causality) + no-pip/uv-note (operator lane), the two tree-side wheel reverts with reachability + residual-symbol evidence, check_dockerfile_wheel_pins re-green, FI-0.7.3 adoption contract restated with post-era facts (upstream code now expects 0.7.3 in one reported JIT path; tree HELD 0.7.2; first boot arbitrates), dist-name/module-name note sglang_kernel/sgl_kernel. ENTRY 328: INCIDENT RECORD 2026-11-10 ~16:4x-16:58, honest narrative -- re-plant waves landed into the EDITABLE-IMPORT working tree while :30000/:30001 ran; 9b died ~pre-16:50 (cause UNVERIFIABLE from the user acct, /root logs unreadable; worker-respawn-importing-changed-code is the LEADING MECHANISM, stated AS SUCH); the operator restart hit the init-order crash = the era's flagship find; recovery = merge6-era-backup @ b069e6a + detached checkout 2d87757, 9b re-booted WARM 16:58:52 all needles hit, 9d NEVER dropped (Nov 3, HTTP 200); NEW ERA LAW: working tree = prod import path, integration by branch ref-move ONLY; b2's emergency-freeze WIP 7ccf76a (author 16:55:02) kept in range and completed as C1 -- provenance note. ENTRY 329: BACKFILL AUDIT RECORD -- window 2026-08-10..2026-10-27, universe 2714 = 1830 in-tree / 255 already-documented / 628 NEW REJECTS / 1 NEW-INTERESTING; method one-liners; the qwen4-main-squashed side-branch class discovery (REJECT-MERGED-TO-SIDE-BRANCH x44 -- future sweeps MUST filter by base; orchestrator's "206" NOT FOUND in the artifacts, 44 recorded as verified); sgl-project#36043 NVFP4 skinny GEMM SM120 full paragraph (closed-unmerged, absent-tree-verified, W4A4 M<=16 = our exact decode cell, BENCHMARK-BEFORE-PORT -> S-36043-BENCH filed); artifact paths. EVALUATED: the 671-line DOCS-BLOCK.txt inserted VERBATIM as its own dated block (165 dropped list-commas tidied at insertion, noted once at the block head; nothing else reflowed). DATED CORRECTIONS where the waves superseded 322/323-era operational text: the header PROD-BOOT-FORBIDDEN bullet and entry 323's WHEEL DRIFT + STATE lines get [11-10 ERA-CLOSE CORRECTION] annotations (classes closed, wheel landed, ban now purely the operator gate); marker-zone math re-verified against the wave ledgers (121-at-merge vs 125 recount stands correct; planted == consumed per zone). WATCH LIST: era-close block appended (sgl-project#36043 bench-gated; sgl-project#42790/sgl-project#40916 arrived-via-merge6 noted; sgl-project#43703 revert-trap cross-ref from entry 323; sgl-project#45520 STILL-OPEN re-confirmed at era close -- zero in-window refs, b5 verified the fail-safe layer intact; bootfix's report-only get_tokenizer_worker_class/entrypoints wiring carry). Docs-only: NO .py touched; check_registered_tests rc=0 re-run against this tree. NOTHING PUSHED; the live tree was NOT touched by this wave (ref-move integration follows with HEAD/status equality proofs).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #48468 (bump to FlashInfer 0.7.3). Merge after it lands.
This PR needs flashinfer-ai/flashinfer#6360, which first shipped in FlashInfer 0.7.3. On 0.7.2 the BF16 tuner times each candidate kernel in isolation, so every small GEMM measures at about 10 µs and its low-M picks are close to a tie-break. #6360 makes the tuner rank tactics on a cold launch chain instead, which is what lets the autotuned picks replace the hand tables. On 0.7.2 this PR would delete the hand rows without a reliable replacement.
Motivation
Low-M BF16 decode GEMMs on Blackwell leave measurable performance on the table versus cuBLAS. So far that gap has been closed by hand-measured per-shape tables (
_BF16_SPLITK_TUNED_TACTICS,_BF16_WARP_SPLITK_TUNED_TACTICS), which need new rows for every model, weight shape and M. This PR lets FlashInfer pick the kernel per shape and M bucket instead, and removes the tables.FlashInfer's
mm_bf16(backend="auto")autotunes across its BF16 runners; withpdl=Truethose are the CuTe DSL low-M kernels (direct, warp split-K, cluster split-K), cuDNN, TGV and TinyGEMM. sglang's existing FlashInfer autotune pass drives the tuning, so no per-model or per-shape code is needed.Modifications
mm_bf16(pdl=True, backend="auto"), in both_bf16_gemm_dispatch_implandapply_into.--disable-flashinfer-autotuneturns it off.should_run_flashinfer_autotunenow also fires when this path is active.initialize_bf16_gemm_config._MUSE_SPARK_TGV_M_RANGES) and the Kimi-K3 tables are untouched; above M = 32, and for K3's directuse_cutedsl_bf16_gemmcall, they still apply.Accuracy Tests
Qwen3-VL-4B-Instruct, TP1, 1x B300, FlashInfer 0.7.3. GSM8K via
sgl-eval run gsm8k(0.1.3), all 1319 questions, 16 threads, temperature 0, top_p 0.95,--max-tokens 4096:--disable-flashinfer-autotuneOutputs are not bitwise identical to cuBLAS (different accumulation order). Truncations are temperature-0 repetition loops in both arms.
Real dispatch on B300: M in {1, 3, 8, 16, 24, 32} takes the autotuned kernel for the plain, biased,
apply_intoand addend paths, while M in {33, 64} does not; all outputs are within 1e-2 relative error of an FP32 reference.Speed Tests and Profiling
Same model, GPU and FlashInfer 0.7.3.
bench_one_batch_server, input 1024 / output 512, steady-state output tok/s, mean of two runs per arm in ON/OFF/OFF/ON order (the two runs of each arm agree within 0.6%, 2.0% for ON at bs 16):Startup: the first boot tunes the 4 decoder shapes; scheduler startup is 195 s, vs 16 s with autotune off. Cached boots run no tuning: 22 s vs 16 s.
Kernel level (B300 sm_103, same SM as GB300; FlashInfer 0.7.3; cold L2: each timed CUDA graph rotates through ≥1 GiB of distinct weights; PDL on). 32 weight shapes (every hand-table shape, the MuseSpark/DFLASH TGV shapes, Qwen3-VL-4B) at M = 1, 4, 8, 12, 16, 24, 32: 224 cells, each run through main's real dispatch (hand row, TGV or cuBLAS), against this PR's autotuned pick (tuned once per shape at M = 32, as the server does) and cuBLAS:
This PR vs cuBLAS: 1.39x geomean, never more than 3% slower. It beats main by more than 5% in 119 cells. The 4 cells where it is more than 5% slower:
Per-shape tuning takes 6–64 s (median 31 s).
Known limitations
autoabove that was not measured against the existing TGV/cuBLAS dispatch.--disable-flashinfer-autotune, the removed hand rows no longer apply, and those shapes use TGV/cuBLAS.SGLANG_CACHE_DIRpersists.Checklist
test_unquant_apply_with_addend.pypasses on B300.apply_into, addend) match an FP32 reference.🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ✅ Run #42311285608
Latest PR Test (Extra): ❌ Run #42311285675
Latest PR Test (AMD ROCm 10): ⏳ Run #42311285570
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: