Skip to content

[Perf] lm-head: Add an opt-in batch-invariant BF16 GEMM for decode-sized batches - #42867

Open
ormandj wants to merge 6 commits into
sgl-project:mainfrom
ormandj:bf16-skinny-lm-head
Open

ormandj wants to merge 6 commits into
sgl-project:mainfrom
ormandj:bf16-skinny-lm-head

Conversation

@ormandj

@ormandj ormandj commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

At decode and speculative-verify batch sizes, an unquantized BF16 LM head is a bandwidth-bound GEMM: the whole vocab shard is read for a handful of rows. cuBLAS can pick a different kernel as the row count changes, so a row's logits can depend on how many other rows share the batch.

Modifications

  • New sglang/kernels/ops/gemm/bf16_skinny_gemm.py: a Triton kernel for x[M, K] @ weight[N, K].T with 1 <= M <= 16, BF16 inputs, K % 128 == 0, unit stride along K and arbitrary row strides, both tensors on the same CUDA GPU. ROCm builds are not accepted, as the kernel has not been validated there.
    • Each CTA streams 64 weight rows over all of K, with no split-K.
    • A row's tiling and FP32 accumulation order depend only on (N, K), so its result is the same alone or in any batch of up to 16.
    • Row offsets are 64-bit, so a weight whose rows start beyond 2^31 elements (a large head, or a large row stride) is addressed correctly.
    • The launch runs under torch.cuda.device(x.device), so operands on a GPU other than the current one run on that GPU's current stream.
  • The kernel is registered in the gemm kernel inventory (gemm.bf16_skinny_gemm, CUDA only). Unsupported operands raise ValueError.
  • bf16_lm_head_matmul() in logits_processor.py uses the kernel when SGLANG_ENABLE_BF16_SKINNY_LM_HEAD=1 (default off) and the inputs qualify. Otherwise it uses the existing torch.matmul.
    • It serves the unquantized BF16 LM head in LogitsProcessor._compute_lm_head (outside the AMX, FP32 and RL on-policy branches) and project_through_lm_head (the DSpark draft logits).

Accuracy Tests

test/registered/kernels/ops/gemm/test_bf16_skinny_gemm.py (CUDA, registered in the base-b-kernel-unit stage on the 1-gpu-small runner):

  • Error bound. Every output is within the error bound of an FP32-accumulated dot product rounded once to BF16, measured against FP64: gamma_K * (|x| @ |w|.T) plus one BF16 ulp. Tested at M = 1, 6 and 16 on the TP4 DeepSeek-V4.1 shard (32320 x 4096) and on a 1000 x 256 shape.
  • Batch invariance. Every row is bit-identical (compared as int16) alone and in batches of 2..16.
  • CUDA graphs. A captured call replays on new activations with the eager result.
  • Strided rows. Operands with padded row strides give the same bits as compact ones.
  • 64-bit row offsets. A strided weight whose last row starts past 2^31 elements gives the same bits as a compact copy. The test needs about 5 GiB of free GPU memory and is skipped otherwise.
  • Non-current device (test_bf16_skinny_gemm_multi_gpu.py, registered on the 2-GPU runner). Operands on cuda:1 called while cuda:0 is current (and busy) give a result on cuda:1 within the error bound.
  • Unsupported inputs are rejected: more than 16 rows, FP16, K % 128 != 0, strided columns of either operand, a weight on the CPU, and a ROCm build. The kernel raises ValueError on them.

test/registered/unit/layers/test_logits_processor_bf16_lm_head.py (CPU, base-a-test-cpu): both production callers (the default branch of LogitsProcessor._compute_lm_head, and project_through_lm_head) and bf16_lm_head_matmul call the kernel, with the hidden states cast to BF16, only when the flag is set and bf16_skinny_supported accepts the operands; otherwise they return torch.matmul's result. The kernel and the predicate are stubbed; a separate case checks that the real predicate rejects CPU operands. test/registered/unit/kernels/test_kernels_namespace.py checks the registration's target and that it is CUDA only.

On 4x RTX PRO 6000 Blackwell (torch 2.14.1+cu130, Triton 3.8.0), test_bf16_skinny_gemm.py, test_bf16_skinny_gemm_multi_gpu.py and the CPU dispatch test pass with no skips (10 tests, 53 subtests).

Speed Tests and Profiling

RTX PRO 6000 Blackwell Max-Q (SM120), torch 2.13.0+cu130, triton 3.7.1. DeepSeek-V4.1-Flash LM head shard at TP4 (N=32320, K=4096), BF16. Script: bench_bf16_skinny_lm_head.py (below), run as CUDA_VISIBLE_DEVICES=1 python bench_bf16_skinny_lm_head.py.

  • Each call is captured 20 times in one CUDA graph.
  • 10 timed replays.
  • Mean time per call.
M cuBLAS this kernel
1 172.2 us 162.6 us
2 165.1 us 162.8 us
4 165.4 us 163.1 us
6 165.4 us 163.4 us
8 179.9 us 165.9 us
12 186.7 us 167.9 us
16 191.8 us 164.4 us

In two earlier runs of the same script, cells differed by up to 7%, and the kernel was faster than cuBLAS at every M each time. Both read the full 265 MB shard. The kernel's time stays flat in M, while cuBLAS's grows above M = 6.

Serving A/B with the flag on (SGLANG_ENABLE_BF16_SKINNY_LM_HEAD=1) vs off. DeepSeek-V4.1-Flash on 4x RTX PRO 6000 Blackwell Max-Q (PCIe, 250 W power limit), TP4, DSpark speculative decoding, temperature 1, top_p 0.95. All runs use the same server build and configuration, which also carries changes that are not part of this PR; only this flag differs between runs. The flag-on configuration was measured before and after the flag-off run.

Metric: steps/s is computed from SGLang's TP0 Decode batch log lines as gen throughput (token/s) / accept len, grouped by the logged running batch size, and taken as the median over a measurement window of about 30 s after warmup. It counts sequence decode steps per second (batch decode steps per second multiplied by the batch size), so it does not depend on the acceptance length. Each configuration ran 3 repetitions; each cell gives the median of the 3 repetitions followed by the 3 values.

The target LM head sees M = batch x verify width rows, with a verify width of up to 6 tokens per sequence, and the DSpark draft projection sees batch x draft block rows. The kernel serves only calls with at most 16 rows: every target verify call at batch 1-2, and at batch 4 and above only calls with at most 4, 2 or 1 tokens per sequence (batch 4, 8, 16).

decode batch on, before off on, after off vs mean of on medians
1 90.7 [92.5, 89.3, 90.7] 89.7 [89.3, 90.5, 89.7] 90.7 [89.3, 90.7, 91.2] -1.1%
2 146.9 [152.0, 146.7, 146.9] 144.5 [148.1, 144.5, 142.7] 147.4 [148.6, 147.4, 144.2] -1.8%
4 212.7 [216.8, 212.7, 211.4] 212.9 [215.8, 209.1, 212.9] 219.4 [225.8, 219.4, 218.4] -1.5%
8 339.1 [346.8, 339.1, 329.9] 326.7 [328.1, 326.7, 317.9] 339.3 [341.8, 339.3, 329.4] -3.7%
16 479.4 [479.4, 474.8, 479.4] 475.1 [475.1, 476.7, 464.9] 478.0 [478.0, 480.8, 471.6] -0.8%

At batch 8, every flag-off repetition is below every flag-on repetition and the flag-off median is 3.7% lower. At batch 1, 2, 4 and 16 the flag-off median is 0.8-1.8% lower, but the flag-off repetitions overlap the flag-on ones (and the two flag-on medians differ by up to 3.1% at batch 4), so this sample does not resolve a difference at those batch sizes. The flag's purpose is batch-invariant LM-head rows at decode sizes; the measured serving speed difference is small.

bench_bf16_skinny_lm_head.py
"""Per-call time of the BF16 skinny LM-head GEMM against cuBLAS (x @ w.T).

Each call is captured 20 times in one CUDA graph; the graph is warmed once and replayed
10 times; the reported value is the mean time per call. Shape: the DeepSeek-V4.1-Flash
LM head shard at TP4 (N=32320, K=4096), BF16.

usage: python bench_bf16_skinny_lm_head.py   (with the change on PYTHONPATH)
"""

import torch
import triton

from sglang.kernels.ops.gemm.bf16_skinny_gemm import bf16_skinny_gemm

CALLS, REPLAYS = 20, 10


def per_call_us(fn):
    stream = torch.cuda.Stream()
    stream.wait_stream(torch.cuda.current_stream())
    with torch.cuda.stream(stream):
        fn()
    torch.cuda.current_stream().wait_stream(stream)
    graph = torch.cuda.CUDAGraph()
    with torch.cuda.graph(graph):
        for _ in range(CALLS):
            fn()
    graph.replay()
    torch.cuda.synchronize()
    start, end = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
    start.record()
    for _ in range(REPLAYS):
        graph.replay()
    end.record()
    torch.cuda.synchronize()
    return start.elapsed_time(end) / (CALLS * REPLAYS) * 1000


def main():
    n, k = 32320, 4096
    print(f"{torch.cuda.get_device_name()}, torch {torch.__version__}, triton {triton.__version__}")
    w = (torch.randn(n, k, device="cuda") / k**0.5).bfloat16()
    for m in (1, 2, 4, 6, 8, 12, 16):
        x = torch.randn(m, k, device="cuda").bfloat16()
        cublas = per_call_us(lambda: x @ w.T)
        skinny = per_call_us(lambda: bf16_skinny_gemm(x, w))
        print(f"M={m:2d}: cuBLAS {cublas:6.1f} us, skinny {skinny:6.1f} us")


if __name__ == "__main__":
    main()

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

Developed with AI assistance.


CI States

Latest PR Test (Base): ❌ Run #40964972249
Latest PR Test (Extra): ❌ Run #40964971988
Latest PR Test (AMD ROCm 10): ❌ Run #40964972158

@ormandj
ormandj force-pushed the bf16-skinny-lm-head branch from 6eb1220 to ca8c338 Compare October 7, 2026 05:43
@ormandj ormandj mentioned this pull request Oct 7, 2026
17 of 56 tasks
@ormandj ormandj changed the title [Perf] Opt-in batch-invariant BF16 LM head GEMM for decode-sized batches [Perf] lm-head: Add an opt-in batch-invariant BF16 GEMM for decode-sized batches Oct 19, 2026
@ormandj
ormandj force-pushed the bf16-skinny-lm-head branch from ca8c338 to fc727bb Compare October 19, 2026 17:33
detain added a commit to detain/sglang that referenced this pull request Oct 20, 2026
…/deferred/avoid map

- 3A [24-35]: NVFP4 MoE boot sgl-project#39807/sgl-project#40173, block-FP8 TP refine sgl-project#42566, k-pool indexer split sgl-project#43619,
  SM120 skinny GEMM sgl-project#42865/sgl-project#42867, DeepGEMM warmup guards sgl-project#32686/sgl-project#43712, autotune sgl-project#43363/sgl-project#40420,
  mHC overlap sgl-project#42985, FP8-einsum precompile sgl-project#42821.
- 3B [36-43]: PCIe CustomAllreduce VMM sgl-project#44448, EP local expert ids sgl-project#44480, shm TP broadcast sgl-project#44202,
  PCIe-IPC all-gather sgl-project#42863 + multicast guard sgl-project#41888, TP-sampling consistency sgl-project#41916/sgl-project#38565,
  draft embed/lm_head VRAM sgl-project#37155.
- sgl-project#39807 toolchain commit verified safe against our CUDA_HOME=12.9 pin (cu13 fallback needs CUDA_HOME unset AND glibc>=2.41; we are 2.39).
- stack_want.txt: execution-status banner. Branch tip 5dd6ec7.
With SGLANG_ENABLE_BF16_SKINNY_LM_HEAD=1, an unquantized BF16 LM head (target
logits and the DSpark draft projection) runs batches of up to 16 rows
through a Triton kernel in which each CTA streams BN weight columns over
all of K. A row's accumulation order depends only on (N, K), so its
logits are identical alone or in any batch.
The kernel now launches under torch.cuda.device(x.device), so operands on
a non-current GPU run on their own device and stream, and the predicate
rejects ROCm builds, where the kernel is not validated.

Kernel tests move to the base-b-kernel-unit stage and add a 64-bit row
offset case (a strided weight whose last row starts past 2**31 elements,
gated on free memory), a two-GPU non-current-device case, and the ROCm
rejection. The LM-head dispatch tests become CPU unit tests under
test/registered/unit/layers with the kernel and predicate stubbed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant