Repository navigation
Conversation
ormandj
requested review from
BBuf,
DarkSharpness,
Edwardf0t1,
Fridge003,
HaiShaw,
HydraQYH,
Ying1123,
celve,
ch-wan,
ispobock,
merrymercy and
yuan-luo
as code owners
October 7, 2026 03:15
ormandj
force-pushed
the
bf16-skinny-lm-head
branch
from
October 7, 2026 05:43
6eb1220 to
ca8c338
Compare
17 of 56 tasks
ormandj
force-pushed
the
bf16-skinny-lm-head
branch
from
October 19, 2026 17:33
ca8c338 to
fc727bb
Compare
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 20, 2026
…/deferred/avoid map - 3A [24-35]: NVFP4 MoE boot sgl-project#39807/sgl-project#40173, block-FP8 TP refine sgl-project#42566, k-pool indexer split sgl-project#43619, SM120 skinny GEMM sgl-project#42865/sgl-project#42867, DeepGEMM warmup guards sgl-project#32686/sgl-project#43712, autotune sgl-project#43363/sgl-project#40420, mHC overlap sgl-project#42985, FP8-einsum precompile sgl-project#42821. - 3B [36-43]: PCIe CustomAllreduce VMM sgl-project#44448, EP local expert ids sgl-project#44480, shm TP broadcast sgl-project#44202, PCIe-IPC all-gather sgl-project#42863 + multicast guard sgl-project#41888, TP-sampling consistency sgl-project#41916/sgl-project#38565, draft embed/lm_head VRAM sgl-project#37155. - sgl-project#39807 toolchain commit verified safe against our CUDA_HOME=12.9 pin (cu13 fallback needs CUDA_HOME unset AND glibc>=2.41; we are 2.39). - stack_want.txt: execution-status banner. Branch tip 5dd6ec7.
With SGLANG_ENABLE_BF16_SKINNY_LM_HEAD=1, an unquantized BF16 LM head (target logits and the DSpark draft projection) runs batches of up to 16 rows through a Triton kernel in which each CTA streams BN weight columns over all of K. A row's accumulation order depends only on (N, K), so its logits are identical alone or in any batch.
The kernel now launches under torch.cuda.device(x.device), so operands on a non-current GPU run on their own device and stream, and the predicate rejects ROCm builds, where the kernel is not validated. Kernel tests move to the base-b-kernel-unit stage and add a 64-bit row offset case (a strided weight whose last row starts past 2**31 elements, gated on free memory), a two-GPU non-current-device case, and the ROCm rejection. The LM-head dispatch tests become CPU unit tests under test/registered/unit/layers with the kernel and predicate stubbed.
…nsupported operands with ValueError
ormandj
force-pushed
the
bf16-skinny-lm-head
branch
from
October 29, 2026 22:19
fc727bb to
0d74e2e
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
At decode and speculative-verify batch sizes, an unquantized BF16 LM head is a bandwidth-bound GEMM: the whole vocab shard is read for a handful of rows. cuBLAS can pick a different kernel as the row count changes, so a row's logits can depend on how many other rows share the batch.
Modifications
sglang/kernels/ops/gemm/bf16_skinny_gemm.py: a Triton kernel forx[M, K] @ weight[N, K].Twith 1 <= M <= 16, BF16 inputs, K % 128 == 0, unit stride along K and arbitrary row strides, both tensors on the same CUDA GPU. ROCm builds are not accepted, as the kernel has not been validated there.torch.cuda.device(x.device), so operands on a GPU other than the current one run on that GPU's current stream.gemmkernel inventory (gemm.bf16_skinny_gemm, CUDA only). Unsupported operands raiseValueError.bf16_lm_head_matmul()inlogits_processor.pyuses the kernel whenSGLANG_ENABLE_BF16_SKINNY_LM_HEAD=1(default off) and the inputs qualify. Otherwise it uses the existingtorch.matmul.LogitsProcessor._compute_lm_head(outside the AMX, FP32 and RL on-policy branches) andproject_through_lm_head(the DSpark draft logits).Accuracy Tests
test/registered/kernels/ops/gemm/test_bf16_skinny_gemm.py(CUDA, registered in thebase-b-kernel-unitstage on the 1-gpu-small runner):gamma_K * (|x| @ |w|.T)plus one BF16 ulp. Tested at M = 1, 6 and 16 on the TP4 DeepSeek-V4.1 shard (32320 x 4096) and on a 1000 x 256 shape.test_bf16_skinny_gemm_multi_gpu.py, registered on the 2-GPU runner). Operands oncuda:1called whilecuda:0is current (and busy) give a result oncuda:1within the error bound.ValueErroron them.test/registered/unit/layers/test_logits_processor_bf16_lm_head.py(CPU,base-a-test-cpu): both production callers (the default branch ofLogitsProcessor._compute_lm_head, andproject_through_lm_head) andbf16_lm_head_matmulcall the kernel, with the hidden states cast to BF16, only when the flag is set andbf16_skinny_supportedaccepts the operands; otherwise they returntorch.matmul's result. The kernel and the predicate are stubbed; a separate case checks that the real predicate rejects CPU operands.test/registered/unit/kernels/test_kernels_namespace.pychecks the registration's target and that it is CUDA only.On 4x RTX PRO 6000 Blackwell (torch 2.14.1+cu130, Triton 3.8.0),
test_bf16_skinny_gemm.py,test_bf16_skinny_gemm_multi_gpu.pyand the CPU dispatch test pass with no skips (10 tests, 53 subtests).Speed Tests and Profiling
RTX PRO 6000 Blackwell Max-Q (SM120), torch 2.13.0+cu130, triton 3.7.1. DeepSeek-V4.1-Flash LM head shard at TP4 (N=32320, K=4096), BF16. Script:
bench_bf16_skinny_lm_head.py(below), run asCUDA_VISIBLE_DEVICES=1 python bench_bf16_skinny_lm_head.py.In two earlier runs of the same script, cells differed by up to 7%, and the kernel was faster than cuBLAS at every M each time. Both read the full 265 MB shard. The kernel's time stays flat in M, while cuBLAS's grows above M = 6.
Serving A/B with the flag on (
SGLANG_ENABLE_BF16_SKINNY_LM_HEAD=1) vs off. DeepSeek-V4.1-Flash on 4x RTX PRO 6000 Blackwell Max-Q (PCIe, 250 W power limit), TP4, DSpark speculative decoding, temperature 1, top_p 0.95. All runs use the same server build and configuration, which also carries changes that are not part of this PR; only this flag differs between runs. The flag-on configuration was measured before and after the flag-off run.Metric: steps/s is computed from SGLang's TP0
Decode batchlog lines asgen throughput (token/s) / accept len, grouped by the logged running batch size, and taken as the median over a measurement window of about 30 s after warmup. It counts sequence decode steps per second (batch decode steps per second multiplied by the batch size), so it does not depend on the acceptance length. Each configuration ran 3 repetitions; each cell gives the median of the 3 repetitions followed by the 3 values.The target LM head sees M = batch x verify width rows, with a verify width of up to 6 tokens per sequence, and the DSpark draft projection sees batch x draft block rows. The kernel serves only calls with at most 16 rows: every target verify call at batch 1-2, and at batch 4 and above only calls with at most 4, 2 or 1 tokens per sequence (batch 4, 8, 16).
At batch 8, every flag-off repetition is below every flag-on repetition and the flag-off median is 3.7% lower. At batch 1, 2, 4 and 16 the flag-off median is 0.8-1.8% lower, but the flag-off repetitions overlap the flag-on ones (and the two flag-on medians differ by up to 3.1% at batch 4), so this sample does not resolve a difference at those batch sizes. The flag's purpose is batch-invariant LM-head rows at decode sizes; the measured serving speed difference is small.
bench_bf16_skinny_lm_head.py
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciDeveloped with AI assistance.
CI States
Latest PR Test (Base): ❌ Run #40964972249
Latest PR Test (Extra): ❌ Run #40964971988
Latest PR Test (AMD ROCm 10): ❌ Run #40964972158