Repository navigation
feat(lm_head):Triton skinny GEMV for the bf16 lm_head on gfx942 - #46
Merged
sammysun0711 merged 3 commits intoOct 26, 2026
Conversation
The lm_head GEMM is a pure weight-streaming workload at decode (M = batch size, N = vocab shard). On MI308X hipBLASLt picks a generic Tensile tile that reaches ~2.2 TB/s; for the Qwen3.8-Flash-Next TP2 shard [124160, 2560] this costs ~250 us per step. Add a Triton skinny GEMM (python/sglang/kernels/ops/gemm/rocm_bf16_gemv.py) with three fixed, capture-safe configs: - M == 1: row-reduction GEMV on the vector ALUs - 2 <= M <= 16: 16-row MFMA tile, kpack=2, waves_per_eu=2 - 17 <= M <= 32: 32-row MFMA tile with a narrower N tile M > 32 and non-bf16 / unaligned-K inputs keep using torch.matmul. LogitsProcessor routes the plain bf16 lm_head matmul through it on gfx942; SGLANG_ENABLE_ROCM_LM_HEAD_GEMV=0 opts out. MI308X, K=2560, us (triton / hipBLASLt): N=124160: M=1 199/269, M=8 230/286, M=32 278/320 N=31040: M=1 37/52, M=8 49/53, M=32 60/65 End-to-end TP2 (pyhip config, 12k in / 5 out, conc 1): lm_head kernel 250.7 -> 179.3 us, TPOT 8.49 -> 8.38 ms, GSM8K (chat, 200, greedy) 0.975 -> 0.980.
yicyang
force-pushed
the
qwen38-lmhead-triton-gemv-rocm
branch
from
October 14, 2026 02:06
f8c04b3 to
962bf72
Compare
Replace the hard-coded launch configs with a per-(N, K) table in
rocm_bf16_gemv.py (_GFX942_CONFIGS, M upper-bound buckets) and add
cache_modifier on the weight load as a tunable.
Only shapes in the table use the Triton kernels; everything else keeps
hipBLASLt. The previous generic configs were slower than hipBLASLt on
untuned shapes for some M (e.g. N <= 24576 at K=2560, or K=5120), and no
single N threshold separated wins from losses across K.
Tuned on MI308X for the Qwen3.8-Flash-Next lm_head shards
(N in {248320, 124160, 62080, 31040}, K=2560) with
benchmark/kernels/rocm_bf16_gemv/tune_rocm_bf16_gemv.py. Tuned vs previous
configs: 1.02-1.14x; vs hipBLASLt: 1.11-1.49x. ".cg" and narrower
1-warp MFMA tiles (BLOCK_N=64) account for most of the gain. Split-K was
evaluated and dropped: no shape improved beyond noise.
TP2 end to end: lm_head 249 -> 165 us (previous configs: 173.5 us).
…-tune Replace the hand-rolled CUDA-event loop in tune_rocm_bf16_gemv.py with triton.testing.do_bench(warmup=25, rep=100), the setting aiter's Triton op_benchmarks use (e.g. bench_gemm_a8w8_per_token_scale.py). Candidates, hipBLASLt and the shipped config are all timed the same way, with L2 flushed between reps. Re-tuned all four Qwen3.8-Flash-Next lm_head shards on MI308X. The GPUs were shared with other jobs, so every bucket where the tuner proposed a different config was re-checked with an interleaved A/B (20 rounds of do_bench, min and median); identical configs varied by up to ~4.6%, so a config is replaced only when it wins by >5% on both: - N=31040, M<=2..32: BLOCK_K 128 -> 256 (BLOCK_K 128 kept for M<=24) and ".cg"; 1.11-1.49x vs the previous entries. The previous M<=32 entry was no faster than hipBLASLt. - N=124160, M<=2: 1-warp BLOCK_N=64 tile, 1.09x. - N=62080, M<=4: ".cg" on the weight load, 1.12x. Every shipped entry beats hipBLASLt by 1.21-1.83x (do_bench min).
yicyang
marked this pull request as ready for review
October 19, 2026 05:18
apinge
self-requested a review
October 22, 2026 08:20
apinge
reviewed
Oct 26, 2026
apinge
approved these changes
Oct 26, 2026
sammysun0711
approved these changes
Oct 26, 2026
sammysun0711
merged commit Oct 26, 2026
c996fcb
into
apinge:qwen38-flash-next-amd-current-fix-install
1 check failed
This was referenced Nov 10, 2026
sammysun0711
added a commit
that referenced
this pull request
Nov 11, 2026
…#101) * [AMD] Triton skinny GEMV for the bf16 lm_head on gfx942 The lm_head GEMM is a pure weight-streaming workload at decode (M = batch size, N = vocab shard). On MI308X hipBLASLt picks a generic Tensile tile that reaches ~2.2 TB/s; for the Qwen3.8-Flash-Next TP2 shard [124160, 2560] this costs ~250 us per step. Add a Triton skinny GEMM (python/sglang/kernels/ops/gemm/rocm_bf16_gemv.py) with three fixed, capture-safe configs: - M == 1: row-reduction GEMV on the vector ALUs - 2 <= M <= 16: 16-row MFMA tile, kpack=2, waves_per_eu=2 - 17 <= M <= 32: 32-row MFMA tile with a narrower N tile M > 32 and non-bf16 / unaligned-K inputs keep using torch.matmul. LogitsProcessor routes the plain bf16 lm_head matmul through it on gfx942; SGLANG_ENABLE_ROCM_LM_HEAD_GEMV=0 opts out. MI308X, K=2560, us (triton / hipBLASLt): N=124160: M=1 199/269, M=8 230/286, M=32 278/320 N=31040: M=1 37/52, M=8 49/53, M=32 60/65 End-to-end TP2 (pyhip config, 12k in / 5 out, conc 1): lm_head kernel 250.7 -> 179.3 us, TPOT 8.49 -> 8.38 ms, GSM8K (chat, 200, greedy) 0.975 -> 0.980. * [AMD] Tune the gfx942 lm_head GEMV and route only tuned shapes Replace the hard-coded launch configs with a per-(N, K) table in rocm_bf16_gemv.py (_GFX942_CONFIGS, M upper-bound buckets) and add cache_modifier on the weight load as a tunable. Only shapes in the table use the Triton kernels; everything else keeps hipBLASLt. The previous generic configs were slower than hipBLASLt on untuned shapes for some M (e.g. N <= 24576 at K=2560, or K=5120), and no single N threshold separated wins from losses across K. Tuned on MI308X for the Qwen3.8-Flash-Next lm_head shards (N in {248320, 124160, 62080, 31040}, K=2560) with benchmark/kernels/rocm_bf16_gemv/tune_rocm_bf16_gemv.py. Tuned vs previous configs: 1.02-1.14x; vs hipBLASLt: 1.11-1.49x. ".cg" and narrower 1-warp MFMA tiles (BLOCK_N=64) account for most of the gain. Split-K was evaluated and dropped: no shape improved beyond noise. TP2 end to end: lm_head 249 -> 165 us (previous configs: 173.5 us). * [AMD] Time the lm_head GEMV tuner with triton.testing.do_bench and re-tune Replace the hand-rolled CUDA-event loop in tune_rocm_bf16_gemv.py with triton.testing.do_bench(warmup=25, rep=100), the setting aiter's Triton op_benchmarks use (e.g. bench_gemm_a8w8_per_token_scale.py). Candidates, hipBLASLt and the shipped config are all timed the same way, with L2 flushed between reps. Re-tuned all four Qwen3.8-Flash-Next lm_head shards on MI308X. The GPUs were shared with other jobs, so every bucket where the tuner proposed a different config was re-checked with an interleaved A/B (20 rounds of do_bench, min and median); identical configs varied by up to ~4.6%, so a config is replaced only when it wins by >5% on both: - N=31040, M<=2..32: BLOCK_K 128 -> 256 (BLOCK_K 128 kept for M<=24) and ".cg"; 1.11-1.49x vs the previous entries. The previous M<=32 entry was no faster than hipBLASLt. - N=124160, M<=2: 1-warp BLOCK_N=64 tile, 1.09x. - N=62080, M<=4: ".cg" on the weight load, 1.12x. Every shipped entry beats hipBLASLt by 1.21-1.83x (do_bench min). --------- (cherry picked from commit c996fcb) Co-authored-by: yicyang <yicyang@users.noreply.github.com> Co-authored-by: Xiake Sun <xisun@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The lm_head of Qwen3.8-Flash-Next is an unquantized bf16 GEMM. On MI308X with TP2 it computes
[M, 2560] x [2560, 124160]each decode step.Technical details
New file
python/sglang/kernels/ops/gemm/rocm_bf16_gemv.pyuses fixed configs chosen by M:torch.matmul.New environment variable
SGLANG_ENABLE_ROCM_LM_HEAD_GEMVis added to trigger Triton skinny GEMV for M<=32 bf16 lm_head on gfx942.Performance
MI308X, K=2560, in us (Triton / hipBLASLt):
Triton is 1.20-1.78x faster than hipBLASLt and 1.04-1.49x faster than the first, untuned version of this PR.
End to end on TP2 (12k input, 5 output tokens, concurrency 1):
Prefill trace(before):
Prefill trace(after):
Decode trace(before):
Decode trace(after):
Test plan
test/registered/kernels/ops/gemm/test_rocm_bf16_gemv.pycovers the M bucket edges, N tail tiles, strided rows, predicate negatives, CUDA graph replay, and lm_head routing with the env var opt-out.test/registered/kernels/benchmark/gemm/bench_rocm_bf16_gemv.py.test/registered/rl/test_fp32_lm_head.py, plus GSM8K accuracy with the kernel on and off.Test results
test_fp32_lm_head.py: 5 passed.