Skip to content

feat(lm_head):Triton skinny GEMV for the bf16 lm_head on gfx942 - #46

Merged
sammysun0711 merged 3 commits into
apinge:qwen38-flash-next-amd-current-fix-installfrom
yicyang:qwen38-lmhead-triton-gemv-rocm
Oct 26, 2026
Merged

sammysun0711 merged 3 commits into
apinge:qwen38-flash-next-amd-current-fix-installfrom
yicyang:qwen38-lmhead-triton-gemv-rocm

Conversation

@yicyang

@yicyang yicyang commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

The lm_head of Qwen3.8-Flash-Next is an unquantized bf16 GEMM. On MI308X with TP2 it computes [M, 2560] x [2560, 124160] each decode step.

Technical details

New file python/sglang/kernels/ops/gemm/rocm_bf16_gemv.py uses fixed configs chosen by M:

  • M=1: vector-ALU row-reduction GEMV.
  • M=2-32: MFMA tile.
  • M>32: falls back to torch.matmul.

New environment variable SGLANG_ENABLE_ROCM_LM_HEAD_GEMV is added to trigger Triton skinny GEMV for M<=32 bf16 lm_head on gfx942.

Performance

MI308X, K=2560, in us (Triton / hipBLASLt):

N (TP shard) M=1 M=8 M=16 M=32
248320 (TP1) 360 / 540 402 / 578 436 / 589 537 / 644
124160 (TP2) 172 / 278 194 / 295 211 / 303 269 / 332
62080 (TP4) 86 / 154 126 / 163 129 / 167 136 / 185
31040 (TP8) 65 / 88 59 / 93 57 / 96 76 / 108

Triton is 1.20-1.78x faster than hipBLASLt and 1.04-1.49x faster than the first, untuned version of this PR.

End to end on TP2 (12k input, 5 output tokens, concurrency 1):

  • lm_head: 249 -> 165 us
  • TPOT: 8.51 -> 8.26 ms
  • TTFT: 794 -> 785 ms

Prefill trace(before):

image

Prefill trace(after):

image

Decode trace(before):

image

Decode trace(after):

image

Test plan

  • Unit test: test/registered/kernels/ops/gemm/test_rocm_bf16_gemv.py covers the M bucket edges, N tail tiles, strided rows, predicate negatives, CUDA graph replay, and lm_head routing with the env var opt-out.
  • Benchmark: test/registered/kernels/benchmark/gemm/bench_rocm_bf16_gemv.py.
  • Regression: existing test/registered/rl/test_fp32_lm_head.py, plus GSM8K accuracy with the kernel on and off.

Test results

  • New unit tests: 37 passed. test_fp32_lm_head.py: 5 passed.
  • GSM8K (chat, 200 examples, greedy): 0.980 with the kernel vs 0.975 without.

The lm_head GEMM is a pure weight-streaming workload at decode
(M = batch size, N = vocab shard). On MI308X hipBLASLt picks a generic
Tensile tile that reaches ~2.2 TB/s; for the Qwen3.8-Flash-Next TP2
shard [124160, 2560] this costs ~250 us per step.

Add a Triton skinny GEMM (python/sglang/kernels/ops/gemm/rocm_bf16_gemv.py)
with three fixed, capture-safe configs:
  - M == 1: row-reduction GEMV on the vector ALUs
  - 2 <= M <= 16: 16-row MFMA tile, kpack=2, waves_per_eu=2
  - 17 <= M <= 32: 32-row MFMA tile with a narrower N tile
M > 32 and non-bf16 / unaligned-K inputs keep using torch.matmul.

LogitsProcessor routes the plain bf16 lm_head matmul through it on gfx942;
SGLANG_ENABLE_ROCM_LM_HEAD_GEMV=0 opts out.

MI308X, K=2560, us (triton / hipBLASLt):
  N=124160: M=1 199/269, M=8 230/286, M=32 278/320
  N=31040:  M=1 37/52,   M=8 49/53,   M=32 60/65
End-to-end TP2 (pyhip config, 12k in / 5 out, conc 1):
  lm_head kernel 250.7 -> 179.3 us, TPOT 8.49 -> 8.38 ms,
  GSM8K (chat, 200, greedy) 0.975 -> 0.980.
@yicyang
yicyang force-pushed the qwen38-lmhead-triton-gemv-rocm branch from f8c04b3 to 962bf72 Compare October 14, 2026 02:06
Replace the hard-coded launch configs with a per-(N, K) table in
rocm_bf16_gemv.py (_GFX942_CONFIGS, M upper-bound buckets) and add
cache_modifier on the weight load as a tunable.

Only shapes in the table use the Triton kernels; everything else keeps
hipBLASLt. The previous generic configs were slower than hipBLASLt on
untuned shapes for some M (e.g. N <= 24576 at K=2560, or K=5120), and no
single N threshold separated wins from losses across K.

Tuned on MI308X for the Qwen3.8-Flash-Next lm_head shards
(N in {248320, 124160, 62080, 31040}, K=2560) with
benchmark/kernels/rocm_bf16_gemv/tune_rocm_bf16_gemv.py. Tuned vs previous
configs: 1.02-1.14x; vs hipBLASLt: 1.11-1.49x. ".cg" and narrower
1-warp MFMA tiles (BLOCK_N=64) account for most of the gain. Split-K was
evaluated and dropped: no shape improved beyond noise.

TP2 end to end: lm_head 249 -> 165 us (previous configs: 173.5 us).
…-tune

Replace the hand-rolled CUDA-event loop in tune_rocm_bf16_gemv.py with
triton.testing.do_bench(warmup=25, rep=100), the setting aiter's Triton
op_benchmarks use (e.g. bench_gemm_a8w8_per_token_scale.py). Candidates,
hipBLASLt and the shipped config are all timed the same way, with L2
flushed between reps.

Re-tuned all four Qwen3.8-Flash-Next lm_head shards on MI308X. The GPUs
were shared with other jobs, so every bucket where the tuner proposed a
different config was re-checked with an interleaved A/B (20 rounds of
do_bench, min and median); identical configs varied by up to ~4.6%, so a
config is replaced only when it wins by >5% on both:

- N=31040, M<=2..32: BLOCK_K 128 -> 256 (BLOCK_K 128 kept for M<=24) and
  ".cg"; 1.11-1.49x vs the previous entries. The previous M<=32 entry was
  no faster than hipBLASLt.
- N=124160, M<=2: 1-warp BLOCK_N=64 tile, 1.09x.
- N=62080, M<=4: ".cg" on the weight load, 1.12x.

Every shipped entry beats hipBLASLt by 1.21-1.83x (do_bench min).
@yicyang
yicyang marked this pull request as ready for review October 19, 2026 05:18
@apinge
apinge self-requested a review October 22, 2026 08:20
Comment thread python/sglang/srt/environ.py Resolved
@sammysun0711
sammysun0711 merged commit c996fcb into apinge:qwen38-flash-next-amd-current-fix-install Oct 26, 2026
1 check failed
sammysun0711 added a commit that referenced this pull request Nov 11, 2026
…#101)

* [AMD] Triton skinny GEMV for the bf16 lm_head on gfx942

The lm_head GEMM is a pure weight-streaming workload at decode
(M = batch size, N = vocab shard). On MI308X hipBLASLt picks a generic
Tensile tile that reaches ~2.2 TB/s; for the Qwen3.8-Flash-Next TP2
shard [124160, 2560] this costs ~250 us per step.

Add a Triton skinny GEMM (python/sglang/kernels/ops/gemm/rocm_bf16_gemv.py)
with three fixed, capture-safe configs:
  - M == 1: row-reduction GEMV on the vector ALUs
  - 2 <= M <= 16: 16-row MFMA tile, kpack=2, waves_per_eu=2
  - 17 <= M <= 32: 32-row MFMA tile with a narrower N tile
M > 32 and non-bf16 / unaligned-K inputs keep using torch.matmul.

LogitsProcessor routes the plain bf16 lm_head matmul through it on gfx942;
SGLANG_ENABLE_ROCM_LM_HEAD_GEMV=0 opts out.

MI308X, K=2560, us (triton / hipBLASLt):
  N=124160: M=1 199/269, M=8 230/286, M=32 278/320
  N=31040:  M=1 37/52,   M=8 49/53,   M=32 60/65
End-to-end TP2 (pyhip config, 12k in / 5 out, conc 1):
  lm_head kernel 250.7 -> 179.3 us, TPOT 8.49 -> 8.38 ms,
  GSM8K (chat, 200, greedy) 0.975 -> 0.980.

* [AMD] Tune the gfx942 lm_head GEMV and route only tuned shapes

Replace the hard-coded launch configs with a per-(N, K) table in
rocm_bf16_gemv.py (_GFX942_CONFIGS, M upper-bound buckets) and add
cache_modifier on the weight load as a tunable.

Only shapes in the table use the Triton kernels; everything else keeps
hipBLASLt. The previous generic configs were slower than hipBLASLt on
untuned shapes for some M (e.g. N <= 24576 at K=2560, or K=5120), and no
single N threshold separated wins from losses across K.

Tuned on MI308X for the Qwen3.8-Flash-Next lm_head shards
(N in {248320, 124160, 62080, 31040}, K=2560) with
benchmark/kernels/rocm_bf16_gemv/tune_rocm_bf16_gemv.py. Tuned vs previous
configs: 1.02-1.14x; vs hipBLASLt: 1.11-1.49x. ".cg" and narrower
1-warp MFMA tiles (BLOCK_N=64) account for most of the gain. Split-K was
evaluated and dropped: no shape improved beyond noise.

TP2 end to end: lm_head 249 -> 165 us (previous configs: 173.5 us).

* [AMD] Time the lm_head GEMV tuner with triton.testing.do_bench and re-tune

Replace the hand-rolled CUDA-event loop in tune_rocm_bf16_gemv.py with
triton.testing.do_bench(warmup=25, rep=100), the setting aiter's Triton
op_benchmarks use (e.g. bench_gemm_a8w8_per_token_scale.py). Candidates,
hipBLASLt and the shipped config are all timed the same way, with L2
flushed between reps.

Re-tuned all four Qwen3.8-Flash-Next lm_head shards on MI308X. The GPUs
were shared with other jobs, so every bucket where the tuner proposed a
different config was re-checked with an interleaved A/B (20 rounds of
do_bench, min and median); identical configs varied by up to ~4.6%, so a
config is replaced only when it wins by >5% on both:

- N=31040, M<=2..32: BLOCK_K 128 -> 256 (BLOCK_K 128 kept for M<=24) and
  ".cg"; 1.11-1.49x vs the previous entries. The previous M<=32 entry was
  no faster than hipBLASLt.
- N=124160, M<=2: 1-warp BLOCK_N=64 tile, 1.09x.
- N=62080, M<=4: ".cg" on the weight load, 1.12x.

Every shipped entry beats hipBLASLt by 1.21-1.83x (do_bench min).

---------


(cherry picked from commit c996fcb)

Co-authored-by: yicyang <yicyang@users.noreply.github.com>
Co-authored-by: Xiake Sun <xisun@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants