Skip to content

[Kernel] Add H800 W8A8 block-FP8 GEMM tuning configs - #43976

Open
jijic137 wants to merge 2 commits into
sgl-project:mainfrom
jijic137:perf/h800-block-fp8-configs
Open

jijic137 wants to merge 2 commits into
sgl-project:mainfrom
jijic137:perf/h800-block-fp8-configs

Conversation

@jijic137

@jijic137 jijic137 commented Oct 13, 2026 •

Copy link
Copy Markdown

Motivation

get_w8a8_block_fp8_configs resolves a per-shape tuning file keyed by
(N, K, device_name, dtype, block_shape). DeepSeek-V4.1-Flash TP8 runs its
dense projections as block-FP8 [32, 32], and the tree already ships that
table for NVIDIA_H200 and NVIDIA_H20 — but there are no NVIDIA_H800
entries. On H800 every one of these shapes therefore misses the lookup and
falls back to the default tile config:

{"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 32, "BLOCK_SIZE_K": 32,
 "GROUP_SIZE_M": 32, "num_warps": 4, "num_stages": 3}

With BLOCK_SIZE_N == 32 and no SPLIT_K, the launch grid is pinned at
N/32. On H800 (132 SMs) that is far below the machine: N=576 launches only
18 CTAs, and measured occupancy is 1–8 % across the six shapes. The kernel is
parallelism-starved rather than compute- or bandwidth-bound.

Modifications

Add six NVIDIA_H800, fp8_w8a8, block [32, 32] config files covering the
(N, K) shapes observed on DeepSeek-V4.1-Flash TP8:

  • N=576,K=5120
  • N=1792,K=5120
  • N=4096,K=1280
  • N=5120,K=1024
  • N=5120,K=288
  • N=25600,K=6144

Each file carries three M entries. "1" and "8" (single-request and batched
decode) use SWAP_AB plus a shape-specific SPLIT_K; "512" is byte-identical
to the previous default config, so larger M (prefill / chunked prefill) keeps
stock behaviour and cannot regress. The split-K values were swept on H800
(6 shapes × SPLIT_K ∈ {1,2,4,8,16} × SWAP_AB × BLOCK_SIZE_M ∈ {16,32,64},
with a 256 MB L2 flush before each timed iteration to emulate a cold kernel
launch; two independent repetitions agree within ~2 %):

shape (N, K) SPLIT_K rationale
576, 5120 16 small N, parallelism-starved
1792, 5120 16 small N, parallelism-starved
4096, 1280 1 enough CTAs; split-K measured 17–28 % slower
5120, 1024 1 enough CTAs; split-K measured 17–28 % slower
5120, 288 1 enough CTAs; split-K measured 17–28 % slower
25600, 6144 16 large N, benefits from split-K here

No runtime kernel or model code is changed. Deleting the six files restores the
previous behaviour exactly.

Accuracy Tests

SPLIT_K changes the fp32 accumulation tree, so these entries are not
bitwise-identical to the single-accumulator path — this is already documented
for, and true of, the existing H200/H20 split-K entries (see the comment above
_w8a8_block_fp8_matmul_hopper and _reduce_block_fp8_split_k).

The "512" entry in every file is the unchanged default, so large-M numerics
are untouched. We did not run a dedicated accuracy suite for the H800 split-K
entries; the sibling H20 PR (#41301, #41098) validated equivalent entries
against an fp64 dequantized reference with max relative L2 ≤ 1.7e-3.

Speed Tests and Profiling

H800, 8×GPU single node, DeepSeek-V4.1-Flash, TP8/EP8, no speculation.

Single-kernel (_w8a8_block_fp8_matmul) and single-step numbers, from
torch-profiler traces with a unique prefix and prefix caching disabled so the
two arms see comparable token counts:

metric default + these configs
fp8 GEMM total / decode step (bs=1) 15.34 ms 1.80 ms
decode step time (bs=1) 20.96 ms 9.42 ms
fp8 GEMM total / decode step (bs=8) 10.25 ms 2.45 ms
decode step time (bs=8) 21.4 ms 16.2 ms

Per-shape speedup at M=1 ranges 6.6×–18× across the six shapes.

End-to-end with the standard aiperf scenario on the real cc-traces dataset
(conc=1, 30 min, 393 entries, seed 42):

metric default + these configs
ITL avg 21.52 ms 10.87 ms
output token throughput (avg) 31.84 tok/s 54.56 tok/s
output token throughput per user (p90) 47.00 tok/s 100.42 tok/s

Checklist

  • Format the changed files with pre-commit.
  • Validate numerical behaviour for the changed configs (the "512" entry is unchanged).
  • Provide end-to-end speed results with the config files as the isolated A/B variable.
  • No documentation update is needed for device-specific tuning data.
  • No new CI test is needed because this PR changes data consumed by the existing Hopper kernel path, with no runtime code changes.

CI States

Latest PR Test (Base): ❌ Run #38693825122
Latest PR Test (Extra): ❌ Run #38693825118
Latest PR Test (AMD ROCm 10): ❌ Run #38693825176

Add per-shape W8A8 block-FP8 (32x32) GEMM tuning configs for
device_name=NVIDIA_H800 so the Triton block-FP8 kernel stops falling back to
the default tile config on H800.

Without a device-specific config the kernel uses BLOCK_SIZE_N=32 with no
SPLIT_K, which fixes the launch grid at N/32. On H800 (132 SMs) this leaves
most of the device idle at decode shapes, e.g. only 18 CTAs for N=576 (~1%
occupancy). These entries use SPLIT_K / SWAP_AB to widen the grid and lift
occupancy.

Six (N, K) shapes observed on DeepSeek-V4.1-Flash TP8 are covered. Each file
carries the bs=1 and bs=8 decode buckets plus a large-M fallback entry that is
exactly the previous default config, so prefill-sized M keeps stock behaviour.

No runtime kernel or model code is changed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant