Repository navigation
Conversation
Add per-shape W8A8 block-FP8 (32x32) GEMM tuning configs for device_name=NVIDIA_H800 so the Triton block-FP8 kernel stops falling back to the default tile config on H800. Without a device-specific config the kernel uses BLOCK_SIZE_N=32 with no SPLIT_K, which fixes the launch grid at N/32. On H800 (132 SMs) this leaves most of the device idle at decode shapes, e.g. only 18 CTAs for N=576 (~1% occupancy). These entries use SPLIT_K / SWAP_AB to widen the grid and lift occupancy. Six (N, K) shapes observed on DeepSeek-V4.1-Flash TP8 are covered. Each file carries the bs=1 and bs=8 decode buckets plus a large-M fallback entry that is exactly the previous default config, so prefill-sized M keeps stock behaviour. No runtime kernel or model code is changed.
jijic137
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
celve and
yuan-luo
as code owners
October 13, 2026 03:27
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
get_w8a8_block_fp8_configsresolves a per-shape tuning file keyed by(N, K, device_name, dtype, block_shape). DeepSeek-V4.1-Flash TP8 runs itsdense projections as block-FP8
[32, 32], and the tree already ships thattable for
NVIDIA_H200andNVIDIA_H20— but there are noNVIDIA_H800entries. On H800 every one of these shapes therefore misses the lookup and
falls back to the default tile config:
{"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 32, "BLOCK_SIZE_K": 32, "GROUP_SIZE_M": 32, "num_warps": 4, "num_stages": 3}With
BLOCK_SIZE_N == 32and noSPLIT_K, the launch grid is pinned atN/32. On H800 (132 SMs) that is far below the machine:N=576launches only18 CTAs, and measured occupancy is 1–8 % across the six shapes. The kernel is
parallelism-starved rather than compute- or bandwidth-bound.
Modifications
Add six
NVIDIA_H800,fp8_w8a8, block[32, 32]config files covering the(N, K) shapes observed on DeepSeek-V4.1-Flash TP8:
N=576,K=5120N=1792,K=5120N=4096,K=1280N=5120,K=1024N=5120,K=288N=25600,K=6144Each file carries three M entries.
"1"and"8"(single-request and batcheddecode) use
SWAP_ABplus a shape-specificSPLIT_K;"512"is byte-identicalto the previous default config, so larger M (prefill / chunked prefill) keeps
stock behaviour and cannot regress. The split-K values were swept on H800
(6 shapes ×
SPLIT_K∈ {1,2,4,8,16} ×SWAP_AB×BLOCK_SIZE_M∈ {16,32,64},with a 256 MB L2 flush before each timed iteration to emulate a cold kernel
launch; two independent repetitions agree within ~2 %):
No runtime kernel or model code is changed. Deleting the six files restores the
previous behaviour exactly.
Accuracy Tests
SPLIT_Kchanges the fp32 accumulation tree, so these entries are notbitwise-identical to the single-accumulator path — this is already documented
for, and true of, the existing H200/H20 split-K entries (see the comment above
_w8a8_block_fp8_matmul_hopperand_reduce_block_fp8_split_k).The
"512"entry in every file is the unchanged default, so large-M numericsare untouched. We did not run a dedicated accuracy suite for the H800 split-K
entries; the sibling H20 PR (#41301, #41098) validated equivalent entries
against an fp64 dequantized reference with max relative L2 ≤ 1.7e-3.
Speed Tests and Profiling
H800, 8×GPU single node, DeepSeek-V4.1-Flash, TP8/EP8, no speculation.
Single-kernel (
_w8a8_block_fp8_matmul) and single-step numbers, fromtorch-profiler traces with a unique prefix and prefix caching disabled so the
two arms see comparable token counts:
Per-shape speedup at M=1 ranges 6.6×–18× across the six shapes.
End-to-end with the standard aiperf scenario on the real cc-traces dataset
(conc=1, 30 min, 393 entries, seed 42):
Checklist
"512"entry is unchanged).CI States
Latest PR Test (Base): ❌ Run #38693825122
Latest PR Test (Extra): ❌ Run #38693825118
Latest PR Test (AMD ROCm 10): ❌ Run #38693825176