Repository navigation
[Perf] [NVIDIA] quantization: Add tuned W8A8 block-FP8 GEMM configs for H20 - #41301
Conversation
b7930eb to
ec93af2
Compare
|
CI is parked on the Could someone with label rights add One note in case it saves a round: #41098/#41035 add H20 configs under |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Added a CPU-only test for the tables (
Two notes for review:
Verified by mutation: flipping one entry to |
This comment has been minimized.
This comment has been minimized.
dbe5db7 to
78182fd
Compare
|
Rebased onto Two things were red beyond the
On the new head |
|
@BBuf this is the tuned H20 W8A8 block-FP8 config table. The files live in |
78182fd to
3a2365a
Compare
| @@ -0,0 +1,189 @@ | |||
| """Schema checks for the tuned W8A8 block-FP8 GEMM tables. | |||
|
[BBuf Review Agent] @nankingjing The loader path and exact NVIDIA_H20 key check out. Accepting these tuned tables still needs BBuf's call; the CPU test also remains present despite the inline removal request. |
|
Deleted The tables target |
BBuf
left a comment
There was a problem hiding this comment.
[BBuf Review Agent] One thing to fix before this can go in.
python/sglang/kernels/ops/gemm/configs/N=1536,K=5120,device_name=NVIDIA_H20,dtype=fp8_w8a8,block_shape=[32, 32].json:36With deterministic inference enabled on H20 and (N,K)=(1536,5120), M=96 uses split-K while M=168 uses a single accumulator, so the same row can change with the batch. Could we bypass these M-dependent reduction choices in deterministic mode?
BBuf
left a comment
There was a problem hiding this comment.
[BBuf Review Agent] A few things to fix before this can go in.
| # accumulates in different orders at different batch sizes. Under | ||
| # deterministic inference, keep every M on the single-accumulator | ||
| # kernel so a row's result cannot depend on the batch it ran in. | ||
| config = {**config, "SPLIT_K": 1, "SWAP_AB": False} |
There was a problem hiding this comment.
[BBuf Review Agent] With deterministic inference enabled and a tuned config found, this passes SPLIT_K and SWAP_AB to _w8a8_block_fp8_matmul, which accepts neither, so the matmul fails at launch. Could we remove these keys from the copied config instead of setting them to disabled values?
There was a problem hiding this comment.
Fixed in dfd2e9c: the keys are dropped from the copied config instead of being set to disabled values:
config = {k: v for k, v in config.items() if k not in ("SPLIT_K", "SWAP_AB")}With SPLIT_K/SWAP_AB gone, hopper_tuned is false, so _w8a8_block_fp8_matmul is selected and never receives meta-parameters it does not declare.
There was a problem hiding this comment.
[BBuf Review Agent] Confirmed the copied config drops both keys before dispatch, so the generic kernel no longer receives unsupported arguments. This fixes the launch issue.
| # If an optimal configuration map has been found, look up the | ||
| # optimal config | ||
| config = configs[min(configs.keys(), key=lambda x: abs(x - M))] | ||
| if get_exec().deterministic.enable_deterministic_inference: |
There was a problem hiding this comment.
[BBuf Review Agent] Standalone calls now raise ValueError("config namespace 'exec' not published") whenever a tuned table exists, even when deterministic inference is unused. Could we default this check to false when the exec config hasn't been published?
There was a problem hiding this comment.
Fixed in dfd2e9c: the flag is read only when the namespace is published, so an unpublished context keeps the previous behaviour rather than raising:
context = get_context()
if context.is_config_namespace_published("exec") and (
get_exec().deterministic.enable_deterministic_inference
):
...This mirrors the existing guard in python/sglang/srt/kernel_plugins/amd/binding.py::_selection.
There was a problem hiding this comment.
[BBuf Review Agent] Confirmed the publication check short-circuits before get_exec(), so standalone calls keep the tuned path without that exception.
Address review feedback: - Drop SPLIT_K/SWAP_AB from the tuned config under deterministic inference instead of passing disabled values to kernels that do not accept them. - Default the deterministic check to disabled when the exec config namespace is not published, so standalone callers keep working. - Remove the CPU config-table test. The tuned tables keep BLOCK_SIZE_K constant within a file, so dropping the reduction meta-parameters makes every M run the same single-accumulator kernel and keeps a row's result independent of the batch it ran in.
e13565f to
dfd2e9c
Compare
|
All four points are addressed in dfd2e9c, and the branch is rebased onto today's
Verification I ran against the actual module (stubbed kernel launch, real table):
The non-deterministic path is unchanged, so the speed numbers in the description still apply.
|
|
[BBuf Review Agent] @nankingjing The fixes check out at dfd2e9c, and BLOCK_SIZE_K is 32 across all 270 entries. Accepting these H20 tuning tables still needs BBuf's call. |
|
/tag-and-rerun-ci |
|
@BBuf thank you for the review and for running CI. One note on the red:
|
|
[BBuf Review Agent] @nankingjing Agreed, the server-args failure is unrelated to this PR's GEMM changes. The unrecognized-flag output is expected in |
|
/rerun-failed-ci |
3 similar comments
|
/rerun-failed-ci |
|
/rerun-failed-ci |
|
/rerun-failed-ci |
Motivation
python/sglang/kernels/ops/gemm/configsalready treats H20 as a supported device(26 config files for
[128,128]blocks and channelwise FP8), but the block-[32,32]W8A8 pathhas only the ten
NVIDIA_H200entries for these shapes. On H20,get_w8a8_block_fp8_configstherefore missesfor every shape and each call falls back to the default tile config:
With
BLOCK_SIZE_K == block_k == 32the K loop is fully serial, and at the small M this kernelactually sees (decode M≈6, speculative-verify M=384, prefill M≤8192) the grid has too few CTAs to
fill the machine. The fallback is not silent — it logs a warning on every miss:
On our 8×H20 deployment that warning fired 80 times per server start before this change.
Modifications
Adds ten new config files, one per (N, K) shape, each covering 27 M tiers (
M = 1 … 8192):The ten tables are purely additive: deleting them restores the previous behaviour. One existing
file is touched —
kernels/ops/gemm/fp8_kernel.py— so that enabling these tables cannot breakdeterministic inference (see Deterministic inference below). Schema is identical to the existing
NVIDIA_H200files for the same shapes, including the use ofSWAP_AB/SPLIT_Kwhere they win.Produced with the in-tree tuner:
python -m benchmark.kernels.quantization.tuning_block_wise_kernel \ --block-n 32 --block-k 32 --input-type fp8 \ --save-path python/sglang/kernels/ops/gemm/configsWhen is this table read?
On SM90, block-
[32,32]W8A8 GEMMs take the native DeepGEMM path by default since#42604 (merged 2026-10-10, after this PR was opened):
dispatch_w8a8_block_fp8_linear(srt/layers/quantization/fp8_utils.py) returnsdeepgemm_w8a8_block_fp8_linear_with_fallbackwhenweight_block_size == [32, 32],SGLANG_ENABLE_JIT_DEEPGEMMis on (the default), and the backend isautoordeep_gemm._deepgemm_block_fp8_supportedaccepts any[32, 32]shape withn % 8 == 0 and k % 32 == 0on SM90. All ten shapes here satisfy that.
So the tables are read when the Triton block-FP8 backend is in use: with
--fp8-gemm-backend=triton, withSGLANG_ENABLE_JIT_DEEPGEMM=0, when DeepGEMM is unavailablefor the build, or when its runtime fallback triggers. The Triton backend is the default on
non-SM90 hardware and on any SM90 build without DeepGEMM.
This matters for reading the numbers below: they are Triton-backend numbers, measured on the
default at the time this PR was opened. On today's H20 default path this PR changes nothing that
runs — the same is true of the ten
NVIDIA_H200block-[32,32]tables already onmain.One case reaches these tables without an explicit backend choice:
triton_w8a8_block_fp8_lineardeliberately bypasses its bf16-weight expansion when
is_batch_invariant_mode_enabled(), which--enable-deterministic-inferenceturns on (model_runner.maybe_enable_batch_invariant_mode).Under that flag the block-32 path runs the Triton matmul, reads this table, and therefore needs
the guard described next.
Deterministic inference
enable_deterministic_inferencepromises batch-invariant ops. The lookup picks an entry by nearestM, and in these files the reduction strategy changes withM— e.g. for(N,K)=(1536,5120),M=96usesSPLIT_K=4whileM=168uses a single accumulator — so the same row would accumulateK in a different order depending on the batch it ran in.
w8a8_block_fp8_matmul_tritonnow dropsSPLIT_KandSWAP_ABfrom the selected entry whendeterministic inference is enabled.
BLOCK_SIZE_Kis constant (32) across every entry of each fileadded here, so once the reduction meta-parameters are gone every
Mdispatches to the samesingle-accumulator kernel and a row's result is independent of its batch. The check reads
exec.deterministiconly when theexecnamespace is published, so standalone callers (tools, unittests) keep their previous behaviour instead of raising
ValueError("config namespace 'exec' not published").Which kernel path each tier takes. Of the 270 entries, 77 set
SWAP_AB(10) orSPLIT_K > 1(67) and so dispatch to
_w8a8_block_fp8_matmul_hopper; the remaining 193 set neither and stay onselect_w8a8_block_fp8_matmul_kernel. No entry sets both. This mirrors how #39657 populates theH200 tables: the transpose/split-K variants are used only where they measure faster, not uniformly.
Accuracy Tests
fp32 block scales and matmuls in fp64, so each candidate is checked for accuracy, not merely for
agreement with the deployed default.
accept_len— the speculative-decode accept length, asensitive end-to-end correctness signal — is unchanged at 2.66–2.67 across all three arms measured
(no configs / M≤64 configs / 27-tier configs).
the deployed configuration: 35 bit-identical, 3 differing by 1 bf16 ulp.
SPLIT_K > 1accumulate K in separate fp32partials, so they are not bit-identical to the single-accumulator kernel by construction; this is
why (1) and (2) exist. The remaining 203 entries use a single accumulator. The H200 files use
SPLIT_K/SWAP_ABin the same way.No model-output regression is claimed beyond the above.
The CPU config-table test from the first revision
(
test/registered/cpu/quant/test_w8a8_block_fp8_configs.py) has been removed at review request; itis not part of this PR.
Speed Tests and Profiling
8× NVIDIA H20 (SM90, 96 GB),
torch 2.11.0+cu130,triton 3.6.0,torch.cuda.get_device_name()=NVIDIA H20, DeepSeek-V4.1-Flash at TP=8 / EP=8.Workload:
vllm bench serve, sharegpt 100 prompts, seed 42, temperature 0, closed-loop concurrency.These figures come from one deployment, not a general claim.
Microbenchmark, per shape, M=1 (CUDA-graph capture+replay timing): 4.3–6.7×; 3258 → 989 µs summed
over the ten shapes.
End-to-end, before → after (output tok/s):
Extending the tables from
M ≤ 64to 27 tiers (M ≤ 8192) moved the high-concurrency tier further,where the lookup had been picking a small-M config for large-M calls:
ttft_p50M ≤ 64tableThe other six concurrency tiers moved within ±1.5% of the pre-extension values.
Runtime confirmation. After the change the server logs
Using configuration from ... NVIDIA_H20 ...for all ten shapes and the "default config / sub-optimal" warning count drops from 80 to 0 per start.
Relation to other open PRs
#41098, #41035 and #39306 add block-
[32,32]tables fordevice_name=NVIDIA_H20-3e. Those keys aredisjoint from
NVIDIA_H20— the config filename is an exact-match lookup ontorch.cuda.get_device_name(), so neither set is reachable from the other's hosts. This PR adds theNVIDIA_H20set and touches no file those PRs touch.If you would rather have a single table keyed by one name, say so and I will fold this in; the two
are otherwise independent.
Worth flagging for whoever reviews them: all three write to
python/sglang/kernels/ops/quantization/configs/, which does not exist onmainand has no reader.The block-FP8 tables are loaded from
python/sglang/kernels/ops/gemm/configs/— seeget_w8a8_block_fp8_configsatkernels/ops/gemm/fp8_kernel.py:553, which joinsos.path.dirname(os.path.realpath(__file__))with"configs"(:576). Files placed under the oldpath would never be opened.
Checklist
(i.e.
pre-commitis clean on them);fp8_kernel.pyis formatted with the repo'spre-commit.configs/README.mdalready documents the filename schema.NVIDIA_H200files for the same shapes.CI States
Latest PR Test (Base): ❌ Run #41646490252
Latest PR Test (Extra): ❌ Run #41646489572
Latest PR Test (AMD ROCm 10): ✅ Run #41646489594
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent:Note for whoever reviews the H20 lane
Rebased onto current
maintoday, socheck-changeswill not block a re-run.While re-checking that these files land somewhere the runtime actually reads, I confirmed the loader is:
so the only directory that can ever be read is
python/sglang/kernels/ops/gemm/configs/. Flagging this because of the other open-3etables: #39306 places them there (correct), while #41098 and #41035 place them underpython/sglang/kernels/ops/quantization/configs/, which does not exist onmainand is not opened by any loader — those two would merge cleanly and then never be used. Not my call to make, but it seemed worth saying before anyone spends review time on the-3eSKU.This PR is orthogonal to all three: it adds
NVIDIA_H20tables, which is an exact-matchdevice_namedistinct fromNVIDIA_H20-3e.