Skip to content

[Perf] [NVIDIA] quantization: Add tuned W8A8 block-FP8 GEMM configs for H20 - #41301

Merged
BBuf merged 1 commit into
sgl-project:mainfrom
nankingjing:h20-blockwise-fp8-tile-configs
Nov 6, 2026
Merged

BBuf merged 1 commit into
sgl-project:mainfrom
nankingjing:h20-blockwise-fp8-tile-configs

Conversation

@nankingjing

@nankingjing nankingjing commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

python/sglang/kernels/ops/gemm/configs already treats H20 as a supported device
(26 config files for [128,128] blocks and channelwise FP8), but the block-[32,32] W8A8 path
has only the ten NVIDIA_H200 entries for these shapes. On H20, get_w8a8_block_fp8_configs therefore misses
for every shape and each call falls back to the default tile config:

config = {
    "BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": block_size[0], "BLOCK_SIZE_K": block_size[1],
    "GROUP_SIZE_M": 32, "num_warps": 4, "num_stages": 3,
}

With BLOCK_SIZE_K == block_k == 32 the K loop is fully serial, and at the small M this kernel
actually sees (decode M≈6, speculative-verify M=384, prefill M≤8192) the grid has too few CTAs to
fill the machine. The fallback is not silent — it logs a warning on every miss:

Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal!

On our 8×H20 deployment that warning fired 80 times per server start before this change.

Modifications

Adds ten new config files, one per (N, K) shape, each covering 27 M tiers (M = 1 … 8192):

configs/N={512,576,1280,1536,1792,4096,5120,25600},K={288,1024,1280,5120,6144,15360},
device_name=NVIDIA_H20,dtype=fp8_w8a8,block_shape=[32, 32].json

The ten tables are purely additive: deleting them restores the previous behaviour. One existing
file is touched — kernels/ops/gemm/fp8_kernel.py — so that enabling these tables cannot break
deterministic inference (see Deterministic inference below). Schema is identical to the existing
NVIDIA_H200 files for the same shapes, including the use of SWAP_AB / SPLIT_K where they win.

Produced with the in-tree tuner:

python -m benchmark.kernels.quantization.tuning_block_wise_kernel \
    --block-n 32 --block-k 32 --input-type fp8 \
    --save-path python/sglang/kernels/ops/gemm/configs

When is this table read?

On SM90, block-[32,32] W8A8 GEMMs take the native DeepGEMM path by default since
#42604 (merged 2026-10-10, after this PR was opened):

  • dispatch_w8a8_block_fp8_linear (srt/layers/quantization/fp8_utils.py) returns
    deepgemm_w8a8_block_fp8_linear_with_fallback when weight_block_size == [32, 32],
    SGLANG_ENABLE_JIT_DEEPGEMM is on (the default), and the backend is auto or deep_gemm.
  • _deepgemm_block_fp8_supported accepts any [32, 32] shape with n % 8 == 0 and k % 32 == 0
    on SM90. All ten shapes here satisfy that.

So the tables are read when the Triton block-FP8 backend is in use: with
--fp8-gemm-backend=triton, with SGLANG_ENABLE_JIT_DEEPGEMM=0, when DeepGEMM is unavailable
for the build, or when its runtime fallback triggers. The Triton backend is the default on
non-SM90 hardware and on any SM90 build without DeepGEMM.

This matters for reading the numbers below: they are Triton-backend numbers, measured on the
default at the time this PR was opened. On today's H20 default path this PR changes nothing that
runs — the same is true of the ten NVIDIA_H200 block-[32,32] tables already on main.

One case reaches these tables without an explicit backend choice: triton_w8a8_block_fp8_linear
deliberately bypasses its bf16-weight expansion when is_batch_invariant_mode_enabled(), which
--enable-deterministic-inference turns on (model_runner.maybe_enable_batch_invariant_mode).
Under that flag the block-32 path runs the Triton matmul, reads this table, and therefore needs
the guard described next.

Deterministic inference

enable_deterministic_inference promises batch-invariant ops. The lookup picks an entry by nearest
M, and in these files the reduction strategy changes with M — e.g. for (N,K)=(1536,5120),
M=96 uses SPLIT_K=4 while M=168 uses a single accumulator — so the same row would accumulate
K in a different order depending on the batch it ran in.

w8a8_block_fp8_matmul_triton now drops SPLIT_K and SWAP_AB from the selected entry when
deterministic inference is enabled. BLOCK_SIZE_K is constant (32) across every entry of each file
added here, so once the reduction meta-parameters are gone every M dispatches to the same
single-accumulator kernel and a row's result is independent of its batch. The check reads
exec.deterministic only when the exec namespace is published, so standalone callers (tools, unit
tests) keep their previous behaviour instead of raising
ValueError("config namespace 'exec' not published").

Which kernel path each tier takes. Of the 270 entries, 77 set SWAP_AB (10) or SPLIT_K > 1
(67) and so dispatch to _w8a8_block_fp8_matmul_hopper; the remaining 193 set neither and stay on
select_w8a8_block_fp8_matmul_kernel. No entry sets both. This mirrors how #39657 populates the
H200 tables: the transpose/split-K variants are used only where they measure faster, not uniformly.

Accuracy Tests

  1. fp64 reference. An independent microbenchmark dequantizes the same fp8 operands with the same
    fp32 block scales and matmuls in fp64, so each candidate is checked for accuracy, not merely for
    agreement with the deployed default.
  2. Numerically lossless end-to-end. accept_len — the speculative-decode accept length, a
    sensitive end-to-end correctness signal — is unchanged at 2.66–2.67 across all three arms measured
    (no configs / M≤64 configs / 27-tier configs).
  3. Candidate-vs-deployed parity. For the 27-tier tables, 38 candidate configs were compared against
    the deployed configuration: 35 bit-identical, 3 differing by 1 bf16 ulp.
  4. Caveat, stated plainly. The 67 entries with SPLIT_K > 1 accumulate K in separate fp32
    partials, so they are not bit-identical to the single-accumulator kernel by construction; this is
    why (1) and (2) exist. The remaining 203 entries use a single accumulator. The H200 files use
    SPLIT_K / SWAP_AB in the same way.

No model-output regression is claimed beyond the above.

The CPU config-table test from the first revision
(test/registered/cpu/quant/test_w8a8_block_fp8_configs.py) has been removed at review request; it
is not part of this PR.

Speed Tests and Profiling

8× NVIDIA H20 (SM90, 96 GB), torch 2.11.0+cu130, triton 3.6.0,
torch.cuda.get_device_name() = NVIDIA H20, DeepSeek-V4.1-Flash at TP=8 / EP=8.
Workload: vllm bench serve, sharegpt 100 prompts, seed 42, temperature 0, closed-loop concurrency.
These figures come from one deployment, not a general claim.

Microbenchmark, per shape, M=1 (CUDA-graph capture+replay timing): 4.3–6.7×; 3258 → 989 µs summed
over the ten shapes.

End-to-end, before → after (output tok/s):

concurrency 1 2 4 8 16 32 64
default fallback 127.53 204.86 316.27 458.63 586.42 757.83 999.78
with H20 tables 172.79 258.46 368.74 507.44 640.63 814.80 1133.32
delta +35.5% +26.2% +16.6% +10.6% +9.2% +7.5% +13.4%

Extending the tables from M ≤ 64 to 27 tiers (M ≤ 8192) moved the high-concurrency tier further,
where the lookup had been picking a small-M config for large-M calls:

concurrency 64 ttft_p50
M ≤ 64 table 1048.83 tok/s 1539.4 ms
27-tier table 1133.32 tok/s (+8.1%, median of 7 samples) 1052.2 ms (−31.7%)

The other six concurrency tiers moved within ±1.5% of the pre-extension values.

Runtime confirmation. After the change the server logs Using configuration from ... NVIDIA_H20 ...
for all ten shapes and the "default config / sub-optimal" warning count drops from 80 to 0 per start.

Relation to other open PRs

#41098, #41035 and #39306 add block-[32,32] tables for device_name=NVIDIA_H20-3e. Those keys are
disjoint from NVIDIA_H20 — the config filename is an exact-match lookup on
torch.cuda.get_device_name(), so neither set is reachable from the other's hosts. This PR adds the
NVIDIA_H20 set and touches no file those PRs touch.

If you would rather have a single table keyed by one name, say so and I will fold this in; the two
are otherwise independent.

Worth flagging for whoever reviews them: all three write to
python/sglang/kernels/ops/quantization/configs/, which does not exist on main and has no reader.
The block-FP8 tables are loaded from python/sglang/kernels/ops/gemm/configs/ — see
get_w8a8_block_fp8_configs at kernels/ops/gemm/fp8_kernel.py:553, which joins
os.path.dirname(os.path.realpath(__file__)) with "configs" (:576). Files placed under the old
path would never be opened.

Checklist

  • Format: the ten JSON files are LF, end with a newline, and carry no trailing whitespace
    (i.e. pre-commit is clean on them); fp8_kernel.py is formatted with the repo's
    pre-commit.
  • Unit tests: none added — the CPU config-table test was removed at review request.
  • Documentation: no doc change needed; configs/README.md already documents the filename schema.
  • Accuracy and speed benchmarks: provided above.
  • Code style: matches the existing NVIDIA_H200 files for the same shapes.

CI States

Latest PR Test (Base): ❌ Run #41646490252
Latest PR Test (Extra): ❌ Run #41646489572
Latest PR Test (AMD ROCm 10): ✅ Run #41646489594

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

Note for whoever reviews the H20 lane

Rebased onto current main today, so check-changes will not block a re-run.

While re-checking that these files land somewhere the runtime actually reads, I confirmed the loader is:

# python/sglang/kernels/ops/gemm/fp8_kernel.py, get_w8a8_block_fp8_configs
config_file_path = os.path.join(
    os.path.dirname(os.path.realpath(__file__)), "configs", json_file_name
)

so the only directory that can ever be read is python/sglang/kernels/ops/gemm/configs/. Flagging this because of the other open -3e tables: #39306 places them there (correct), while #41098 and #41035 place them under python/sglang/kernels/ops/quantization/configs/, which does not exist on main and is not opened by any loader — those two would merge cleanly and then never be used. Not my call to make, but it seemed worth saying before anyone spends review time on the -3e SKU.

This PR is orthogonal to all three: it adds NVIDIA_H20 tables, which is an exact-match device_name distinct from NVIDIA_H20-3e.

@nankingjing

Copy link
Copy Markdown
Contributor Author

CI is parked on the run-ci label gate (only Require run-ci label fails; every test job is skipped), so this has never been exercised. The 10 config files live under kernels/ops/gemm/configs/, which is the directory the loader in kernels/ops/gemm/fp8_kernel.py reads — I re-verified that against current main.

Could someone with label rights add run-ci?

One note in case it saves a round: #41098/#41035 add H20 configs under kernels/ops/quantization/configs/, which does not exist on main, and use the NVIDIA_H20-3e key. This PR uses the exact-match NVIDIA_H20 key, so the two sets are disjoint rather than duplicates.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@nankingjing

Copy link
Copy Markdown
Contributor Author

Added a CPU-only test for the tables (dbe5db7), and updated the description accordingly — the previous checklist said "unit tests: not applicable", which is no longer true.

test/registered/cpu/quant/test_w8a8_block_fp8_configs.py, registered to stage-a-test-cpu-intel, so it runs on the CPU stage without a GPU:

  • filename contract — the exact N=…,K=…,device_name=…,dtype=fp8_w8a8,block_shape=[…] name get_w8a8_block_fp8_configs builds, since a mismatch means the file is never opened;
  • the fields the loader dereferences (BLOCK_SIZE_M/N/K, GROUP_SIZE_M, num_warps, num_stages);
  • the block_k % BLOCK_SIZE_K == 0 rule that decides whether fp8_utils silently clamps a tier;
  • the ten H20 grids themselves: 27 strictly ascending M tiers reaching 8192, BLOCK_SIZE_K == 32, and no entry setting both SWAP_AB and SPLIT_K.

Two notes for review:

  1. It is scoped to every block-FP8 table in the directory, not only the ten added here, so it also guards the existing H200 and ROCm tables.
  2. It deliberately does not assert BLOCK_SIZE_K >= block_k, which is what test_fp8_block_gemm_configs.py proposed in [Kernel][DS V4.1] Add W8A8 block-FP8 GEMM configs for H20-3e (DS-V4.1, 32x32) #41035 does. Existing tables in this directory violate that (N=1024,K=4096,device_name=AMD_Instinct_MI455X,block_shape=[128, 128] has BLOCK_SIZE_K: 64 at M=1536), and the loader keeps them as-is because 128 % 64 == 0. That assertion would fail on main today.

Verified by mutation: flipping one entry to BLOCK_SIZE_K=40 fails two of the checks. ruff check and ruff format --check are clean.

@github-actions

This comment has been minimized.

@nankingjing nankingjing changed the title [quantization] Add tuned W8A8 block-FP8 GEMM configs for NVIDIA H20 (block_shape 32x32) [Perf] [NVIDIA] quantization: Add tuned W8A8 block-FP8 GEMM configs for H20 Oct 22, 2026
@nankingjing
nankingjing force-pushed the h20-blockwise-fp8-tile-configs branch from dbe5db7 to 78182fd Compare October 22, 2026 14:21
@nankingjing

Copy link
Copy Markdown
Contributor Author

Rebased onto main (78182fd), and fixed the title.

Two things were red beyond the run-ci gate:

  • check-changes — the previous head was 1,727 commits behind and tripped the "Rebase Required Before CI Can Run" check.
  • Check PR title — [quantization] is not one of the accepted types; it is now [Perf] [NVIDIA] quantization: ….

On the new head check-changes, Check PR title and lint are green. The only remaining red is the run-ci gate.

@nankingjing

Copy link
Copy Markdown
Contributor Author

@BBuf this is the tuned H20 W8A8 block-FP8 config table. The files live in kernels/ops/gemm/configs/, which is the directory fp8_kernel.py reads (I re-checked that against current main today), and there is a CPU-only test that checks the filename contract and that every table is reachable from the loader. What I am unsure about is whether you want tuned configs of this shape checked in at all, and if so what would make it landable — happy to restructure either way.

@nankingjing
nankingjing force-pushed the h20-blockwise-fp8-tile-configs branch from 78182fd to 3a2365a Compare November 2, 2026 15:08
@@ -0,0 +1,189 @@
"""Schema checks for the tuned W8A8 block-FP8 GEMM tables.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please delete this file

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in dfd2e9c. The PR is now the ten config files plus the deterministic-mode guard in fp8_kernel.py.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BBuf Review Agent] Confirmed the test file is gone at dfd2e9c.

@BBuf

BBuf commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

[BBuf Review Agent] @nankingjing The loader path and exact NVIDIA_H20 key check out. Accepting these tuned tables still needs BBuf's call; the CPU test also remains present despite the inline removal request.

@nankingjing

Copy link
Copy Markdown
Contributor Author

Deleted test/registered/cpu/quant/test_w8a8_block_fp8_configs.py as requested — it is gone from the branch (head is now acbeadc1). The PR is down to the 10 config tables and nothing else, +2257/−0.

The tables target NVIDIA_H20 explicitly; say the word if you would rather see a different key shape or a narrower set before this is considered for landing.

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BBuf Review Agent] One thing to fix before this can go in.

  • python/sglang/kernels/ops/gemm/configs/N=1536,K=5120,device_name=NVIDIA_H20,dtype=fp8_w8a8,block_shape=[32, 32].json:36 With deterministic inference enabled on H20 and (N,K)=(1536,5120), M=96 uses split-K while M=168 uses a single accumulator, so the same row can change with the batch. Could we bypass these M-dependent reduction choices in deterministic mode?

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BBuf Review Agent] A few things to fix before this can go in.

# accumulates in different orders at different batch sizes. Under
# deterministic inference, keep every M on the single-accumulator
# kernel so a row's result cannot depend on the batch it ran in.
config = {**config, "SPLIT_K": 1, "SWAP_AB": False}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BBuf Review Agent] With deterministic inference enabled and a tuned config found, this passes SPLIT_K and SWAP_AB to _w8a8_block_fp8_matmul, which accepts neither, so the matmul fails at launch. Could we remove these keys from the copied config instead of setting them to disabled values?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dfd2e9c: the keys are dropped from the copied config instead of being set to disabled values:

config = {k: v for k, v in config.items() if k not in ("SPLIT_K", "SWAP_AB")}

With SPLIT_K/SWAP_AB gone, hopper_tuned is false, so _w8a8_block_fp8_matmul is selected and never receives meta-parameters it does not declare.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BBuf Review Agent] Confirmed the copied config drops both keys before dispatch, so the generic kernel no longer receives unsupported arguments. This fixes the launch issue.

# If an optimal configuration map has been found, look up the
# optimal config
config = configs[min(configs.keys(), key=lambda x: abs(x - M))]
if get_exec().deterministic.enable_deterministic_inference:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BBuf Review Agent] Standalone calls now raise ValueError("config namespace 'exec' not published") whenever a tuned table exists, even when deterministic inference is unused. Could we default this check to false when the exec config hasn't been published?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dfd2e9c: the flag is read only when the namespace is published, so an unpublished context keeps the previous behaviour rather than raising:

context = get_context()
if context.is_config_namespace_published("exec") and (
    get_exec().deterministic.enable_deterministic_inference
):
    ...

This mirrors the existing guard in python/sglang/srt/kernel_plugins/amd/binding.py::_selection.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[BBuf Review Agent] Confirmed the publication check short-circuits before get_exec(), so standalone calls keep the tuned path without that exception.

Address review feedback:
- Drop SPLIT_K/SWAP_AB from the tuned config under deterministic inference
  instead of passing disabled values to kernels that do not accept them.
- Default the deterministic check to disabled when the exec config
  namespace is not published, so standalone callers keep working.
- Remove the CPU config-table test.

The tuned tables keep BLOCK_SIZE_K constant within a file, so dropping the
reduction meta-parameters makes every M run the same single-accumulator
kernel and keeps a row's result independent of the batch it ran in.
@nankingjing
nankingjing force-pushed the h20-blockwise-fp8-tile-configs branch from e13565f to dfd2e9c Compare November 3, 2026 13:40
@nankingjing

Copy link
Copy Markdown
Contributor Author

All four points are addressed in dfd2e9c, and the branch is rebased onto today's main (99fd010). @BBuf

  1. Test file — deleted.
  2. SPLIT_K / SWAP_AB passed to a kernel that does not accept them — the keys are now removed from the copied config rather than set to disabled values. With them gone, hopper_tuned is false, the generic _w8a8_block_fp8_matmul is selected, and it receives only the meta-parameters it declares.
  3. ValueError("config namespace 'exec' not published") for standalone calls — the flag is read only when get_context().is_config_namespace_published("exec") is true, so tools and unit tests keep the previous behaviour.
  4. M-dependent split-K under deterministic inference — same change fixes it at the source. For (N,K)=(1536,5120), M=96 was SPLIT_K=4 while M=168 was single-accumulator. BLOCK_SIZE_K is 32 for every entry in each of the ten files, so once the reduction meta-parameters are dropped every M takes the same single-accumulator path and a row's result no longer depends on the batch it ran in.

Verification I ran against the actual module (stubbed kernel launch, real table):

scenario M dispatched kernel meta-parameters
determinism on, exec published 96 / 168 / 1024 generic no SPLIT_K, no SWAP_AB, BLOCK_SIZE_K=32
determinism off, exec published 96 hopper SPLIT_K=4 (tuned path unchanged)
exec not published 96 hopper tuned path, no exception
no tuned table any generic default config, untouched

The non-deterministic path is unchanged, so the speed numbers in the description still apply.

python -m py_compile is clean on the modified file; pre-commit is clean on the JSON files.

@BBuf

BBuf commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

[BBuf Review Agent] @nankingjing The fixes check out at dfd2e9c, and BLOCK_SIZE_K is 32 across all 270 entries. Accepting these H20 tuning tables still needs BBuf's call.

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@BBuf

BBuf commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Nov 5, 2026
@nankingjing

Copy link
Copy Markdown
Contributor Author

@BBuf thank you for the review and for running CI.

One note on the red: build-test (xeon-spr, spr, ...) fails at test_server_args.py::test_sharded_padded_cp_kv_layout_refuses_unified_memory with error: unrecognized arguments: --enable-prefill-context-parallel. That test was added by #48002 (merged 2026-11-04), and those CP flags are not defined anywhere on main — code search for enable_prefill_context_parallel returns 0 hits. This PR touches only kernels/ops/gemm/fp8_kernel.py and the ten config JSONs, so it cannot affect that test. The pr-test-extra-finish red is the same cascade.

Check PR title passes, and every job this PR's files can influence is green. Ready to merge from my side.

@BBuf

BBuf commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

[BBuf Review Agent] @nankingjing Agreed, the server-args failure is unrelated to this PR's GEMM changes. The unrecognized-flag output is expected in test_generic_v1_cp_options_are_not_public_cli, though, so that line alone doesn't identify the failure cause.

@BBuf

BBuf commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

3 similar comments
@BBuf

BBuf commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@BBuf

BBuf commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@BBuf

BBuf commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@BBuf
BBuf merged commit 732cd76 into sgl-project:main Nov 6, 2026
499 of 561 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jit-kernel run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants