Repository navigation
[DSV4.1] Add tuned H20-3e block-FP8 GEMM configs - #39306
Open
freedomkk-qfeng wants to merge 2 commits into
Open
freedomkk-qfeng wants to merge 2 commits into
freedomkk-qfeng wants to merge 2 commits into
Conversation
freedomkk-qfeng
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
celve and
yuan-luo
as code owners
September 13, 2026 14:56
Port the existing Hopper implementation unchanged from dsv4.1 at 2ee4570. Add H20-specific configs, numerical regression, and reproducible benchmarks.
freedomkk-qfeng
force-pushed
the
h20/dsv41-fp8-decode-20260913
branch
from
September 13, 2026 15:14
8dde1d7 to
aeb3411
Compare
freedomkk-qfeng
requested review from
CatherineSue,
FlamingoPg,
Jialin,
JustinTong0323,
Qiaolin-Yu,
Ying1123,
alphabetc1,
hnyls2002,
hzh0425,
ispobock,
key4ng,
kpham-sgl,
merrymercy,
mickqian,
ping1jing2,
pyc96,
slin1237,
sundar24295s,
yhyang201 and
yizhang2077
as code owners
September 13, 2026 15:14
freedomkk-qfeng
requested review from
AgainstEntropy,
Alisehen,
ByronHsu,
Duyi-Wang,
Kangyan-Zhou,
OrangeRedeng,
YAMY1234,
billishyahao,
bingxche,
hebiao064,
iforgetmyname,
ishandhanani,
liusy58,
rainj-me,
sogalin,
whybeyoung,
wisclmy0611,
yeahdongcn,
yingluosanqian and
zijiexia
as code owners
September 13, 2026 15:14
This was referenced Sep 16, 2026
4 of 5 tasks
This was referenced Oct 1, 2026
…paths The Hopper kernel prerequisite is already upstream in sgl-project#39657. Keep all six original H20-3e tuning tables and both benchmarks; update their module/config paths without changing the tuning values. Document direct-file unittest invocation to avoid a stdlib test package collision.
17 of 56 tasks
5 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
DeepSeek-V4.1-Flash TP8/EP8 uses six dense block-FP8 projection shapes for which NVIDIA H20-3e lacks tuned decode configurations. This PR supplies the H20-3e tables and reproducible numerical/benchmark coverage.
This revision resolves the conflicts with
mainatd7190b0122c5ae181731d61fc6573f883b7a94b5. The Hopper SWAP_AB/Split-K prerequisite originally ported by this PR has since merged in #39657, so this revision uses the upstream kernel. All six original tuning tables are preserved unchanged; their files and benchmark/test imports move fromops/quantizationtoops/gemm. No overlapping unmerged tuning PR is required or incorporated.Modifications
The tables match the exact
NVIDIA H20-3edevice name. Other H20 variants have not been measured.Accuracy Tests
Rerun on 2026-10-07 with the full Python source from the pinned
mainplus this revision:The rerun uses an isolated official dev runtime, pinned to image ID
3a1e3789da6f70c1af7d4c05b1e4310fab79ae3e42c89428b38def1dd90a41f7: PyTorch 2.14.1+cu130, Triton 3.8.0, FlashInfer 0.7.0.post1, sglang-kernel 0.4.9, Transformers 5.17.0, and NVIDIA driver 580.173.02. The mounted Python source is the pinned main plus this revision. Production dependencies were not changed.Speed Tests and Profiling
Current-main isolated GEMM rerun
Measured on one otherwise idle H20-3e. The benchmark covers six shapes and 12 M values (72 points); every variant is validated before timing. Timings use CUDA Graph replay with five 50 ms trials in rotating variant order. The table shows M=1, microseconds (lower is better).
Worst relative L2 error among the microbenchmark variants: 0.000537974. H200 results explicitly select the H200 table while running on H20. These are individual kernel timings, not end-to-end model speedups.
Across all 72 shape/batch points, the unchanged H20 tables have 69 faster / 3 within 3% / 0 slower cases versus the generic configuration. Against the available H200 tables, the counts are 49 / 12 / 11. This is a descriptive ±3% band, not a statistical confidence interval; some selected configurations are identical. The H20 tables are not universally optimal, and the rerun includes cases where the H200 table is faster. This conflict-repair revision preserves the original tuning values.
Historical serving validation (unchanged tables)
The serving measurements below were taken for the original PR in September. They are retained as historical evidence for these unchanged tables and are not a new end-to-end current-main measurement.
Hardware: 8×NVIDIA H20-3e (78 SMs and approximately 141 GiB per GPU). The model-level comparison uses DeepSeek-V4.1-Flash, TP8/EP8, no CP and no speculative decoding. Engram remains on the GPUs. Both variants use identical model files, dependencies, launch arguments, and synthetic input token IDs.
The following serving A/B results were measured in a pinned
dsv4.1integration, not by launching the full model onmain. The serving control isdsv4.1at2ee4570a5f1414e6980ea3eeeaf05cc5a8a0b0d0plus #39114 at0138071bf0011d160154a6a97c7714bed9a9a7da; the serving candidate adds only the six H20 JSON tables. Both serving variants already contain the Hopper implementation. The original PR revision was based onmainat6220f45d8e9a932857b5490e81d1fe9bfc70e841and additionally ported that prerequisite. The resulting FP8 file is byte-identical to the integration-tested kernel (blobda92c781bc2ce1646d7c481436f9c6e6516a9b75). The original full-main numerical test verified that the port imported and ran, but the 91.3% serving gain is not presented as a fresh full-main measurement.Runtime: PyTorch 2.13.0+cu130, Triton 3.7.1, FlashInfer 0.6.18, Transformers 5.12.1, NVIDIA driver 580.173.02. The runtime image was pinned to the
dev-dsv41build atda64c5cbb8cf6bfd39be19da43573fdfd484c43a; the serving source was replaced with the control/candidate trees described above.Single-request decode improves from 44.16 to 84.48 tok/s (+91.3%) at 8192 input / 1000 output tokens. All six measured cases improve; higher-concurrency gains are smaller.
At 8192/1000 C1, median TTFT remains 0.764 s and TPOT drops from 22.64 to 11.84 ms. This is a decode configuration change; no prefill speedup is claimed. The serving control uses the generic H20 configurations. Transplanting H200 tables onto H20 is compared only in the isolated GEMM benchmark above.
Each variant completed 153 measured requests with matching input hashes and token counts. These are finite concurrent batches, not a sustained-load capacity test.
Each serving case is warmed up before measurement, the prefix cache is cleared before every batch, and results are medians of three rounds. Decode throughput excludes the first token; aggregate output throughput includes prefill and scheduling. All measured requests assert zero cached input tokens.
Historical serving command used for both variants:
Run the serving benchmark only on an idle test server: it clears the prefix cache and generates a finite set of requests.
Checklist
CI States
Latest PR Test (Base): ❌ Run #37567580724
Latest PR Test (Extra): ❌ Run #37567580538
Latest PR Test (AMD ROCm 10): ❌ Run #37567580686