Skip to content

[DSV4.1] Add tuned H20-3e block-FP8 GEMM configs - #39306

Open
freedomkk-qfeng wants to merge 2 commits into
sgl-project:mainfrom
freedomkk-qfeng:h20/dsv41-fp8-decode-20260913
Open

freedomkk-qfeng wants to merge 2 commits into
sgl-project:mainfrom
freedomkk-qfeng:h20/dsv41-fp8-decode-20260913

Conversation

@freedomkk-qfeng

@freedomkk-qfeng freedomkk-qfeng commented Sep 13, 2026 •

Copy link
Copy Markdown

Motivation

DeepSeek-V4.1-Flash TP8/EP8 uses six dense block-FP8 projection shapes for which NVIDIA H20-3e lacks tuned decode configurations. This PR supplies the H20-3e tables and reproducible numerical/benchmark coverage.

This revision resolves the conflicts with main at d7190b0122c5ae181731d61fc6573f883b7a94b5. The Hopper SWAP_AB/Split-K prerequisite originally ported by this PR has since merged in #39657, so this revision uses the upstream kernel. All six original tuning tables are preserved unchanged; their files and benchmark/test imports move from ops/quantization to ops/gemm. No overlapping unmerged tuning PR is required or incorporated.

Modifications

  • Retain the six H20-3e configuration tables for 32×32 block-FP8, covering decode batches through 64.
  • Retain the standalone GEMM/serving benchmarks and manual numerical regression test, using the current GEMM module path.
  • Preserve default entries at M=0 and M=65, so nearest-M lookup keeps the generic configuration outside the tuned range.
  • Resolve the branch with a merge commit that preserves the original PR history; no model-forward or CP changes are included.

The tables match the exact NVIDIA H20-3e device name. Other H20 variants have not been measured.

Accuracy Tests

Rerun on 2026-10-07 with the full Python source from the pinned main plus this revision:

  • Manual numerical regression: 2 tests passed, covering 3,120 numerical subcases (six shapes × M=1..65 × two seeds × two activation-scale layouts × BF16/FP16 output).
  • Each comparison requires finite outputs and relative L2 error below 0.01 against dequantized FP32 matmul rounded to the output dtype. Bitwise equality is not claimed because Split-K changes reduction order.
  • The out-of-range test verifies that the default configuration remains selected for M=0, M=65..8192 and selected larger batches.
  • Full repository pre-commit checks passed for all changed files.

The rerun uses an isolated official dev runtime, pinned to image ID 3a1e3789da6f70c1af7d4c05b1e4310fab79ae3e42c89428b38def1dd90a41f7: PyTorch 2.14.1+cu130, Triton 3.8.0, FlashInfer 0.7.0.post1, sglang-kernel 0.4.9, Transformers 5.17.0, and NVIDIA driver 580.173.02. The mounted Python source is the pinned main plus this revision. Production dependencies were not changed.

python test/manual/quant/test_h20_block_fp8_configs.py -v

Speed Tests and Profiling

Current-main isolated GEMM rerun

Measured on one otherwise idle H20-3e. The benchmark covers six shapes and 12 M values (72 points); every variant is validated before timing. Timings use CUDA Graph replay with five 50 ms trials in rotating variant order. The table shows M=1, microseconds (lower is better).

N K Generic H200 table on H20 H20-3e table
1792 5120 72.61 10.18 8.27
25600 6144 645.10 79.97 76.97
4096 1280 63.89 6.48 6.02
5120 1024 71.51 8.00 6.78
5120 288 25.22 4.90 3.96
576 5120 43.85 6.14 6.15

Worst relative L2 error among the microbenchmark variants: 0.000537974. H200 results explicitly select the H200 table while running on H20. These are individual kernel timings, not end-to-end model speedups.

Across all 72 shape/batch points, the unchanged H20 tables have 69 faster / 3 within 3% / 0 slower cases versus the generic configuration. Against the available H200 tables, the counts are 49 / 12 / 11. This is a descriptive ±3% band, not a statistical confidence interval; some selected configurations are identical. The H20 tables are not universally optimal, and the rerun includes cases where the H200 table is faster. This conflict-repair revision preserves the original tuning values.

python benchmark/kernels/deepseek/benchmark_h20_block_fp8.py \
  --output h20_block_fp8.jsonl

Historical serving validation (unchanged tables)

The serving measurements below were taken for the original PR in September. They are retained as historical evidence for these unchanged tables and are not a new end-to-end current-main measurement.

Hardware: 8×NVIDIA H20-3e (78 SMs and approximately 141 GiB per GPU). The model-level comparison uses DeepSeek-V4.1-Flash, TP8/EP8, no CP and no speculative decoding. Engram remains on the GPUs. Both variants use identical model files, dependencies, launch arguments, and synthetic input token IDs.

The following serving A/B results were measured in a pinned dsv4.1 integration, not by launching the full model on main. The serving control is dsv4.1 at 2ee4570a5f1414e6980ea3eeeaf05cc5a8a0b0d0 plus #39114 at 0138071bf0011d160154a6a97c7714bed9a9a7da; the serving candidate adds only the six H20 JSON tables. Both serving variants already contain the Hopper implementation. The original PR revision was based on main at 6220f45d8e9a932857b5490e81d1fe9bfc70e841 and additionally ported that prerequisite. The resulting FP8 file is byte-identical to the integration-tested kernel (blob da92c781bc2ce1646d7c481436f9c6e6516a9b75). The original full-main numerical test verified that the port imported and ran, but the 91.3% serving gain is not presented as a fresh full-main measurement.

Runtime: PyTorch 2.13.0+cu130, Triton 3.7.1, FlashInfer 0.6.18, Transformers 5.12.1, NVIDIA driver 580.173.02. The runtime image was pinned to the dev-dsv41 build at da64c5cbb8cf6bfd39be19da43573fdfd484c43a; the serving source was replaced with the control/candidate trees described above.

Single-request decode improves from 44.16 to 84.48 tok/s (+91.3%) at 8192 input / 1000 output tokens. All six measured cases improve; higher-concurrency gains are smaller.

Input / output tokens Concurrency Decode tok/s, before → after Decode gain Aggregate output tok/s, before → after Aggregate gain
8K / 1000 1 44.16 → 84.48 +91.3% 42.75 → 79.39 +85.7%
1K / 128 1 44.27 → 84.73 +91.4% 42.04 → 76.58 +82.1%
1K / 128 8 25.65 → 29.63 +15.5% 176.80 → 199.63 +12.9%
1K / 128 32 8.50 → 8.78 +3.2% 235.87 → 242.22 +2.7%
8K / 128 8 18.67 → 20.71 +10.9% 93.43 → 99.44 +6.4%
32K / 128 1 44.23 → 84.54 +91.1% 20.69 → 26.59 +28.5%

At 8192/1000 C1, median TTFT remains 0.764 s and TPOT drops from 22.64 to 11.84 ms. This is a decode configuration change; no prefill speedup is claimed. The serving control uses the generic H20 configurations. Transplanting H200 tables onto H20 is compared only in the isolated GEMM benchmark above.

Each variant completed 153 measured requests with matching input hashes and token counts. These are finite concurrent batches, not a sustained-load capacity test.

Each serving case is warmed up before measurement, the prefix cache is cleared before every batch, and results are medians of three rounds. Decode throughput excludes the first token; aggregate output throughput includes prefill and scheduling. All measured requests assert zero cached input tokens.

Historical serving command used for both variants:

export SGLANG_RUST_SERVER=0 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=8
export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1

sglang serve \
  --trust-remote-code \
  --enable-cache-report \
  --enable-metrics \
  --model-path "$MODEL_PATH" \
  --served-model-name deepseek-v4-flash \
  --tp 8 \
  --ep-size 8 \
  --mem-fraction-static 0.78 \
  --context-length 1048576 \
  --max-running-requests 256 \
  --chunked-prefill-size 8192 \
  --attention-backend dsv4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --flashinfer-mxfp4-moe-precision fp8 \
  --tool-call-parser auto \
  --reasoning-parser auto \
  --enable-decoder-swa-bounded-replay \
  --cuda-graph-backend-prefill disabled \
  --cuda-graph-max-bs-decode 64 \
  --host 0.0.0.0 \
  --port 8000

python benchmark/serving/benchmark_h20_decode.py \
  --model-path "$MODEL_PATH" --output before.jsonl
# Repeat on the candidate with --output after.jsonl.

Run the serving benchmark only on an idle test server: it clears the prefix cache and generates a finite set of requests.

Checklist

  • Format your code according to the SGLang pre-commit configuration.
  • Add accuracy tests and reproducible speed benchmarks.
  • Document the measured hardware, scope, and reproduction commands.
  • Provide accuracy and speed benchmark results.
  • Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ❌ Run #37567580724
Latest PR Test (Extra): ❌ Run #37567580538
Latest PR Test (AMD ROCm 10): ❌ Run #37567580686

@freedomkk-qfeng freedomkk-qfeng changed the title [DSV4.1] Tune block-FP8 decode configs for H20-3e [Quantization] Add Hopper block-FP8 path and tuned H20-3e configs Sep 13, 2026
Port the existing Hopper implementation unchanged from dsv4.1 at 2ee4570. Add H20-specific configs, numerical regression, and reproducible benchmarks.
…paths

The Hopper kernel prerequisite is already upstream in sgl-project#39657. Keep all six original H20-3e tuning tables and both benchmarks; update their module/config paths without changing the tuning values.

Document direct-file unittest invocation to avoid a stdlib test package collision.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant