Skip to content

[ROCm] Port PR #8 hipBLASLt linears to qwen4_v0.5.21 - #63

Open
TianHao65 wants to merge 3 commits into
apinge:qwen4_v0.5.21from
TianHao65:qwen4-v0.5.21-hipblaslt-pr8
Open

TianHao65 wants to merge 3 commits into
apinge:qwen4_v0.5.21from
TianHao65:qwen4-v0.5.21-hipblaslt-pr8

Conversation

@TianHao65

@TianHao65 TianHao65 commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

What

Port PR #8 hipBLASLt linears to qwen4_v0.5.21 and fix DFlash startup with packed QKV weights. Based on c0a27e48, which includes merged CI PR #65.

commit change
f5ac45b7da AITER hipb_mm for PTPC-FP8 and BF16 linears, with per-weight layout guards and v0.5.21 API adaptation
4183a98ec5 Reject dense slicing of AITER-packed QKV in both DFlash KV paths; use full linear dispatch and split K/V; add regression tests

Enable with:

export SGLANG_USE_AITER=1
export SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB=1
export SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE=1

The two linear flags default off; hipBLASLt output is BF16.

Data

Historical port measurements (2026-10-20): e00930c54 → 0bcf65b8f, Qwen3.5-397B-A17B-PTPC-FP8, TP8, 8× MI308X, ROCm 7.2 / lmsysorg/sglang:v0.5.21-rocm720-mi30x. AITER attention, Triton GDN, FP8 KV, radix disabled; 512 input / 256 output, seed 42. One startup per side, discarded warmups, six formal rounds of ≥60s, 7,168 successful requests. Cross-start noise was not estimated. These comparisons were not rerun for the current DFlash fix.

concurrency output tok/s before → after TTFT ms before → after TPOT ms before → after ETE ms before → after
16 911.14 → 902.65 (-0.93%) 547.01 → 524.90 (-4.04%) 15.464 → 15.716 (+1.63%) 4490.30 → 4532.35 (+0.94%)
64 1727.85 → 1750.89 (+1.33%) 1703.30 → 1569.44 (-7.86%) 30.442 → 30.475 (+0.11%) 9465.90 → 9340.66 (-1.32%)
128 2453.45 → 2511.14 (+2.35%) 2469.64 → 2335.35 (-5.44%) 42.552 → 41.895 (-1.54%) 13320.51 → 13018.58 (-2.27%)

Benchmarking and Profiling

Historical TP0 profiles from the same baseline/port pair, collected separately at concurrency 1. The four screenshots use complete original traces and native Perfetto selections. CK FP8 preshuffle calls changed 240 → 0, hipBLASLt FP8 0 → 240, and hipBLASLt BF16 0 → 210.

sampled GPU kernel before (µs) after (µs)
Prefill FP8 42.319 37.319
Prefill BF16 11.639 9.519
Decode FP8 10.119 9.279
Decode BF16 5.359 9.039

These are sampled single-call durations. The Decode pair increased 15.478 → 18.318 µs (+18.35%), including a BF16 regression; this is not an ETE comparison.

Prefill

Before:

Prefill before: original FP8 and BF16 GEMMs

Open prefill before in Perfetto.

After:

Prefill after: original FP8 and BF16 GEMMs

Open prefill after in Perfetto.

Decode

Before:

Decode before: original FP8 and BF16 GEMMs

Open decode before in Perfetto.

After:

Decode after: original FP8 and BF16 GEMMs

Open decode after in Perfetto.

Complete original traces, hashes and capture manifest.

Validation

Local validation of 4183a98ec5 (2026-11-04): MI308X, Torch 2.12 / ROCm 7.14 / Triton 3.8, AITER 05420f166. Five CPU regression tests, 21 CI driver tests, six real packed-weight numerical cases, and two changed-input CUDA Graph replays passed (maximum relative L2 0.001850 against dense FP32).

All six normal CI configurations passed: each completed 128 performance requests and 200 GSM8K examples, with zero terminal request errors. The 35B/TP1/DFlash result was reused from the same repaired source; the other five were run once.

configuration GSM8K
Qwen3.8 / TP2 / Base 197/200 (98.5%)
Qwen3.8 / TP2 / MTP 196/200 (98%)
Qwen3.5-35B / TP4 / Base 194/200 (97%)
Qwen3.5-35B / TP1 / DFlash 195/200 (97.5%)
Qwen3.5-397B / TP8 / Base 198/200 (99%)
Qwen3.5-397B / TP8 / DFlash 197/200 (98.5%)

The CI workload and gates were unchanged: performance concurrency 1, ten warmups, target input/output 12000/350 for Qwen3.8 and 8000/500 for Qwen3.5; GSM8K permits a maximum 3% relative drop. Test services were stopped and GPU memory release verified.

Daily pressure, full 1319-example GSM8K, and a controlled before/after comparison of the fix remain unmeasured. Packed fallback computes full QKV, so its performance cost remains to be measured.

@TianHao65 TianHao65 changed the title [ROCm] Port PR #8 hipBLASLt swizzle linears to qwen4_v0.5.21 [ROCm] Port PR #8 hipBLASLt linears and optimize small-batch Decode Oct 23, 2026
@sammysun0711

Copy link
Copy Markdown
Collaborator

Here are some information need to be clarify to continue this work:

  1. Why add small-batch decode optimization in this PR?
  2. Did you check all related Qwen3.5 & Qwen3,5 model performance impact for new small-batch decode kernel?
  3. Why the Perfetto trace are not screenshot from original trace, instead of make from the artificial trace?

@TianHao65
TianHao65 force-pushed the qwen4-v0.5.21-hipblaslt-pr8 branch from a8cd545 to ddccab6 Compare October 26, 2026 06:18
@TianHao65 TianHao65 changed the title [ROCm] Port PR #8 hipBLASLt linears and optimize small-batch Decode [ROCm] Port PR #8 hipBLASLt linears to qwen4_v0.5.21 Oct 26, 2026
@TianHao65 TianHao65 closed this Oct 26, 2026
@TianHao65 TianHao65 reopened this Oct 26, 2026
@TianHao65

Copy link
Copy Markdown
Collaborator Author

Here are some information need to be clarify to continue this work:

  1. Why add small-batch decode optimization in this PR?
  2. Did you check all related Qwen3.5 & Qwen3,5 model performance impact for new small-batch decode kernel?
  3. Why the Perfetto trace are not screenshot from original trace, instead of make from the artificial trace?
  • 1–2: The small-batch Decode optimization was added to address the Decode regression observed after the PR [ROCm] Use hipBLASLt swizzle GEMM for fp8/bf16 linears #8 port, but its two commits and associated data have now been split out. This PR contains only the pure port, and the current results primarily cover Qwen3.5-397B-A17B, rather than all Qwen3.5 models.
  • 3: All Perfetto figures have been replaced with screenshots taken directly from the complete, unmodified original traces.

@TianHao65
TianHao65 force-pushed the qwen4-v0.5.21-hipblaslt-pr8 branch 2 times, most recently from 948bd95 to affd9cb Compare November 2, 2026 07:11
@TianHao65
TianHao65 changed the base branch from qwen4_v0.5.21 to qwen4-ci-base November 2, 2026 08:42
@TianHao65
TianHao65 force-pushed the qwen4-v0.5.21-hipblaslt-pr8 branch from affd9cb to e6ff953 Compare November 2, 2026 08:42
@TianHao65
TianHao65 changed the base branch from qwen4-ci-base to qwen4_v0.5.21 November 3, 2026 02:15
@TianHao65
TianHao65 force-pushed the qwen4-v0.5.21-hipblaslt-pr8 branch from e6ff953 to affd9cb Compare November 3, 2026 02:15
Port apinge/sglang PR sgl-project#8 (a6f5d30)
using the previously tested v0.5.21 adaptation.

Route PTPC-FP8 and unquantized BF16 linears through AITER hipb_mm behind
the existing swizzle flags. Track weight layouts per Parameter and retain
fallback dispatch for weights that fail the swizzle shape guard.

Handle v0.5.21 prequantized three-element tuples and leading dimensions,
and support BF16 caller-owned output buffers.

(cherry picked from commit 0bcf65b)
Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
Signed-off-by: TianHao65 <Hao.Tian@amd.com>
@TianHao65
TianHao65 force-pushed the qwen4-v0.5.21-hipblaslt-pr8 branch from affd9cb to f5ac45b Compare November 3, 2026 07:34
@TianHao65
TianHao65 marked this pull request as ready for review November 3, 2026 09:34
Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
return unfused_matches[0] if all(unfused_matches) else None


class AiterHipblaslt:

@sammysun0711 sammysun0711 Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we need cherry pick this PR as is? I see unconditionally call hipblaslt will trigger the decode perf regression.

Can we consider choose aiter.tgemm instead
Example: sammysun0711@ca9c322

Current this example only cover FP8 linear in LLM case, should add BF16 linear in VIT part for compute bound case.

For decode case memory bound case, we can either

cc @apinge, @yicyang

I create a follow up issue: #88

@sammysun0711

Copy link
Copy Markdown
Collaborator

Data

Historical port measurements (2026-10-20): e00930c54 → 0bcf65b8f, Qwen3.5-397B-A17B-PTPC-FP8, TP8, 8× MI308X, ROCm 7.2 / lmsysorg/sglang:v0.5.21-rocm720-mi30x. AITER attention, Triton GDN, FP8 KV, radix disabled; 512 input / 256 output, seed 42. One startup per side, discarded warmups, six formal rounds of ≥60s, 7,168 successful requests. Cross-start noise was not estimated. These comparisons were not rerun for the current DFlash fix.

concurrency output tok/s before → after TTFT ms before → after TPOT ms before → after ETE ms before → after
16 911.14 → 902.65 (-0.93%) 547.01 → 524.90 (-4.04%) 15.464 → 15.716 (+1.63%) 4490.30 → 4532.35 (+0.94%)
64 1727.85 → 1750.89 (+1.33%) 1703.30 → 1569.44 (-7.86%) 30.442 → 30.475 (+0.11%) 9465.90 → 9340.66 (-1.32%)
128 2453.45 → 2511.14 (+2.35%) 2469.64 → 2335.35 (-5.44%) 42.552 → 41.895 (-1.54%) 13320.51 → 13018.58 (-2.27%)

We may not reach so high con (con64/128) to see the per gain.

Please use standard benchmark scripts (con=1) to report before/after performance on target machine with PTL on
https://github.com/apinge/sglang/blob/qwen4_v0.5.21/evaluation/run_qwen35_pure_text_bench.sh
https://github.com/apinge/sglang/blob/qwen4_v0.5.21/evaluation/run_qwen38_pure_text_bench.sh

@sammysun0711

sammysun0711 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator
  • Base / before: run 42149584665, tested SGLang c8cb75cecb2e, “Base” denotes this selected before run.
  • PR / after: run 42367555977, tested SGLang 8511084989b4.
  • Shared sources: AITER 514c166e9abad59acde74309923c2d923a89eee0, pyhip 8027faad29415747707670db7776528c6b72db81, and benchmark client fc6ddbbf7aac80167ea04cc00bf200af8adf2b85. Installed dependency versions match across the selected artifacts; SGLang differs.
  • Performance workload: random inputs, configured input/output lengths of 12,000/350 tokens for Qwen3.8 and 8,000/500 for Qwen3.5.

Performance changes are (PR / base - 1) × 100%, calculated from unrounded artifact values. Negative TTFT/TPOT changes mean lower latency; positive throughput/QPS changes mean higher throughput. GSM8K differences are percentage points (pp).

Case GSM8K (base → PR; Δ pp) TTFT ms (base → PR; Δ %) TPOT ms (base → PR; Δ %)
Qwen3.8-Flash-TP2-Base 99.00% → 98.50% (-0.50 pp) 965.853 → 961.761 (-0.4%) 12.787 → 12.767 (-0.2%)
Qwen3.8-Flash-TP2-MTP 99.00% → 98.00% (-1.00 pp) 1043.977 → 1047.015 (+0.3%) 8.495 → 8.383 (-1.3%)
Qwen3.5-35B-TP4-Base 98.00% → 97.50% (-0.50 pp) 170.021 → 168.023 (-1.2%) 4.101 → 4.264 (+4.0%)
Qwen3.5-35B-TP1-DFlash 98.50% → 97.00% (-1.50 pp) 333.475 → 336.070 (+0.8%) 2.754 → 2.773 (+0.7%)
Qwen3.5-397B-TP8-Base 98.50% → 98.00% (-0.50 pp) 362.035 → 334.141 (-7.7%) 6.317 → 6.587 (+4.3%)
Qwen3.5-397B-TP8-DFlash 98.50% → 98.50% (0.00 pp) 375.879 → 349.114 (-7.1%) 3.011 → 3.043 (+1.0%)

The remaining summary metrics and the unchanged GSM8K reference/minimum are retained below. Acceptance length applies to speculative cases.

Case Output tokens/s (base → PR; Δ %) QPS (base → PR; Δ %) Accept length (base → PR; Δ %) GSM8K reference / minimum
Qwen3.8-Flash-TP2-Base 64.464 → 64.599 (+0.2%) 0.184 → 0.185 (+0.2%) — 98.00% / 95.06%
Qwen3.8-Flash-TP2-MTP 87.294 → 88.085 (+0.9%) 0.249 → 0.252 (+0.9%) 3.503 → 3.548 (+1.3%) 98.50% / 95.54%
Qwen3.5-35B-TP4-Base 225.527 → 217.746 (-3.5%) 0.451 → 0.435 (-3.5%) — 97.00% / 94.09%
Qwen3.5-35B-TP1-DFlash 292.613 → 290.603 (-0.7%) 0.585 → 0.581 (-0.7%) 6.171 → 6.168 (-0.1%) 97.00% / 94.09%
Qwen3.5-397B-TP8-Base 142.235 → 138.050 (-2.9%) 0.284 → 0.276 (-2.9%) — 99.00% / 96.03%
Qwen3.5-397B-TP8-DFlash 266.062 → 267.644 (+0.6%) 0.532 → 0.535 (+0.6%) 7.838 → 7.855 (+0.2%) 98.50% / 95.54%

The observed tradeoffs are:

  • Qwen3.5 397B TTFT decreases by 7.7% for Base and 7.1% for DFlash. TPOT increases by 4.3% and 1.0%, respectively.
  • Qwen3.5 35B TP4 Base TPOT increases by 4.0%, with output throughput down 3.5%.
  • GSM8K decreases by 0.50–1.50 percentage points in five cases and is unchanged for 397B DFlash. All six remain above the recorded CI minimum.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants