Skip to content

[ROCm] Integrate AITER tgemm, CK decode and FP8/BF16 tuning - #107

Draft
TianHao65 wants to merge 2 commits into
apinge:qwen4_v0.5.21from
TianHao65:qwen4-pr63-tgemm-issue88
Draft

TianHao65 wants to merge 2 commits into
apinge:qwen4_v0.5.21from
TianHao65:qwen4-pr63-tgemm-issue88

Conversation

@TianHao65

@TianHao65 TianHao65 commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

What

Closes #88; follows #63 on qwen4_v0.5.21 (9ac4ebe). Route supported FP8/BF16 prefill through AITER tgemm, retain CK for FP8 M <= 16 decode, and preserve dense BF16 weights for DFlash. Include validated tuning and reproduction tools.

commit change
f315e8d FP8/BF16 routing, CK decode, DFlash layout and packed-tuple fixes
05cd615 Five accepted tuning rows, sixteen shapes, table merge, benchmark and six-case CI entrypoints
export SGLANG_USE_AITER=1
export SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB=1
export SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE=1

The two linear flags default off. Use the tuning kit to merge selected rows into complete AITER tables.

Data

Nov12 local prototype CI: PR63 8511084989b4 → same source plus tgemm and accepted tables. MI308X/PTL34 and PTL52; Torch 2.12 / ROCm 7.14, AITER 514c166, pyhip 8027faa, client fc6ddbb. Con1, random inputs, configured 12000/350 for Qwen3.8 and 8000/500 for Qwen3.5; 10 warmups,128 performance requests and200 GSM8K questions per case/arm. One startup/arm; cross-start noise unestimated. Both arms use the flags above. These results predate final integration/tuple fixes; percentages use unrounded values.

Case Output tok/s before → after (Δ) TTFT ms TPOT ms ETE ms GSM8K correct/200 (Δ pp)
Qwen3.8 TP2 Base 63.110 → 65.396 (+3.62%) 934.786 → 924.615 (-1.09%) 13.210 → 12.684 (-3.98%) 5545.063 → 5351.246 (-3.50%) 196/200 → 197/200 (+0.50 pp)
Qwen3.8 TP2 MTP 89.920 → 91.805 (+2.10%) 1008.787 → 1002.395 (-0.63%) 8.260 → 8.050 (-2.55%) 3891.627 → 3811.752 (-2.05%) 198/200 → 197/200 (-0.50 pp)
Qwen3.5 35B TP4 Base 221.117 → 200.478 (-9.33%) 171.425 → 189.339 (+10.45%) 4.186 → 4.617 (+10.28%) 2260.441 → 2493.081 (+10.29%) 194/200 → 193/200 (-0.50 pp)
Qwen3.5 35B TP1 DFlash 280.805 → 284.802 (+1.42%) 340.043 → 321.651 (-5.41%) 2.885 → 2.872 (-0.45%) 1779.750 → 1754.866 (-1.40%) 197/200 → 197/200 (+0.00 pp)
Qwen3.5 397B TP8 Base 143.223 → 150.996 (+5.43%) 320.585 → 319.966 (-0.19%) 6.352 → 5.993 (-5.65%) 3490.246 → 3310.598 (-5.15%) 198/200 → 198/200 (+0.00 pp)
Qwen3.5 397B TP8 DFlash 277.447 → 281.024 (+1.29%) 337.593 → 338.728 (+0.34%) 2.933 → 2.885 (-1.65%) 1801.303 → 1778.348 (-1.27%) 198/200 → 197/200 (-0.50 pp)

35B/TP4's +10.29% mean ETE regression includes a sustained late slowdown; its trigger remains unresolved.

Follow-ups: two starts/arm in ABBA order, con1. Nov10 changes only tuning tables on image 4d63dea / PTL34: Qwen3.8 TP2, generator means 11000/350, sigma=4, prefix=0.35, MTP/radix on. Nov16 repeats the frozen851 source pair on PTL52: 35B TP4, 8000/500, seed42, radix off, decode Graph on. Both use pool262144; values average the two startup metrics.

Comparison Output tok/s before → after TTFT ms TPOT ms ETE ms
Nov10 Qwen3.8 TP2, base → tuned tables 156.941 → 168.326 (+7.25%) 476.562 → 479.689 (+0.66%) 5.032 → 4.598 (-8.62%) 2226.815 → 2076.229 (-6.76%)
Nov16 35B TP4, PR63 → prototype 184.945 → 191.823 (+3.72%) 196.138 → 189.617 (-3.32%) 5.023 → 4.842 (-3.61%) 2702.621 → 2605.640 (-3.59%)

The initial untuned Qwen3.8 TP2 prototype regressed ETE by 3.52% versus transplanted PR63. The after-only large-pool 35B control measured 2620.343ms ETE; it does not establish the old slowdown's cause. Full comparisons, QPS/acceptance, gates and protocols retain these distinct baselines, all tuning candidates and unmeasured configurations. Custom 11K/350 specifies generator means, not exact token lengths or cache hit rates.

Benchmarking and Profiling

Complete original traces: frozen851 before/prototype after, Quark Qwen3.5-35B, TP1/rank0/PTL52 GPU6, identical 8000 input IDs/64 output tokens, con1, two warmups/start, decode Graph on, radix off. These operator samples are separate from model ETE and final-head CI.

GDN layer0 sample M/N/K GPU µs before → after
Prefill QKVZ, CK → tgemm-selected hipBLASLt 8000/12288/2048 1169.323 → 853.602
Decode BA, BF16 hipBLASLt → AITER skinny 1/64/2048 9.199 → 4.959

Prefill shapes use CPU/GPU External-id matching; decode shapes/layer attribution are inferred from the original graph/source order. Read each GPU row, not summed overlapping scopes.

Prefill

Before:

Prefill before

Open prefill before in Perfetto.

After:

Prefill after

Open prefill after in Perfetto.

Decode

Before:

Decode before

Open decode before in Perfetto.

After:

Decode after

Open decode after in Perfetto.

Complete originals, sample positions and SHA256.

Validation

  • Runtime f315e8d, unchanged in 05cd615: 8 CPU regressions,16 native caller cases/32 changed-input Graph replays passed (max relative L2 0.002740). The tuning kit's 10 CPU checks and changed-file syntax/isort/Ruff/registry checks passed. Repository-wide registry retains an unchanged base taxonomy error in test_pyhip_qsa.py.
  • Both prototype arms passed all six regular local CI cases: 1536 performance requests, 2400 GSM8K evaluations using the same 200-question subset, zero terminal errors. Every case passed reference × 0.97; performance has no speedup gate.
  • Earlier 16-shape/caller/Graph checks passed; repeated ETE controls are in full evidence. Owned test services stopped; GPU release verified.

Final-head six-model GPU CI, daily pressure and full 1319-question GSM8K remain unmeasured.

Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
… entrypoints

Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
@TianHao65 TianHao65 changed the title [ROCm] Use AITER tgemm for PTPC prefill and preserve CK decode [ROCm] Integrate AITER tgemm, CK decode and FP8/BF16 tuning Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[GEMM] Use aiter.gemm to handle hipblaslt linear routing

1 participant