Skip to content

[ROCm] Add decode swizzled GEMV and safe caller-owned outputs - #76

Draft
TianHao65 wants to merge 3 commits into
apinge:qwen4_v0.5.21from
TianHao65:qwen4-pr8-decode-validated-20261028
Draft

TianHao65 wants to merge 3 commits into
apinge:qwen4_v0.5.21from
TianHao65:qwen4-pr8-decode-validated-20261028

Conversation

@TianHao65

@TianHao65 TianHao65 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

What

Two Decode follow-ups to PR63, retaining its rebased prerequisite f5ac45b7da807ed3965633518111b40ba8517dfc. Rebased onto apinge:qwen4_v0.5.21 at c0a27e481010cd2f285fe426eed35860ae1920a5. CI PR65 is merged into this target; all separately cherry-picked CI and stacked-CI compatibility commits were removed. The PR now targets qwen4_v0.5.21 directly so the merged CI matches its target. The comparison contains three commits / nine files: one prerequisite port and two Decode commits.

Decode-only comparison.

Commit Change
446e570f0a Swizzled BF16 GEMV for Decode N=16, K=4096, M=1..32; other shapes and prefill retain the existing path.
c68affd18b Direct writes to eligible apply_into() buffers, including the corrected overlapping DLPack storage guard.

Enable GEMV with SGLANG_ROCM_HIPB_DECODE_GEMV=1 (off by default). Tuned solutions were disabled in these tests.

Data

Historical measurements and original screenshots below were collected on 2026-10-28 using ddccab6e0d / e0e3b3271a / 176150ccc3. The optimization blobs are unchanged by this rebase; no new full-model measurements were run for these rebased commits.

Qwen3.5-397B-A17B-PTPC-FP8, 8×MI308X / TP8, SGLang 0.5.21, Torch 2.9.1 / ROCm 7.2. Measured on 2026-10-28 UTC with AITER attention, Triton GDN, FP8 KV, and radix/speculative decoding disabled.

c1 uses three independent starts per version; c16/c64/c128 use one. Warmups were discarded, each formal round lasted ≥60s, and all 7,128 measured requests succeeded with matching actual lengths. Values are means; percentages compare with the pure port.

Concurrency 1: 8000 input / 500 output; concurrency 16/64/128: 512 / 256.

Pure port → GEMV

concurrency output tok/s before → after TTFT ms before → after TPOT ms before → after ETE ms before → after
1 94.48 → 96.54 (+2.18%) 385.41 → 385.13 (-0.07%) 9.829 → 9.604 (-2.29%) 5290.25 → 5177.50 (-2.13%)
16 905.82 → 912.20 (+0.70%) 524.93 → 519.06 (-1.12%) 15.651 → 15.550 (-0.65%) 4515.91 → 4484.22 (-0.70%)
64 1755.64 → 1755.55 (-0.01%) 1557.25 → 1553.23 (-0.26%) 30.411 → 30.429 (+0.06%) 9312.09 → 9312.65 (+0.01%)
128 2519.46 → 2512.22 (-0.29%) 2329.42 → 2306.38 (-0.99%) 41.721 → 41.907 (+0.45%) 12968.29 → 12992.62 (+0.19%)

Pure port → GEMV + direct output

concurrency output tok/s before → after TTFT ms before → after TPOT ms before → after ETE ms before → after
1 94.48 → 96.40 (+2.03%) 385.41 → 385.40 (-0.003%) 9.829 → 9.619 (-2.14%) 5290.25 → 5185.09 (-1.99%)
16 905.82 → 911.57 (+0.64%) 524.93 → 523.79 (-0.22%) 15.651 → 15.544 (-0.68%) 4515.91 → 4487.43 (-0.63%)
64 1755.64 → 1753.19 (-0.14%) 1557.25 → 1570.29 (+0.84%) 30.411 → 30.413 (+0.01%) 9312.09 → 9325.66 (+0.15%)
128 2519.46 → 2517.35 (-0.08%) 2329.42 → 2304.26 (-1.08%) 41.721 → 41.783 (+0.15%) 12968.29 → 12958.87 (-0.07%)

High-concurrency differences are single-start observations, including small regressions; full batches at c64/c128 use the M>32 fallback.

Direct apply_into() reduced M1 operator time from 5.490 to 2.865µs (−47.81% vs GEMV). This is an API benefit: the measured Qwen3.5 caller uses apply(), so no additional model benefit from direct output is established.

Full results, per-start values, runtime configuration, and microbenchmarks.

Benchmarking and Profiling

Complete untouched model traces, rank 0, first in_proj_ba sample: BF16 N16/K4096; prefill M8000, Decode M1/graph bs1. Layer 0 is inferred from execution order. These are individual kernel samples, separate from ETE.

The four screenshots below use the original Perfetto views. Open the complete original traces using the links below each image; full SHA256 manifest.

Prefill

BF16 hipBLASLt fallback: 32.120 → 31.919µs, both at a 100µs scale.

Before

Prefill Before

Open prefill before in Perfetto.

After

Prefill After

Open prefill after in Perfetto.

Decode

hipBLASLt → GEMV: 8.959 → 5.279µs, both at a 40µs scale.

Before

Decode Before

Open decode before in Perfetto.

After

Decode After

Open decode after in Perfetto.

Validation

Current rebase checks: preserved feature patches/model code, author identities/dates, Python syntax and clean diff; 21 CI driver tests and 10 evaluation shell syntax checks passed. The original five CPU output-guard regressions /35 subtests are reused because the kernel and test blobs are unchanged. No new model or GPU measurements were run; the results below retain their measured revisions.

  • Passed: 5 CPU regressions, 15 GPU output/padding/pointer/graph cases, and 10 DLPack overlap rejections. Maximum relative L2: 0.00199431, below the 0.02 limit.
  • GEMV is present on all eight TP ranks; the direct-output operator profile contains no copy_. Source/syntax/diff checks passed.
  • Quality remains unresolved: all 27 short-answer probes reached the 64-token cap, including the baseline. No accuracy or quality-equivalence claim is made.
  • All task-owned test servers and containers stopped; process, port, and GPU release were verified.

Validation reports · Final protocol/quality/resource audit.

CI execution scope: the merged workflow is inherited unchanged, and this PR now targets qwen4_v0.5.21. It remains Draft; six-model GPU CI has not run for this rebased head.

@TianHao65
TianHao65 force-pushed the qwen4-pr8-decode-validated-20261028 branch from 176150c to 71f9a0e Compare November 2, 2026 06:58
@TianHao65
TianHao65 force-pushed the qwen4-pr8-port-base branch from ddccab6 to 948bd95 Compare November 2, 2026 06:59
@TianHao65
TianHao65 force-pushed the qwen4-pr8-decode-validated-20261028 branch from 71f9a0e to 7ec66bc Compare November 2, 2026 07:11
@TianHao65
TianHao65 force-pushed the qwen4-pr8-port-base branch from 948bd95 to affd9cb Compare November 2, 2026 07:12
@TianHao65
TianHao65 force-pushed the qwen4-pr8-decode-validated-20261028 branch from 7ec66bc to bd2e055 Compare November 2, 2026 08:42
@TianHao65
TianHao65 force-pushed the qwen4-pr8-port-base branch from affd9cb to e6ff953 Compare November 2, 2026 08:42
@TianHao65
TianHao65 force-pushed the qwen4-pr8-decode-validated-20261028 branch from bd2e055 to 7ec66bc Compare November 3, 2026 02:16
@TianHao65
TianHao65 force-pushed the qwen4-pr8-port-base branch from e6ff953 to affd9cb Compare November 3, 2026 02:16
TianHao65 and others added 3 commits November 3, 2026 15:29
Port apinge/sglang PR sgl-project#8 (a6f5d30)
using the previously tested v0.5.21 adaptation.

Route PTPC-FP8 and unquantized BF16 linears through AITER hipb_mm behind
the existing swizzle flags. Track weight layouts per Parameter and retain
fallback dispatch for weights that fail the swizzle shape guard.

Handle v0.5.21 prequantized three-element tuples and leading dimensions,
and support BF16 caller-owned output buffers.

(cherry picked from commit 0bcf65b)
Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
Signed-off-by: TianHao65 <Hao.Tian@amd.com>
Read swizzled BF16 weights directly for N16/K4096 and M1..32 during decode. Keep hipBLASLt for unsupported shapes, bias, prefill and other phases. Allow an optional offline BF16 solution table for the measured runtime.

Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
Support explicit output strides and reject aliasing or overlapping buffers. Reuse the same shape and dtype guard in apply and apply_into. Write directly to eligible caller-owned BF16 outputs during decode, avoiding a temporary allocation and copy.

Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
@TianHao65
TianHao65 changed the base branch from qwen4-pr8-port-base to qwen4_v0.5.21 November 3, 2026 07:34
@TianHao65
TianHao65 force-pushed the qwen4-pr8-decode-validated-20261028 branch from 7ec66bc to c68affd Compare November 3, 2026 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant