Skip to content

[ROCm] Add guarded swizzled GEMV and direct outputs for Qwen3.5 Decode - #72

Closed
TianHao65 wants to merge 3 commits into
apinge:qwen4_v0.5.21from
TianHao65:qwen4-pr8-decode-optimization
Closed

TianHao65 wants to merge 3 commits into
apinge:qwen4_v0.5.21from
TianHao65:qwen4-pr8-decode-optimization

Conversation

@TianHao65

@TianHao65 TianHao65 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

What

Adds a guarded swizzled BF16 GEMV path for the small in_proj_ba linear during ordinary Qwen3.5 Decode, and a direct caller-owned output API for eligible uses.

Depends on #63. Rebased onto apinge:qwen4_v0.5.21 at c0a27e481010cd2f285fe426eed35860ae1920a5. CI PR65 is merged into this target; all separately cherry-picked CI and stacked-CI compatibility commits were removed. This branch retains the rebased prerequisite f5ac45b7da807ed3965633518111b40ba8517dfc: three commits / eight files, with the two Decode commits changing six files.

Review the Decode-only diff. Recheck/rebase the stack when #63 lands, particularly if it is squash-merged.

Commit Source / purpose Change
d6d2fab540 Decode GEMV Read the existing swizzled BF16 weights directly for N16/K4096, M1–32; scope activation to Qwen3.5 ordinary Decode.
829fc2a707 Caller-owned output Reuse the guard in apply/apply_into, support valid output strides and write directly into the supplied buffer.

All three relevant flags default off. Enable SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB=1, SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE=1 and SGLANG_ROCM_HIPB_DECODE_GEMV=1 together. The guard requires BF16 inputs/weights, the expected swizzled layout, N16/K4096, M1–32, no bias and ordinary Decode. Other shapes/phases use the existing dispatch. Prefill, verification and mixed modes do not activate this GEMV path. SGLANG_ROCM_HIPB_TUNED_SOLUTIONS stays empty in these measurements; offline solution IDs are runtime-specific.

Output-buffer scope: the current Qwen3.5 BA projection is MergedColumnParallelLinear and calls allocating apply. The direct apply_into gain below applies when the caller actually supplies eligible output storage. The existing model traces establish the allocating Decode GEMV switch; they do not establish an additional model-level copy-removal gain from the second commit.

Data

Historical measurements from 2026-10-26 on Qwen3.5-397B-A17B-PTPC-FP8, TP8, 8× MI308X, Torch 2.9.1+rocm7.2.0.git7e1940d4, AITER acf8fdf93074. Pure PR8 ddccab6e0d → GEMV e0e3b3271a → direct-output API a8cd545783.

AITER attention/MoE, Triton GDN, FP8 KV, page1/NHD, fixed token capacity1310720, maxrequests256/Mamba512, radix/speculation disabled. Source manifests and actual imports were pinned. Offline solution table disabled.

8000 input / 500 output, concurrency1

Two completed independent starts per variant, 24 measured requests per start, 4-request warmup discarded. Orders: port/GEMV/direct, then direct/GEMV/port. All formal runs exceed60s with token-ID submission and exact recorded token lengths, expected completion and zero request errors. Entries are the mean of the two starts, with the observed min–max in parentheses; the range is not a confidence interval. The scheduled third matched set is incomplete: an extra GEMV-only formal round completed, but port/direct do not form a third pair, so it is excluded from the matched means.

Metric PR8 port +GEMV +Direct-output API Final vs port
Output tok/s 112.002 (111.173–112.831) 114.180 (113.254–115.106) 114.147 (113.275–115.020) +1.92%
TTFT ms 324.707 (323.906–325.508) 325.002 (323.518–326.486) 324.830 (323.166–326.494) +0.04%
TPOT ms 8.293 (8.228–8.357) 8.122 (8.054–8.190) 8.125 (8.061–8.188) -2.03%
ETE ms 4462.879 (4429.861–4495.897) 4377.794 (4342.255–4413.332) 4378.982 (4345.433–4412.530) -1.88%

The completed two-start sample shows about 1.9% higher output throughput and1.9% lower ETE for the final branch. The ordinary model uses allocating GEMV in both optimized variants; their close means do not demonstrate a model benefit from the optional output-buffer API. This is limited to the recorded runtime/configuration, with incomplete three-start coverage.

512 input / 256 output, first-start concurrency check

One completed start per side; 512/768/1024 measured requests at concurrency16/64/128. Warmup requests32/128/256 were discarded. All runs exceed60s and complete with exact lengths and zero request errors. M>32 falls back; larger concurrency does not imply this GEMV is active throughout the run.

Concurrency tok/s port → final TTFT ms port → final TPOT ms port → final ETE ms port → final
16 1047.026 → 1058.028 (+1.05%) 491.398 → 485.391 (-1.22%) 13.393 → 13.259 (-1.00%) 3906.682 → 3866.399 (-1.03%)
64 2041.695 → 2041.872 (+0.01%) 1344.154 → 1358.780 (+1.09%) 26.135 → 26.077 (-0.22%) 8008.550 → 8008.298 (-0.00%)
128 2906.663 → 2914.771 (+0.28%) 2058.285 → 2049.792 (-0.41%) 35.952 → 35.924 (-0.08%) 11226.151 → 11210.286 (-0.14%)

Actual single-operator entrypoints, N16/K4096

MI308X, BF16; actual UnquantizedLinearMethod.apply/apply_into, 50 calls per captured graph, 100 warm replays and median of5GPU-event samples. Units:µs/call. These measurements are separate from model ETE.

M Port apply GEMV apply Port apply_into GEMV + copy apply_into Direct apply_into Direct vs port apply_into
1 5.792 2.309 7.814 6.700 2.337 −70.09%
8 5.791 2.860 7.885 4.914 2.896 −63.27%
16 5.665 3.542 7.895 5.521 3.490 −55.79%
32 5.898 4.852 8.204 6.924 4.889 −40.41%

Benchmarking and Profiling

Four screenshots open the complete, untouched original TP0 model traces from the port/direct first-start pair. Profiling uses8000 input/500 output/concurrency1, configured33forward steps; the originals retain CPU EXTEND8000 and32Decode annotations and all captured GPU prefill/Decode events. No events are extracted, translated, relabeled or reconstructed.

The selected BF16 BA projection is the 30th GDN call (layer 38). Prefill is matched through original CPU shapes [8000,4096] × [4096,16], external ID and GPU launch correlation. Decode is matched through the original stream7 kernel sequence and six identical neighboring kernels;45eligible BF16 calls change to _swizzled_gemv per captured Decode pass.

Native blue Area Selection/duration and Current Selection > Slices are visible. Before/after use the same scale within each phase:100µs Prefill,40µs Decode. The native list includes overlapping enclosing phase annotations; do not add those clipped wall durations to GPU kernel duration. Both untouched files yield 36 overlapping-complete-event diagnostics in Perfetto v59.1; original overlap conditions and events are preserved, and selected timestamps/durations match the original JSON.

Prefill

Selected GPU call: 25.559 → 25.600µs. The prefill hipBLASLt path is unchanged. These are single-call samples, not aggregate stage time.

Before:

Prefill before, complete original trace

Open complete original trace in Perfetto · SHA256 and selection manifest

After:

Prefill after, complete original trace

Open complete original trace in Perfetto · SHA256 and selection manifest

Decode

Selected GPU call: 7.519 → 4.759µs. The Decode call switches from hipBLASLt to allocating swizzled GEMV. These are single-call samples, not aggregate stage time.

Before:

Decode before, complete original trace

Open complete original trace in Perfetto · SHA256 and selection manifest

After:

Decode after, complete original trace

Open complete original trace in Perfetto · SHA256 and selection manifest

Validation

Current rebase checks passed: preserved feature patches/model code, 21 CI driver tests, 10 evaluation shell syntax checks, Python syntax and clean diff. No fresh GPU/model measurements were run.

  • All 12 shape/version cases (24 apply/apply_into outputs) pass the original FP32-reference checks; maximum relative L2≈0.001653. Additional 15 GPU output-layout/alias/graph checks and 12 CPU interpreter layout checks pass for the direct-output API.
  • Unsupported shapes/phases, bias and dtype retain their existing guards. This rebase preserves the legacy output-buffer implementation; the overlapping DLPack storage guard is corrected in PR76. Prior layout checks do not establish that correction for this PR.
  • Complete original model traces show port0 → final 45 swizzled GEMV calls, BF16 hipBLASLt210 → 165 calls, while FP8 hipBLASLt stays 240 calls in the captured workload. Source/callsite inspection separates allocating model execution from the optional output-buffer microbenchmark.
  • Formal requests, actual lengths/concurrency, source manifests and recorded owned process/port/VRAM cleanup were checked. No interrupted third-start data is included in the means. No fresh GPU experiment was started solely to fill this Draft template.
  • Model quality remains an open limitation: short semantic probes are abnormal in the port, GEMV and direct variants of this 397B reference configuration. Numerical kernel checks and fixed-length performance completion do not establish model-level quality. Quality investigation is separate; these results do not assert all-Qwen performance/quality coverage.
  • Source remains8files against the unmerged target branch, including the unchanged [ROCm] Port PR #8 hipBLASLt linears to qwen4_v0.5.21 #63 prerequisite. PR63 remains the pure port. Larger router prototypes and unrelated optimizations are excluded.

Original full traces, figures and hashes.

@apinge
apinge requested a review from sammysun0711 October 28, 2026 06:54
TianHao65 and others added 3 commits November 3, 2026 15:29
Port apinge/sglang PR sgl-project#8 (a6f5d30)
using the previously tested v0.5.21 adaptation.

Route PTPC-FP8 and unquantized BF16 linears through AITER hipb_mm behind
the existing swizzle flags. Track weight layouts per Parameter and retain
fallback dispatch for weights that fail the swizzle shape guard.

Handle v0.5.21 prequantized three-element tuples and leading dimensions,
and support BF16 caller-owned output buffers.

(cherry picked from commit 0bcf65b)
Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
Signed-off-by: TianHao65 <Hao.Tian@amd.com>
Read swizzled BF16 weights directly for N16/K4096 and M1..32 during decode. Keep hipBLASLt for unsupported shapes, bias, prefill and other phases. Allow an optional offline BF16 solution table for the measured runtime.

Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
Support explicit output strides and reject aliasing or overlapping buffers. Reuse the same shape and dtype guard in apply and apply_into. Write directly to eligible caller-owned BF16 outputs during decode, avoiding a temporary allocation and copy.

Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
@TianHao65
TianHao65 force-pushed the qwen4-pr8-decode-optimization branch from a8cd545 to 829fc2a Compare November 3, 2026 07:34
@TianHao65 TianHao65 closed this Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant