Repository navigation
Conversation
Port apinge/sglang PR sgl-project#8 (a6f5d30) using the previously tested v0.5.21 adaptation. Route PTPC-FP8 and unquantized BF16 linears through AITER hipb_mm behind the existing swizzle flags. Track weight layouts per Parameter and retain fallback dispatch for weights that fail the swizzle shape guard. Handle v0.5.21 prequantized three-element tuples and leading dimensions, and support BF16 caller-owned output buffers. (cherry picked from commit 0bcf65b) Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com> Signed-off-by: TianHao65 <Hao.Tian@amd.com>
Read swizzled BF16 weights directly for N16/K4096 and M1..32 during decode. Keep hipBLASLt for unsupported shapes, bias, prefill and other phases. Allow an optional offline BF16 solution table for the measured runtime. Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
Support explicit output strides and reject aliasing or overlapping buffers. Reuse the same shape and dtype guard in apply and apply_into. Write directly to eligible caller-owned BF16 outputs during decode, avoiding a temporary allocation and copy. Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
TianHao65
force-pushed
the
qwen4-pr8-decode-optimization
branch
from
November 3, 2026 07:34
a8cd545 to
829fc2a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a guarded swizzled BF16 GEMV path for the small
in_proj_balinear during ordinary Qwen3.5 Decode, and a direct caller-owned output API for eligible uses.Depends on #63. Rebased onto
apinge:qwen4_v0.5.21atc0a27e481010cd2f285fe426eed35860ae1920a5. CI PR65 is merged into this target; all separately cherry-picked CI and stacked-CI compatibility commits were removed. This branch retains the rebased prerequisitef5ac45b7da807ed3965633518111b40ba8517dfc: three commits / eight files, with the two Decode commits changing six files.Review the Decode-only diff. Recheck/rebase the stack when #63 lands, particularly if it is squash-merged.
d6d2fab540829fc2a707apply/apply_into, support valid output strides and write directly into the supplied buffer.All three relevant flags default off. Enable
SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB=1,SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE=1andSGLANG_ROCM_HIPB_DECODE_GEMV=1together. The guard requires BF16 inputs/weights, the expected swizzled layout, N16/K4096, M1–32, no bias and ordinary Decode. Other shapes/phases use the existing dispatch. Prefill, verification and mixed modes do not activate this GEMV path.SGLANG_ROCM_HIPB_TUNED_SOLUTIONSstays empty in these measurements; offline solution IDs are runtime-specific.Output-buffer scope: the current Qwen3.5 BA projection is
MergedColumnParallelLinearand calls allocatingapply. The directapply_intogain below applies when the caller actually supplies eligible output storage. The existing model traces establish the allocating Decode GEMV switch; they do not establish an additional model-level copy-removal gain from the second commit.Data
Historical measurements from 2026-10-26 on Qwen3.5-397B-A17B-PTPC-FP8, TP8, 8× MI308X, Torch
2.9.1+rocm7.2.0.git7e1940d4, AITERacf8fdf93074. Pure PR8ddccab6e0d→ GEMVe0e3b3271a→ direct-output APIa8cd545783.AITER attention/MoE, Triton GDN, FP8 KV, page1/NHD, fixed token capacity1310720, maxrequests256/Mamba512, radix/speculation disabled. Source manifests and actual imports were pinned. Offline solution table disabled.
8000 input / 500 output, concurrency1
Two completed independent starts per variant, 24 measured requests per start, 4-request warmup discarded. Orders: port/GEMV/direct, then direct/GEMV/port. All formal runs exceed60s with token-ID submission and exact recorded token lengths, expected completion and zero request errors. Entries are the mean of the two starts, with the observed min–max in parentheses; the range is not a confidence interval. The scheduled third matched set is incomplete: an extra GEMV-only formal round completed, but port/direct do not form a third pair, so it is excluded from the matched means.
The completed two-start sample shows about 1.9% higher output throughput and1.9% lower ETE for the final branch. The ordinary model uses allocating GEMV in both optimized variants; their close means do not demonstrate a model benefit from the optional output-buffer API. This is limited to the recorded runtime/configuration, with incomplete three-start coverage.
512 input / 256 output, first-start concurrency check
One completed start per side; 512/768/1024 measured requests at concurrency16/64/128. Warmup requests32/128/256 were discarded. All runs exceed60s and complete with exact lengths and zero request errors. M>32 falls back; larger concurrency does not imply this GEMV is active throughout the run.
Actual single-operator entrypoints, N16/K4096
MI308X, BF16; actual
UnquantizedLinearMethod.apply/apply_into, 50 calls per captured graph, 100 warm replays and median of5GPU-event samples. Units:µs/call. These measurements are separate from model ETE.Benchmarking and Profiling
Four screenshots open the complete, untouched original TP0 model traces from the port/direct first-start pair. Profiling uses8000 input/500 output/concurrency1, configured33forward steps; the originals retain CPU EXTEND8000 and32Decode annotations and all captured GPU prefill/Decode events. No events are extracted, translated, relabeled or reconstructed.
The selected BF16 BA projection is the 30th GDN call (layer 38). Prefill is matched through original CPU shapes
[8000,4096] × [4096,16], external ID and GPU launch correlation. Decode is matched through the original stream7 kernel sequence and six identical neighboring kernels;45eligible BF16 calls change to_swizzled_gemvper captured Decode pass.Native blue Area Selection/duration and
Current Selection > Slicesare visible. Before/after use the same scale within each phase:100µs Prefill,40µs Decode. The native list includes overlapping enclosing phase annotations; do not add those clipped wall durations to GPU kernel duration. Both untouched files yield 36 overlapping-complete-event diagnostics in Perfetto v59.1; original overlap conditions and events are preserved, and selected timestamps/durations match the original JSON.Prefill
Selected GPU call: 25.559 → 25.600µs. The prefill hipBLASLt path is unchanged. These are single-call samples, not aggregate stage time.
Before:
Open complete original trace in Perfetto · SHA256 and selection manifest
After:
Open complete original trace in Perfetto · SHA256 and selection manifest
Decode
Selected GPU call: 7.519 → 4.759µs. The Decode call switches from hipBLASLt to allocating swizzled GEMV. These are single-call samples, not aggregate stage time.
Before:
Open complete original trace in Perfetto · SHA256 and selection manifest
After:
Open complete original trace in Perfetto · SHA256 and selection manifest
Validation
Current rebase checks passed: preserved feature patches/model code, 21 CI driver tests, 10 evaluation shell syntax checks, Python syntax and clean diff. No fresh GPU/model measurements were run.
Original full traces, figures and hashes.