Repository navigation
Conversation
TianHao65
force-pushed
the
qwen4-pr8-decode-validated-20261028
branch
from
November 2, 2026 06:58
176150c to
71f9a0e
Compare
TianHao65
force-pushed
the
qwen4-pr8-port-base
branch
from
November 2, 2026 06:59
ddccab6 to
948bd95
Compare
TianHao65
force-pushed
the
qwen4-pr8-decode-validated-20261028
branch
from
November 2, 2026 07:11
71f9a0e to
7ec66bc
Compare
TianHao65
force-pushed
the
qwen4-pr8-port-base
branch
from
November 2, 2026 07:12
948bd95 to
affd9cb
Compare
TianHao65
force-pushed
the
qwen4-pr8-decode-validated-20261028
branch
from
November 2, 2026 08:42
7ec66bc to
bd2e055
Compare
TianHao65
force-pushed
the
qwen4-pr8-port-base
branch
from
November 2, 2026 08:42
affd9cb to
e6ff953
Compare
TianHao65
force-pushed
the
qwen4-pr8-decode-validated-20261028
branch
from
November 3, 2026 02:16
bd2e055 to
7ec66bc
Compare
TianHao65
force-pushed
the
qwen4-pr8-port-base
branch
from
November 3, 2026 02:16
e6ff953 to
affd9cb
Compare
Port apinge/sglang PR sgl-project#8 (a6f5d30) using the previously tested v0.5.21 adaptation. Route PTPC-FP8 and unquantized BF16 linears through AITER hipb_mm behind the existing swizzle flags. Track weight layouts per Parameter and retain fallback dispatch for weights that fail the swizzle shape guard. Handle v0.5.21 prequantized three-element tuples and leading dimensions, and support BF16 caller-owned output buffers. (cherry picked from commit 0bcf65b) Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com> Signed-off-by: TianHao65 <Hao.Tian@amd.com>
Read swizzled BF16 weights directly for N16/K4096 and M1..32 during decode. Keep hipBLASLt for unsupported shapes, bias, prefill and other phases. Allow an optional offline BF16 solution table for the measured runtime. Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
Support explicit output strides and reject aliasing or overlapping buffers. Reuse the same shape and dtype guard in apply and apply_into. Write directly to eligible caller-owned BF16 outputs during decode, avoiding a temporary allocation and copy. Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
TianHao65
force-pushed
the
qwen4-pr8-decode-validated-20261028
branch
from
November 3, 2026 07:34
7ec66bc to
c68affd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two Decode follow-ups to PR63, retaining its rebased prerequisite
f5ac45b7da807ed3965633518111b40ba8517dfc. Rebased ontoapinge:qwen4_v0.5.21atc0a27e481010cd2f285fe426eed35860ae1920a5. CI PR65 is merged into this target; all separately cherry-picked CI and stacked-CI compatibility commits were removed. The PR now targetsqwen4_v0.5.21directly so the merged CI matches its target. The comparison contains three commits / nine files: one prerequisite port and two Decode commits.Decode-only comparison.
446e570f0aN=16,K=4096,M=1..32; other shapes and prefill retain the existing path.c68affd18bapply_into()buffers, including the corrected overlapping DLPack storage guard.Enable GEMV with
SGLANG_ROCM_HIPB_DECODE_GEMV=1(off by default). Tuned solutions were disabled in these tests.Data
Historical measurements and original screenshots below were collected on 2026-10-28 using
ddccab6e0d/e0e3b3271a/176150ccc3. The optimization blobs are unchanged by this rebase; no new full-model measurements were run for these rebased commits.Qwen3.5-397B-A17B-PTPC-FP8, 8×MI308X / TP8, SGLang 0.5.21, Torch 2.9.1 / ROCm 7.2. Measured on 2026-10-28 UTC with AITER attention, Triton GDN, FP8 KV, and radix/speculative decoding disabled.
c1 uses three independent starts per version; c16/c64/c128 use one. Warmups were discarded, each formal round lasted ≥60s, and all 7,128 measured requests succeeded with matching actual lengths. Values are means; percentages compare with the pure port.
Concurrency 1: 8000 input / 500 output; concurrency 16/64/128: 512 / 256.
Pure port → GEMV
Pure port → GEMV + direct output
High-concurrency differences are single-start observations, including small regressions; full batches at c64/c128 use the
M>32fallback.Direct
apply_into()reduced M1 operator time from 5.490 to 2.865µs (−47.81% vs GEMV). This is an API benefit: the measured Qwen3.5 caller usesapply(), so no additional model benefit from direct output is established.Full results, per-start values, runtime configuration, and microbenchmarks.
Benchmarking and Profiling
Complete untouched model traces, rank 0, first
in_proj_basample: BF16 N16/K4096; prefill M8000, Decode M1/graph bs1. Layer 0 is inferred from execution order. These are individual kernel samples, separate from ETE.The four screenshots below use the original Perfetto views. Open the complete original traces using the links below each image; full SHA256 manifest.
Prefill
BF16 hipBLASLt fallback: 32.120 → 31.919µs, both at a 100µs scale.
Before
Open prefill before in Perfetto.
After
Open prefill after in Perfetto.
Decode
hipBLASLt → GEMV: 8.959 → 5.279µs, both at a 40µs scale.
Before
Open decode before in Perfetto.
After
Open decode after in Perfetto.
Validation
Current rebase checks: preserved feature patches/model code, author identities/dates, Python syntax and clean diff; 21 CI driver tests and 10 evaluation shell syntax checks passed. The original five CPU output-guard regressions /35 subtests are reused because the kernel and test blobs are unchanged. No new model or GPU measurements were run; the results below retain their measured revisions.
copy_. Source/syntax/diff checks passed.Validation reports · Final protocol/quality/resource audit.
CI execution scope: the merged workflow is inherited unchanged, and this PR now targets
qwen4_v0.5.21. It remains Draft; six-model GPU CI has not run for this rebased head.