Repository navigation
[AMD] Run Qwen3.5 GDN Triton decode on strided projection views - #43366
Closed
vorapolsiloai wants to merge 1 commit into
Closed
vorapolsiloai wants to merge 1 commit into
vorapolsiloai wants to merge 1 commit into
Conversation
Batched GDN decode on ROCm still launches the qkvzba split kernel (or, for v/k head ratios outside _GDN_FUSED_QKVZBA_RATIOS such as the ratio-3 dense 27B layout, two .contiguous() copies plus a torch.cat) just to hand the Conv1D update, the packed recurrence and the gated RMSNorm contiguous tensors. All three Triton consumers already take row strides, and on the contiguous Qwen3.5 layout the split is an identity copy, so decode can read mixed_qkv, z, b and a in place from the projection outputs. - qwen3_5.py: when the decode backend is Triton on CUDA/HIP, decode reuses qwen3_5_gdn_prefill_projection_views (the zero-copy helper CUDA prefill already uses) for any head ratio. The fused decode projection+Conv1D path keeps priority when enabled, so default CUDA decode is unchanged. - The small-M FP8 out_proj producer now counts tokens from z_flat_shape, so it keeps working with the 3D strided z instead of being skipped. - Tests: bit-exact decode chain (Conv1D update, packed decode, gated norm, conv/ssm state) on views vs the split kernel for ratio 3 (27B TP2) and ratio 2; the gfx950 FP8 producer on a strided 3D z.
3 of 5 tasks
vorapolsiloai
force-pushed
the
gdn-decode-projection-views-hip
branch
from
October 12, 2026 06:49
7825503 to
ac6f70f
Compare
Contributor
|
This PR is duplicate, there was already opened PR for this, please refer #41677 |
Contributor
Author
|
Thanks. Agreed that they overlap. As the description says, this was opened as an alternative to #37113 / #41677 for the HIP decode split, and it becomes the fallback once the fused decode projection+Conv1D kernel is enabled on HIP. On our Qwen3.8-27B MI355X/MI350X stacks, with your gate applied, decode takes the fused kernel at TP1 and TP2, and the view path here is never reached. So I'll close this in favour of #41677. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On ROCm, batched Qwen3.5/3.6/3.8 GDN decode still materializes the projection outputs before the recurrence:
_GDN_FUSED_QKVZBA_RATIOS((1, 2, 4, 8)with aiter) launchfused_qkvzba_split_reshape_cat_contiguous_kernelper GDN layer per step;torch.split+ two.contiguous()copies +torch.cat.On the contiguous Qwen3.5 layout (
qkvz = [q|k|v|z],ba = [b|a]) this split is an identity copy: each output is a column slice of a projection. Every decode consumer downstream already takes row strides:causal_conv1d_update(Triton) readsx.stride()and writes a fresh contiguous output;fused_recurrent_gated_delta_rule_packed_decodeand the GDN ReplaySSM decode kernel takestride_a_tok/stride_b_tok;_layer_norm_fwd) reads a 3Dzthroughstride_z_token/stride_z_head, in both the plain and the per-token FP8 producer variants.CUDA prefill already relies on this through
qwen3_5_gdn_prefill_projection_views. This PR applies the same zero-copy views to batched Triton decode.Modifications
python/sglang/srt/models/qwen3_5.py:_gdn_decode_reads_projection_views()holds when the resolved linear-attention decode backend istritonon CUDA or HIP. It is evaluated once inQwen3_5GatedDeltaNet.__init__.qwen3_5_gdn_prefill_projection_viewsfor every head ratio. Views need no head-group kernel, so ratios outside_GDN_FUSED_QKVZBA_RATIOSare covered as well.use_strided_prefill_zbecomesuse_projection_views, which covers both the existing CUDA prefill case and this decode case.SGLANG_ENABLE_GDN_DECODE_FUSED_PROJ_CONV) keeps priority whenever it is enabled. Default CUDA decode is therefore unchanged. On CUDA, this path runs only with that fusion disabled.out_projproducer (_fp8_tuple_input, gfx950 AttnFP8) now counts tokens fromz_flat_shape. It keeps working with the 3D stridedzinstead of being skipped. On CUDA this is a no-op, because_fp8_tuple_inputrequires aiter.python/sglang/kernels/ops/attention/triton_gdn_fused_proj.py: docstring only.Not changed: CPU, NPU, XPU, FlashInfer/CuTe DSL decode backends, prefill, extend and target-verify.
Relation to open PRs
SGLANG_ENABLE_GDN_DECODE_FUSED_PROJ_CONV=0. A head-to-head on MI355X is listed under Speed Tests below. Our earlier screen of [AMD] Enable the GDN decode projection/Conv1D fusion on ROCm #37113 on Qwen3.8-27B TP2 MI355X was inconclusive (+0.5% / -0.4%), while the views variant measured +5% on the same setup.shape[0] == 1). It avoids batched views becausez.reshape(-1, head_v)would insert a copy. This PR keeps the 3Dz(as CUDA prefill does), so batched decode stays zero-copy.V_POW2, Qwen3.8-27B Model Support #34859), while the aiter whitelist still omits 3. This PR is orthogonal: it skips the split in decode for any ratio. Prefill and extend still need the kernel or whitelist fix.Accuracy Tests
The change is a pure layout change; outputs are expected to be bit-identical.
TestGdnDecodeOnProjectionViewsintest/registered/attention/test_gdn_noncontiguous_stride.py(CUDA + AMD CI). It runs the full decode chain, from Conv1D update through packed recurrence to the gated RMSNorm, on views and on the split-kernel outputs. It assertstorch.equalon the outputs, the Conv1D state and the SSM state, for ratio 3 (27B TP2: 8 k-heads / 24 v-heads, batch 7) and ratio 2.test_fused_quant_reads_projection_view_zintest/registered/amd/test_smallm_fp8_proj_gfx950.py. It checks that the gfx950 FP8 producer is bit-exact on a row-strided 3Dzagainst a contiguousz.amd/Qwen3.8-27B-Quark-AWQ-MXFP4TP2 MI355X, main vs this PR (pending)A downstream MI355X stack carrying this change alongside other changes was compared with its control on Qwen3.8-27B TP2. Evals were unchanged within noise: GSM8K gate 0.955 vs 0.955, GPQA 0.803 vs 0.813 (stack first, control second; one standard error at 198 questions is about 2.8 points).
Speed Tests and Profiling
Downstream measurement on MI355X:
amd/Qwen3.8-27B-Quark-AWQ-MXFP4, TP2, aiter attention, ratio-3 GDN layout, 5 serving rows (8K/1K c4 and c32, 128K/1K c4, two image rows).direct_copy145 -> 1 andCatArrayBatchedCopy48 -> 0. That saves about 0.83-0.86 ms per decode step at 8K c4/c32.These numbers come from a downstream tree, not from this branch on main. Still to do before review:
SGLANG_ENABLE_GDN_DECODE_FUSED_PROJ_CONVon vs views)Checklist
CI States
Latest PR Test (Base): ❌ Missing
run-cilabel -- add it to run CI tests.Latest PR Test (Extra): ❌ Blocked --
run-ciis required first.Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.