Skip to content

[AMD] Run Qwen3.5 GDN Triton decode on strided projection views - #43366

Closed
vorapolsiloai wants to merge 1 commit into
sgl-project:mainfrom
vorapolsiloai:gdn-decode-projection-views-hip
Closed

vorapolsiloai wants to merge 1 commit into
sgl-project:mainfrom
vorapolsiloai:gdn-decode-projection-views-hip

Conversation

@vorapolsiloai

@vorapolsiloai vorapolsiloai commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

On ROCm, batched Qwen3.5/3.6/3.8 GDN decode still materializes the projection outputs before the recurrence:

  • v/k head ratios in _GDN_FUSED_QKVZBA_RATIOS ((1, 2, 4, 8) with aiter) launch fused_qkvzba_split_reshape_cat_contiguous_kernel per GDN layer per step;
  • other ratios, such as the ratio-3 dense 27B layout (16 k-heads / 48 v-heads), fall back to torch.split + two .contiguous() copies + torch.cat.

On the contiguous Qwen3.5 layout (qkvz = [q|k|v|z], ba = [b|a]) this split is an identity copy: each output is a column slice of a projection. Every decode consumer downstream already takes row strides:

  • causal_conv1d_update (Triton) reads x.stride() and writes a fresh contiguous output;
  • fused_recurrent_gated_delta_rule_packed_decode and the GDN ReplaySSM decode kernel take stride_a_tok / stride_b_tok;
  • the gated RMSNorm (_layer_norm_fwd) reads a 3D z through stride_z_token / stride_z_head, in both the plain and the per-token FP8 producer variants.

CUDA prefill already relies on this through qwen3_5_gdn_prefill_projection_views. This PR applies the same zero-copy views to batched Triton decode.

Modifications

python/sglang/srt/models/qwen3_5.py:

  • _gdn_decode_reads_projection_views() holds when the resolved linear-attention decode backend is triton on CUDA or HIP. It is evaluated once in Qwen3_5GatedDeltaNet.__init__.
  • In decode, the layer takes qwen3_5_gdn_prefill_projection_views for every head ratio. Views need no head-group kernel, so ratios outside _GDN_FUSED_QKVZBA_RATIOS are covered as well. use_strided_prefill_z becomes use_projection_views, which covers both the existing CUDA prefill case and this decode case.
  • The fused decode projection+Conv1D path (SGLANG_ENABLE_GDN_DECODE_FUSED_PROJ_CONV) keeps priority whenever it is enabled. Default CUDA decode is therefore unchanged. On CUDA, this path runs only with that fusion disabled.
  • The small-M FP8 out_proj producer (_fp8_tuple_input, gfx950 AttnFP8) now counts tokens from z_flat_shape. It keeps working with the 3D strided z instead of being skipped. On CUDA this is a no-op, because _fp8_tuple_input requires aiter.

python/sglang/kernels/ops/attention/triton_gdn_fused_proj.py: docstring only.

Not changed: CPU, NPU, XPU, FlashInfer/CuTe DSL decode backends, prefill, extend and target-verify.

Relation to open PRs

Accuracy Tests

The change is a pure layout change; outputs are expected to be bit-identical.

  • New: TestGdnDecodeOnProjectionViews in test/registered/attention/test_gdn_noncontiguous_stride.py (CUDA + AMD CI). It runs the full decode chain, from Conv1D update through packed recurrence to the gated RMSNorm, on views and on the split-kernel outputs. It asserts torch.equal on the outputs, the Conv1D state and the SSM state, for ratio 3 (27B TP2: 8 k-heads / 24 v-heads, batch 7) and ratio 2.
  • New: test_fused_quant_reads_projection_view_z in test/registered/amd/test_smallm_fp8_proj_gfx950.py. It checks that the gfx950 FP8 producer is bit-exact on a row-strided 3D z against a contiguous z.
  • Run both on MI355X and in CUDA CI (pending)
  • GSM8K / GPQA on amd/Qwen3.8-27B-Quark-AWQ-MXFP4 TP2 MI355X, main vs this PR (pending)

A downstream MI355X stack carrying this change alongside other changes was compared with its control on Qwen3.8-27B TP2. Evals were unchanged within noise: GSM8K gate 0.955 vs 0.955, GPQA 0.803 vs 0.813 (stack first, control second; one standard error at 198 questions is about 2.8 points).

Speed Tests and Profiling

Downstream measurement on MI355X: amd/Qwen3.8-27B-Quark-AWQ-MXFP4, TP2, aiter attention, ratio-3 GDN layout, 5 serving rows (8K/1K c4 and c32, 128K/1K c4, two image rows).

  • Views alone, against a control without the ratio-3 aiter whitelist entry (so control decode used the split/contiguous/cat fallback): +5.2% output-throughput geomean.
  • Combined with other changes, a TraceLens breakdown shows the decode copy/cat launches dropping from 204 to 12 per step: direct_copy 145 -> 1 and CatArrayBatchedCopy 48 -> 0. That saves about 0.83-0.86 ms per decode step at 8K c4/c32.

These numbers come from a downstream tree, not from this branch on main. Still to do before review:

Checklist


CI States

Latest PR Test (Base): ❌ Missing run-ci label -- add it to run CI tests.
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

Batched GDN decode on ROCm still launches the qkvzba split kernel (or, for
v/k head ratios outside _GDN_FUSED_QKVZBA_RATIOS such as the ratio-3 dense
27B layout, two .contiguous() copies plus a torch.cat) just to hand the
Conv1D update, the packed recurrence and the gated RMSNorm contiguous
tensors. All three Triton consumers already take row strides, and on the
contiguous Qwen3.5 layout the split is an identity copy, so decode can read
mixed_qkv, z, b and a in place from the projection outputs.

- qwen3_5.py: when the decode backend is Triton on CUDA/HIP, decode reuses
  qwen3_5_gdn_prefill_projection_views (the zero-copy helper CUDA prefill
  already uses) for any head ratio. The fused decode projection+Conv1D path
  keeps priority when enabled, so default CUDA decode is unchanged.
- The small-M FP8 out_proj producer now counts tokens from z_flat_shape, so
  it keeps working with the 3D strided z instead of being skipped.
- Tests: bit-exact decode chain (Conv1D update, packed decode, gated norm,
  conv/ssm state) on views vs the split kernel for ratio 3 (27B TP2) and
  ratio 2; the gfx950 FP8 producer on a strided 3D z.
@sushildubey171

Copy link
Copy Markdown
Contributor

This PR is duplicate, there was already opened PR for this, please refer #41677

@vorapolsiloai

Copy link
Copy Markdown
Contributor Author

Thanks. Agreed that they overlap. As the description says, this was opened as an alternative to #37113 / #41677 for the HIP decode split, and it becomes the fallback once the fused decode projection+Conv1D kernel is enabled on HIP. On our Qwen3.8-27B MI355X/MI350X stacks, with your gate applied, decode takes the fused kernel at TP1 and TP2, and the view path here is never reached. So I'll close this in favour of #41677.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants