Repository navigation
[Perf] [AMD] Qwen3.5/Qwen3.8: Opt-in AITER FlyDSL causal conv1d prefill - #41879
amd-nprotaso wants to merge 15 commits into
Conversation
Co-Authored-By: Claude <noreply@anthropic.com>
…10 image with patches Image: lmsysorg/sglang-rocm:v0.5.21-rocm10-mi35x-20261014 (pinned by digest). The recipe's setup_script patches its editable AITER (e7d2453f) and SGLang (bab9c81f) checkouts before the server starts: - ROCm/aiter#5992: FlyDSL causal conv1d prefill and GDN decode/verify - sgl-project/sglang#41879, ported to bab9c81f: opt-in SGLANG_AITER_GDN_CONV/_DECODE hooks - fused Triton QK Gemma-RMSNorm + M-RoPE + gate on HIP for Qwen3.5/Qwen3.8 A patch that no longer applies fails the worker. Local check on MI355X (TP1, recipe server args at conc 16, CI's lm-eval GSM8K, 5-shot, full test set): 0.9636 with GDN conv on and GDN decode off, as in the recipe; 0.9697 with GDN decode on too. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Hi, @amd-nprotaso I am a bit confused by the decode part change. I am not sure the decode kernel will be called or not? # gdn_triton.py
supports_packed_decode = not is_cpu() and not is_npu()
# gdn_backend.py
# early return
if self.kernel_dispatcher.supports_packed_decode:
core_attn_out = self.kernel_dispatcher.packed_decode(
......
)
return
# your change is inside this decode()
core_attn_out = self.kernel_dispatcher.decode(
......
)Also, the current results look kernel-level only, so we cannot tell whether sglang actually runs these kernels. Could we add an e2e test? We may see improvement on TTFT for |
|
Hi, @amd-nprotaso Some other notes:
Thanks. |
This comment has been minimized.
This comment has been minimized.
Resolve conflict in gdn_backend.py: route the FlyDSL prefill conv dispatch through upstream's new _convolve_prefill helper, keeping the Triton-only block-table launch on causal_conv1d_fn. Co-Authored-By: Claude <noreply@anthropic.com>
Correct. I move it to TritonGDNKernel.packed_decode, so new kernels, will be use even for packed scenario. End-to-end gains are small (1–2%), for conc 16, ISL=32k OSL=32k |
|
Hi, @amd-nprotaso I mentioned this in previous comments. Maybe overlook. XD! Could we add a table for e2e perf in PR head? Could Thanks. |
I measure again with updated SGLang:
And see Interactivity degradation for conc=12, so maybe will close this PR. |
|
All pr-test-base passed. |
End-to-end A/B on MI355X showed no decode-side gain from SGLANG_AITER_GDN_DECODE, and it only ran with a bf16 SSM state, which is not the model default. The PR now carries just the opt-in FlyDSL prefill causal conv (SGLANG_AITER_GDN_CONV). Co-Authored-By: Claude <noreply@anthropic.com>
Updated PR, now it's only about convolution kernel. Add e2e results for different scenarios. |
|
Unofficial InferenceX run: https://inferencex.semianalysis.com/inference/deepseek-v41-flash?unofficialRun=42050314712 |
|
E2E performance on InferenceX agentixX benchmark, c16
|
|
Hi @amd-nprotaso . There's already Fine to skip if you'd rather not. |
Added unit test, better to have |
chuyeh
left a comment
There was a problem hiding this comment.
LGTM. This adds an opt-in AITER FlyDSL causal conv1d for GDN prefill on gfx950 (Qwen3.5/Qwen3.8), with Triton as the fallback.
|
@ChangLiu0709 The InferenceX result seems to combine many PRs, so it cannot show how much performance gain this PR brings. Please remove it, or replace it with a correct InferenceX result that includes only PR #41879. c.c @amd-nprotaso |
|
@amd-nprotaso Please resolve conflict issue, code change LGTM. |
This comment has been minimized.
This comment has been minimized.
Fresh mainline results for conc 16 and 48:
|
Requires AITER with ROCm/aiter#5992 (merged), which adds
causal_conv1d_prefill_flydsl_fn.Motivation
On ROCm, the GDN layers of Qwen3.5/Qwen3.8 run the prefill causal conv1d through Triton. AITER has a FlyDSL kernel for it on gfx950 that is 1.1x to 1.8x faster. This PR adds it behind an opt-in flag. Triton stays the default and the fallback.
Modifications
layers/attention/linear/gdn_backend.pyonly:_prefill_causal_conv1d. WithSGLANG_USE_AITER=1andSGLANG_AITER_GDN_CONV=1it calls AITER'scausal_conv1d_prefill_flydsl_fn, which has the same contract ascausal_conv1d_fn: one output,conv_statesupdated in place.ValueError(unsupported shape or dtype) falls back to Tritoncausal_conv1d_fn. When the flags are off or the import fails, that choice is cached and Triton is used.conv_block_tableset) stay on Triton.Nothing changes on CUDA or when the flags are unset.
Accuracy Tests
Kernel level, on Qwen3.8 shapes, against Triton
causal_conv1d_fn:conv_states: bit-identical.Speed Tests and Profiling
MI355X (gfx950), BF16, Qwen3.8 TP1 shapes (dim 10240, width 4), eager timing:
End to end
Setup:
amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8.E2E performance on InferenceX agentixX benchmark, c16
aiperf profile
CI States
Latest PR Test (Base): ✅ Run #43865775653
Latest PR Test (Extra): ❌ Run #43865775433
Latest PR Test (AMD ROCm 10): ❌ Run #43865775510
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: