Repository navigation
[Perf] attention: Enable tensor descriptors in the Triton extend-attention kernel - #46034
Closed
rahulvijayaraghavan wants to merge 1 commit into
Closed
rahulvijayaraghavan wants to merge 1 commit into
rahulvijayaraghavan wants to merge 1 commit into
Conversation
rahulvijayaraghavan
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
celve and
yuan-luo
as code owners
October 23, 2026 10:32
This comment has been minimized.
This comment has been minimized.
Contributor
|
/tag-run-ci-label |
Load the Q / extend-K / extend-V tiles through host-side descriptors, which lower to 2D block copies on XPU; the tensor-of-pointer path is unchanged and stays the default everywhere else.
rahulvijayaraghavan
force-pushed
the
xpu_extend_attention_tensor_desc_v2
branch
from
October 23, 2026 10:56
1407985 to
ccbbcb4
Compare
Contributor
Author
|
Superseded by #46041 (same commit, moved to a device-agnostic branch name). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #29639 (auto-closed as a stale draft; rebased onto current
main).Motivation
Triton's tensor descriptors let a kernel describe a tile by its shape and strides instead of computing a tensor of per-element pointers. Where the backend has a hardware fast path for descriptor loads (on NVIDIA, TMA on Hopper and newer), it can use it, and the per-element address arithmetic and boundary masks go away.
_fwd_kernelloads the Q / extend-K / extend-V tiles with tensor-of-pointer loads. Those tiles are contiguous rows of a(token, head, head_dim)buffer, so they can be described by a host-sideTensorDescriptorinstead.The host-side descriptors follow the existing MoE implementation (#10567). The approach matches the vLLM RFC on tensor descriptor adoption: vllm-project/vllm#42545
What this changes
The descriptor path is behind a tri-state env var. The tensor-of-pointer path is unchanged.
SGLANG_USE_TRITON_ATTN_TENSOR_DESC1/0Three files: the kernel (
extend_attention.py), the env var (environ.py), and a new A/B benchmark (bench_extend_attention_tdesc.py) that reports per-call time and numerical agreement.Descriptors are gated per tile, so a tensor whose layout doesn't qualify keeps the pointer path while the others still use descriptors.
Correctness
Descriptors on vs off, same build, same device. Outputs are asserted exactly equal (
atol=0, rtol=0), not within a tolerance, because both paths read the same tiles in the same order.max err 0.0e+00max err 0.0e+00Pooled across two implementations × two reps per platform, the on-vs-off GSM8K difference is -0.25 pp on CUDA and +0.22 pp on XPU. The sign flips between platforms and both are well within run-to-run spread (CUDA 0.780–0.795, XPU 0.790–0.797), so there's no accuracy change.
With descriptors disabled,
_fwd_kernelandextend_attention_fwdare AST-identical to the parent commit, so the pointer path is provably untouched, not just tested.Performance
NVIDIA CUDA (B200, sm_100, triton 3.7.1),
autodefault: offAt the shipped
num_stages=1it's a wash: within ±1%, with the sign depending on shape. The measurement is tight (off-arm spread 0.2% and 0.003%), so this is a real result, not noise. The kernel runs unpipelined, so the descriptors' async transfer never overlaps with compute.With a local, unshipped edit forcing
num_stages=2, the same kernel benchmark goes from 1.009x to 1.098x geomean. Most of that gain comes from pipelining itself: the pointer path alone goes from 1x to 1.26x atnum_stages=2. Even so, descriptors beat the pipelined pointer path on 4 of 6 shapes:num_stages=1(shipped)num_stages=2num_stages=2num_stages=2Intel XPU (Arc Pro B60, triton 3.8.0),
autodefault: onPer-shape kernel ratios range from 0.98x to 1.51x. The workload gain is larger because serving issues the shapes that benefit most far more often, and prefill dominates this request mix.
Reproduce
Use
--disable-radix-cachefor output-equality checks. With the cache on and duplicated prompts, outputs differ from run to run even with descriptors off in both arms.Scope and limitations
autodefault turns descriptors on only where they were a measured win in the tables above. Everywhere else the pointer path stays the default unless the env var forces it, so no existing deployment changes behaviour.CI States
Latest PR Test (Base): ⏳ Run #39942579233
Latest PR Test (Extra): ❌ Run #39942579180
Latest PR Test (AMD ROCm 10): ⏳ Run #39942579094