Repository navigation
[Perf] attention: Enable tensor descriptors in the Triton extend-attention kernel - #46041
rahulvijayaraghavan wants to merge 3 commits into
Conversation
|
/tag-run-ci-label |
ccbbcb4 to
ed0f70d
Compare
BBuf
left a comment
There was a problem hiding this comment.
Please address the [P1] items, then re-ping.
Agent-assisted pre-review, findings verified against ed0f70da3a97.
|
@rahulvijayaraghavan I pushed Automated pre-review for |
BBuf
left a comment
There was a problem hiding this comment.
Please address the [P1] items, then re-ping.
Agent-assisted pre-review, findings verified against a6424db04177.
|
/rerun-test test/registered/e2e/xpu/test_dflash_intel_xpu.py Targeted validation for |
|
⛔ |
|
/rerun-failed-ci |
Load the Q / extend-K / extend-V tiles through host-side descriptors, which lower to 2D block copies on XPU; the tensor-of-pointer path is unchanged and stays the default everywhere else.
Mask descriptor-loaded V rows at request boundaries and add NaN/Inf isolation regression coverage.
f219ef3 to
d1f6920
Compare
|
/tag-and-rerun-ci |
2 similar comments
|
/tag-and-rerun-ci |
|
/tag-and-rerun-ci |
|
/rerun-failed-ci |
BBuf
left a comment
There was a problem hiding this comment.
LGTM. The remaining benchmark note is non-blocking.
|
@rahulvijayaraghavan Thanks for the work and the fixes. After looking at the benchmarks again, the performance difference in the current shipped NVIDIA configuration is too small (around ±1%) to move this PR forward for now. Let's hold off on merging and revisit when there is a clearer performance benefit. |
Supersedes #29639 (auto-closed as a stale draft; rebased onto current
main).Motivation
Triton's tensor descriptors let a kernel describe a tile by its shape and strides instead of computing a tensor of per-element pointers. Where the backend has a hardware fast path for descriptor loads (on NVIDIA, TMA on Hopper and newer), it can use it, and the per-element address arithmetic and boundary masks go away.
_fwd_kernelloads the Q / extend-K / extend-V tiles with tensor-of-pointer loads. Those tiles are contiguous rows of a(token, head, head_dim)buffer, so they can be described by a host-sideTensorDescriptorinstead.The host-side descriptors follow the existing MoE implementation (#10567). The approach matches the vLLM RFC on tensor descriptor adoption: vllm-project/vllm#42545
What this changes
The descriptor path is behind a tri-state env var. The tensor-of-pointer path is unchanged.
SGLANG_USE_TRITON_ATTN_TENSOR_DESC1/0Four files: the kernel (
extend_attention.py), the env var (environ.py), a new A/B benchmark (bench_extend_attention_tdesc.py) that reports per-call time and numerical agreement, and a new XPU test (test_extend_attention_tensor_descriptor.py) that checks NaN/Inf in one packed request can't leak into another.Descriptors are gated per tile, so a tensor whose layout doesn't qualify keeps the pointer path while the others still use descriptors.
Correctness
Descriptors on vs off, same build, same device. Outputs are asserted exactly equal (
atol=0, rtol=0), not within a tolerance, because both paths read the same tiles in the same order.max err 0.0e+00max err 0.0e+00Pooled across two implementations × two reps per platform, the on-vs-off GSM8K difference is -0.25 pp on CUDA and +0.22 pp on XPU. The sign flips between platforms and both are well within run-to-run spread (CUDA 0.780–0.795, XPU 0.790–0.797), so there's no accuracy change.
With descriptors disabled,
_fwd_kernelandextend_attention_fwdare AST-identical to the parent commit, so the pointer path is provably untouched, not just tested.Performance
NVIDIA CUDA (B200, sm_100, triton 3.7.1),
autodefault: offAt the shipped
num_stages=1it's a wash: within ±1%, with the sign depending on shape. The measurement is tight (off-arm spread 0.2% and 0.003%), so this is a real result, not noise. The kernel runs unpipelined, so the descriptors' async transfer never overlaps with compute.With a local, unshipped edit forcing
num_stages=2, the same kernel benchmark goes from 1.009x to 1.098x geomean. Most of that gain comes from pipelining itself: the pointer path alone goes from 1x to 1.26x atnum_stages=2. Even so, descriptors beat the pipelined pointer path on 4 of 6 shapes:num_stages=1(shipped)num_stages=2num_stages=2num_stages=2Intel XPU (Arc Pro B60, triton 3.8.0),
autodefault: offPer-shape kernel ratios range from 0.98x to 1.51x. The workload gain is larger because serving issues the shapes that benefit most far more often, and prefill dominates this request mix.
Reproduce
Use
--disable-radix-cachefor output-equality checks. With the cache on and duplicated prompts, outputs differ from run to run even with descriptors off in both arms.Scope and limitations
autodefault is off on every platform, so descriptors run only when the env var forces them and no existing deployment changes behaviour.CI States
Latest PR Test (Base): ✅ Run #41680454980
Latest PR Test (Extra): ❌ Run #41680454410
Latest PR Test (AMD ROCm 10): ✅ Run #41680454807
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: