Skip to content

[Perf] attention: Enable tensor descriptors in the Triton extend-attention kernel - #46034

Closed
rahulvijayaraghavan wants to merge 1 commit into
sgl-project:mainfrom
rahulvijayaraghavan:xpu_extend_attention_tensor_desc_v2
Closed

rahulvijayaraghavan wants to merge 1 commit into
sgl-project:mainfrom
rahulvijayaraghavan:xpu_extend_attention_tensor_desc_v2

Conversation

@rahulvijayaraghavan

@rahulvijayaraghavan rahulvijayaraghavan commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Supersedes #29639 (auto-closed as a stale draft; rebased onto current main).

Motivation

Triton's tensor descriptors let a kernel describe a tile by its shape and strides instead of computing a tensor of per-element pointers. Where the backend has a hardware fast path for descriptor loads (on NVIDIA, TMA on Hopper and newer), it can use it, and the per-element address arithmetic and boundary masks go away.

_fwd_kernel loads the Q / extend-K / extend-V tiles with tensor-of-pointer loads. Those tiles are contiguous rows of a (token, head, head_dim) buffer, so they can be described by a host-side TensorDescriptor instead.

The host-side descriptors follow the existing MoE implementation (#10567). The approach matches the vLLM RFC on tensor descriptor adoption: vllm-project/vllm#42545

What this changes

The descriptor path is behind a tri-state env var. The tensor-of-pointer path is unchanged.

SGLANG_USE_TRITON_ATTN_TENSOR_DESC behaviour
unset (default) auto: on only where descriptors are a measured win (see Performance)
1 / 0 force descriptors on / off on any platform, for A/B

Three files: the kernel (extend_attention.py), the env var (environ.py), and a new A/B benchmark (bench_extend_attention_tdesc.py) that reports per-call time and numerical agreement.

Descriptors are gated per tile, so a tensor whose layout doesn't qualify keeps the pointer path while the others still use descriptors.

Correctness

Descriptors on vs off, same build, same device. Outputs are asserted exactly equal (atol=0, rtol=0), not within a tolerance, because both paths read the same tiles in the same order.

check NVIDIA CUDA (B200, sm_100) Intel XPU (Arc Pro B60)
Kernel, 5 shapes covering every gate branch bit-exact bit-exact
Benchmark, 6 shapes max err 0.0e+00 max err 0.0e+00
Llama-3.1-8B greedy output, 24 requests byte-identical byte-identical
GSM8K, 5-shot, 400 questions, 2 reps 0.792 / 0.790 off vs 0.790 / 0.790 on 0.790 / 0.790 off vs 0.797 / 0.795 on

Pooled across two implementations × two reps per platform, the on-vs-off GSM8K difference is -0.25 pp on CUDA and +0.22 pp on XPU. The sign flips between platforms and both are well within run-to-run spread (CUDA 0.780–0.795, XPU 0.790–0.797), so there's no accuracy change.

With descriptors disabled, _fwd_kernel and extend_attention_fwd are AST-identical to the parent commit, so the pointer path is provably untouched, not just tested.

Performance

NVIDIA CUDA (B200, sm_100, triton 3.7.1), auto default: off

on vs off
Kernel, 6 shapes (geomean) 1.009x
Workload, 32K input (~35% attention FLOPs) 1.007x
Workload, 120K input (~69% attention FLOPs) 0.997x

At the shipped num_stages=1 it's a wash: within ±1%, with the sign depending on shape. The measurement is tight (off-arm spread 0.2% and 0.003%), so this is a real result, not noise. The kernel runs unpipelined, so the descriptors' async transfer never overlaps with compute.

With a local, unshipped edit forcing num_stages=2, the same kernel benchmark goes from 1.009x to 1.098x geomean. Most of that gain comes from pipelining itself: the pointer path alone goes from 1x to 1.26x at num_stages=2. Even so, descriptors beat the pipelined pointer path on 4 of 6 shapes:

shape off, num_stages=1 (shipped) off, num_stages=2 on, num_stages=2 on vs off at num_stages=2
serving 1st chunk 0.24 ms 0.20 ms 0.19 ms 1.05x
serving later chunk 0.56 ms 0.41 ms 0.41 ms 1.00x
batched prefill 1.36 ms 1.09 ms 0.86 ms 1.27x
MHA, mixed 0.35 ms 0.25 ms 0.26 ms 0.96x
long cold prefill 4.60 ms 3.80 ms 2.78 ms 1.37x
small head dim 0.21 ms 0.18 ms 0.18 ms 1.00x

Intel XPU (Arc Pro B60, triton 3.8.0), auto default: on

descriptors off descriptors on on vs off
Workload: Llama-3.1-8B, 2048-in / 32-out, 64 prompts 119.61 s 52.69 s 2.27x
Input token throughput 1095 tok/s 2486 tok/s 2.27x
Median TTFT 64.3 s 28.7 s 2.24x
Kernel, 6 shapes (geomean) — — 1.118x

Per-shape kernel ratios range from 0.98x to 1.51x. The workload gain is larger because serving issues the shapes that benefit most far more often, and prefill dominates this request mix.

Reproduce

# kernel A/B, any platform (the benchmark forces both arms itself)
python benchmark/kernels/extend_attention_triton/bench_extend_attention_tdesc.py --repeat 31

# workload / GSM8K A/B: one server per arm, same client against each
SGLANG_USE_TRITON_ATTN_TENSOR_DESC=0 python -m sglang.launch_server \
    --model-path <Llama-3.1-8B-Instruct> --attention-backend triton \
    --max-total-tokens 32768 --disable-radix-cache --port 30000

python -m sglang.bench_serving --backend sglang --port 30000 \
    --model <Llama-3.1-8B-Instruct> --dataset-name random \
    --random-input-len 2048 --random-output-len 32 --random-range-ratio 1.0 \
    --num-prompts 64

python -m sglang.test.few_shot_gsm8k --num-shots 5 --num-questions 400 \
    --temperature 0.0 --port 30000

Use --disable-radix-cache for output-equality checks. With the cache on and duplicated prompts, outputs differ from run to run even with descriptors off in both arms.

Scope and limitations

  • Extend attention only. Decode attention keeps the pointer path, because its K/V reads are paged gathers that have no contiguous-row descriptor form. The output store also stays on the masked pointer path, because descriptors zero-pad reads but don't mask stores.
  • 16-bit element types only. The fp8 prefill paths aren't validated here, so the gate declines them.
  • The auto default turns descriptors on only where they were a measured win in the tables above. Everywhere else the pointer path stays the default unless the env var forces it, so no existing deployment changes behaviour.

CI States

Latest PR Test (Base): ⏳ Run #39942579233
Latest PR Test (Extra): ❌ Run #39942579180
Latest PR Test (AMD ROCm 10): ⏳ Run #39942579094

@github-actions

This comment has been minimized.

@rahulvijayaraghavan rahulvijayaraghavan changed the title Enable tensor descriptors in the Triton extend-attention kernel [Perf] [Intel] attention: Enable tensor descriptors in the Triton extend-attention kernel Oct 23, 2026
@rahulvijayaraghavan rahulvijayaraghavan changed the title [Perf] [Intel] attention: Enable tensor descriptors in the Triton extend-attention kernel [Perf] attention: Enable tensor descriptors in the Triton extend-attention kernel Oct 23, 2026
@polisettyvarma

Copy link
Copy Markdown
Contributor

/tag-run-ci-label

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 23, 2026
Load the Q / extend-K / extend-V tiles through host-side descriptors, which lower
to 2D block copies on XPU; the tensor-of-pointer path is unchanged and stays the
default everywhere else.
@rahulvijayaraghavan

Copy link
Copy Markdown
Contributor Author

Superseded by #46041 (same commit, moved to a device-agnostic branch name).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jit-kernel run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants