Skip to content

[Perf] attention: Enable tensor descriptors in the Triton extend-attention kernel - #46041

Open
rahulvijayaraghavan wants to merge 3 commits into
sgl-project:mainfrom
rahulvijayaraghavan:triton-extend-attn-tensor-desc
Open

rahulvijayaraghavan wants to merge 3 commits into
sgl-project:mainfrom
rahulvijayaraghavan:triton-extend-attn-tensor-desc

Conversation

@rahulvijayaraghavan

@rahulvijayaraghavan rahulvijayaraghavan commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Supersedes #29639 (auto-closed as a stale draft; rebased onto current main).

Motivation

Triton's tensor descriptors let a kernel describe a tile by its shape and strides instead of computing a tensor of per-element pointers. Where the backend has a hardware fast path for descriptor loads (on NVIDIA, TMA on Hopper and newer), it can use it, and the per-element address arithmetic and boundary masks go away.

_fwd_kernel loads the Q / extend-K / extend-V tiles with tensor-of-pointer loads. Those tiles are contiguous rows of a (token, head, head_dim) buffer, so they can be described by a host-side TensorDescriptor instead.

The host-side descriptors follow the existing MoE implementation (#10567). The approach matches the vLLM RFC on tensor descriptor adoption: vllm-project/vllm#42545

What this changes

The descriptor path is behind a tri-state env var. The tensor-of-pointer path is unchanged.

SGLANG_USE_TRITON_ATTN_TENSOR_DESC behaviour
unset (default) auto: off on every platform
1 / 0 force descriptors on / off on any platform, for A/B

Four files: the kernel (extend_attention.py), the env var (environ.py), a new A/B benchmark (bench_extend_attention_tdesc.py) that reports per-call time and numerical agreement, and a new XPU test (test_extend_attention_tensor_descriptor.py) that checks NaN/Inf in one packed request can't leak into another.

Descriptors are gated per tile, so a tensor whose layout doesn't qualify keeps the pointer path while the others still use descriptors.

Correctness

Descriptors on vs off, same build, same device. Outputs are asserted exactly equal (atol=0, rtol=0), not within a tolerance, because both paths read the same tiles in the same order.

check NVIDIA CUDA (B200, sm_100) Intel XPU (Arc Pro B60)
Kernel, 5 shapes covering every gate branch bit-exact bit-exact
Benchmark, 6 shapes max err 0.0e+00 max err 0.0e+00
Llama-3.1-8B greedy output, 24 requests byte-identical byte-identical
GSM8K, 5-shot, 400 questions, 2 reps 0.792 / 0.790 off vs 0.790 / 0.790 on 0.790 / 0.790 off vs 0.797 / 0.795 on

Pooled across two implementations × two reps per platform, the on-vs-off GSM8K difference is -0.25 pp on CUDA and +0.22 pp on XPU. The sign flips between platforms and both are well within run-to-run spread (CUDA 0.780–0.795, XPU 0.790–0.797), so there's no accuracy change.

With descriptors disabled, _fwd_kernel and extend_attention_fwd are AST-identical to the parent commit, so the pointer path is provably untouched, not just tested.

Performance

NVIDIA CUDA (B200, sm_100, triton 3.7.1), auto default: off

on vs off
Kernel, 6 shapes (geomean) 1.009x
Workload, 32K input (~35% attention FLOPs) 1.007x
Workload, 120K input (~69% attention FLOPs) 0.997x

At the shipped num_stages=1 it's a wash: within ±1%, with the sign depending on shape. The measurement is tight (off-arm spread 0.2% and 0.003%), so this is a real result, not noise. The kernel runs unpipelined, so the descriptors' async transfer never overlaps with compute.

With a local, unshipped edit forcing num_stages=2, the same kernel benchmark goes from 1.009x to 1.098x geomean. Most of that gain comes from pipelining itself: the pointer path alone goes from 1x to 1.26x at num_stages=2. Even so, descriptors beat the pipelined pointer path on 4 of 6 shapes:

shape off, num_stages=1 (shipped) off, num_stages=2 on, num_stages=2 on vs off at num_stages=2
serving 1st chunk 0.24 ms 0.20 ms 0.19 ms 1.05x
serving later chunk 0.56 ms 0.41 ms 0.41 ms 1.00x
batched prefill 1.36 ms 1.09 ms 0.86 ms 1.27x
MHA, mixed 0.35 ms 0.25 ms 0.26 ms 0.96x
long cold prefill 4.60 ms 3.80 ms 2.78 ms 1.37x
small head dim 0.21 ms 0.18 ms 0.18 ms 1.00x

Intel XPU (Arc Pro B60, triton 3.8.0), auto default: off

descriptors off descriptors on on vs off
Workload: Llama-3.1-8B, 2048-in / 32-out, 64 prompts 119.61 s 52.69 s 2.27x
Input token throughput 1095 tok/s 2486 tok/s 2.27x
Median TTFT 64.3 s 28.7 s 2.24x
Kernel, 6 shapes (geomean) — — 1.118x

Per-shape kernel ratios range from 0.98x to 1.51x. The workload gain is larger because serving issues the shapes that benefit most far more often, and prefill dominates this request mix.

Reproduce

# kernel A/B, any platform (the benchmark forces both arms itself)
python benchmark/kernels/extend_attention_triton/bench_extend_attention_tdesc.py --repeat 31

# workload / GSM8K A/B: one server per arm, same client against each
SGLANG_USE_TRITON_ATTN_TENSOR_DESC=0 python -m sglang.launch_server \
    --model-path <Llama-3.1-8B-Instruct> --attention-backend triton \
    --max-total-tokens 32768 --disable-radix-cache --port 30000

python -m sglang.bench_serving --backend sglang --port 30000 \
    --model <Llama-3.1-8B-Instruct> --dataset-name random \
    --random-input-len 2048 --random-output-len 32 --random-range-ratio 1.0 \
    --num-prompts 64

python -m sglang.test.few_shot_gsm8k --num-shots 5 --num-questions 400 \
    --temperature 0.0 --port 30000

Use --disable-radix-cache for output-equality checks. With the cache on and duplicated prompts, outputs differ from run to run even with descriptors off in both arms.

Scope and limitations

  • Extend attention only. Decode attention keeps the pointer path, because its K/V reads are paged gathers that have no contiguous-row descriptor form. The output store also stays on the masked pointer path, because descriptors zero-pad reads but don't mask stores.
  • 16-bit element types only. The fp8 prefill paths aren't validated here, so the gate declines them.
  • The auto default is off on every platform, so descriptors run only when the env var forces them and no existing deployment changes behaviour.

CI States

Latest PR Test (Base): ✅ Run #41680454980
Latest PR Test (Extra): ❌ Run #41680454410
Latest PR Test (AMD ROCm 10): ✅ Run #41680454807

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@rahulvijayaraghavan

Copy link
Copy Markdown
Contributor Author

@HaiShaw @BBuf could you please review and add run-ci? Opt-in tensor-descriptor path for the Triton extend-attention kernel; default path unchanged.

@polisettyvarma

Copy link
Copy Markdown
Contributor

/tag-run-ci-label

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 23, 2026
@rahulvijayaraghavan
rahulvijayaraghavan force-pushed the triton-extend-attn-tensor-desc branch from ccbbcb4 to ed0f70d Compare October 28, 2026 02:59

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the [P1] items, then re-ping.

Agent-assisted pre-review, findings verified against ed0f70da3a97.

@BBuf

BBuf commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator

@rahulvijayaraghavan I pushed a6424db0 to your branch: Mask descriptor-loaded V rows at request boundaries and add NaN/Inf isolation regression coverage.
Please pull before your next push. The review runs again on the new head.

Automated pre-review for ed0f70da3a97; it re-checks when the PR is updated.

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the [P1] items, then re-ping.

Agent-assisted pre-review, findings verified against a6424db04177.

@BBuf

BBuf commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/e2e/xpu/test_dflash_intel_xpu.py

Targeted validation for a6424db04177.

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

⛔ test/registered/e2e/xpu/test_dflash_intel_xpu.py: test/registered/e2e/xpu/test_dflash_intel_xpu.py is registered for XPU (suite nightly-xpu-kernel-main-1-gpu), not for CUDA or CPU; rerun-test.yml has no XPU job. Rerun it with /rerun-failed-ci, or dispatch the XPU workflow manually.

@BBuf

BBuf commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

rahulvijayaraghavan and others added 3 commits November 3, 2026 22:33
Load the Q / extend-K / extend-V tiles through host-side descriptors, which lower
to 2D block copies on XPU; the tensor-of-pointer path is unchanged and stays the
default everywhere else.
Mask descriptor-loaded V rows at request boundaries and add NaN/Inf isolation regression coverage.
@rahulvijayaraghavan
rahulvijayaraghavan force-pushed the triton-extend-attn-tensor-desc branch from f219ef3 to d1f6920 Compare November 3, 2026 17:03
@rahulvijayaraghavan

Copy link
Copy Markdown
Contributor Author

@BBuf
Thanks for the review!
P1s are addressed in f219ef3 (on top of your a6424db), and the description is updated. Please take a look.

@BBuf

BBuf commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

2 similar comments
@BBuf

BBuf commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@BBuf

BBuf commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@BBuf

BBuf commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The remaining benchmark note is non-blocking.

BBuf commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

@rahulvijayaraghavan Thanks for the work and the fixes. After looking at the benchmarks again, the performance difference in the current shipped NVIDIA configuration is too small (around ±1%) to move this PR forward for now. Let's hold off on merging and revisit when there is a clearer performance benefit.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jit-kernel run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants