Repository navigation
[Triton/Gluon] [Perf] unified_attention: split KV by actual sequence lengths under CUDA graphs (gfx950, head 64, bf16 Q / fp8 KV) - #7088
Conversation
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
One backend per PR: PR title tags & labels: |
There was a problem hiding this comment.
🔵 Needs a closer look
The gfx950 split-count retune needs exact-head CUDA-graph correctness and latency validation because the supplied measurements predate this PR and GPU execution was unavailable during review.
No open issues 🎉
What changed in this PR
This config-only PR tunes unified attention on gfx950 for head dimensions up to 64 with bf16 queries and fp8 KV, targeting long-context decode under CUDA graphs.
Changes:
- Adds a dtype-specific entry with 128 KV segments and fixed segment and tile sizes of 64.
- Leaves kernel code unchanged. Reported measurements show faster long-context decode, with a 2.6 µs slowdown for the short-sequence case.
| File | Description |
|---|---|
aiter/ops/triton/configs/gfx950/triton/attention/unified_attention/DEFAULT.json |
Adds the D_LEQ_64.DT_bf16_fp8 KV-split configuration. |
Review effort: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
The gfx950 default’s reported mixed-length gains come with uniform-batch slowdowns, requiring maintainer judgment supported by target-GPU correctness and latency measurements unavailable in this review.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
The gfx950 tuning’s reported long-context gains come with uniform-batch latency regressions, requiring maintainer judgment on the workload trade-off because GPU correctness and performance results could not be independently reproduced.
No open issues 🎉
Review effort: Balanced
4948cd3 to
be2a245
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The packed work-item mapping and slot-major reduction require exact-head gfx950 correctness and performance validation under CUDA graph replay, which this source-only review cannot perform.
No open issues 🎉
Review effort: Balanced
… reduce - reduce_segments: keep tiles_per_segment >= 1, so a zero-length sequence does not compute cdiv(0, 0) - Length-aware reduce: cap the merged slots at NUM_SEGMENTS_PER_SEQ, the most slots the 3D kernel writes - Length-aware reduce: get num_tokens from the launcher for the slot stride, as the 3D kernel does - Tests: add skewed batches with zero-length (padded) requests, and a padded replay to the graph test - Tests: fill the output with NaN before the call, and require zero rows for a zero-length request
- _length_aware_split needs all kv_split, attn_3d and reduce keys the split reads, else it returns None - Remove the NUM_SEGMENTS fallback for SPLIT_NUM_SEGMENTS and the tuning values in the SEQ_BLOCK defaults - Return a _LengthAwareSplit with the attn_3d and reduce entries, so the launchers do not read them again - Share the BLOCK_M rule of the 3D launch and the query-block guard in _attn_3d_block_m - Eager host time per in-scope call on MI355X: 8x2K q1 102.3 -> 103.4 us, skewed32 q4 108.0 -> 104.3 us - Tests: check the fallback for each missing key, and pin the attn_3d and reduce lookups to gfx950
- Replace use_sinks with an option parameter: sinks, shuffled_kv_cache, softcap or qq_bias - Add skewed q4 cases for shuffled_kv_cache, softcap and qq_bias, which the length-aware guard does not exclude - Use softcap 0.1: the scores here are ~0.1, so a dropped softcap moves each request ~5%, past the 2e-2 bound
- Size the cache as the sum of per-request pages plus 64 spare pages instead of requests x the longest request - skewed_300 drops from 60.2 GiB to 1.3 GiB peak GPU memory (it allocated ~37 GB of fp32 K/V for ~5K used pages) - Each request still gets distinct random pages; the graph replay test keeps each request's pages across replays
There was a problem hiding this comment.
🔵 Needs a closer look
The new GPU work-item mapping and reduction layout require exact-head gfx950 correctness and CUDA-graph replay validation, which this source-only review cannot provide and no supplied report establishes.
1 open issue
Review effort: Balanced
- The Triton review rules allow one or two sentences per docstring
There was a problem hiding this comment.
🔵 Needs a closer look
The new GPU work-item mapping and slot-major reduction require independent exact-head gfx950 correctness, latency, and changing-length graph-replay validation that this source-only review could not perform.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
…on gfx942 - The length-aware split tests pin the gfx950 kv_split tables on gfx942, like test_length_aware_split_scope does - Its kernels are plain Triton, so gfx942 CI now checks the split math the gfx950 entry relies on - test_triton_unified_attn_3d_long_kv accepts gfx942; all 19 cases pass on MI300X
|
gfx942 (MI300X) validation: validate-kernel-pr PASS. 1149 passed / 0 failed; independent grid 317 pass. The length-aware split is gated off on gfx942 by its config and runs correctly there when the gfx950 tables are pinned; the split and long-KV tests now run on gfx942. Bench latency within ±0.8% of base. New commits for gfx942: 4efe461. Reports were generated with the repo's validate-kernel-pr skill on a local head, so |
- One conflict in unified_attention.py, both sides added a block after _split_launch_config - Keep the length-aware split (Triton 3D path) and main's gfx12 FILL_MIN_SEGMENTS widening (Gluon 3D path) - The two apply to disjoint paths, so their order does not matter
There was a problem hiding this comment.
🔵 Needs a closer look
The packed GPU scheduling and partial-buffer changes require independent exact-head gfx950 correctness and latency validation, especially for length changes during CUDA-graph replay, which this environment cannot perform.
No open issues 🎉
Review effort: Balanced
- reduce_segments: length-aware num_segments first, then main's split-rebalance / sliding-window / uniform tiling - reduce_segments: issue main's output-gate load before the length-aware merge loop - _reduce_segments_triton takes the length-aware split plus main's output_gate, output_rows and segm_tiles - _unified_attention_3d_triton keeps the length-aware config handling and adds main's NVFP4 KV rejection - Gluon reduce stays off both for an output epilogue and for a length-aware split
There was a problem hiding this comment.
🔵 Needs a closer look
The packed GPU work-item mapping and slot-major partial layout require exact-head gfx950 correctness and base/head latency checks under graph replay, which this source-only environment cannot provide.
No open issues 🎉
Review effort: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
The new GPU work-item mapping and slot-major reduction require exact-head gfx950 CUDA-graph correctness and latency validation, which this source-only review cannot perform and no head-matched report was supplied to establish.
1 open issue
Review effort: Balanced
- List the SPLIT_* keys and the uniform fallback conditions in configs/AGENTS.md - Point to them from the triton README and the review instructions
There was a problem hiding this comment.
🔵 Needs a closer look
The packed GPU mapping and slot-major reduction lack exact-head gfx950 validation for CUDA-graph replay correctness and base-versus-head latency, which static review alone cannot establish.
1 open issue
Review effort: Balanced
- Keep the purpose and the reason for the relative-error check, as aiter/ops/triton/README.md rule 18 asks
There was a problem hiding this comment.
🔵 Needs a closer look
The new GPU work-item mapping and slot-major partial-buffer layout require exact-head gfx950 CUDA-graph correctness and latency validation, but no matching report was supplied and this environment cannot run those checks.
No open issues 🎉
1 resolved since last review
Review effort: Balanced


Summary
seqused_k, read on the GPU) instead of the capturedmax_seqlen_kbound.Motivation
max_seqlen_k = 1M, so the uniform split gives every request the same segment count.Changes
seqused_k(int32 or int64) and run a flat grid of (sequence, segment) work items per KV head.split_budget + num_seqs - 1work items, the most the per-sequence shares can sum to.ITEM_RUNshrinks for small batches.compute_segment_paramsapplies main'sSPLIT_FIT_BUDGET/SPLIT_SEQS_PER_TILEin this PR's helper.New gfx950 config keys (read only on a length-aware launch):
kv_split D_EQ_64.DT_bf16_fp8SPLIT_PROGRAMS_PER_CUkv_split D_EQ_64.DT_bf16_fp8SPLIT_SEGMENTS_PER_CUkv_split D_EQ_64.DT_bf16_fp8SPLIT_SEQ_BLOCK_MIN/MAXattn_3d(any,D_LEQ_128.Q_GEQ_4)SPLIT_RUNS_PER_XCDattn_3d D_LEQ_128.Q_GEQ_4SPLIT_BLOCK_MreduceSPLIT_SEGMENT_BLOCK_MINThe uniform split stays for:
(32,64)/(64,32)skip_reducenum_seqs >= 2**15,split_budget + num_seqs >= 2**16, int32 overflow boundsMerged main (#6889 window-aware split, #6983 Q-axis configs): #6983's
attn_3d D_LEQ_128.Q_GEQ_4(BLOCK_M 32) getsSPLIT_RUNS_PER_XCD(without it, q ≥ 4 length-aware calls hit the launcher assert) andSPLIT_BLOCK_M: 16(BLOCK_M 32 padded q4 blocks to twice their rows; the uniform split keeps BLOCK_M 32).Performance (if applicable)
Author run: MI355X; bf16 Q / fp8 KV, head 64, page 128; default 8 Q / 2 KV heads. One CUDA graph per workload at
max_seqlen_k = 1M; 1 GiB write before each of 40 replays (evicts L2 and MALL); median per round, best of 3 alternating rounds on one GPU. Before = maine4973c4bed, after =12f033ff57(later commits change only tests). Kernel latency only.SPLIT_BLOCK_M(same method, length-aware launch at BLOCK_M 32 vs 16):Review re-run at
acd39a137: same script and method, one MI355X, 3 alternating rounds, best of 3. Before = previous headb6997bdce, after =acd39a137(review fixes). Outputs are bit-identical (same checksum on every workload).Benchmark scripts
Run:
./run_bench.sh <gpu> <main-tree> <head-tree> 3bench_ua_split.py:run_bench.sh:Testing
MI355X (gfx950),
op_tests/triton_tests/attention/test_unified_attention.py:e4973c4bedf732d9af02f732d9af02-k "long_kv or length_aware or graph_replay"e7fcdde3eeacd39a137(review run)acd39a137(review run)-k "long_kv or length_aware or graph_replay or split_scope"¹ Deselects
test_triton_unified_attn_qq_bias_padded_query_rows(5 cases, from #6983); it aborts the process on unmodified main in this environment too.Documentation
Dependencies
Breaking Changes
None. Calls outside the scope above keep the uniform split and its outputs.
PR Self-Review
Reviewed revision
9c02c030a7d30a7fd447be4ad6e142035f9739ab9c02c030aonly shortens the_assert_request_rel_l2docstring to two sentences, the same text as [Triton/Gluon] [Feature] unified_attention: support causal=False for bidirectional draft verify #7244. MI355X:pytest test_unified_attention.py -k "length_aware or rel_l2"gives 17 passed, 0 failed.c05579a79(Copilot review) documents theSPLIT_*keys inaiter/ops/triton/configs/AGENTS.md. The results below are atd8b4be493.d8b4be493andede8629bbonly shorten comments/docstrings (Copilot review); no code change sinceacd39a137.Behavior and intent
seqused_kinstead of the capturedmax_seqlen_k.Review findings (AITER
review-pr+code-review, atb6997bdce)find_work_itemcounted empty lanes pastnum_seqsas sequences before a padding item; near 2**15 sequences the packed count overflowsacd39a137(before & valid). Latent today: the 3D path runs only whennum_seqs × KV heads ≤ 4 × CUsuse_gluon_reducerequiressplit_budget is None(latent on gfx950, which has no Gluon reduce)num_seqs * max_seqlen_kcould wrap for a numpymax_seqlen_kint()castSPLIT_SEQ_BLOCK_MIN/MAXin the gfx950 JSONkv_splitentry can opt inITEM_RUNmapping with the sharedremap_xcdhelpermax_seqlen_qsmaller than a real query length drops rowsmax_seqlen_qis the documented upper bound; the uniform path also relies on it (2D/3D choice,ALL_DECODE)Validation and unverified scope
Commands:
pytest op_tests/triton_tests/attention/test_unified_attention.py -k "not qq_bias_padded_query_rows"and-k "long_kv or length_aware or graph_replay or split_scope"on one MI355X; results in Testing.Benchmarks:
run_bench.sh(attached) on one MI355X; results in Performance.End-to-end accuracy through SGLang (Kimi-K3, DSpark draft attention on this path),
sgl-eval 0.1.3 run gsm8k --load-preset-from-model-id moonshotai/Kimi-K3(full 1319, temperature 1.0, top_p 0.95, thinking on, reasoning_effort max):SPLIT_PROGRAMS_PER_CUin the gfx950 config)Within sampling noise (single-run stderr ~0.45 pp). Setup: 8× MI355X,
Kimi-K3-Quark-MXFP4-AttnFP8, AgentX c16 recipe (TP8/DCP8, DSpark block 4, decode graphs bs 1-32), simulated acceptance off. Build: SGLangk3-agentx-pr-stack@39d944c57d, aiter stack @f9b5308b2(this PR ate2355e700; the later commits are the review fixes and comment edits listed above).Unverified: other architectures.
PR-specific question
kv_split D_EQ_64.DT_bf16_fp8setsSPLIT_PROGRAMS_PER_CU; its uniform values equalD_LEQ_64.attn_3dkeys (SPLIT_RUNS_PER_XCD,SPLIT_BLOCK_M) andreducekey (SPLIT_SEGMENT_BLOCK_MIN) are read only on a length-aware launch.skip_reduce, query block, integer bounds) return the uniform split.test_length_aware_split_scopechecks this and that attn_3d/reduce entries for q = 1, 4, 8 carry their keys.Changed op tests
op_tests/triton_tests/attention/test_unified_attention.py:test_triton_unified_attn_3d_long_kv: actual vs captured KV bound.test_triton_unified_attn_3d_length_aware_split: skewed, uniform, 300-request, int64seqused_kand q=8 batches againstref_paged_attn, plus per-sequence segment counts.test_triton_unified_attn_3d_length_aware_split_graph_replay: changesseqused_kbetween graph replays.test_length_aware_split_scope: excluded calls, asymmetric heads, sliding window, integer bounds, the query-block guard on verify params (q = 4 and 8 split, q = 9 falls back), config keys, and theSEQ_BLOCKthe entry gives.