Skip to content

gfx942 unified attention: more KV-split segments for bf16 decode and spec-decode verify - #1

Open
ajassani wants to merge 1 commit into
upstream-mainfrom
gfx942-ua-kv-split-config
Open

ajassani wants to merge 1 commit into
upstream-mainfrom
gfx942-ua-kv-split-config

Conversation

@ajassani

@ajassani ajassani commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

On gfx942, the unified attention split-KV (3D) path caps MAX_SEGMENTS at 128 for every KV dtype. At batch 1 with a long context, that launches too few programs to fill the GPU. For example, one sequence with a few KV heads per GPU gets 128 segments of 2 waves on 304 CUs, and each segment walks about 46 tiles serially. In a Qwen3.5 decode trace, the spec-decode verify call took 240 µs and was 17.8% of decode kernel time.

This PR changes only gfx942/triton/attention/unified_attention/DEFAULT.json:

  • Adds a Q axis to the kv_split schema. attn_2d already uses it.
  • Q_LEQ_1.DT_bf16_bf16 (single-token decode, e.g. the MTP draft): 2 segments per CU.
  • DT_bf16_bf16 (multi-token queries, e.g. spec-decode verify): 4 segments per CU.
  • Both entries: MAX_SEGMENTS 512, SEGMENT_TILE_MIN/MAX 256 (a 256-token floor per segment), MIN_SEGMENTS 8, TILE_SIZE 1–64.
  • The fp8 KV entries are unchanged.

No code change is needed: the lookup is schema-driven, and the kv_split reader already accepts SEGMENT_TILE_MIN.

Results

MI325X, Qwen3.5-397B-A17B-FP8, TP4, head_dim 256, bf16 KV, MTP speculative decoding.

Kernel microbenchmark. Run through aiter's config lookup with CUDA graphs. Outputs match the default config.

Call, batch 1 Context Default This PR
Verify 131k 128 µs 92 µs
Verify 196k 184 µs 115 µs
Draft 131k 120 µs 69 µs
Draft 196k 181 µs 83 µs

At batch 2–32 it is equal or up to 10% faster. One exception: +2% at batch 2–4 with a 65k context.

End to end. SGLang prefill/decode disaggregated serving, multi-turn agentic workload, with this config as the only change:

Concurrency Batch-1 decode step (trimmed mean) Tokens/s/user p50
2 13.93 → 13.51 ms 203 → 211
8 16.71 → 14.95 ms 172 → 177

Accuracy. Full GSM8K (1,319 questions) on the same server, everything else equal: 0.727 with this config (1.4% invalid) and 0.703 with the default (2.6% invalid).

Related

Scope and open questions

  • The new keys aren't qualified by head size. D is the leftmost schema axis, so D_GEQ_512 still wins for head_size >= 512, and the new entries cover head sizes below 512 with bf16 or fp16 KV. Only head_dim 256 was measured. I can narrow them to D_GEQ_256 if that's preferred.
  • MI300X wasn't measured. It's the same gfx942 target with the same CU count.
  • op_tests for unified attention haven't been run on this branch yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant