Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On gfx942, the unified attention split-KV (3D) path caps
MAX_SEGMENTSat 128 for every KV dtype. At batch 1 with a long context, that launches too few programs to fill the GPU. For example, one sequence with a few KV heads per GPU gets 128 segments of 2 waves on 304 CUs, and each segment walks about 46 tiles serially. In a Qwen3.5 decode trace, the spec-decode verify call took 240 µs and was 17.8% of decode kernel time.This PR changes only
gfx942/triton/attention/unified_attention/DEFAULT.json:Qaxis to thekv_splitschema.attn_2dalready uses it.Q_LEQ_1.DT_bf16_bf16(single-token decode, e.g. the MTP draft): 2 segments per CU.DT_bf16_bf16(multi-token queries, e.g. spec-decode verify): 4 segments per CU.MAX_SEGMENTS512,SEGMENT_TILE_MIN/MAX256 (a 256-token floor per segment),MIN_SEGMENTS8,TILE_SIZE1–64.No code change is needed: the lookup is schema-driven, and the kv_split reader already accepts
SEGMENT_TILE_MIN.Results
MI325X, Qwen3.5-397B-A17B-FP8, TP4, head_dim 256, bf16 KV, MTP speculative decoding.
Kernel microbenchmark. Run through aiter's config lookup with CUDA graphs. Outputs match the default config.
At batch 2–32 it is equal or up to 10% faster. One exception: +2% at batch 2–4 with a 65k context.
End to end. SGLang prefill/decode disaggregated serving, multi-turn agentic workload, with this config as the only change:
Accuracy. Full GSM8K (1,319 questions) on the same server, everything else equal: 0.727 with this config (1.4% invalid) and 0.703 with the default (2.6% invalid).
Related
max_seqlen_q, for head_size <= 128. This PR applies the idea to gfx942.Scope and open questions
Dis the leftmost schema axis, soD_GEQ_512still wins for head_size >= 512, and the new entries cover head sizes below 512 with bf16 or fp16 KV. Only head_dim 256 was measured. I can narrow them toD_GEQ_256if that's preferred.op_testsfor unified attention haven't been run on this branch yet.