Repository navigation
Conversation
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
PR title tags & labels: |
dce39a9 to
24246ef
Compare
|
Rebased onto main as #4538 moved the split heuristic into fp8_mqa_logits_kernels.py and turned the multiplier into _SplitPolicy.cu_oversub, so this is now a one-line change on the gfx942 entry (gfx950 left at 4x, different tuning). This is orthogonal to #4963 and vllm-project/vllm#58366 . Those pick the variant (rows_per_block), this picks grid.y splits, and neither touches _auto_num_splits. They stack rather than overlap, so this can land on its own and doesn't need to queue behind either. |
…6x CU _auto_num_splits aims for 4x CU worth of blocks, which leaves the device under-filled on the indexer prefill shapes. At seq_len 1024 / seq_len_kv 131072 it picks 3 splits: 1536 blocks over 304 CUs, 84.2% occupancy. A 16x target picks 10 splits, 99.1%. logits[m,n] are independent across n, so this only changes how the work is spread over grid.y. Output is bit-identical. One side effect: the early-out for an already-oversubscribed row grid moves with the target, so shapes with a row grid between 4x and 16x CU now get split where they did not before. ROCm#4538 moved this heuristic into fp8_mqa_logits_kernels.py and made the multiplier a _SplitPolicy field, so the change is now cu_oversub on the gfx942 entry. gfx950 stays at 4x, it was tuned separately on MI355X. MI325X (gfx942, 304 CU): 1.10x on the fp8_mqa_logits region, work-weighted over the top 8 call shapes (81% of the region's GPU time). Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com>
24246ef to
209e625
Compare
|
If #5921 supersedes this, then should this be closed? |
|
Yep superceded by #5921 |
Summary
_auto_num_splits targets 4x CU worth of blocks, which underfills the device on the shapes vLLM issues for DeepSeek-V4 indexer prefill. Raising the target to 16x fills it.
Suggested fix
At seq_len 1024 / seq_len_kv 131072 the 4x target picks 3 splits: 1536 blocks over 304 CUs, six launched waves at 84.2% occupancy, the last holding 16 blocks. At 16x it picks 10 splits — 5120 blocks, 17 waves, 99.1%.
logits[m,n] are independent across n, so splitting a row's window across grid.y is pure parallelism: same work, same numerics. The early-out for an already-oversubscribed row grid still fires first, so large row grids are untouched.
Perf
MI325X (gfx942, 304 CU): 1.10x on the fp8_mqa_logits region, work-weighted over the top 8 call shapes, which are 81% of the region's GPU time. Output is bit-identical.
Composes with #4963. Once _auto_variant can pick rows_per_block, this shape selects r4 and grid_x halves to 256, so the same target moves it 5 → 19 splits. Combined effect measured at 1.82x.