Repository navigation
perf(minimax-m3): tune long-context decode topk tiles #2979
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -26,7 +26,7 @@ | |
| # which couples to vLLM internals. ATOM imports triton directly everywhere else. | ||
| import triton | ||
| import triton.language as tl | ||
| from aiter.jit.utils.chip_info import get_gfx | ||
| from aiter.jit.utils.chip_info import get_gfx, get_gfx_runtime | ||
|
|
||
| try: | ||
| # aiter's chunk-narrowing per-row selector: same forced-block pins and tie | ||
|
|
@@ -588,14 +588,9 @@ def _score_tile_keys( | |
| # because the next shape down (68 blocks) prefers 128 over 64 by 10%, so | ||
| # lowering it is not a one-line change; it is worth ~0.1ms of a prefill step. | ||
| # | ||
| # Decode cannot derive it, because it is captured: the wrapper's Python runs | ||
| # once, at capture, and capture is handed max_seqlen_k = max_model_len so the | ||
| # grid covers every replay. A derived width would therefore be the | ||
| # longest-context one on every replay -- (1024, 8) here, which is 1.4-1.7x off | ||
| # the per-shape best at 64 blocks. So decode takes a constant, chosen by total | ||
| # regret over the deployed shapes. That regret is real and mostly irreducible: | ||
| # the per-shape optima span (64, 1) to (1024, 8), and the best single pair, | ||
| # (256, 4), is only 0.6% better than this one over 16 shapes. | ||
| # Decode can only use the capture-time max-block bound, never the live replay | ||
| # length. Its width is selected from that bound and remains fixed for every | ||
| # replay of the graph. | ||
| # | ||
| # Neither phase may autotune, and not for the usual reason: triton's key here | ||
| # would be BLOCK_SIZE_T == next_pow2(topk) == 16, a constant, so the first shape | ||
|
|
@@ -614,6 +609,16 @@ def _score_tile_keys( | |
| DECODE_TOPK_NUM_WARPS = 8 | ||
|
|
||
|
|
||
| def _decode_topk_block_size_k(arch: str, max_block: int) -> int: | ||
| if arch != "gfx1250" or max_block <= 512: | ||
| return DECODE_TOPK_BLOCK_SIZE_K | ||
| # At 2048+ blocks the production emit path normally selects aiter first. | ||
| # This tile serves calls without published bounds or shapes aiter declines. | ||
| if max_block >= 2048: | ||
| return 2048 | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. at 2048 with emit the shipped path already takes aiter, so this tile is only the no-aiter fallback? leave a comment so we know.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added the fallback-path comment in 40af5ad. The fixed-bound retest also confirms that at 2048+ the production emit path normally selects aiter, so this tile is not a production optimization there. |
||
| return 1024 | ||
|
|
||
|
|
||
| @functools.cache | ||
| def _compute_units(device_index: int) -> int: | ||
| return torch.cuda.get_device_properties(device_index).multi_processor_count | ||
|
|
@@ -1341,7 +1346,7 @@ def minimax_m3_index_topk_decode( | |
| num_idx_heads, | ||
| rows_per_req=max_query_len, | ||
| decode_max_q=max_query_len, | ||
| block_size_k=DECODE_TOPK_BLOCK_SIZE_K, | ||
| block_size_k=_decode_topk_block_size_k(get_gfx_runtime(), max_block), | ||
| num_warps=DECODE_TOPK_NUM_WARPS, | ||
| emit=emit_sparse_block_table, | ||
| block_page_stride=block_page_stride, | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nothing pins the new policy.. add a parametrized test for the 512/513 and 2047/2048 boundaries and the non-gfx1250 default?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added parameterized coverage for gfx1250 512/513 and 2047/2048, plus the non-gfx1250 default, in 40af5ad. Focused tests, Black, and Ruff pass.