Skip to content

[Config] [Perf] Tune DeepSeek-V4-Pro small-M FP8 GEMMs on gfx1250 - #7644

Merged
valarLip merged 1 commit into
ROCm:mainfrom
hubertlu-tw:config/gfx1250-dsv4-smallm-fp8-gemm
Nov 10, 2026
Merged

valarLip merged 1 commit into
ROCm:mainfrom
hubertlu-tw:config/gfx1250-dsv4-smallm-fp8-gemm

Conversation

@hubertlu-tw

Copy link
Copy Markdown
Contributor

Summary

Add 15 gfx1250 rows to aiter/configs/model_configs/dsv4_a8w8_blockscale_mxscale_bpreshuffle_tuned_gemm.csv for DeepSeek-V4-Pro TP4 decode GEMMs that run the untuned FlyDSL fallback on main. Also add the matching untuned shape list for the in-tree tuner. Scope is TP4, with and without MTP (multi-token prediction, 3 draft tokens). On main these GEMMs take 25 to 101 us per call; the rows take 3.6 to 9.4 us. In a replay of one decode step's dense FP8 GEMMs, that removes 11 to 12 ms per step at batch 4 and 16 without MTP, and at batch 1 and 4 with MTP. Config only. No existing row changes.

Motivation

A gfx1250 mxfp8_128 bpreshuffle GEMM with no tuned row goes to _flydsl_mxfp8_fallback_kernel. That function sorts by -tile_n, so at small M it picks t16x1024x128 (t32x/t64x1024x128 above M=16). An N=2048 GEMM then launches 2 workgroups on 256 CUs, and N=7168 launches 7. At these M the GEMM is bound by reading the weight from HBM, and a handful of workgroups can't do that.

On main, #7018 covers M=1/2/8/64 for four of these shapes, and the Kimi-K3 table covers the two shared-expert shapes. What still falls back at decode M (ATOM's CUDA-graph batch sizes are 1 to 8, 16, 32, 48, 64, …):

  • wqkv_a 2048x7168, wo_b 7168x4096, wq_b 16384x1536 and the indexer wq_b 8192x1536, at M=3, 4, 12 and 16. Without MTP that is batch 3, 4 and 16. With MTP the target model verifies 4 tokens per sequence, so batch 1 runs at M=4, batch 3 at M=12 and batch 4 at M=16.
  • The MTP draft block's e_proj and h_proj (both 7168x7168), at every M. Each of the 3 draft steps runs e_proj at M = T tokens and h_proj at M = 4T (4 hyper-connection copies). The first draft step has T = 4 x batch and the next two have T = batch. That's 6 calls per decode step.

Changes

  • 15 rows, all gfx1250, cu_num 256, w_scale_block 128x128, split-K 1:
    • 2048x7168, 7168x4096, 8192x1536, 16384x1536: M=4 (also serves M=3) and M=16 (also serves M=12). Each shape uses one kernel for both rows.
    • 7168x7168: M=1, 2, 4, 8, 16, 32, 64. Through padded-M lookup these serve every M from 1 to 64. M=48 goes to the M=64 row, which has the kernel that is fastest at M=48 too.
  • dsv4_a8w8_blockscale_mxscale_bpreshuffle_untuned_gemm.csv: the 15 shapes, in the format of the Kimi-K3 untuned list. The table merge skips untuned files.
  • Columns follow the existing gfx1250 rows:
    • kernelId is the kernel's index in main's kernels_list. Dispatch reads kernelName. Many older gfx1250 rows in this file have ids from an earlier list; this PR doesn't touch them.
    • us is the timing method below. Older rows in the file use other methods, so us is not comparable across row groups.
    • bw is in GB/s over A + B + C bytes.
    • errRatio is checkAllclose(ref, out, rtol=0.01, atol=0.01) against an fp32 dequantized reference.
  • The rows are appended as one block, grouped by (N, K) with ascending M like the file's main gfx1250 block. Re-sorting the file is left for a separate change.

Why not fix the fallback heuristic instead: that's the better long-term fix, and I'll send it separately. It would cover other models, TP sizes and the data-parallel-attention shapes without rows. The shape of it: among split-K 1 sync kernels that fit, prefer the widest tile_n that still launches at least about cu_num / 2 workgroups, instead of the widest tile_n outright. I haven't measured that rule. These rows are still worth having afterwards, because they are the measured best on main.

Performance (if applicable)

Hardware: one gfx1250 GPU (MI455X, 256 CUs), aiter main 0de43e662b + this PR, FlyDSL 0.4.2. The branch is rebased onto a0c4b69c14, whose new commits change only docs and review tooling.

Timing method, used for every number here and for the us column: 50 back-to-back launches in one HIP graph, with weights cycling through more than 1 GiB of copies so each launch reads its weight from HBM, as in decode graph replay. Each number is the median of 5 replays divided by 50, then the median over 3 fresh processes. Across all candidates the spread between processes (max minus min, over median) was 1.2% at the median, 6% at p90 and 11% at most. For the picked kernels it was at most 3.7%.

Per-step dense FP8 GEMM time. The replay sums every dense FP8 linear in one decode step, each call running whatever main's lookup picks for that table:

  • Target model: 61 layers of wqkv_a, wq_b, wo_b, shared expert gate_up 1536x7168 and w2 7168x768, plus the indexer wq_b on its 30 layers.
  • With MTP, the draft block's same five linears, plus e_proj/h_proj, for each of the 3 draft steps.
Hardware Workload (TP4, batch = concurrency) Main Main + PR Change
gfx1250 no MTP, batch 1 1.57 ms 1.57 ms 0
gfx1250 no MTP, batch 4 12.52 ms 1.56 ms -10.96 ms
gfx1250 no MTP, batch 8 1.68 ms 1.67 ms 0
gfx1250 no MTP, batch 16 12.55 ms 1.60 ms -10.95 ms
gfx1250 MTP 3 tokens, batch 1 13.36 ms 1.67 ms -11.69 ms
gfx1250 MTP 3 tokens, batch 4 13.74 ms 1.71 ms -12.02 ms
gfx1250 MTP 3 tokens, batch 8 2.38 ms 1.94 ms -0.44 ms
gfx1250 MTP 3 tokens, batch 16 3.01 ms 2.19 ms -0.82 ms

Each value is the median of 3 runs; the runs spread by at most 1.2%. At batch 8 and 16 with MTP, the remaining gain is the draft GEMMs. h_proj there runs at M=128 and 256, which stay untuned and take 21 and 24 us.

How much of a step that is: I couldn't serve with main's aiter (see unverified scope). The closest measurement is the older ATOM image's aiter with a superset table that tunes all of these GEMMs. Its dense FP8 GEMM time at M=1 is about the same as main + PR's (1.56 ms). There, a decode step took:

  • 9.8 ms at batch 4 and 14.1 ms at batch 16 without MTP.
  • About 8.4 to 11.1 ms at batch 1 and 13.9 ms at batch 4 with MTP (TPOT times 2.84 tokens per step).

So on main the GEMM time these rows remove is roughly 45 to 55% of the decode step at those four points. It is about 2 to 3% at MTP batch 8 and 16. That's an estimate, not a serving measurement on main.

Per row, before (main's fallback) vs after, at every decode M the row serves:

N x K Used by Row M Serves M Before: kernel, us After: kernel, us Speedup
2048 x 7168 wqkv_a 4 / 16 3, 4 / 12, 16 t16x1024x128 100 t32x32x1024_nb4 5.8-6.0 16.7-17.1x
7168 x 4096 wo_b 4 / 16 3, 4 / 12, 16 t16x1024x128 59 t16x64x1024_nb3 5.0-5.2 11.3-11.8x
16384 x 1536 wq_b 4 / 16 3, 4 / 12, 16 t16x1024x128 25.4 t16x64x256_nb4 3.9-4.0 6.2-6.4x
8192 x 1536 indexer wq_b 4 / 16 3, 4 / 12, 16 t16x1024x128 25.2 t32x32x512_nb3 3.6-3.7 6.7-7.0x
7168 x 7168 e_proj, h_proj 1, 2, 4, 8, 16 1-8, 12, 16 t16x1024x128 101.5 t16x64x1024_nb3 6.9-7.1 14.3-14.7x
7168 x 7168 e_proj, h_proj 32 20, 24, 28, 32 t32x1024x128 94.5 t32x32x1024_nb4 7.2-7.5 12.6-13.1x
7168 x 7168 e_proj, h_proj 64 48, 64 t64x1024x128 99.4 t64x32x512_nb4 9.3-9.4 10.6-10.7x

(nbN is the kernel's number of LDS pipeline buffers.)

How the kernels were picked:

  • For each row, every split-K 1, 2, 4 and 8 sync-WMMA kernel on main that fits was timed at the decode M values the row serves: 501 to 897 per shape.
  • The fastest few were re-timed in 3 fresh processes.
  • A kernel counts as faster only if its slowest run beats another kernel's fastest run, averaged over the M values the row serves. Among kernels within that noise, the pick prefers split-K 1, then the kernel shared by the most rows of that shape.

Split-K:

  • Main runs split-K kernels with fp32 partials. For 2048x7168 at M=3/4 the best one (split-K 8) is 1.4% faster than the split-K 1 pick, which is within noise.
  • For 7168x7168 at M=48 and 64, t64x128x256 with split-K 4 is 11% and 8% faster (8.25 vs 9.28 us, 8.71 vs 9.43 us).
  • Split-K outputs are usually not bitwise equal to the fallback's: 97 of the 112 tested were not. Their errRatio is the same as split-K 1 (at most 0.0094).
  • For about 1 us on a GEMM that runs at most once per decode step at those M, I kept split-K 1, so every row stays bitwise equal to the fallback.

Testing

  • Unit tests added or updated
  • Performance benchmarks run
  • Tested on MI250X
  • Tested on MI300X

The rows are keyed to gfx1250 and cu_num 256, so MI250X and MI300X don't read them. No default CI job runs gfx1250 GEMMs. In CI, only the CSV checks (test_csv_validation, test_config_shape_collision) cover this change.

Documentation

  • Docstrings updated
  • User guide updated
  • Performance guide updated
  • No documentation change is needed, with the reason stated below

Tuned-table rows and a shape list only.

Dependencies

  • No new third-party dependencies added
  • New dependencies are listed and justified below

Breaking Changes

None.

Merge order:

  • #7600 bumps FlyDSL to 0.5.0. These rows were picked on 0.4.2 and should be re-timed after that bump; the branches merge cleanly.
  • #7387 and #6696 rewrite gfx950 rows in the same file and merge cleanly with this branch.
  • #6191 already conflicts with main in this file. Its rows and these are both at the end of the file and share no keys.

PR Self-Review

Reviewed revision

061877a8caa2668a84bee2534520bfd5f3e18e61

Behavior and intent

On gfx1250, gemm_a8w8_blockscale_bpreshuffle with e8m0 128x128 scales finds a tuned row for the 15 keys above and the padded M they serve. Before, it ran _flydsl_mxfp8_fallback_kernel, whose widest-N tile launches 2 to 16 workgroups at these M and takes 25 to 101 us. The new kernels take 3.6 to 9.4 us. They are from the same FlyDSL kernel family, use split-K 1, and their output is bitwise equal to the fallback's on every tested input. No existing row, other table or code path changes.

Validation and unverified scope

All on one gfx1250 GPU, aiter main 0de43e662b + this PR, FlyDSL 0.4.2, in a container separate from the serving image:

  • Numerics: each row, at every decode M it serves, ran on the tuner's inputs (random fp8 data, random e8m0 scales). Output was bitwise equal to the fallback kernel's, and errRatio against the fp32 dequantized reference was at most 0.0094. python op_tests/test_gemm_a8w8_blockscale.py --ck_preshuffle True --flydsl -d bf16 -m 1 2 3 4 5 6 7 8 12 16 20 24 28 32 48 64 -nk 7168,7168 2048,7168 7168,4096 8192,1536 16384,1536 passed 160 cases (constant and uniform/auto inputs), max error ratio 0.0024. Real model activations were not used.
  • Lookup: get_mxscale_bpreshuffle_config returns the new row for all 15 keys (AITER_LOG_TUNED_CONFIG=1). The public op returns the same bytes as the row's kernel called directly, and no "falling back" warning appears.
  • Unchanged selection: I resolved find_padded_m_row for every (gfx, cu_num, N, K, w_scale_block) in the merged table at M = 1 to 1024 plus 8 larger M, with main's table and with this PR. Of 104,232 lookups, the only 104 that changed went from no row to a new row. Removing any one of the 15 rows changes the kernel for some decode M.
  • TDM-store hazard: the picked kernels end with the gfx1250 a8w8 epilogue: ds_store of the output tile, then a workgroup barrier, then tensor_store_from_lds. A standalone Gluon repro of that sequence (no aiter kernel involved) returns stale LDS in about 7e-7 of tile stores. The open fixes for other kernels are #7202, #7605 and #7632. A matching fix for these a8w8 GEMMs exists on a branch of mine but is not upstream. Stress check on main: each row's kernel ran 1,000,000 times at 16 (shape, M) points covering the 15 rows (100 calls per HIP graph replay, 10,000 replays) with a bf16 4096^3 matmul on a second stream every replay. Every output was compared bitwise on the device with the first one. Result: 0 mismatches in 16M calls and 2.6e9 workgroup tile stores. At the repro's rate about 1,800 would be expected. The old fallback tiles also gave 0 in 10M calls. This bounds the rate for these kernels and shapes under this load; it doesn't show the race can't happen.
  • python3 -m unittest op_tests.tuning_tests.test_csv_validation op_tests.tuning_tests.test_config_shape_collision: 51 passed.
  • In-tree tuner: python3 csrc/gemm_a8w8_mxfp8_bpreshuffle/gemm_a8w8_mxfp8_bpreshuffle_tune.py --mxscale_table -i aiter/configs/model_configs/dsv4_a8w8_blockscale_mxscale_bpreshuffle_untuned_gemm.csv -o <out.csv> runs on main. It picks the same kernel for 7 of the 15 rows. Seven others are split-K 1 kernels within 2.2% of ours under the method above, which is inside the noise. For 7168x7168 M=64 it picks the split-K 4 kernel this PR declines. Its us column comes from a different timer and is about 6 to 10% higher.
  • Serving context, on the older ATOM image's aiter (0.1.26.dev21) with a 123-row superset table loaded through AITER_CONFIG_GEMM_A8W8_BLOCKSCALE_MXSCALE_BPRESHUFFLE. That aiter predates [Config] [Perf] Tune DeepSeek-V4 M1/M2/M8/M64 FP8 GEMMs on gfx1250 #7018, and many of that table's kernels differ from these rows.
    • GSM8K 3-shot: 0.951 (TP4) and 0.957 (TP4 + MTP).
    • A 20-minute sustained-load check passed for both. It keeps 600 GSM8K requests in flight and fails on bursts of empty, non-text or answerless responses, on rolling accuracy below 0.5, or on failed probe prompts.
    • Conc-1 MTP TPOT there was 3.0 to 3.9 ms over two runs.

Unverified:

  • Serving on aiter main. ATOM 0.1.8 imports against main but fails at model load (ImportError: pa_mqa_logits_mxfp4_block_table_width), so the step savings above are a GEMM replay, not a TPOT measurement.
  • Not covered: TP8 shapes (for example wo_b 7168x2048), data-parallel-attention shapes (6144x7168, 65536x1536, 7168x16384, 7168x3072), the wo_a batched GEMM, and 7168x7168 above M=64 (h_proj at MTP batch 8 and 16).
  • gfx1250-strict (the A0 part), which reads these rows through the base-arch fallback.

PR-specific question

Consumers: gfx1250, cu_num 256, w_scale_block 128x128, (N, K) in {(2048, 7168), (7168, 4096), (8192, 1536), (16384, 1536)} at M = 3 to 4 and 9 to 16, and (7168, 7168) at M = 1 to 64. Of these, ATOM decode reaches M = 3, 4, 12 and 16 for the layer GEMMs, and 1 to 8, 12, 16, 20, 24, 28, 32, 48 and 64 for the draft GEMMs. Evidence that nothing else moved: the before/after comparison of all 104,232 (arch, N, K, block, M) lookups in the merged table changed only those 104 lookups, each from no row to a new row. The diff adds lines and removes none. The 29 keys that #7018 and the Kimi-K3 table already cover were left alone. Under this timing method some of #7018's rows are slower than alternatives, but under #7018's own single-launch method several are faster, so I didn't replace them. #7552 (open) sends M < 16 to the M=16 row only when no power-of-two row exists. Every M < 16 here hits an exact or power-of-two row first, so lookups with both PRs equal this PR alone. If #7552 merges, the M=1/2/4/8 rows of 7168x7168 and the M=4 rows would become redundant with the M=16 rows, because they use the same kernel.

Changed op tests

None.

Add 15 gfx1250 mxfp8_128 bpreshuffle rows to the DeepSeek-V4 table for
TP4 decode GEMMs that run the untuned fallback on main:

- wqkv_a 2048x7168, wo_b 7168x4096, wq_b 16384x1536 and indexer wq_b
  8192x1536 at M=4 and 16 (serve decode M 3, 4, 12, 16)
- MTP draft e_proj / h_proj 7168x7168 at M=1, 2, 4, 8, 16, 32, 64
  (serve every decode M up to 64)

All rows are split-K 1 and bitwise equal to the fallback output. Add the
untuned shape list for the in-tree mxfp8 tuner. No existing row changes.

Signed-off-by: Hubert Lu <55214931+hubertlu-tw@users.noreply.github.com>
@hubertlu-tw
hubertlu-tw requested a review from a team November 9, 2026 06:54
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355 Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows Windows JIT op smoke test on the Radeon R9700 ROCm runner
ci:multi-gpu Aiter multi-GPU tests on the 8-GPU runner
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 7644 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

Reviews:
A reviewer is usually assigned within one working day after the PR is ready for review (see Code Review Process). If nobody has responded for about a week, mention the area contact listed in CONTACTS.md. Want to help review? See REVIEWERS.md.

@github-actions github-actions Bot added the Config label Nov 9, 2026
@aiter-bot

Copy link
Copy Markdown

READY FOR HUMAN REVIEW

PR Self-Review

Revision 061877a8caa2 · mode shadow

Author self-review accepted. Readiness report:

Author answers

  • PASS — current-SHA answers are present and structurally specific
Show the author's self-review answers

Behavior and intent

On gfx1250, `gemm_a8w8_blockscale_bpreshuffle` with e8m0 128x128 scales finds a tuned row for the 15 keys above and the padded M they serve. Before, it ran `_flydsl_mxfp8_fallback_kernel`, whose widest-N tile launches 2 to 16 workgroups at these M and takes 25 to 101 us. The new kernels take 3.6 to 9.4 us. They are from the same FlyDSL kernel family, use split-K 1, and their output is bitwise equal to the fallback's on every tested input. No existing row, other table or code path changes.

Validation and unverified scope

All on one gfx1250 GPU, aiter main `0de43e662b` + this PR, FlyDSL 0.4.2, in a container separate from the serving image:

- Numerics: each row, at every decode M it serves, ran on the tuner's inputs (random fp8 data, random e8m0 scales). Output was bitwise equal to the fallback kernel's, and errRatio against the fp32 dequantized reference was at most 0.0094. `python op_tests/test_gemm_a8w8_blockscale.py --ck_preshuffle True --flydsl -d bf16 -m 1 2 3 4 5 6 7 8 12 16 20 24 28 32 48 64 -nk 7168,7168 2048,7168 7168,4096 8192,1536 16384,1536` passed 160 cases (constant and uniform/auto inputs), max error ratio 0.0024. Real model activations were not used.
- Lookup: `get_mxscale_bpreshuffle_config` returns the new row for all 15 keys (`AITER_LOG_TUNED_CONFIG=1`). The public op returns the same bytes as the row's kernel called directly, and no "falling back" warning appears.
- Unchanged selection: I resolved `find_padded_m_row` for every (gfx, cu_num, N, K, w_scale_block) in the merged table at M = 1 to 1024 plus 8 larger M, with main's table and with this PR. Of 104,232 lookups, the only 104 that changed went from no row to a new row. Removing any one of the 15 rows changes the kernel for some decode M.
- TDM-store hazard: the picked kernels end with the gfx1250 a8w8 epilogue: `ds_store` of the output tile, then a workgroup barrier, then `tensor_store_from_lds`. A standalone Gluon repro of that sequence (no aiter kernel involved) returns stale LDS in about 7e-7 of tile stores. The open fixes for other kernels are [#7202](https://github.com/ROCm/aiter/pull/7202), [#7605](https://github.com/ROCm/aiter/pull/7605) and [#7632](https://github.com/ROCm/aiter/pull/7632). A matching fix for these a8w8 GEMMs exists on a branch of mine but is not upstream. Stress check on main: each row's kernel ran 1,000,000 times at 16 (shape, M) points covering the 15 rows (100 calls per HIP graph replay, 10,000 replays) with a bf16 4096^3 matmul on a second stream every replay. Every output was compared bitwise on the device with the first one. Result: 0 mismatches in 16M calls and 2.6e9 workgroup tile stores. At the repro's rate about 1,800 would be expected. The old fallback tiles also gave 0 in 10M calls. This bounds the rate for these kernels and shapes under this load; it doesn't show the race can't happen.
- `python3 -m unittest op_tests.tuning_tests.test_csv_validation op_tests.tuning_tests.test_config_shape_collision`: 51 passed.
- In-tree tuner: `python3 csrc/gemm_a8w8_mxfp8_bpreshuffle/gemm_a8w8_mxfp8_bpreshuffle_tune.py --mxscale_table -i aiter/configs/model_configs/dsv4_a8w8_blockscale_mxscale_bpreshuffle_untuned_gemm.csv -o <out.csv>` runs on main. It picks the same kernel for 7 of the 15 rows. Seven others are split-K 1 kernels within 2.2% of ours under the method above, which is inside the noise. For 7168x7168 M=64 it picks the split-K 4 kernel this PR declines. Its `us` column comes from a different timer and is about 6 to 10% higher.
- Serving context, on the older ATOM image's aiter (0.1.26.dev21) with a 123-row superset table loaded through `AITER_CONFIG_GEMM_A8W8_BLOCKSCALE_MXSCALE_BPRESHUFFLE`. That aiter predates #7018, and many of that table's kernels differ from these rows.
  - GSM8K 3-shot: 0.951 (TP4) and 0.957 (TP4 + MTP).
  - A 20-minute sustained-load check passed for both. It keeps 600 GSM8K requests in flight and fails on bursts of empty, non-text or answerless responses, on rolling accuracy below 0.5, or on failed probe prompts.
  - Conc-1 MTP TPOT there was 3.0 to 3.9 ms over two runs.

Unverified:
- Serving on aiter main. ATOM 0.1.8 imports against main but fails at model load (`ImportError: pa_mqa_logits_mxfp4_block_table_width`), so the step savings above are a GEMM replay, not a TPOT measurement.
- Not covered: TP8 shapes (for example `wo_b` 7168x2048), data-parallel-attention shapes (6144x7168, 65536x1536, 7168x16384, 7168x3072), the `wo_a` batched GEMM, and 7168x7168 above M=64 (`h_proj` at MTP batch 8 and 16).
- gfx1

[truncated in readiness report; see the PR description for the full answer]

PR-specific question

Consumers: gfx1250, cu_num 256, w_scale_block 128x128, (N, K) in {(2048, 7168), (7168, 4096), (8192, 1536), (16384, 1536)} at M = 3 to 4 and 9 to 16, and (7168, 7168) at M = 1 to 64. Of these, ATOM decode reaches M = 3, 4, 12 and 16 for the layer GEMMs, and 1 to 8, 12, 16, 20, 24, 28, 32, 48 and 64 for the draft GEMMs. Evidence that nothing else moved: the before/after comparison of all 104,232 (arch, N, K, block, M) lookups in the merged table changed only those 104 lookups, each from no row to a new row. The diff adds lines and removes none. The 29 keys that #7018 and the Kimi-K3 table already cover were left alone. Under this timing method some of #7018's rows are slower than alternatives, but under #7018's own single-launch method several are faster, so I didn't replace them. [#7552](https://github.com/ROCm/aiter/pull/7552) (open) sends M < 16 to the M=16 row only when no power-of-two row exists. Every M < 16 here hits an exact or power-of-two row first, so lookups with both PRs equal this PR alone. If #7552 merges, the M=1/2/4/8 rows of 7168x7168 and the M=4 rows would become redundant with the M=16 rows, because they use the same kernel.

Changed op tests

None.

PR state

  • PASS — pull request is marked ready for review

Copilot findings

  • PASS — no active Copilot finding lacks a substantive author reply
  • ADVISORY — Copilot did not review the current head SHA

Code and op-test hygiene

  • PASS — no committed artifact or conflict marker detected

Required CI

  • PASS — Black: passed
  • PASS — Ruff: passed
  • PASS — Repository dependency: passed
  • PASS — Aiter Test Gate: passed

Shadow mode reports what would block but keeps the status successful. Set AITER_PR_SELF_REVIEW_MODE=enforce only after reviewing real false positives.

@valarLip
valarLip merged commit 541185a into ROCm:main Nov 10, 2026
50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants