[phambinhfin]

Summary

On gfx942, nine decode rows of a8w8_blockscale_tuned_gemm_qwen38_27b.csv (Qwen3.8-27B-FP8, TP1, from #5585) pick a
CK instance that is slower than another instance already in the table when the weights come from HBM, as they do in a
decode step. This switches those nine rows.

Claims

• Cold weights: 1.17-1.33x per call on MI325X and 1.11-1.20x on MI300X for the nine rows; with one hot weight copy the
  new rows are still 1.01-1.36x (MI325X) and 1.04-1.16x (MI300X) faster. Unchanged neighbouring M move < 1%.
• vLLM decode median ITL drops 1.7-3.3% at concurrency 1-16 (MI325X, Qwen3.8-27B-FP8; MI300X, Qwen3.6-27B-FP8, which
  has the same GEMM shapes). GSM8K stays within its standard error.
• Nine rows changed in place, no row added; both instances are already used by other rows of this file, so they are in
  the compiled module_gemm_a8w8_blockscale. gfx942 / 304 CUs only.

Validation

MI325X: vllm/vllm-openai-rocm:nightly-a2bcb565… (AITER 0.1.27). MI300X: nightly-cf5cea2da37d… (AITER 0.1.28.post1).
In both, this CSV is byte-identical to main and "after" only bind-mounts this branch's file.

Kernel

gemm_a8w8_blockscale (production lookup), HIP-graph replay, us per call, median of interleaved rounds; cold = weights
rotated over ≥ 512 MB (MI325X) / 2 GiB (MI300X), hot = one weight copy.

M	N	K	main	this PR	MI325X cold	MI325X hot	MI300X cold	MI300X hot
1	5120	6144	256x16x128x128 sk2	256x16x128x256 sk2	1.28x	1.04x	1.20x	1.05x
4	5120	6144	256x16x64x128 sk2	256x16x128x256 sk2	1.17x	1.36x	1.20x	1.15x
1	14336	5120	256x16x128x256 sk0	256x16x64x256 sk0	1.33x	1.09x	1.17x	1.09x
2	14336	5120	256x16x128x128 sk1	256x16x64x256 sk0	1.28x	1.11x	1.14x	1.09x
4	14336	5120	256x16x128x128 sk1	256x16x64x256 sk0	1.27x	1.06x	1.13x	1.06x
8	14336	5120	256x16x128x128 sk1	256x16x64x256 sk0	1.29x	1.09x	1.15x	1.09x
4	16384	5120	256x16x128x128 sk1	256x16x64x256 sk0	1.22x	1.06x	1.12x	1.07x
8	16384	5120	256x16x128x128 sk1	256x16x64x256 sk0	1.23x	1.08x	1.11x	1.08x
16	34816	5120	256x16x128x128 sk0	256x16x128x256 sk0	1.24x	1.01x	1.18x	1.07x

N = 5120 / K = 6144 is o_proj / out_proj, 14336 qkv, 16384 in_proj_qkvz, 34816 gate_up. Through the
getPaddedM walk the new rows serve out_proj M = 1/3/4, qkv M = 1-8, in_proj_qkvz M = 3-8 and gate_up M = 9-16.
Relative L2 against an fp32 reference is unchanged or lower on every served M; where split-K is dropped it falls from
2.5e-3 to 1.7e-3, the bf16 output-rounding floor. CUDA-graph replay with the split-K memset inside matches eager.

Serving

Median ITL in ms, ISL 1024 / OSL 1024, fresh server per run; vLLM pads decode batches to capture sizes 1, 2, 4, 8, 16,
so the changed rows are hit (server log, AITER_LOG_TUNED_CONFIG=1).

concurrency	MI325X before → after	MI300X before → after (2 runs each, ABBA)
1	10.20 → 9.93 (−2.6%)	11.31 → 11.12 (−1.7%)
4	11.12 → 10.74 (−3.3%)	12.33 → 12.01 (−2.6%)
8	11.86 → 11.55 (−2.6%)	13.46 → 13.21 (−1.9%)
16	12.73 → 12.51 (−1.7%)	14.59 → 14.23 (−2.5%)
32	15.31 → 15.24 (−0.4%)	18.37 → 18.31 (−0.3%)

GSM8K, 5-shot, 1319 questions. MI325X (Qwen3.8-27B-FP8, lm_eval completions): flexible-extract 0.7225 ± 0.0123 →
0.7195 ± 0.0124, strict-match 0.6929 ± 0.0127 → 0.6899 ± 0.0127. MI300X (Qwen3.6-27B-FP8, vLLM's
tests/evals/gsm8k/gsm8k_eval.py, chat API, thinking off): 0.9682 → 0.9689 (± 0.0048).

Tests

test_config_shape_collision and test_csv_validation: 51 passed on main and on this branch.

Not run: Qwen3.8-27B-FP8 serving on MI300X (Qwen3.6-27B-FP8 used instead).

Details

The current rows were chosen by tuner timing that reads one resident weight copy; with weights streamed from HBM they
lose. Where a non-split instance is within 2% of the fastest it is used, so the qkv M = 1-8 and in_proj_qkvz M = 4/8
rows drop split-K, removing the output memset and the bf16 atomic accumulation.

us in the changed rows is the cold-weight kernel time (torch profiler, memset excluded), not the tuner-written us
of the other rows. Dispatch reads only kernelName and splitK.

Not added: an M = 32 row for qkv / in_proj_qkvz (256x64x64x256) was 1.13-1.17x faster cold but 0.91x hot and
raised ITL 0.3-0.5% at concurrency 32, so M = 17-32 keep the default, as in #5585. #7043 (M = 4096 rows in this file)
and #7233 (the B-preshuffle file) touch other rows and merge cleanly.

AI assistance was used to find and validate this change.

Testing

• Unit tests added or updated
• Performance benchmarks run
• Tested on MI250X
• Tested on MI300X

Documentation

• No documentation change is needed, with the reason stated below

Nine tuned rows change kernel; the CSV is the only place they are described.

Dependencies

• No new third-party dependencies added

Breaking Changes

None. No row is added or removed, and every new instance is already compiled.

PR Self-Review

Reviewed revision

48041eaebaa584ff5eea54249f11362d4897a581

Behavior and intent

On gfx942 / 304 CUs, nine Qwen3.8-27B TP1 a8w8 blockscale decode rows switch to the CK instance that is fastest with
weights streamed from HBM. Other rows, files and architectures are unchanged.

Validation and unverified scope

Kernel, serving and GSM8K results above on MI325X and MI300X. Not run: Qwen3.8-27B-FP8 itself on MI300X.

PR-specific question

Does changing a row move another M onto it? The getPaddedM walk is exact M, then the next power of two below 16 or
the next multiple of 16 up to 256, then the next power of two; the served M above follow from it and were confirmed in
the lookup log, and neighbouring M (2, 5, 8 of out_proj; 9, 16 of qkv / in_proj_qkvz; 8, 17 of gate_up) keep
their rows and timing.

Changed op tests

None.

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter standard tests on MI35X
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:mi300x	Run Aiter standard tests on MI300X in PRs; main branch always runs them
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355	Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows	Windows JIT op smoke test on the Radeon R9700 ROCm runner
ci:multi-gpu	Aiter multi-GPU tests on the 8-GPU runner
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 7752 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

Reviews:
A reviewer is usually assigned within one working day after the PR is ready for review (see Code Review Process). If nobody has responded for about a week, mention the area contact listed in CONTACTS.md. Want to help review? See REVIEWERS.md.