Skip to content

[HIP] [Perf] fused_qknorm_idxrqknorm: faster on gfx1250, split-K partials as input - #8011

Open
lirui927 wants to merge 5 commits into
ROCm:mainfrom
lirui927:perf/gfx1250-m3-qknorm
Open

lirui927 wants to merge 5 commits into
ROCm:mainfrom
lirui927:perf/gfx1250-m3-qknorm

Conversation

@lirui927

@lirui927 lirui927 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

What it does

Five commits on fused_qknorm_idxrqknorm (qk-norm, partial NeoX RoPE and the KV / index-cache write):

  1. Fix: with an fp8 KV cache, the Python op no longer asks an index-less layer (num_index_heads=0, a dense attention layer) for index operands. The kernel does not use them.
  2. Faster on gfx1250, same outputs. The changes:
    • The warp index is wave-uniform (readfirstlane), so the per-token tables are read with scalar loads.
    • All independent loads are issued first.
    • 16-bit rows are stored with one 64-bit store per lane, and fp8 K / index-cache rows with one 32-bit store.
    • Warp reductions and RoPE partners use DPP / permlanex16, in the original xor pairing order.
    • bf16 rounding uses v_cvt_pk_bf16_f32.
    • Launches of at least 128 rows per CU take 4 rows per warp (every mode except the fp4 caches, which are gfx950-only).
  3. qkv_partials (new optional last argument): the fp32 [splits, >= num_tokens, row] split-K partials of a bf16 qkv (2, 4 or 8 splits).
    • The kernel sums them the way the gfx1250 split-K reduce kernel (gemm_a8w8_splitk_reduce) does: fp32, split order, one round to nearest even.
    • It writes the reduced row back into qkv and continues from it, so the GEMM in front can skip its reduce launch.
    • This module flushes denormals. A lane whose partials or running sums leave the normal range redoes its sum exactly in double, so the results stay bitwise.
    • Fused in the op's own pass: bf16 insert mode with every qkv slot processed, no fp4 cache and no fp8 q_out. Any other call (no insert, skip_index_branch, an fp4 cache, an e4m3 q_out) reduces the partials into qkv first with a small kernel doing the same sum, then runs as on a reduced qkv. So a caller passes the partials of any deferred qkv and does not need to know which modes fuse them.
  4. Errors instead of aborts: arguments the op refuses (head_dim, rotary_dim, qkv / positions / cache layout and dtype) now raise RuntimeError before anything launches, instead of aborting the process. A caller no longer has to pre-check them. The fp8 arch check counts gfx1250-strict as gfx1250 (base_gfx_arch) and also covers fp8 KV and index caches, which opus would store as zeros on gfx11, as it already did for an e4m3 q_out. Commit 1 no longer adds a capability flag.
  5. fused_qknorm_idxrqknorm_supports(...) (same arguments as the op) returns whether the op takes a call. It runs the op itself with check_only=True: the Python wrapper's checks and every host check of the C++ entry run as for the real call, then the C++ entry returns before the device guard and any launch. One set of checks serves both the refusal (RuntimeError) and the query (False). A caller with another path (perf(mha): Gemma q/k norm + RoPE + KV write through aiter fused_qknorm_idxrqknorm on gfx1250 ATOM#3405: head_dim 256 models keep their own kernels) asks this instead of restating the op's limits.

Benefit

Kernel device time, one MI455X (gfx1250). MiniMax-M3 TP4 per-rank shapes: 16 q, 1 KV and 1 index head, rotary 64, page-16 SHUFFLE K/V and index caches, bf16 q / index-q outputs.

Method: run_perftest (profiler kernel time, rotating inputs, 200 iterations). main and this branch were interleaved for 3 rounds. The table shows the median, with the min-max over rounds.

cache M main us (min-max) branch us (min-max) speed-up
bf16 1 2.67 (2.66-2.68) 2.19 (2.15-2.21) 1.22x
bf16 8 3.05 (3.01-3.05) 2.54 (2.51-2.55) 1.20x
bf16 16 3.23 (3.19-3.24) 2.76 (2.72-2.76) 1.17x
bf16 64 3.63 (3.61-3.64) 2.68 (2.66-2.68) 1.35x
bf16 128 4.06 (4.04-4.11) 2.95 (2.90-3.05) 1.38x
bf16 512 6.42 (6.39-6.43) 4.78 (4.71-4.79) 1.34x
bf16 2048 11.74 (11.65-11.76) 7.98 (7.95-8.06) 1.47x
bf16 8192 30.78 (30.54-30.98) 18.41 (18.35-18.57) 1.67x
fp8 per-token 1 2.91 (2.86-2.92) 2.29 (2.28-2.31) 1.27x
fp8 per-token 8 3.30 (3.29-3.30) 2.77 (2.72-2.80) 1.19x
fp8 per-token 16 3.45 (3.41-3.46) 2.79 (2.78-2.80) 1.24x
fp8 per-token 64 3.88 (3.88-3.97) 2.85 (2.85-2.86) 1.36x
fp8 per-token 128 4.90 (4.71-4.97) 3.09 (3.06-3.10) 1.59x
fp8 per-token 512 6.26 (6.25-6.29) 4.97 (4.92-5.00) 1.26x
fp8 per-token 2048 12.61 (12.61-12.67) 9.74 (9.67-9.78) 1.29x
fp8 per-token 8192 31.71 (31.64-32.00) 19.67 (19.66-19.83) 1.61x
  • main has no qkv_partials, so it has no baseline for that path. In the op test's partials table on MI455X, run_perftest gives about 2.3–2.9 µs without partials and 3.8–4.3 / 5.0–5.9 / 7.9–8.5 µs with 2 / 4 / 8 splits.
  • Those inputs include many edge-value columns, which take the exact (double) path, so the partials costs are upper bounds.
  • The author's figures (57 layers in one HIP graph) are 0.5 µs at 4 splits and 1.4–2.2 µs at 8 splits. The reduce launch it replaces costs 1.6–2.6 µs ([FlyDSL] [Perf] Let the consumer finish a gfx1250 PTPC split-K GEMM #7449).

The op test's own --minimax_benchmark (event-timed eager calls, same interleaving) mostly measures launch overhead (25–40 µs per call). It shows no change up to M = 2048 (0.97–1.02x) and 1.08–1.25x at M = 8192.

Costs:

  • 1160 kernel instances instead of 680.
  • gfx1250 JIT build of the module: 63 s instead of 23 s, and 15.2 MB instead of 8.1 MB.

Testing

All on one MI455X, image rocm/fw-bringup:gfx1250-minimax-m3-20261112:

  • python3 op_tests/test_fused_qknorm_idxrqknorm.py passes on main and on this branch (exit 0). On gfx1250 the mxfp4 and fp8 q_out cases are skipped (gfx950-only).
    • It is extended, not duplicated. A dense fp8 per-token case covers the fix.
    • qkv_partials* modes cover: sparse and dense layers; M3's page-16 fp8 per-token, bf16 and page-128 caches; 2 / 4 / 8 splits; a split stride longer than the token count; q to q_out and in place; subnormal, overflowing, NaN and tie sums. Every output, qkv included, is compared bitwise against the op on the reduced qkv.
    • qkv_partials_reject checks that a call in each mode that does not fuse the partials (no insert, skip_index_branch, e4m3 q_out) matches the op on the reduced qkv bitwise, and that partials the op cannot read are refused.
    • arg_reject checks that each refused argument raises RuntimeError in process (head_dim 64, rotary_dim 12, a strided qkv, int32 or 2-D positions, a 4-D SHUFFLE cache, a cache dtype that does not match qkv, an fp8 cache on a faked gfx1100), and that gfx1250-strict is accepted. The reject harness now runs in process instead of in a child process, and asks fused_qknorm_idxrqknorm_supports for every case: True for each accepted call, False for each refused one (all arg_reject, qkv_partials_reject and mxfp4 reject cases).
  • The original partials test of this PR, whose reference is main's FlyDSL split-K reduce kernel: 11 passed on this branch.
  • --minimax_tp4: every fused check passes on both. Both main and this branch then exit 1 in the last case, pa_sparse_block_score_decode, which requires gfx950 (an existing failure unrelated to this PR).
  • ISA: device code compiled for gfx1250, gfx942 and gfx950 and compared per instance. This branch's 1160 instances are identical to the PR's original single commit. Commit 2's instances match the final non-partials ones. Commits 4 and 5 leave all 1160 gfx1250 instances identical; commit 4 adds only the reduce kernel, commit 5 is host-only.
    • gfx942 / gfx950 vs main: no scratch, occupancy unchanged; VGPR median 26 → 28.

Not covered

  • gfx942 / gfx950 run different code from main, and it has been neither run nor timed there. The changes are shared by all archs, so their ISA could not stay unchanged without a second copy of the kernel body.
  • HIP-graph timing was not re-measured. The author reports 1.37–1.52x at decode and 1.71–1.90x at prefill in a 57-layer HIP graph; the non-insert mode at M = 16 is 0.07 µs slower there.
  • NVFP4 (gfx950 only) and qkv_partials off gfx1250 are not run.
  • The reduce-first path (modes that do not fuse the partials) is checked for bits only and not timed. No current caller defers the GEMM in front of those modes at decode sizes.

Dependencies

None to merge. #7449 (GEMM side, no shared files) produces the partials this op consumes. ROCm/ATOM#2988 (both) and ROCm/ATOM#3405 (fused_qknorm_idxrqknorm_supports and the op, non-insert mode) call aiter directly once their aiter floor includes this PR.

PR Self-Review

Reviewed revision

f646b7e

Behavior and intent

  • Fix: with an fp8 KV cache, the Python op no longer asks an index-less layer (num_index_heads=0) for index operands the kernel does not use.
  • gfx1250 speed-up with the same outputs: wave-uniform warp index with scalar table loads, loads issued up front, wider stores, DPP / permlanex16 reductions in the original xor order, v_cvt_pk_bf16_f32, 4 rows per warp from 128 rows per CU. Kernel time vs main is 1.17x-1.67x on MiniMax-M3 TP4 shapes (description table, run_perftest, 3 interleaved rounds).
  • New optional qkv_partials (fp32 [splits, >= num_tokens, row], 2/4/8 splits): the kernel sums them as gemm_a8w8_splitk_reduce does, writes the reduced row back into qkv and continues, so the GEMM in front ([FlyDSL] [Perf] Let the consumer finish a gfx1250 PTPC split-K GEMM #7449) can skip its reduce launch. Bitwise results are kept under denormal flushing via an exact double-precision redo. Fused in bf16 insert mode with every qkv slot, no fp4 cache and no fp8 q_out; any other mode reduces first with a small kernel doing the same sum.
  • Refused arguments raise RuntimeError before any launch (AiterThrowGuard) instead of aborting; the fp8 arch check uses the gfx1250 family and covers fp8 caches. The index-less fp8 capability flag is not added. fused_qknorm_idxrqknorm_supports answers whether a call is taken by running the same checks with check_only (no second copy of the rules).
  • Costs: 1160 kernel instances instead of 680; gfx1250 JIT build 63 s instead of 23 s.

Validation and unverified scope

  • One MI455X, rocm/fw-bringup:gfx1250-minimax-m3-20261112:
    • python3 op_tests/test_fused_qknorm_idxrqknorm.py passes on main, the previous head (46fb3f4, 27 cases) and this branch (28 cases, + arg_reject); mxfp4 / fp8 q_out skipped, gfx950-only.
    • The PR's original partials test (reference: main's FlyDSL split-K reduce kernel): 11 passed.
    • --minimax_tp4: all fused checks pass on both; both exit 1 in pa_sparse_block_score_decode (gfx950-only, pre-existing).
    • ISA compared per instance for gfx1250/gfx942/gfx950; gfx942/gfx950 vs main: no scratch, occupancy unchanged, VGPR median 26 -> 28.
  • Unverified: gfx942/gfx950 run different code from main and were neither run nor timed; HIP-graph timing not re-measured here; NVFP4 and qkv_partials off gfx1250 not run.

PR-specific question

The kernel body is shared by all archs; what protects gfx942 / gfx950, which now run changed code that was not executed?

  • The changes are semantic no-ops by construction (same reduction pairing order, same rounding), and the ISA for gfx942/gfx950 compiles without scratch and at unchanged occupancy.
  • That is compile-level evidence only. A reviewer with gfx942/gfx950 access should run op_tests/test_fused_qknorm_idxrqknorm.py there before merge; this is the main open risk of the PR.

Changed op tests

  • op_tests/test_fused_qknorm_idxrqknorm.py (extended, not duplicated)
    • Covers: a dense fp8 per-token case (num_index_heads=0) for the fix; qkv_partials* modes for sparse and dense layers, page-16 fp8 per-token / bf16 / page-128 caches, 2/4/8 splits, split stride longer than the token count, q to q_out and in place, subnormal / overflow / NaN / tie sums, every output (including qkv) bitwise against the op on the reduced qkv; qkv_partials_reject checks the non-fused modes bitwise and refuses unreadable partials; arg_reject checks each refused argument raises in process.
    • Before this PR: the dense fp8 case trips the Python asserts that require index operands; the partials modes fail because qkv_partials does not exist; arg_reject aborts the test process.
    • Not redundant: no other test covers index-less fp8 layers or in-kernel split-K reduction; the perf changes are covered by the existing modes, which must stay bitwise equal.

@lirui927
lirui927 force-pushed the perf/gfx1250-m3-qknorm branch from cd9fc7f to 0acef7c Compare November 16, 2026 05:35
@lirui927
lirui927 changed the base branch from xiaobingsuper/m3-mi455-perf-stack to main November 16, 2026 05:35
@lirui927
lirui927 force-pushed the perf/gfx1250-m3-qknorm branch from 0acef7c to ff79769 Compare November 16, 2026 06:36
@lirui927 lirui927 changed the title [HIP] [Perf] gfx1250: faster fused_qknorm_idxrqknorm for MiniMax-M3 [HIP] [Perf] fused_qknorm_idxrqknorm: faster on gfx1250, several rows per warp for large launches Nov 16, 2026
@lirui927
lirui927 force-pushed the perf/gfx1250-m3-qknorm branch from ff79769 to ea2cca7 Compare November 16, 2026 08:07
@lirui927 lirui927 changed the title [HIP] [Perf] fused_qknorm_idxrqknorm: faster on gfx1250, several rows per warp for large launches [HIP] [Perf] fused_qknorm_idxrqknorm: faster on gfx1250, split-K partials as input Nov 16, 2026
lirui927 pushed a commit that referenced this pull request Nov 16, 2026
…ials as input (#8011)

Squash of #8011 at ea2cca7, by @lirui927.

#8011 brings the split-K partials consumer of 87a9321 (an earlier
#7449) to the optimized kernel, so the kernel and wrapper are
taken whole from #8011; the header and pybind already match.
lirui927 pushed a commit to ROCm/ATOM that referenced this pull request Nov 16, 2026
…m_idxrqknorm on gfx1250 (#3405)

Port of #3405 at 310014b, by @lirui927,
with the prefill split at 512 tokens (the crossover measured with page 128).
Needs ROCm/aiter#8011.

On top of the PR, for the stack's deferred split-K QKV: the decode insert
passes the partials to fused_qknorm_idxrqknorm when the kernel takes them
(aiter_qknorm_takes_partials) and reduces them first otherwise, as before.
@lirui927
lirui927 force-pushed the perf/gfx1250-m3-qknorm branch from ea2cca7 to e0c7c77 Compare November 16, 2026 14:32
@lirui927
lirui927 marked this pull request as ready for review November 16, 2026 14:40
@lirui927
lirui927 requested a review from a team November 16, 2026 14:40
@github-actions github-actions Bot added the HIP label Nov 16, 2026
@aiter-bot

aiter-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

NOT READY FOR HUMAN REVIEW

PR Self-Review

Revision 46fb3f4756e0 · mode shadow

Author self-review accepted. Readiness report:

Author answers

  • PASS — current-SHA answers are present and structurally specific
Show the author's self-review answers

Behavior and intent

- Fix: with an fp8 KV cache, the Python op no longer asks an index-less layer (`num_index_heads=0`) for index operands the kernel does not use.
- gfx1250 speed-up with the same outputs: wave-uniform warp index with scalar table loads, loads issued up front, wider stores, DPP / `permlanex16` reductions in the original xor order, `v_cvt_pk_bf16_f32`, 4 rows per warp from 128 rows per CU. Kernel time vs `main` is 1.17x-1.67x on MiniMax-M3 TP4 shapes (description table, `run_perftest`, 3 interleaved rounds).
- New optional `qkv_partials` (fp32 `[splits, >= num_tokens, row]`, 2/4/8 splits): the kernel sums them as `gemm_a8w8_splitk_reduce` does, writes the reduced row back into `qkv` and continues, so the GEMM in front (#7449) can skip its reduce launch. Bitwise results are kept under denormal flushing via an exact double-precision redo. Only bf16 insert mode with every qkv slot, no fp4 cache and no fp8 q_out; the host refuses anything else.
- Costs: 1160 kernel instances instead of 680; gfx1250 JIT build 63 s instead of 23 s.

Validation and unverified scope

- One MI455X, `rocm/fw-bringup:gfx1250-minimax-m3-20261112`:
  - `python3 op_tests/test_fused_qknorm_idxrqknorm.py` passes on main and this branch (mxfp4 / fp8 q_out skipped, gfx950-only).
  - The PR's original partials test (reference: main's FlyDSL split-K reduce kernel): 11 passed.
  - `--minimax_tp4`: all fused checks pass on both; both exit 1 in `pa_sparse_block_score_decode` (gfx950-only, pre-existing).
  - ISA compared per instance for gfx1250/gfx942/gfx950; gfx942/gfx950 vs main: no scratch, occupancy unchanged, VGPR median 26 -> 28.
- Unverified: gfx942/gfx950 run different code from `main` and were neither run nor timed; HIP-graph timing not re-measured here; NVFP4 and `qkv_partials` off gfx1250 not run.

PR-specific question

The kernel body is shared by all archs; what protects gfx942 / gfx950, which now run changed code that was not executed?
- The changes are semantic no-ops by construction (same reduction pairing order, same rounding), and the ISA for gfx942/gfx950 compiles without scratch and at unchanged occupancy.
- That is compile-level evidence only. A reviewer with gfx942/gfx950 access should run `op_tests/test_fused_qknorm_idxrqknorm.py` there before merge; this is the main open risk of the PR.

Changed op tests

- `op_tests/test_fused_qknorm_idxrqknorm.py` (extended, not duplicated)
  - Covers: a dense fp8 per-token case (`num_index_heads=0`) for the fix; `qkv_partials*` modes for sparse and dense layers, page-16 fp8 per-token / bf16 / page-128 caches, 2/4/8 splits, split stride longer than the token count, q to `q_out` and in place, subnormal / overflow / NaN / tie sums, every output (including `qkv`) bitwise against the op on the reduced `qkv`; `qkv_partials_reject` checks each unsupported call is refused.
  - Before this PR: the dense fp8 case trips the Python asserts that require index operands; the partials modes fail because `qkv_partials` does not exist.
  - Not redundant: no other test covers index-less fp8 layers or in-kernel split-K reduction; the perf changes are covered by the existing modes, which must stay bitwise equal.

PR state

  • PASS — pull request is marked ready for review

Copilot findings

  • PASS — no active Copilot finding lacks a substantive author reply
  • ADVISORY — Copilot did not review the current head SHA

Code and op-test hygiene

  • PASS — no committed artifact or conflict marker detected

Required CI

  • PASS — Black: passed
  • PASS — Ruff: passed
  • PASS — Repository dependency: passed
  • FAIL — Aiter Test Gate: failed
  • Required CI failed; fix it before requesting human review.

Shadow mode reports what would block but keeps the status successful. Set AITER_PR_SELF_REVIEW_MODE=enforce only after reviewing real false positives.

@lirui927
lirui927 requested a review from valarLip November 16, 2026 14:52
lirui927 and others added 3 commits November 17, 2026 02:07
…8 layer

With an fp8 KV cache the Python op asserted the index norm weights and
the index cache even for a layer without index heads (num_index_heads
0, a dense attention layer), where the kernel uses none of them. Only
ask for them when the layer has an index branch.

Adds a dense fp8 per-token case to the op test.

Signed-off-by: lirui927 <ruili@amd.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fusedQKNormIdxrQKNormKernel (qk-norm, partial NeoX RoPE and the KV /
index-cache write), with the same outputs:

- token and slot are wave-uniform on gfx12 (readfirstlane): scalar
  loads of the per-token tables, scalar branches on the slot kind
- every independent load goes out first (row, norm weight, position),
  the cos/sin loads as soon as the position arrives, the cache slot last
- one 64-bit store per lane for 16-bit rows, one 32-bit store for fp8
  K / index-cache rows
- warp reductions and RoPE partners with DPP / permlanex16 on gfx12, in
  the original xor pairing order
- bf16 rounding with v_cvt_pk_bf16_f32 on gfx1250 (the same bits as
  hip_bfloat16 for every non-NaN FP32 result)
- on gfx1250, launches of 128 or more (token, slot) rows per CU take 4
  rows per warp, in every mode but the fp4 caches (gfx950 only); other
  archs get a trap stub for that instantiation, which the launcher never
  selects there

The restructuring is shared by every arch: gfx942 and gfx950 compile to
different (not yet measured) code too.

Signed-off-by: lirui927 <ruili@amd.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
qkv_partials (optional, last argument): the fp32 [splits, >= num_tokens,
row] split-K partials of a bf16 qkv whose GEMM left them unreduced (2,
4 or 8 splits), so that the GEMM can skip its reduce kernel. Each lane
sums its elements as the gfx1250 split-K reduce kernel
(gemm_a8w8_splitk_reduce) does: fp32, split order, one round to nearest
even. It writes them back into qkv and goes on from them, so every
output is what reduce + qknorm produce. This module flushes f32
denormals and the reduce kernel does not, so a lane whose partials or
running sums leave the normal range redoes its sum exactly in double.

Supported: bf16 insert mode with every qkv slot processed, no fp4 cache
or fp8 q_out (refused on the host). The partials load has its own
instances (48, one row per warp); the other instances compile to the
same code as without this change.

The op test checks every output bitwise against the op on the reduced
qkv (a CPU fp32 sum in split order), with edge values (subnormal,
overflowing, NaN and tie sums), q to q_out and in place, and the
refused calls.

Signed-off-by: lirui927 <ruili@amd.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lirui927
lirui927 force-pushed the perf/gfx1250-m3-qknorm branch from e0c7c77 to 46fb3f4 Compare November 17, 2026 02:07
lirui927 pushed a commit to lirui927/ATOM that referenced this pull request Nov 17, 2026
…heir consumers

With ROCm/aiter#7449, aiter.gemm_a8w8_bpreshuffle_splitk_deferred leaves the
fp32 partials unreduced when the tuned gfx1250 PTPC GEMM splits K and M is at
most max_m; the consumer claims them (aiter.splitk_partials) and reduces them
as it reads the output.

- LinearBase.splitk_consumer_max_m (0 = off) routes the per-token FP8
  bpreshuffle GEMM through it. Not while a token graph records: its replay
  would run the consumer, an eager region, without the partials.
- MiniMax-M3 sets it on gfx1250 to ATOM_SPLITK_CONSUMER_REDUCE_MAX_M
  (default 64, 0 = off) for the QKV projections whose attention impl claims
  the partials in rope_cache (not under Ulysses SP, whose head exchange reads
  qkv first) and for the dense MLP's gate/up.
- PagedAttentionImpl.rope_cache hands them to triton_fused_norm_rope_cache;
  SparseMHAPagedAttentionImpl.rope_cache to aiter.fused_qknorm_idxrqknorm
  (qkv_partials, ROCm/aiter#8011) when that op takes them: the index branch
  runs, no fp4 cache, and an aiter with the argument. Every other path
  finishes qkv with the standalone reduce first. A dummy run releases them.
- swiglu_oai_split_reducing, an opaque op, claims a gate/up's partials at run
  time and reduces them in the activation kernel.

An aiter without #7449 never defers; one with #7449 but without #8011 defers
and reduces in the sparse layers with the standalone kernel, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lirui927 pushed a commit to lirui927/ATOM that referenced this pull request Nov 17, 2026
… decode

For a Gemma q/k norm layer with a plain NeoX RoPE and head_dim 128 (the
MiniMax-M3 dense layers), PagedAttentionImpl.rope_cache hands a decode batch
to aiter fused_qknorm_idxrqknorm on gfx1250 instead of
triton_fused_norm_rope_cache: one insert launch writes q and the normed,
roped K and V into the SHUFFLE cache, per-token FP8 or bf16. k and v are
not written back; the decode backends read only the cache.

It needs ROCm/aiter#8011, which lets a layer without index heads insert
into an FP8 cache (earlier versions assert index operands there) and
brings the faster gfx1250 kernel; ATOM reads that off
aiter.FUSED_QKNORM_IDXRQKNORM_SUPPORTS_INDEXLESS_FP8. Without it nothing
changes. The op aborts on arguments it refuses, so
_takes_aiter_gemma_norm_rope checks what the KV pool does not guarantee.

tests/model_ops/test_gemma_qknorm_rope_aiter_gpu.py runs rope_cache against
the Triton kernel (q, the dequantized K/V cache; pages of 16 and 128; CUDA
graph padding rows) and checks the route each batch takes.

Signed-off-by: lirui927 <ruili@amd.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efused arguments

A call whose mode does not sum the split-K partials in the op's own pass (no
insert, the index branch skipped, an fp4 cache, an e4m3 q_out) now reduces
them into qkv first with the same sum, so a caller passes the partials of
any deferred qkv without restating which modes fuse them. The fused
instantiations are unchanged (gfx1250 ISA identical).

Refused arguments (head_dim, rotary_dim, qkv / positions / cache layout and
dtype) now raise RuntimeError before anything launches instead of aborting
the process, so callers need not pre-check them; the test checks the
refusals in process. The fp8 arch check counts gfx1250-strict as gfx1250 and
also covers fp8 KV and index caches, which opus would store as zeros on
gfx11, as it already did an e4m3 q_out.

Drop FUSED_QKNORM_IDXRQKNORM_SUPPORTS_INDEXLESS_FP8: it existed only for a
caller to probe; callers raise their aiter floor instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lirui927

Copy link
Copy Markdown
Contributor Author

Follow-up for ROCm/ATOM#2988 and #3405 (ATOM must not probe aiter or restate its rules), in 8335393:

  • qkv_partials is now accepted in every mode. Where the op cannot sum the partials in its own pass (no insert, skip_index_branch, fp4 caches, an e4m3 q_out), it first reduces them into qkv with a small kernel that does the same sum, then runs as on a reduced qkv. Callers just pass the partials. The fused instantiations are unchanged (all 1160 gfx1250 instances have identical ISA).
  • Refused arguments now raise RuntimeError before anything launches (AiterThrowGuard) instead of aborting the process. This covers head_dim, rotary_dim, qkv / positions / cache layout and dtype. The fp8 arch check counts gfx1250-strict as gfx1250 and now also covers fp8 KV / index caches, which opus would store as zeros on gfx11.
  • FUSED_QKNORM_IDXRQKNORM_SUPPORTS_INDEXLESS_FP8 is dropped. It existed only so ATOM could probe for it; ATOM raises its aiter floor instead. The two existing flags on main stay.
  • Test: qkv_partials_reject now checks the non-fused modes bitwise against the reduced-qkv call. A new arg_reject mode checks that each refused argument raises in process, and the reject harness no longer needs a child process. On MI455X: 28/28 cases pass, versus 27/27 at the previous head.

A caller with another path can ask whether the op takes a call instead of
restating its limits. The query runs the op itself with check_only: the
Python wrapper's checks and every host check of the C++ entry run as for
the real call, then it returns before the device guard and any launch. One
set of checks serves both the refusal (RuntimeError) and the query (False).
The reject harness asks the query for every case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lirui927

Copy link
Copy Markdown
Contributor Author

Added fused_qknorm_idxrqknorm_supports(...) in f646b7e, so a caller that has another path can ask instead of restating the op's limits:

  • It takes the op's own arguments and returns a bool. It runs the op with a new check_only=True: the Python wrapper's checks and every host check of the C++ entry run exactly as for the real call, then the entry returns before the device guard and any launch. One set of checks serves both the refusal (RuntimeError) and the query (False); there is no second copy of the rules.
  • The reject harness asks the query for every case: True for each accepted call, False for each refused one (arg_reject, qkv_partials_reject, the mxfp4 rejects).
  • Host-only change: all 1160 gfx1250 kernel instances keep identical ISA.
  • perf(mha): Gemma q/k norm + RoPE + KV write through aiter fused_qknorm_idxrqknorm on gfx1250 ATOM#3405 is being narrowed (update pushed after its TP4 run) to the long-prefill norm + RoPE route (non-insert mode). It uses this query to keep head_dim-256 layers on their own path. It doesn't depend on the index-less fp8 insert fix in commit 1, which stays as a real fix for fp8 inserts of dense layers.

lirui927 pushed a commit to lirui927/ATOM that referenced this pull request Nov 17, 2026
…easured choice

Drop the FUSED_QKNORM_IDXRQKNORM_SUPPORTS_INDEXLESS_FP8 probe, the copy of
fused_qknorm_idxrqknorm's argument limits (head_dim, rotary_dim, qkv,
positions, cache layout and dtype) and the getattr/hasattr checks on the
rotary module: with ROCm/aiter#8011 the op raises on anything it refuses
instead of aborting. ATOM keeps only its own choices: the route is opt-in
per model (`aiter_gemma_norm_rope`, set by the MiniMax-M3 dense layers it
was measured on), on the gfx1250 family (aiter's is_gfx1250_family, so
gfx1250-strict counts), into its SHUFFLE cache layout. Needs aiter with
#8011.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (5)

Scope is fine: the perf rework, qkv_partials and fused_qknorm_idxrqknorm_supports belong together. ROCm/ATOM#3405 is waiting on this PR. It uses supports() as its only gate, so the query has to reject exactly what the launch path rejects, and do it quietly.

  1. Reject a non-fp16/bf16 qkv before the check_only return. The only dtype gate is AITER_DISPATCH_FLOATING16_TYPES_rmTorch at fused_qknorm_idxrqknorm.cu:2086, after if(check_only) return; at :2058. As a result, supports() returns True for an fp32 qkv and then the op raises. Add AITER_CHECK(qkv.dtype() == AITER_DTYPE_bf16 || qkv.dtype() == AITER_DTYPE_fp16, ...) with the other checks, plus an arg_reject case.
  2. Stop supports() from turning build or load failures into False. fused_qknorm_idxrqknorm.py:221 catches every RuntimeError, which includes the JIT build failure (raise_build_error in aiter/jit/core.py) and the pybind ABI-mismatch error. A broken install then looks like "unsupported", and the caller falls back silently on every call. Either load the module outside the try, or raise a dedicated refusal exception and catch only that.
  3. Keep check_only quiet. check_fail (aiter_hip_common.h) writes [AITER] file:line ... to stderr before it throws, even under AiterThrowGuard. Every False from supports() therefore prints a line: ATOM#3405 asks for each head_dim-256 layer on every long prefill. Skip the print when the check is a query, or print only on the abort path.
  4. Raise RuntimeError instead of bare assert in the fp8-cache branch (fused_qknorm_idxrqknorm.py:171-183). The comment at :130 promises RuntimeError. Under python -O these asserts vanish, so supports() answers True and the C++ entry rejects the call later.
  5. Add a correctness case for the 4-rows-per-warp path. kRows=4 is used only when there are at least 128 * CUs rows (.cu:1393-1398). The largest non-fp4 default case is 64 tokens, and --minimax_tp4 uses 3, so the new path and its tail rows (min(row_token, num_tokens - 1), .cu:957) never run in the op test. Add a gfx1250-gated default case above the threshold whose row count is not a multiple of 4.

Also note: gfx942/gfx950 now run changed shared code that has not been executed there yet. Please merge only after aiter-test (MI300X / MI35X) has run op_tests/test_fused_qknorm_idxrqknorm.py green; it was still pending at review time. Non-blocking: the kRows=4 instances are built for every arch but trap off gfx1250, which accounts for most of the 680 -> 1160 instance growth and the 23 s -> 63 s JIT build. Nit: the message at .cu:1649 does not mention num_index_heads.


Reviewed at f646b7e.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants