Repository navigation
Conversation
cd9fc7f to
0acef7c
Compare
0acef7c to
ff79769
Compare
ff79769 to
ea2cca7
Compare
…m_idxrqknorm on gfx1250 (#3405) Port of #3405 at 310014b, by @lirui927, with the prefill split at 512 tokens (the crossover measured with page 128). Needs ROCm/aiter#8011. On top of the PR, for the stack's deferred split-K QKV: the decode insert passes the partials to fused_qknorm_idxrqknorm when the kernel takes them (aiter_qknorm_takes_partials) and reduces them first otherwise, as before.
ea2cca7 to
e0c7c77
Compare
NOT READY FOR HUMAN REVIEWPR Self-ReviewRevision Author self-review accepted. Readiness report: Author answers
Show the author's self-review answersBehavior and intentValidation and unverified scopePR-specific questionChanged op testsPR state
Copilot findings
Code and op-test hygiene
Required CI
Shadow mode reports what would block but keeps the status successful. Set |
…8 layer With an fp8 KV cache the Python op asserted the index norm weights and the index cache even for a layer without index heads (num_index_heads 0, a dense attention layer), where the kernel uses none of them. Only ask for them when the layer has an index branch. Adds a dense fp8 per-token case to the op test. Signed-off-by: lirui927 <ruili@amd.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fusedQKNormIdxrQKNormKernel (qk-norm, partial NeoX RoPE and the KV / index-cache write), with the same outputs: - token and slot are wave-uniform on gfx12 (readfirstlane): scalar loads of the per-token tables, scalar branches on the slot kind - every independent load goes out first (row, norm weight, position), the cos/sin loads as soon as the position arrives, the cache slot last - one 64-bit store per lane for 16-bit rows, one 32-bit store for fp8 K / index-cache rows - warp reductions and RoPE partners with DPP / permlanex16 on gfx12, in the original xor pairing order - bf16 rounding with v_cvt_pk_bf16_f32 on gfx1250 (the same bits as hip_bfloat16 for every non-NaN FP32 result) - on gfx1250, launches of 128 or more (token, slot) rows per CU take 4 rows per warp, in every mode but the fp4 caches (gfx950 only); other archs get a trap stub for that instantiation, which the launcher never selects there The restructuring is shared by every arch: gfx942 and gfx950 compile to different (not yet measured) code too. Signed-off-by: lirui927 <ruili@amd.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
qkv_partials (optional, last argument): the fp32 [splits, >= num_tokens, row] split-K partials of a bf16 qkv whose GEMM left them unreduced (2, 4 or 8 splits), so that the GEMM can skip its reduce kernel. Each lane sums its elements as the gfx1250 split-K reduce kernel (gemm_a8w8_splitk_reduce) does: fp32, split order, one round to nearest even. It writes them back into qkv and goes on from them, so every output is what reduce + qknorm produce. This module flushes f32 denormals and the reduce kernel does not, so a lane whose partials or running sums leave the normal range redoes its sum exactly in double. Supported: bf16 insert mode with every qkv slot processed, no fp4 cache or fp8 q_out (refused on the host). The partials load has its own instances (48, one row per warp); the other instances compile to the same code as without this change. The op test checks every output bitwise against the op on the reduced qkv (a CPU fp32 sum in split order), with edge values (subnormal, overflowing, NaN and tie sums), q to q_out and in place, and the refused calls. Signed-off-by: lirui927 <ruili@amd.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e0c7c77 to
46fb3f4
Compare
…heir consumers With ROCm/aiter#7449, aiter.gemm_a8w8_bpreshuffle_splitk_deferred leaves the fp32 partials unreduced when the tuned gfx1250 PTPC GEMM splits K and M is at most max_m; the consumer claims them (aiter.splitk_partials) and reduces them as it reads the output. - LinearBase.splitk_consumer_max_m (0 = off) routes the per-token FP8 bpreshuffle GEMM through it. Not while a token graph records: its replay would run the consumer, an eager region, without the partials. - MiniMax-M3 sets it on gfx1250 to ATOM_SPLITK_CONSUMER_REDUCE_MAX_M (default 64, 0 = off) for the QKV projections whose attention impl claims the partials in rope_cache (not under Ulysses SP, whose head exchange reads qkv first) and for the dense MLP's gate/up. - PagedAttentionImpl.rope_cache hands them to triton_fused_norm_rope_cache; SparseMHAPagedAttentionImpl.rope_cache to aiter.fused_qknorm_idxrqknorm (qkv_partials, ROCm/aiter#8011) when that op takes them: the index branch runs, no fp4 cache, and an aiter with the argument. Every other path finishes qkv with the standalone reduce first. A dummy run releases them. - swiglu_oai_split_reducing, an opaque op, claims a gate/up's partials at run time and reduces them in the activation kernel. An aiter without #7449 never defers; one with #7449 but without #8011 defers and reduces in the sparse layers with the standalone kernel, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… decode For a Gemma q/k norm layer with a plain NeoX RoPE and head_dim 128 (the MiniMax-M3 dense layers), PagedAttentionImpl.rope_cache hands a decode batch to aiter fused_qknorm_idxrqknorm on gfx1250 instead of triton_fused_norm_rope_cache: one insert launch writes q and the normed, roped K and V into the SHUFFLE cache, per-token FP8 or bf16. k and v are not written back; the decode backends read only the cache. It needs ROCm/aiter#8011, which lets a layer without index heads insert into an FP8 cache (earlier versions assert index operands there) and brings the faster gfx1250 kernel; ATOM reads that off aiter.FUSED_QKNORM_IDXRQKNORM_SUPPORTS_INDEXLESS_FP8. Without it nothing changes. The op aborts on arguments it refuses, so _takes_aiter_gemma_norm_rope checks what the KV pool does not guarantee. tests/model_ops/test_gemma_qknorm_rope_aiter_gpu.py runs rope_cache against the Triton kernel (q, the dequantized K/V cache; pages of 16 and 128; CUDA graph padding rows) and checks the route each batch takes. Signed-off-by: lirui927 <ruili@amd.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efused arguments A call whose mode does not sum the split-K partials in the op's own pass (no insert, the index branch skipped, an fp4 cache, an e4m3 q_out) now reduces them into qkv first with the same sum, so a caller passes the partials of any deferred qkv without restating which modes fuse them. The fused instantiations are unchanged (gfx1250 ISA identical). Refused arguments (head_dim, rotary_dim, qkv / positions / cache layout and dtype) now raise RuntimeError before anything launches instead of aborting the process, so callers need not pre-check them; the test checks the refusals in process. The fp8 arch check counts gfx1250-strict as gfx1250 and also covers fp8 KV and index caches, which opus would store as zeros on gfx11, as it already did an e4m3 q_out. Drop FUSED_QKNORM_IDXRQKNORM_SUPPORTS_INDEXLESS_FP8: it existed only for a caller to probe; callers raise their aiter floor instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Follow-up for ROCm/ATOM#2988 and #3405 (ATOM must not probe aiter or restate its rules), in 8335393:
|
A caller with another path can ask whether the op takes a call instead of restating its limits. The query runs the op itself with check_only: the Python wrapper's checks and every host check of the C++ entry run as for the real call, then it returns before the device guard and any launch. One set of checks serves both the refusal (RuntimeError) and the query (False). The reject harness asks the query for every case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Added
|
…easured choice Drop the FUSED_QKNORM_IDXRQKNORM_SUPPORTS_INDEXLESS_FP8 probe, the copy of fused_qknorm_idxrqknorm's argument limits (head_dim, rotary_dim, qkv, positions, cache layout and dtype) and the getattr/hasattr checks on the rotary module: with ROCm/aiter#8011 the op raises on anything it refuses instead of aborting. ATOM keeps only its own choices: the route is opt-in per model (`aiter_gemma_norm_rope`, set by the MiniMax-M3 dense layers it was measured on), on the gfx1250 family (aiter's is_gfx1250_family, so gfx1250-strict counts), into its SHUFFLE cache layout. Needs aiter with #8011. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (5)
Scope is fine: the perf rework, qkv_partials and fused_qknorm_idxrqknorm_supports belong together. ROCm/ATOM#3405 is waiting on this PR. It uses supports() as its only gate, so the query has to reject exactly what the launch path rejects, and do it quietly.
- Reject a non-fp16/bf16
qkvbefore thecheck_onlyreturn. The only dtype gate isAITER_DISPATCH_FLOATING16_TYPES_rmTorchatfused_qknorm_idxrqknorm.cu:2086, afterif(check_only) return;at:2058. As a result,supports()returns True for an fp32qkvand then the op raises. AddAITER_CHECK(qkv.dtype() == AITER_DTYPE_bf16 || qkv.dtype() == AITER_DTYPE_fp16, ...)with the other checks, plus anarg_rejectcase. - Stop
supports()from turning build or load failures intoFalse.fused_qknorm_idxrqknorm.py:221catches everyRuntimeError, which includes the JIT build failure (raise_build_errorinaiter/jit/core.py) and the pybind ABI-mismatch error. A broken install then looks like "unsupported", and the caller falls back silently on every call. Either load the module outside thetry, or raise a dedicated refusal exception and catch only that. - Keep
check_onlyquiet.check_fail(aiter_hip_common.h) writes[AITER] file:line ...to stderr before it throws, even underAiterThrowGuard. EveryFalsefromsupports()therefore prints a line: ATOM#3405 asks for each head_dim-256 layer on every long prefill. Skip the print when the check is a query, or print only on the abort path. - Raise
RuntimeErrorinstead of bareassertin the fp8-cache branch (fused_qknorm_idxrqknorm.py:171-183). The comment at:130promisesRuntimeError. Underpython -Othese asserts vanish, sosupports()answers True and the C++ entry rejects the call later. - Add a correctness case for the 4-rows-per-warp path.
kRows=4is used only when there are at least128 * CUsrows (.cu:1393-1398). The largest non-fp4 default case is 64 tokens, and--minimax_tp4uses 3, so the new path and its tail rows (min(row_token, num_tokens - 1),.cu:957) never run in the op test. Add a gfx1250-gated default case above the threshold whose row count is not a multiple of 4.
Also note: gfx942/gfx950 now run changed shared code that has not been executed there yet. Please merge only after aiter-test (MI300X / MI35X) has run op_tests/test_fused_qknorm_idxrqknorm.py green; it was still pending at review time. Non-blocking: the kRows=4 instances are built for every arch but trap off gfx1250, which accounts for most of the 680 -> 1160 instance growth and the 23 s -> 63 s JIT build. Nit: the message at .cu:1649 does not mention num_index_heads.
Reviewed at f646b7e.
What it does
Five commits on
fused_qknorm_idxrqknorm(qk-norm, partial NeoX RoPE and the KV / index-cache write):num_index_heads=0, a dense attention layer) for index operands. The kernel does not use them.readfirstlane), so the per-token tables are read with scalar loads.permlanex16, in the original xor pairing order.v_cvt_pk_bf16_f32.qkv_partials(new optional last argument): the fp32[splits, >= num_tokens, row]split-K partials of a bf16qkv(2, 4 or 8 splits).gemm_a8w8_splitk_reduce) does: fp32, split order, one round to nearest even.qkvand continues from it, so the GEMM in front can skip its reduce launch.skip_index_branch, an fp4 cache, an e4m3 q_out) reduces the partials intoqkvfirst with a small kernel doing the same sum, then runs as on a reducedqkv. So a caller passes the partials of any deferredqkvand does not need to know which modes fuse them.qkv/ positions / cache layout and dtype) now raiseRuntimeErrorbefore anything launches, instead of aborting the process. A caller no longer has to pre-check them. The fp8 arch check counts gfx1250-strict as gfx1250 (base_gfx_arch) and also covers fp8 KV and index caches, which opus would store as zeros on gfx11, as it already did for an e4m3 q_out. Commit 1 no longer adds a capability flag.fused_qknorm_idxrqknorm_supports(...)(same arguments as the op) returns whether the op takes a call. It runs the op itself withcheck_only=True: the Python wrapper's checks and every host check of the C++ entry run as for the real call, then the C++ entry returns before the device guard and any launch. One set of checks serves both the refusal (RuntimeError) and the query (False). A caller with another path (perf(mha): Gemma q/k norm + RoPE + KV write through aiter fused_qknorm_idxrqknorm on gfx1250 ATOM#3405: head_dim 256 models keep their own kernels) asks this instead of restating the op's limits.Benefit
Kernel device time, one MI455X (gfx1250). MiniMax-M3 TP4 per-rank shapes: 16 q, 1 KV and 1 index head, rotary 64, page-16 SHUFFLE K/V and index caches, bf16 q / index-q outputs.
Method:
run_perftest(profiler kernel time, rotating inputs, 200 iterations). main and this branch were interleaved for 3 rounds. The table shows the median, with the min-max over rounds.mainhas noqkv_partials, so it has no baseline for that path. In the op test's partials table on MI455X,run_perftestgives about 2.3–2.9 µs without partials and 3.8–4.3 / 5.0–5.9 / 7.9–8.5 µs with 2 / 4 / 8 splits.The op test's own
--minimax_benchmark(event-timed eager calls, same interleaving) mostly measures launch overhead (25–40 µs per call). It shows no change up to M = 2048 (0.97–1.02x) and 1.08–1.25x at M = 8192.Costs:
Testing
All on one MI455X, image
rocm/fw-bringup:gfx1250-minimax-m3-20261112:python3 op_tests/test_fused_qknorm_idxrqknorm.pypasses on main and on this branch (exit 0). On gfx1250 the mxfp4 and fp8 q_out cases are skipped (gfx950-only).qkv_partials*modes cover: sparse and dense layers; M3's page-16 fp8 per-token, bf16 and page-128 caches; 2 / 4 / 8 splits; a split stride longer than the token count; q toq_outand in place; subnormal, overflowing, NaN and tie sums. Every output,qkvincluded, is compared bitwise against the op on the reducedqkv.qkv_partials_rejectchecks that a call in each mode that does not fuse the partials (no insert,skip_index_branch, e4m3 q_out) matches the op on the reducedqkvbitwise, and that partials the op cannot read are refused.arg_rejectchecks that each refused argument raisesRuntimeErrorin process (head_dim 64, rotary_dim 12, a stridedqkv, int32 or 2-D positions, a 4-D SHUFFLE cache, a cache dtype that does not matchqkv, an fp8 cache on a faked gfx1100), and that gfx1250-strict is accepted. The reject harness now runs in process instead of in a child process, and asksfused_qknorm_idxrqknorm_supportsfor every case: True for each accepted call, False for each refused one (allarg_reject,qkv_partials_rejectand mxfp4 reject cases).--minimax_tp4: every fused check passes on both. Both main and this branch then exit 1 in the last case,pa_sparse_block_score_decode, which requires gfx950 (an existing failure unrelated to this PR).Not covered
main, and it has been neither run nor timed there. The changes are shared by all archs, so their ISA could not stay unchanged without a second copy of the kernel body.qkv_partialsoff gfx1250 are not run.Dependencies
None to merge. #7449 (GEMM side, no shared files) produces the partials this op consumes. ROCm/ATOM#2988 (both) and ROCm/ATOM#3405 (
fused_qknorm_idxrqknorm_supportsand the op, non-insert mode) call aiter directly once their aiter floor includes this PR.PR Self-Review
Reviewed revision
f646b7e
Behavior and intent
num_index_heads=0) for index operands the kernel does not use.permlanex16reductions in the original xor order,v_cvt_pk_bf16_f32, 4 rows per warp from 128 rows per CU. Kernel time vsmainis 1.17x-1.67x on MiniMax-M3 TP4 shapes (description table,run_perftest, 3 interleaved rounds).qkv_partials(fp32[splits, >= num_tokens, row], 2/4/8 splits): the kernel sums them asgemm_a8w8_splitk_reducedoes, writes the reduced row back intoqkvand continues, so the GEMM in front ([FlyDSL] [Perf] Let the consumer finish a gfx1250 PTPC split-K GEMM #7449) can skip its reduce launch. Bitwise results are kept under denormal flushing via an exact double-precision redo. Fused in bf16 insert mode with every qkv slot, no fp4 cache and no fp8 q_out; any other mode reduces first with a small kernel doing the same sum.RuntimeErrorbefore any launch (AiterThrowGuard) instead of aborting; the fp8 arch check uses the gfx1250 family and covers fp8 caches. The index-less fp8 capability flag is not added.fused_qknorm_idxrqknorm_supportsanswers whether a call is taken by running the same checks withcheck_only(no second copy of the rules).Validation and unverified scope
rocm/fw-bringup:gfx1250-minimax-m3-20261112:python3 op_tests/test_fused_qknorm_idxrqknorm.pypasses on main, the previous head (46fb3f4, 27 cases) and this branch (28 cases, +arg_reject); mxfp4 / fp8 q_out skipped, gfx950-only.--minimax_tp4: all fused checks pass on both; both exit 1 inpa_sparse_block_score_decode(gfx950-only, pre-existing).mainand were neither run nor timed; HIP-graph timing not re-measured here; NVFP4 andqkv_partialsoff gfx1250 not run.PR-specific question
The kernel body is shared by all archs; what protects gfx942 / gfx950, which now run changed code that was not executed?
op_tests/test_fused_qknorm_idxrqknorm.pythere before merge; this is the main open risk of the PR.Changed op tests
op_tests/test_fused_qknorm_idxrqknorm.py(extended, not duplicated)num_index_heads=0) for the fix;qkv_partials*modes for sparse and dense layers, page-16 fp8 per-token / bf16 / page-128 caches, 2/4/8 splits, split stride longer than the token count, q toq_outand in place, subnormal / overflow / NaN / tie sums, every output (includingqkv) bitwise against the op on the reducedqkv;qkv_partials_rejectchecks the non-fused modes bitwise and refuses unreadable partials;arg_rejectchecks each refused argument raises in process.qkv_partialsdoes not exist;arg_rejectaborts the test process.