Repository navigation
Conversation
2c8902d to
3499407
Compare
9ecd054 to
546a3e8
Compare
At decode sizes the tuned gfx1250 PTPC GEMM splits K (MiniMax-M3 TP4 sparse QKV, M16 x N2560 x K6144: 40 output tiles on 256 CUs, split 4), and a second kernel sums the fp32 partials into bf16. That reduce reads 0.7 MB and costs a fixed ~1.3 us per launch in a HIP graph from 46 KB to 2.9 MB (1.7 us of the GEMM + reduce pair at M16), 57 times per M3 decode step. gemm_a8w8_bpreshuffle_splitk_deferred runs the same tuned kernel but, when it splits K (split 2/4/8, M <= max_m, gfx1250), leaves the partials unreduced in the allocation of the bf16 output it returns. A consumer claims them with splitk_partials() and sums them as gemm_a8w8_splitk_reduce does (fp32, split order, one round to nearest even), writing the reduced row back, so it sees and leaves behind exactly what the two-kernel path produces. reduce_splitk_partials() finishes the output with the standalone kernel for a consumer that cannot. Every other call returns gemm_a8w8_bpreshuffle's output. run_preshuffle_gemm_a8_gfx1250 and run_gemm_a8w8_bpreshuffle_gfx1250 take a caller's partials buffer (partials=) and then skip the reduce, which reduce_splitk_partials_gfx1250 launches on its own. Without partials= they behave as before. op_tests/test_gemm_a8w8_splitk_deferred.py checks the deferred GEMM + reduce bitwise against gemm_a8w8_bpreshuffle at the M3 TP4 decode shapes, M 1..64 (sparse QKV, dense QKV, dense gate/up), and the hand-off rules. Signed-off-by: lirui927 <ruili@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
49f0052 to
9f82222
Compare
READY FOR HUMAN REVIEWPR Self-ReviewRevision Author self-review accepted. Readiness report: Author answers
Show the author's self-review answersBehavior and intentValidation and unverified scopePR-specific questionChanged op testsPR state
Copilot findings
Code and op-test hygiene
Required CI
Shadow mode reports what would block but keeps the status successful. Set |
…heir consumers With ROCm/aiter#7449, aiter.gemm_a8w8_bpreshuffle_splitk_deferred leaves the fp32 partials unreduced when the tuned gfx1250 PTPC GEMM splits K and M is at most max_m; the consumer claims them (aiter.splitk_partials) and reduces them as it reads the output. - LinearBase.splitk_consumer_max_m (0 = off) routes the per-token FP8 bpreshuffle GEMM through it. Not while a token graph records: its replay would run the consumer, an eager region, without the partials. - MiniMax-M3 sets it on gfx1250 to ATOM_SPLITK_CONSUMER_REDUCE_MAX_M (default 64, 0 = off) for the QKV projections whose attention impl claims the partials in rope_cache (not under Ulysses SP, whose head exchange reads qkv first) and for the dense MLP's gate/up. - PagedAttentionImpl.rope_cache hands them to triton_fused_norm_rope_cache; SparseMHAPagedAttentionImpl.rope_cache to aiter.fused_qknorm_idxrqknorm (qkv_partials, ROCm/aiter#8011) when that op takes them: the index branch runs, no fp4 cache, and an aiter with the argument. Every other path finishes qkv with the standalone reduce first. A dummy run releases them. - swiglu_oai_split_reducing, an opaque op, claims a gate/up's partials at run time and reduces them in the activation kernel. An aiter without #7449 never defers; one with #7449 but without #8011 defers and reduces in the sparse layers with the standalone kernel, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review routing reminder: @aoli26 is the primary reviewer for the FlyDSL GEMM scope on this PR, with @coderfeli as backup. Please route the next technical review/action to the component owner; CI status remains a separate merge gate. |
…er decide Drop the aiter probes: the try-import of the deferred GEMM API and the inspect.signature check for fused_qknorm_idxrqknorm's qkv_partials. ATOM needs aiter with ROCm/aiter#7449 and #8011 (docs/environment_variables.md). Stop restating aiter's rules: the sparse rope_cache passes qkv's partials to fused_qknorm_idxrqknorm in every mode (the op reduces them itself where it cannot fuse them), and the gfx1250 gate goes: the deferred GEMM decides per call, arch included, whether it defers. Only ATOM's own Triton kernel path still reduces first. claims_splitk_partials is opt-in per class: PagedAttentionImpl and SparseMHAPagedAttentionImpl set it, a subclass does not inherit it. The model checks ATOM's own attention layer and impl by type instead of getattr defaults. The env doc names both deferred GEMMs (QKV and the dense MLP gate/up into SwiGLU-OAI). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (3)
- Make a missed claim fail loudly instead of returning the "already reduced"
None.aiter/ops/gemm_op_a8w8.pysplitk_partials(~L1053-1067) returnsNoneboth for "y holds its values" and for a still-deferredyit does not recognise: a reshaped view (y.view(m, h, d)), a column slice, a non-leading row slice, or an entry pushed out of the 64-entry table (~L1041). The consumer then reads unwritten bf16, andreduce_splitk_partials(y)returnsyunwritten too. A CPU run of the PR's own code reproduces all four cases. Fix: when the address/storage matches a live entry but the view check fails, raise. On eviction, launch the standalone reduce for the evicted entry (or raise); do not drop it silently. - Share the kernel selection with
_gemm_a8w8_bpreshuffle_intoinstead of re-deriving it in_splitk_deferrable._splitk_deferrable(~L979) re-implements the config lookup, the libtype check and the arch gate, and the deferred path also skipsgemm_a8w8_bpreshuffle's input asserts (e.g.WQ.dtype == fp8). Main has already changed that dispatch since this branch's base (INT8 bpreshuffle), so the two copies can drift and break the "bitwise equal" promise. Fix: factor one(config, kernelName)resolver and call it from both paths. - Add a CPU test of the hand-off table that CI actually runs.
op_tests/test_gemm_a8w8_splitk_deferred.pyskips everywhere except gfx1250, so CI (MI300X/MI35X) never executes any of the new logic. The table is pure host code. Fix: stub the two launchers and cover claim-once, the leading-row-slice claim, the rejected views (they should raise after item 1) and eviction.
Minor (non-blocking):
_SPLITK_DEFERRED.pop(Y.data_ptr(), None)in the non-deferred branch is dead code. A live entry holds its allocation, so a freshYcan never land on that address.- The eviction
whilecan be anif. - The comment "its address cannot be handed out again until the consumer claims it" ignores eviction.
- The test docstring says "Every check is bitwise", but the
errcolumn is an allclose check against the fp32 reference.
Reviewed at 9f82222. CI: 30 pass, 33 skipped, 0 failing (the new test skips on CI archs).
|
|
Based on
main, GEMM only, with no dependency on other PRs. The consumer side is separate: #8011 letsfused_qknorm_idxrqknormtake the partials (qkv_partials). ROCm/ATOM#2988 uses both.What this PR does
At decode sizes (M ≤ 64 per rank), the tuned gfx1250 PTPC FP8 GEMM (
gemm_a8w8_bpreshuffle, FlyDSL backend) splits K. A separategemm_a8w8_splitk_reducelaunch then sums the fp32 partials into bf16. That reduce moves very little data, so its cost is mostly the launch plus one store/reload round trip. This PR lets the kernel that reads the GEMM output do the reduction itself.gemm_a8w8_bpreshuffle_splitk_deferred(...)runs the same tuned kernel but skips the reduce. It applies when the config is the gfx1250 FlyDSL kernel with split 2/4/8, bf16 output and M ≤max_m(default 64). In every other case it returns the normal output unchanged.splitk_partials(y)hands the partials to a consumer. The consumer sums them as the reduce kernel does (fp32, split order, one round to nearest even) and writes the reduced rows back, so it sees, and leaves behind, exactly what the two-kernel path produces.reduce_splitk_partials(y)finishesyin place with the standalone reduce kernel, for a consumer that cannot take partials.run_preshuffle_gemm_a8_gfx1250(..., partials=)(andrun_gemm_a8w8_bpreshuffle_gfx1250) accepts a caller-provided partials buffer and leaves it unreduced;reduce_splitk_partials_gfx1250launches the reduce on its own. Without that argument they behave as before.Benefit
One MI455X, MiniMax-M3 TP4 decode GEMMs. 16 GEMMs with independent inputs are captured in one HIP graph; the number is the average per GEMM over replays, median of 5 rounds (a benchmark script outside this PR).
The deferred GEMM followed by
reduce_splitk_partialstakes as long asgemm_a8w8_bpreshuffle(within ±0.4 µs run to run), so the new path costs nothing by itself.What a layer saves is the last column minus what the consumer spends on summing the partials. For
fused_qknorm_idxrqknormwith #8011 that is 0.5 µs at 4 splits and 1.4–2.2 µs at 8 splits, so the MiniMax-M3 sparse attention layer saves about 1.5 µs at M 16, 0.7 µs at M 32 and 0.2 µs at M 64.Testing
All runs on MI455X.
op_tests/test_gemm_a8w8_splitk_deferred.py(works withpython3and with pytest): 21 cases, sparse QKV, dense QKV and gate/up at M 1–64, split 4 and 8. The deferred GEMM + reduce is bitwise equal togemm_a8w8_bpreshufflein every case. The test also checks the hand-off: the partials are handed out once, andreduce_splitk_partialsclaims them itself and returns a finished tensor as it is. It fails if no case actually splits K.op_tests/test_gemm_a8w8.py --no-legacyruns every FlyDSL shape of the gfx1250 tuned bpreshuffle CSV throughgemm_a8w8_bpreshuffle, whose split-K reduce now goes through the shared launcher. All 451 shapes pass. 181 are within atol/rtol 1e-2 of the fp32 reference; 270 have up to 2.4% of elements outside it from bf16 rounding (largest absolute difference 0.094). None fails the catastrophic check.Not covered
PR Self-Review
Reviewed revision
9f82222
Behavior and intent
gemm_a8w8_bpreshuffle_splitk_deferred(...): for the gfx1250 FlyDSL PTPC FP8 GEMM with split 2/4/8, bf16 output and M <=max_m(default 64), it skips thegemm_a8w8_splitk_reducelaunch and leaves the fp32 partials for the consumer (splitk_partials(y)), or finishes in place withreduce_splitk_partials(y). Every other config returns the normal output.run_preshuffle_gemm_a8_gfx1250/run_gemm_a8w8_bpreshuffle_gfx1250accept a callerpartials=buffer; without it they behave as before.Validation and unverified scope
op_tests/test_gemm_a8w8_splitk_deferred.py, 21 cases (sparse QKV, dense QKV, gate/up; M 1-64; split 4 and 8): deferred GEMM + reduce bitwise equal togemm_a8w8_bpreshuffle; hand-off claim semantics checked.op_tests/test_gemm_a8w8.py --no-legacyover all 451 FlyDSL shapes of the gfx1250 tuned CSV (reduce now via the shared launcher): all pass.PR-specific question
What behavior would each changed op test fail on before this PR, and why is that coverage not already provided by an existing canonical test?
test_gemm_a8w8_splitk_deferred.pyis new and calls APIs added here (gemm_a8w8_bpreshuffle_splitk_deferred,splitk_partials,reduce_splitk_partials), so it cannot run before this PR.test_gemm_a8w8.pychecks the finished GEMM against an fp32 reference with tolerance; it never exposes unreduced partials or the claim/hand-off contract, and it does not check bitwise equality between the deferred and two-kernel paths.Changed op tests
op_tests/test_gemm_a8w8_splitk_deferred.py(new)gemm_a8w8_bpreshuffleon MiniMax-M3 decode shapes at split 4/8; partials handed out once;reduce_splitk_partialsclaims them itself and passes a finished tensor through unchanged.test_gemm_a8w8.pyonly checks finished outputs within tolerance.