Skip to content

[FlyDSL] [Perf] Let the consumer finish a gfx1250 PTPC split-K GEMM - #7449

Open
lirui927 wants to merge 1 commit into
ROCm:mainfrom
lirui927:fuse/f2-splitk-consumer
Open

lirui927 wants to merge 1 commit into
ROCm:mainfrom
lirui927:fuse/f2-splitk-consumer

Conversation

@lirui927

@lirui927 lirui927 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Based on main, GEMM only, with no dependency on other PRs. The consumer side is separate: #8011 lets fused_qknorm_idxrqknorm take the partials (qkv_partials). ROCm/ATOM#2988 uses both.

What this PR does

At decode sizes (M ≤ 64 per rank), the tuned gfx1250 PTPC FP8 GEMM (gemm_a8w8_bpreshuffle, FlyDSL backend) splits K. A separate gemm_a8w8_splitk_reduce launch then sums the fp32 partials into bf16. That reduce moves very little data, so its cost is mostly the launch plus one store/reload round trip. This PR lets the kernel that reads the GEMM output do the reduction itself.

  • gemm_a8w8_bpreshuffle_splitk_deferred(...) runs the same tuned kernel but skips the reduce. It applies when the config is the gfx1250 FlyDSL kernel with split 2/4/8, bf16 output and M ≤ max_m (default 64). In every other case it returns the normal output unchanged.
    • The returned bf16 tensor shares one allocation with the fp32 partials.
    • splitk_partials(y) hands the partials to a consumer. The consumer sums them as the reduce kernel does (fp32, split order, one round to nearest even) and writes the reduced rows back, so it sees, and leaves behind, exactly what the two-kernel path produces.
    • reduce_splitk_partials(y) finishes y in place with the standalone reduce kernel, for a consumer that cannot take partials.
  • run_preshuffle_gemm_a8_gfx1250(..., partials=) (and run_gemm_a8w8_bpreshuffle_gfx1250) accepts a caller-provided partials buffer and leaves it unreduced; reduce_splitk_partials_gfx1250 launches the reduce on its own. Without that argument they behave as before.

Benefit

One MI455X, MiniMax-M3 TP4 decode GEMMs. 16 GEMMs with independent inputs are captured in one HIP graph; the number is the average per GEMM over replays, median of 5 rounds (a benchmark script outside this PR).

GEMM (N × K) M split GEMM + reduce (µs) deferred GEMM (µs) left to the consumer (µs)
sparse QKV (2560 × 6144) 1 4 5.36 3.73 1.63
sparse QKV (2560 × 6144) 16 4 5.83 3.85 1.98
sparse QKV (2560 × 6144) 64 8 6.53 4.14 2.39
dense QKV (2304 × 6144) 1 8 5.59 3.67 1.92
dense QKV (2304 × 6144) 16 8 5.78 3.62 2.17
dense QKV (2304 × 6144) 64 8 6.64 4.32 2.32
gate/up (6144 × 6144) 1 8 6.32 4.58 1.74
gate/up (6144 × 6144) 16 8 7.11 4.85 2.26
gate/up (6144 × 6144) 64 4 7.94 5.38 2.56

The deferred GEMM followed by reduce_splitk_partials takes as long as gemm_a8w8_bpreshuffle (within ±0.4 µs run to run), so the new path costs nothing by itself.

What a layer saves is the last column minus what the consumer spends on summing the partials. For fused_qknorm_idxrqknorm with #8011 that is 0.5 µs at 4 splits and 1.4–2.2 µs at 8 splits, so the MiniMax-M3 sparse attention layer saves about 1.5 µs at M 16, 0.7 µs at M 32 and 0.2 µs at M 64.

Testing

All runs on MI455X.

  • op_tests/test_gemm_a8w8_splitk_deferred.py (works with python3 and with pytest): 21 cases, sparse QKV, dense QKV and gate/up at M 1–64, split 4 and 8. The deferred GEMM + reduce is bitwise equal to gemm_a8w8_bpreshuffle in every case. The test also checks the hand-off: the partials are handed out once, and reduce_splitk_partials claims them itself and returns a finished tensor as it is. It fails if no case actually splits K.
  • op_tests/test_gemm_a8w8.py --no-legacy runs every FlyDSL shape of the gfx1250 tuned bpreshuffle CSV through gemm_a8w8_bpreshuffle, whose split-K reduce now goes through the shared launcher. All 451 shapes pass. 181 are within atol/rtol 1e-2 of the fp32 reference; 270 have up to 2.4% of elements outside it from bf16 rounding (largest absolute difference 0.094). None fails the catastrophic check.

Not covered

  • CI runs on MI300X/MI35X, where the new test exits early.
  • No end-to-end accuracy A/B.
  • A deferred output that is never claimed drops out of the 64-entry hand-off table. Callers must claim or reduce every deferred output before reading it.
  • The hand-off is host state: under graph capture, the producer and its consumer must be captured together.

PR Self-Review

Reviewed revision

9f82222

Behavior and intent

  • Adds gemm_a8w8_bpreshuffle_splitk_deferred(...): for the gfx1250 FlyDSL PTPC FP8 GEMM with split 2/4/8, bf16 output and M <= max_m (default 64), it skips the gemm_a8w8_splitk_reduce launch and leaves the fp32 partials for the consumer (splitk_partials(y)), or finishes in place with reduce_splitk_partials(y). Every other config returns the normal output.
  • run_preshuffle_gemm_a8_gfx1250 / run_gemm_a8w8_bpreshuffle_gfx1250 accept a caller partials= buffer; without it they behave as before.
  • Why: at decode sizes the separate reduce is mostly launch cost plus a store/reload round trip; the per-GEMM table in the description shows 1.6-2.6 us left to the consumer vs GEMM + reduce on main's path.

Validation and unverified scope

  • MI455X: op_tests/test_gemm_a8w8_splitk_deferred.py, 21 cases (sparse QKV, dense QKV, gate/up; M 1-64; split 4 and 8): deferred GEMM + reduce bitwise equal to gemm_a8w8_bpreshuffle; hand-off claim semantics checked.
  • MI455X: op_tests/test_gemm_a8w8.py --no-legacy over all 451 FlyDSL shapes of the gfx1250 tuned CSV (reduce now via the shared launcher): all pass.
  • Unverified: CI (MI300X/MI35X, where the new test exits early), end-to-end accuracy A/B.
  • Known limits: an unclaimed deferred output drops out of the 64-entry hand-off table; under graph capture producer and consumer must be captured together.

PR-specific question

What behavior would each changed op test fail on before this PR, and why is that coverage not already provided by an existing canonical test?

  • test_gemm_a8w8_splitk_deferred.py is new and calls APIs added here (gemm_a8w8_bpreshuffle_splitk_deferred, splitk_partials, reduce_splitk_partials), so it cannot run before this PR.
  • It also fails if no case actually splits K, so a tuned-CSV change that disables split-K cannot make it pass vacuously.
  • test_gemm_a8w8.py checks the finished GEMM against an fp32 reference with tolerance; it never exposes unreduced partials or the claim/hand-off contract, and it does not check bitwise equality between the deferred and two-kernel paths.

Changed op tests

  • op_tests/test_gemm_a8w8_splitk_deferred.py (new)
    • Covers: deferred GEMM + reduce bitwise equal to gemm_a8w8_bpreshuffle on MiniMax-M3 decode shapes at split 4/8; partials handed out once; reduce_splitk_partials claims them itself and passes a finished tensor through unchanged.
    • Before this PR: fails at import/call, since the deferred API does not exist.
    • Not redundant: no existing test covers the partials hand-off or bitwise parity of the deferred path; test_gemm_a8w8.py only checks finished outputs within tolerance.

@lirui927
lirui927 force-pushed the xiaobingsuper/m3-mi455-perf-stack branch from 2c8902d to 3499407 Compare November 10, 2026 07:46
@lirui927
lirui927 force-pushed the fuse/f2-splitk-consumer branch 2 times, most recently from 9ecd054 to 546a3e8 Compare November 12, 2026 05:08
@lirui927
lirui927 changed the base branch from xiaobingsuper/m3-mi455-perf-stack to main November 12, 2026 05:08
At decode sizes the tuned gfx1250 PTPC GEMM splits K (MiniMax-M3 TP4 sparse
QKV, M16 x N2560 x K6144: 40 output tiles on 256 CUs, split 4), and a second
kernel sums the fp32 partials into bf16. That reduce reads 0.7 MB and costs a
fixed ~1.3 us per launch in a HIP graph from 46 KB to 2.9 MB (1.7 us of the
GEMM + reduce pair at M16), 57 times per M3 decode step.

gemm_a8w8_bpreshuffle_splitk_deferred runs the same tuned kernel but, when it
splits K (split 2/4/8, M <= max_m, gfx1250), leaves the partials unreduced in
the allocation of the bf16 output it returns. A consumer claims them with
splitk_partials() and sums them as gemm_a8w8_splitk_reduce does (fp32, split
order, one round to nearest even), writing the reduced row back, so it sees
and leaves behind exactly what the two-kernel path produces.
reduce_splitk_partials() finishes the output with the standalone kernel for a
consumer that cannot. Every other call returns gemm_a8w8_bpreshuffle's output.

run_preshuffle_gemm_a8_gfx1250 and run_gemm_a8w8_bpreshuffle_gfx1250 take a
caller's partials buffer (partials=) and then skip the reduce, which
reduce_splitk_partials_gfx1250 launches on its own. Without partials= they
behave as before.

op_tests/test_gemm_a8w8_splitk_deferred.py checks the deferred GEMM + reduce
bitwise against gemm_a8w8_bpreshuffle at the M3 TP4 decode shapes, M 1..64
(sparse QKV, dense QKV, dense gate/up), and the hand-off rules.

Signed-off-by: lirui927 <ruili@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@lirui927
lirui927 force-pushed the fuse/f2-splitk-consumer branch from 49f0052 to 9f82222 Compare November 16, 2026 08:07
@lirui927
lirui927 marked this pull request as ready for review November 16, 2026 08:25
@lirui927
lirui927 requested a review from a team November 16, 2026 08:25
@github-actions github-actions Bot changed the title [Perf] Let the consumer finish a gfx1250 PTPC split-K GEMM [FlyDSL] [Perf] Let the consumer finish a gfx1250 PTPC split-K GEMM Nov 16, 2026
@aiter-bot

aiter-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

READY FOR HUMAN REVIEW

PR Self-Review

Revision 9f82222b6078 · mode shadow

Author self-review accepted. Readiness report:

Author answers

  • PASS — current-SHA answers are present and structurally specific
Show the author's self-review answers

Behavior and intent

- Adds `gemm_a8w8_bpreshuffle_splitk_deferred(...)`: for the gfx1250 FlyDSL PTPC FP8 GEMM with split 2/4/8, bf16 output and M <= `max_m` (default 64), it skips the `gemm_a8w8_splitk_reduce` launch and leaves the fp32 partials for the consumer (`splitk_partials(y)`), or finishes in place with `reduce_splitk_partials(y)`. Every other config returns the normal output.
- `run_preshuffle_gemm_a8_gfx1250` / `run_gemm_a8w8_bpreshuffle_gfx1250` accept a caller `partials=` buffer; without it they behave as before.
- Why: at decode sizes the separate reduce is mostly launch cost plus a store/reload round trip; the per-GEMM table in the description shows 1.6-2.6 us left to the consumer vs GEMM + reduce on main's path.

Validation and unverified scope

- MI455X: `op_tests/test_gemm_a8w8_splitk_deferred.py`, 21 cases (sparse QKV, dense QKV, gate/up; M 1-64; split 4 and 8): deferred GEMM + reduce bitwise equal to `gemm_a8w8_bpreshuffle`; hand-off claim semantics checked.
- MI455X: `op_tests/test_gemm_a8w8.py --no-legacy` over all 451 FlyDSL shapes of the gfx1250 tuned CSV (reduce now via the shared launcher): all pass.
- Unverified: CI (MI300X/MI35X, where the new test exits early), end-to-end accuracy A/B.
- Known limits: an unclaimed deferred output drops out of the 64-entry hand-off table; under graph capture producer and consumer must be captured together.

PR-specific question

What behavior would each changed op test fail on before this PR, and why is that coverage not already provided by an existing canonical test?
- `test_gemm_a8w8_splitk_deferred.py` is new and calls APIs added here (`gemm_a8w8_bpreshuffle_splitk_deferred`, `splitk_partials`, `reduce_splitk_partials`), so it cannot run before this PR.
- It also fails if no case actually splits K, so a tuned-CSV change that disables split-K cannot make it pass vacuously.
- `test_gemm_a8w8.py` checks the finished GEMM against an fp32 reference with tolerance; it never exposes unreduced partials or the claim/hand-off contract, and it does not check bitwise equality between the deferred and two-kernel paths.

Changed op tests

- `op_tests/test_gemm_a8w8_splitk_deferred.py` (new)
  - Covers: deferred GEMM + reduce bitwise equal to `gemm_a8w8_bpreshuffle` on MiniMax-M3 decode shapes at split 4/8; partials handed out once; `reduce_splitk_partials` claims them itself and passes a finished tensor through unchanged.
  - Before this PR: fails at import/call, since the deferred API does not exist.
  - Not redundant: no existing test covers the partials hand-off or bitwise parity of the deferred path; `test_gemm_a8w8.py` only checks finished outputs within tolerance.

PR state

  • PASS — pull request is marked ready for review

Copilot findings

  • PASS — no active Copilot finding lacks a substantive author reply
  • ADVISORY — Copilot did not review the current head SHA

Code and op-test hygiene

  • PASS — no committed artifact or conflict marker detected

Required CI

  • PASS — Black: passed
  • PASS — Ruff: passed
  • PASS — Repository dependency: passed
  • PASS — Aiter Test Gate: passed

Shadow mode reports what would block but keeps the status successful. Set AITER_PR_SELF_REVIEW_MODE=enforce only after reviewing real false positives.

lirui927 pushed a commit that referenced this pull request Nov 16, 2026
…ials as input (#8011)

Squash of #8011 at ea2cca7, by @lirui927.

#8011 brings the split-K partials consumer of 87a9321 (an earlier
#7449) to the optimized kernel, so the kernel and wrapper are
taken whole from #8011; the header and pybind already match.
@lirui927
lirui927 requested review from coderfeli and zufayu November 16, 2026 14:51
lirui927 pushed a commit to lirui927/ATOM that referenced this pull request Nov 17, 2026
…heir consumers

With ROCm/aiter#7449, aiter.gemm_a8w8_bpreshuffle_splitk_deferred leaves the
fp32 partials unreduced when the tuned gfx1250 PTPC GEMM splits K and M is at
most max_m; the consumer claims them (aiter.splitk_partials) and reduces them
as it reads the output.

- LinearBase.splitk_consumer_max_m (0 = off) routes the per-token FP8
  bpreshuffle GEMM through it. Not while a token graph records: its replay
  would run the consumer, an eager region, without the partials.
- MiniMax-M3 sets it on gfx1250 to ATOM_SPLITK_CONSUMER_REDUCE_MAX_M
  (default 64, 0 = off) for the QKV projections whose attention impl claims
  the partials in rope_cache (not under Ulysses SP, whose head exchange reads
  qkv first) and for the dense MLP's gate/up.
- PagedAttentionImpl.rope_cache hands them to triton_fused_norm_rope_cache;
  SparseMHAPagedAttentionImpl.rope_cache to aiter.fused_qknorm_idxrqknorm
  (qkv_partials, ROCm/aiter#8011) when that op takes them: the index branch
  runs, no fp4 cache, and an aiter with the argument. Every other path
  finishes qkv with the standalone reduce first. A dummy run releases them.
- swiglu_oai_split_reducing, an opaque op, claims a gate/up's partials at run
  time and reduces them in the activation kernel.

An aiter without #7449 never defers; one with #7449 but without #8011 defers
and reduces in the sparse layers with the standalone kernel, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zufayu

zufayu commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Review routing reminder: @aoli26 is the primary reviewer for the FlyDSL GEMM scope on this PR, with @coderfeli as backup. Please route the next technical review/action to the component owner; CI status remains a separate merge gate.

lirui927 pushed a commit to lirui927/ATOM that referenced this pull request Nov 17, 2026
…er decide

Drop the aiter probes: the try-import of the deferred GEMM API and the
inspect.signature check for fused_qknorm_idxrqknorm's qkv_partials. ATOM
needs aiter with ROCm/aiter#7449 and #8011 (docs/environment_variables.md).

Stop restating aiter's rules: the sparse rope_cache passes qkv's partials to
fused_qknorm_idxrqknorm in every mode (the op reduces them itself where it
cannot fuse them), and the gfx1250 gate goes: the deferred GEMM decides per
call, arch included, whether it defers. Only ATOM's own Triton kernel path
still reduces first.

claims_splitk_partials is opt-in per class: PagedAttentionImpl and
SparseMHAPagedAttentionImpl set it, a subclass does not inherit it. The
model checks ATOM's own attention layer and impl by type instead of getattr
defaults. The env doc names both deferred GEMMs (QKV and the dense MLP
gate/up into SwiGLU-OAI).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zufayu
zufayu requested a review from aoli26 November 17, 2026 07:49

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (3)

  1. Make a missed claim fail loudly instead of returning the "already reduced" None. aiter/ops/gemm_op_a8w8.py splitk_partials (~L1053-1067) returns None both for "y holds its values" and for a still-deferred y it does not recognise: a reshaped view (y.view(m, h, d)), a column slice, a non-leading row slice, or an entry pushed out of the 64-entry table (~L1041). The consumer then reads unwritten bf16, and reduce_splitk_partials(y) returns y unwritten too. A CPU run of the PR's own code reproduces all four cases. Fix: when the address/storage matches a live entry but the view check fails, raise. On eviction, launch the standalone reduce for the evicted entry (or raise); do not drop it silently.
  2. Share the kernel selection with _gemm_a8w8_bpreshuffle_into instead of re-deriving it in _splitk_deferrable. _splitk_deferrable (~L979) re-implements the config lookup, the libtype check and the arch gate, and the deferred path also skips gemm_a8w8_bpreshuffle's input asserts (e.g. WQ.dtype == fp8). Main has already changed that dispatch since this branch's base (INT8 bpreshuffle), so the two copies can drift and break the "bitwise equal" promise. Fix: factor one (config, kernelName) resolver and call it from both paths.
  3. Add a CPU test of the hand-off table that CI actually runs. op_tests/test_gemm_a8w8_splitk_deferred.py skips everywhere except gfx1250, so CI (MI300X/MI35X) never executes any of the new logic. The table is pure host code. Fix: stub the two launchers and cover claim-once, the leading-row-slice claim, the rejected views (they should raise after item 1) and eviction.

Minor (non-blocking):

  • _SPLITK_DEFERRED.pop(Y.data_ptr(), None) in the non-deferred branch is dead code. A live entry holds its allocation, so a fresh Y can never land on that address.
  • The eviction while can be an if.
  • The comment "its address cannot be handed out again until the consumer claims it" ignores eviction.
  • The test docstring says "Every check is bitwise", but the err column is an allclose check against the fp32 reference.

Reviewed at 9f82222. CI: 30 pass, 33 skipped, 0 failing (the new test skips on CI archs).

@aoli26

aoli26 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

gemm_a8w8_256x256_gfx1250.py already has a cluster-based fused split-K epilogue (_fsk) that reduces in-kernel, so the output is final with no extra launch and no caller hand-off. Have you tried extending that to the PTPC kernel? For decode shapes (M ≤ 64, split 2/4/8) the cluster constraints look satisfiable, and it would avoid the deferred path's caveats (unclaimed outputs reading uninitialized data, 64-entry eviction, extra memory pinned, eager-only).
If you hit a blocker, could you note it in the PR? A comparison against a fused PTPC variant on the MiniMax-M3 shapes would also help justify the added complexity.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants