Skip to content

perf(minimax-m3): raise decode score memory throughput - #2982

Draft
XiaobingSuper wants to merge 5 commits into
ROCm:mainfrom
XiaobingSuper:xiaobingsuper/m3-gfx1250-decode-score-warps
Draft

XiaobingSuper wants to merge 5 commits into
ROCm:mainfrom
XiaobingSuper:xiaobingsuper/m3-gfx1250-decode-score-warps

Conversation

@XiaobingSuper

@XiaobingSuper XiaobingSuper commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Depends on #2873 and remains draft until it lands; review commits a1fbfaed1 and eaa36f8 in the meantime. Part of https://github.com/wuhuikx/LLM-Booster/issues/341.

Summary

Two exact gfx1250 decode-score optimizations:

  1. Use two wave32s instead of Triton's four-wave default for batch >=4 and context >64K.
  2. For measured wide-grid shapes, transpose the independent query/position axes so BF16 K is WMMA's N operand, and bypass caches for the streaming K load. The D reduction order, FP32 accumulation, scores, selector, and graph bounds are unchanged. Launch-limited shapes retain the prior kernel byte-for-byte.

Dispatch depends only on arch and capture-time batch/context/query bounds, so it is graph-safe and model-name independent.

Roofline and performance

MI455X e15-05, graph replay, disjoint request pages and rotating working sets larger than the 192 MB L2.

At B4/Q4/193K, score-only same-run ABBA:

path us useful K TB/s
original 4-wave 16.059 6.15
2-wave 13.691 7.22
2-wave + transposed/bypass 12.759 7.75

Final vs original: -20.5%. Final vs the first commit in this PR: -6.8%. Complete exact score + ordered top-k + sparse-table emit improves 26.676 -> 25.285 us (-5.2%) over the first commit.

Across the full 70-cell matrix (S=1,2,4,8,16,32,64,128; q1/q4; 32K/64K/128K/193K/256K), all 33 transposed-dispatch cells have no >2% regression, 30 improve by >2%, and their geometric-mean score reduction is 6.3% over the first commit. Fallback cells keep the first commit unchanged. Representative incremental score gains: 193K S32/q4 -8.8%, 256K S32/q4 -15.0%, 128K S16/q1 -11.1%.

The supplied native HIP non-temporal read benchmark reaches 20.0 TB/s at the exact 98.8 MB scorer traffic size (4.94 us), and 19.2-22.3 TB/s over 64 MB-3 GB. MI455X theoretical HBM peak is 23.3 TB/s. The final scorer is 38.7% of measured same-size read roofline, so meaningful low-N/LDS/pipeline headroom remains; this PR does not claim roofline.

Correctness

  • all 70 matrix cells: bitwise-identical FP32 score buffers and exact ordered top-k outputs
  • random inputs, ties, causal tails, q1/q4, and graph replay covered
  • production-shape topk_idx, sparse_bt, and sparse_ctx bit-exact
  • `gfx1250 GPU + CPU-policy suites: 259 passed, 1 skipped on gfx1250
  • Ruff, Black, IDE diagnostics, py_compile, and git diff --check clean

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2982 --add-label <label>

@XiaobingSuper

XiaobingSuper commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

Corrected roofline follow-up using the supplied native HIP benchmark:

  • Official MI455X theoretical HBM peak: 23.3 TB/s.
  • Native non-temporal read at the scorer's exact 98.8 MB traffic size: 20.0 TB/s (4.94 us); 64 MB-3 GB sustains 19.2-22.3 TB/s.
  • Optimized scorer: 7.27 TB/s / 13.586 us, only 36.4% of the measured same-size read roofline. The earlier torch.copy_ result was an untuned software-copy result and is not a valid ceiling.
  • Cold counters show 95.5 MB DRAM fetch for 98.8 MB useful K traffic, almost entirely 256-byte requests, so transaction coalescing is already good.
  • Generated code uses async global-to-LDS + BF16 WMMA with 34,944 B LDS and 120 VGPR. q4 fills 4 of 16 WMMA columns; q1 fills 1. Avoiding that low-N padding while preserving exact score/top-k semantics is the main kernel-level follow-up.

The PR keeps the robust bit-exact 14.8% score / 7.8% full-path gain, but does not claim roofline.

@XiaobingSuper

Copy link
Copy Markdown
Contributor Author

Rejected low-N follow-ups (kept out of production):

  • Triton vector scorer: 139.17 us / 0.71 TB/s and substantial FP32 drift.
  • Native one-wave rocWMMA + non-temporal loads: 90.84 us / 1.09 TB/s; 651 of 24,128 scores differed, max 2 ULP.
  • Exact baseline in the same isolated setup: 13.81-13.93 us / about 7.1 TB/s.

gfx1250 BF16 WMMA requires a 16x16x32 tile, and manually replacing the exact reduction with low-N vector/native paths both lost performance and exactness. These prototypes remain benchmark-only and are not dispatched. The exact transposed-WMMA path in this PR remains the selected implementation.

lirui927 pushed a commit to lirui927/ATOM that referenced this pull request Nov 9, 2026
…tion

On gfx1250 the CP scorer takes Q as WMMA's M operand and the converted K
as its N operand, with streaming K loads, as the gfx1250 TP decode
scorer of ROCm#2982. The D reduction order and FP32 accumulation
are unchanged, so the scores keep their bits. Other architectures keep
the previous orientation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@XiaobingSuper
XiaobingSuper force-pushed the xiaobingsuper/m3-gfx1250-decode-score-warps branch from 0f84959 to eaa36f8 Compare November 10, 2026 08:22
lirui927 pushed a commit to lirui927/ATOM that referenced this pull request Nov 16, 2026
…tion

On gfx1250 the CP scorer takes Q as WMMA's M operand and the converted K
as its N operand, with streaming K loads, as the gfx1250 TP decode
scorer of ROCm#2982. The D reduction order and FP32 accumulation
are unchanged, so the scores keep their bits. Other architectures keep
the previous orientation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lirui927 pushed a commit to lirui927/ATOM that referenced this pull request Nov 16, 2026
…tion

On gfx1250 the CP scorer takes Q as WMMA's M operand and the converted K
as its N operand, with streaming K loads, as the gfx1250 TP decode
scorer of ROCm#2982. The D reduction order and FP32 accumulation
are unchanged, so the scores keep their bits. Other architectures keep
the previous orientation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants