[FlyDSL] MiniMax-M3 MXFP8 1x32 tuned GEMM configs by andrei-kochin · Pull Request #6441 · ROCm/aiter

andrei-kochin · 2026-10-12T16:08:19Z

Motivation

Part 3 of 3 split out of #6060 (author: @sshlyapn), so each PR covers one concern.

Important

Stacked on #6439 and #6440. The tuned rows use 1x32 weight scales (#6439) and many use the grouped XCD order _xcd4 (#6440), so this cannot merge before both. Because the branches live on a fork, this PR targets main and carries their commits; review only the last commit ("MiniMax-M3 MXFP8 1x32 tuned GEMM configs"). Once #6439 and #6440 merge, the diff shrinks to just that commit.

Changes

Tuned configs: model_configs/minimax_m3_a8w8_blockscale_mxscale_bpreshuffle_tuned_gemm.csv
- 342 rows: 18 MiniMax-M3 dense N x K shapes (TP1/TP2/TP4) x 19 M buckets (1..512, 1k..64k), gfx950.
Tests: test_mxscale_bpreshuffle_1x32_weight_scales adds the two _xcd4 layouts with 1x32 weight scales (including a partial swizzle group and a padded grid).

Performance (MI355, gfx950, 256 CU), from #6060

vLLM MiniMaxAI/MiniMax-M3-MXFP8 performance (TP4, 8k/1k random dataset):

Metric	C=1	C=16	C=64
Output throughput	+4.4%	+4.1%	+5.0%
TTFT mean	-5.8%	-8.0%	-8.8%
TTFT p99	-4.7%	-8.9%	-9.1%
TPOT mean	-4.2%	-3.4%	-4.3%
TPOT p99	-3.9%	-3.4%	-5.3%
E2E latency mean	-4.2%	-3.9%	-4.8%

Speedup of the full stack (tuned CSV) over each MXFP8 1x32 baseline: geomean over the 18 M3 shapes.
Above 1 means faster.

M	vs vLLM Triton MXFP8	vs AITER group32 Triton (#5750/#5862), tuned
1-16	1.30-1.35x	0.99-1.03x
32-256	1.54-2.07x	1.05-1.15x
1k	2.06x	1.42x
2k-16k	1.57-1.70x	1.30-1.34x
24k-128k	1.74-2.18x	1.33-1.35x

vLLM: rocm_native._mxfp8_linear_kernel with vLLM main's own tile selection, Triton 3.7.1.
group32: gemm_afp8wfp8 as of [Triton/Gluon] Unify afp8wfp8 GEMM kernels #5862, which unified [Triton/Gluon] [JIT] [Feature] Add tuned native group32 A8W8 Triton GEMM for gfx950 #5750's group32 kernel into it. AITER ships no MiniMax-M3 tables for it, so its configs were tuned on the same shapes with the same protocol, in an earlier interleaved run.

Protocol: weights rotate between 2 copies; each point is the median of 3 interleaved rounds; every timed config was checked against a dequantized fp32 reference (rel err < 3e-3).

Tests

The CSV is byte-identical to #6060's; all 342 kernelNames parse with the #6440 name parser. Combined, #6439 + #6440 + this PR equal #6060's code exactly, plus the new 32x32 _xcd4 tests in #6440.

Accept a 1x32 w_scale [B, N, K/32] (one e8m0 per weight row per 32 K) in the gfx950 mxscale bmm, as OCP MX checkpoints such as MiniMax-M3 carry. The kernel reads the B-scale panel one row per B row, the same way it reads A's; the layout is row-major like x_scale, so no repack. - W_SCALE_ROWS = (1, 32, 128); check_bmm_config allows a 1-row w_scale only with 32-wide K blocks. - Wired through _RUNNERS, gemm_a8w8_blockscale_bpreshuffle and AOT. - 32x32 / 128x128 are unchanged. - Tests: a 1x32 case in test_flydsl_batched_gemm.py (weight magnitudes change every row, each a8w8 candidate gets its own reference); test_gemm_group32_interface.py routes 1x32, checks 1x128 is still rejected, and checks 1x32 accuracy against a dequantized reference. Split out of ROCm#6060. Signed-off-by: Sergei Shliapnikov <sergei.shliapnikov@amd.com> Signed-off-by: Andrei Kochin <andrei.kochin@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>

…nly when split - xcd_swizzle (_xcd<g> in kernel names): inside each XCD's contiguous run of tiles (xcd_order), re-order tiles M-major in groups of g M tiles, as xcd_remap_bx_by does. Fewer B column blocks are live per XCD, so L2 reuse improves. The b_direct heuristic now picks _xcd4 instead of _xcd. Existing _xcd CSV rows parse and run unchanged; xcd_swizzle without xcd_order is rejected, since the digits would parse back as _sk. - LDS: reserve the 16-byte split-K flag only when K is actually split, which lets a few large tiles fit. - Tests: _xcd4 accuracy with 32x32 scales (including a partial swizzle group and a padded grid), kernel-name round trip, and the invalid xcd_swizzle-without-xcd_order name. Split out of ROCm#6060. Signed-off-by: Sergei Shliapnikov <sergei.shliapnikov@amd.com> Signed-off-by: Andrei Kochin <andrei.kochin@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Add model_configs/minimax_m3_a8w8_blockscale_mxscale_bpreshuffle_tuned_gemm.csv: 342 rows, 18 MiniMax-M3 dense N x K shapes (TP1/TP2/TP4) x 19 M buckets (1..512, 1k..64k), gfx950. The rows use 1x32 weight scales and many use the grouped XCD order (_xcd4), so this needs both of those changes. Also cover the two _xcd4 layouts with 1x32 weight scales in test_mxscale_bpreshuffle_1x32_weight_scales. Split out of ROCm#6060. Signed-off-by: Sergei Shliapnikov <sergei.shliapnikov@amd.com> Signed-off-by: Andrei Kochin <andrei.kochin@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-10-12T16:08:44Z

🏷️ CI Guide

Runs automatically on every PR:

✅ Pre-checks (submodule verification, code formatting)
✅ Aiter op tests (gfx942 + gfx950)
✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
`ci:gfx1250-ffm-triton`	Run the five-shard gfx1250 FFM Triton test suite
`ci:triton-300x`	Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
`ci:triton-355`	Run the full Triton test suite on MI35X, not only the tests the change affects
`ci:windows`	Windows JIT op smoke test on the gfx1201 ROCm runner
`ci:multi-gpu`	Aiter multi-GPU tests on the 8-GPU runner
`ci:sglang`	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
`ci:atom`	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
`ci:atom_full`	ATOM accuracy suite for PR and main models from ATOM `models_accuracy.json`
`ci:vllm`	vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
`ci:all`	All standard extended tests (excludes `ci:atom_full`)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 6441 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

Copilot

🟡 Changes recommended

Split-K configurations can encode xcd_swizzle, but that option is silently ignored by the kernel.

1 open finding

Reject xcd_swizzle with xcd_order when splits > 1 · New

What changed in this PR

Extends the gfx950 FlyDSL MXFP8 GEMM stack with per-row 1x32 weight scales, grouped XCD ordering, and MiniMax-M3 tuned configurations.

Changes:

Adds 1x32 scale dispatch, kernel reads, validation, and AOT support.
Adds _xcd4 tile ordering and reduces non-split LDS usage.
Adds 342 tuned rows and expanded correctness tests.

File	Description
`op_tests/test_gemm_group32_interface.py`	Tests 1x32 routing, accuracy, and XCD swizzling.
`op_tests/test_flydsl_batched_gemm.py`	Extends batched GEMM validation to 1x32 scales.
`aiter/ops/flydsl/kernels/bmm_a8w8_mxscale_gfx950.py`	Implements scale-row and XCD-order kernel changes.
`aiter/ops/flydsl/batched_gemm_a8w8.py`	Registers gfx950 1x32 dispatch.
`aiter/ops/flydsl/batched_gemm_a8w8_gfx950.py`	Adds parsing, validation, and launch support.
`aiter/ops/batched_gemm_op_a8w8.py`	Documents the public 1x32 operand contract.
`aiter/configs/model_configs/minimax_m3_a8w8_blockscale_mxscale_bpreshuffle_tuned_gemm.csv`	Adds MiniMax-M3 tuning rows.
`aiter/aot/flydsl/gemm.py`	Wires 1x32 configurations into AOT compilation.

🧠 Review effort: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    if xcd_swizzle < 0 or (xcd_swizzle and not xcd_order):
+        raise ValueError("[FlyDSL gfx950] xcd_swizzle re-tiles the xcd_order run")


andrei-kochin · 2026-10-12T16:41:10Z

Opened by mistake against the wrong PR; closing.

sshlyapn and others added 3 commits October 12, 2026 20:06

andrei-kochin requested review from a team and a balanced review from Copilot October 12, 2026 16:08

This was referenced Oct 12, 2026

[FlyDSL] gfx950 MXFP8 bmm: accept 1x32 (OCP MX) weight scales #6439

Closed

[FlyDSL] gfx950 MXFP8 bmm: grouped XCD tile order, split-K LDS flag only when split #6440

Closed

github-actions Bot added the FlyDSL label Oct 12, 2026

andrei-kochin mentioned this pull request Oct 12, 2026

[HIP] [OPUS] [FlyDSL] gfx950 MXFP8 bmm: 1x32 weight scales, grouped XCD order, MiniMax-M3 configs #6060

Merged

Copilot started reviewing on behalf of andrei-kochin October 12, 2026 16:09

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Comment thread aiter/ops/flydsl/kernels/bmm_a8w8_mxscale_gfx950.py

Comment on lines +226 to +227

if xcd_swizzle < 0 or (xcd_swizzle and not xcd_order):

raise ValueError("[FlyDSL gfx950] xcd_swizzle re-tiles the xcd_order run")

andrei-kochin closed this Oct 12, 2026

andrei-kochin deleted the split/mxfp8-minimax-m3-configs branch October 12, 2026 16:41

Navigation Menu

Repository navigation

[FlyDSL] MiniMax-M3 MXFP8 1x32 tuned GEMM configs - #6441

[FlyDSL] MiniMax-M3 MXFP8 1x32 tuned GEMM configs#6441
andrei-kochin wants to merge 3 commits into
ROCm:mainfrom
andrei-kochin:split/mxfp8-minimax-m3-configs

andrei-kochin commented Oct 12, 2026

Uh oh!

github-actions Bot commented Oct 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

andrei-kochin commented Oct 12, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		if xcd_swizzle < 0 or (xcd_swizzle and not xcd_order):
		raise ValueError("[FlyDSL gfx950] xcd_swizzle re-tiles the xcd_order run")

Conversation

andrei-kochin commented Oct 12, 2026

Motivation

Changes

Performance (MI355, gfx950, 256 CU), from #6060

Tests

Uh oh!

github-actions Bot commented Oct 12, 2026

🏷️ CI Guide

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

andrei-kochin commented Oct 12, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants