You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Part 3/3, follows #5795 and #5932 (both merged). Optimize MoE decoding (1–256 tokens) on MI308X/gfx942. Earlier revisions also covered gfx950 MXFP4/SiTUv2 decoding; that part is dropped and this PR is gfx942-only.
Changes
Direct decoding for 2–8 tokens (idea from @coderfeli). Routes run straight from topk_ids without sorting, on batched batch-1 kernels (grid over tokens × routes). It is tunable as 16_16_16_False_True; the fifth tile field is optional, so existing config strings keep their meaning, and test_config_strings covers the new flag. Routes with expert ID −1 (padding rows of topk with num_valid_rows, [ASM] [HIP] [FlyDSL] feat: pad-row routing count, custom all_gatherv, MoE quant-sort race fix #6916) are skipped.
Fused Down-output clear. The batch-1 Gate/Up kernel zeroes the Down output, removing the separate fill launch for batch 1 and direct decoding.
BN128 sorted-decoding tile. 16_128_64 computes 128 output columns per workgroup next to the BN64 16_16_16, halving the workgroups and activation re-reads. Split-K always uses BK64, so no other BN64 tile is offered. Direct decoding is tuned only for 2–8 tokens, and batch 1 only as 16_16_16.
Prefill sort and reduction. The Opus aux sort returns each route's sorted row, replacing the invert_sorted_ids kernel. Routes the sort drops (repeated expert IDs of a token, −1 padding) are left out of the final sum, and per-tensor FP8 zeroes the Gate/Up output so they cannot raise the Down input absmax. The sort no longer clears the output, which sorted_sum writes in full.
Retune the decoding rows of the six model CSVs (below).
Performance
MI308X (gfx942)
MI308X/gfx942 (80 CUs, PTL enabled), ROCm 7.2.4, FlyDSL 0.4.0, default MoE sorting. Measured on 7b58f3a457; baseline is main 2bafed4728 (the merge base) with its shipped CSVs. The first three value columns use the default rounding (AITER_FLYDSL_MOE_BF16_RTA_SIMPLIFIED unset), the RTA columns AITER_FLYDSL_MOE_BF16_RTA_SIMPLIFIED=1 on both trees. Values are the per-shape full-MoE GPU kernel-time sums that --run_config reports (10 warmups, 51 iterations, rotating inputs, IQR filtering); each is the geometric mean over shapes of the median of three alternating-order rounds. Speedup is baseline / PR3 − 1; these are not full-model throughput claims.
Batch
Shapes
Baseline us
PR3 us
Speedup
Baseline us (RTA)
PR3 us (RTA)
Speedup (RTA)
1
12
19.979
18.339
+8.94%
20.050
18.038
+11.16%
2
12
38.330
26.704
+43.54%
37.979
26.364
+44.06%
4
12
55.539
43.886
+26.55%
55.089
43.169
+27.61%
8
12
84.745
77.000
+10.06%
84.141
75.762
+11.06%
1/2/4/8 combined
48
43.572
35.867
+21.48%
43.345
35.314
+22.74%
Batch 1 keeps 16_16_16_False; its gain comes from the batch-1 kernel changes alone.
Tokens
Shapes
Baseline us
PR3 us
Speedup
Baseline us (RTA)
PR3 us (RTA)
Speedup (RTA)
16–256
60
261.930
258.197
+1.45%
260.903
256.043
+1.90%
1–256 (all decoding)
108
118.025
107.386
+9.91%
117.494
106.153
+10.68%
>256 (CSV unchanged)
94
1836.385
1821.785
+0.80%
1790.142
1778.693
+0.64%
All
202
423.314
400.972
+5.57%
417.312
394.089
+5.89%
Runtime recommendation: set AITER_FLYDSL_MOE_BF16_RTA_SIMPLIFIED=1. The decoding rows are tuned with it, and with this PR it is 1.16% faster than the default rounding on the decoding rows (1.75% on all rows). Simplified RTA rounds BF16 halfway cases away from zero instead of to even and does not preserve NaN payloads; op_tests/test_flydsl_moe_gfx942.py passes with error 0 either way.
Rows above 256 tokens keep main's kernels. The 71 FlyDSL prefill rows are +1.09% (+0.89% with RTA), mostly because the sort no longer clears the output; the 23 CK and ASM prefill rows, whose code is unchanged, move by −0.07% (−0.10%), which is run-to-run noise.
Selection changes. 58 of 202 rows change kernel.
2–8 tokens: 34 rows move to direct decoding (30 from BN64, 3 from CK 2-stage, 1 from ASM 1-stage).
16–256 tokens: 24 rows move to BN128 (15 from CK 2-stage, 9 from BN64). On the 24 changed rows above 8 tokens the speedup is +3.48% (+4.62% with RTA); 22 are faster and the other 2 are within 0.9%.
Rows above 256 tokens are unchanged.
A row tuned at M tokens also serves M/2+1 to M−1 tokens (get_padded_M). Every token count the retuned rows serve, 1476 points from 2 to 256 tokens, was measured as well: +6.4% on average, worst −1.3%, none more than 2% slower than main.
Tuning. Main's tuned CSVs were the starting point and only decoding shapes (≤256 tokens, 108 of 202) were retuned. The whole-MoE stage (--e2e_tune) ran with the environment of #5932 (TUNE_ONLY=flydslv2,cktile,flydsl,asm, AITER_FMOE_TUNE_EXCLUDE_KERNELS=fmoe_stage1, simplified BF16 RTA). Each decoding shape was then confirmed by a one-off script that re-timed main's selection and every eligible whole-MoE config over three interleaved rounds and kept the lowest median. Three picks that were more than 2% slower than main with the default rounding keep main's selection, and so do seven BN128 picks that were more than 2% slower than main at a token count they serve, or less than 1% faster on average.
Reproduce (MI308X, gfx942, 80 CUs)
Run on this PR. For the baseline, run the --run_config line on main (2bafed4728).
All standard extended tests (excludes ci:atom_full)
Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 5973 --add-label <label>
One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.
PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.
These checks accept BLOCK_N=16 and BLOCK_K=16, but the rejection text says only BN64/BN128 and BK64. When an invalid config hits these branches, the diagnostic contradicts the actual accepted configuration space and can mislead tuning/debugging; include the 16 variants in both messages.
The reason will be displayed to describe this comment to others. Learn more.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
A moderate unresolved validation issue can mark untuned direct-decoding tile configurations as valid, and the performance-sensitive kernel changes warrant final human review.
The performance results cannot be tied to the final HEAD:
No measured SHA is given. The last two commits (c8e8158, 4641dd7) changed the prefill sort/reduce path after the CSV retune, so it is unclear whether the 71 published prefill rows were measured with the current code.
All tuning and benchmarking used AITER_FLYDSL_MOE_BF16_RTA_SIMPLIFIED=1, while the default is 0.
Routing is uniform random (randn scores), which favors direct decoding over sorted expert grouping. Only power-of-two token buckets were measured, while 3, 5–7 and 9–15 tokens also resolve to these configs at runtime.
Only per-group geometric means are reported, and CI has no gfx942 performance check.
…tput
Since ROCm#6916, routing gives rows past num_valid_rows expert -1 and weight 0
and expects sorting to drop them.
- Batch-1 and direct-decoding kernels read topk_ids without sorting; they now
skip -1 routes instead of reading weights and scales in front of expert 0.
- Per-tensor prefill zeroes the Gate/Up output: its Down absmax reads every
route row, including routes sorting dropped (merged repeated IDs, -1 ids).
Tests: pad_rows routing (-1 ids, zero weights). Weights and scales sit behind
a NaN expert and the Gate/Up output starts as inf, so reading expert -1 or a
skipped row fails the check. New edge cases: per-tensor prefill and B2 direct
decoding with pad rows.
Decoding rows (<= 256 tokens) start from main's selections. --e2e_tune ran with
TUNE_ONLY=flydslv2,cktile,flydsl,asm, AITER_FMOE_TUNE_EXCLUDE_KERNELS=fmoe_stage1
and AITER_FLYDSL_MOE_BF16_RTA_SIMPLIFIED=1 on MI308X with PTL enabled
(VECTOR,F8); each shape then keeps the lowest median of main's selection and
every eligible FlyDSL config over three interleaved rounds. Three picks that
are slower than main with the default rounding keep main's selection. 65 rows
change kernel versus main; rows above 256 tokens are main's.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
A critical regression test is recursive, and several shipped direct-decoding selections violate the stated tuning safeguards or accept misleading tile configurations.
The performance results cannot be tied to the final HEAD:
No measured SHA is given. The last two commits (c8e8158, 4641dd7) changed the prefill sort/reduce path after the CSV retune, so it is unclear whether the 71 published prefill rows were measured with the current code.
All tuning and benchmarking used AITER_FLYDSL_MOE_BF16_RTA_SIMPLIFIED=1, while the default is 0.
Routing is uniform random (randn scores), which favors direct decoding over sorted expert grouping. Only power-of-two token buckets were measured, while 3, 5–7 and 9–15 tokens also resolve to these configs at runtime.
Only per-group geometric means are reported, and CI has no gfx942 performance check.
The description now shows results re-measured on 6f9f5f9557 against main 2bafed4728 (the merge base), with FlyDSL 0.4.0. That SHA includes c8e8158 and 4641dd7, so all 202 rows, including the prefill rows, were measured with the current sort/reduce path. The later af98038f0 only rejects direct-decoding configs with tiles other than 16_16_16, so the shipped configs and kernels are unchanged. The description covers both AITER_FLYDSL_MOE_BF16_RTA_SIMPLIFIED=1 and the default (0); the tables below use the default rounding.
Skewed routing. The router scores get a fixed per-expert bias -alpha * log(1 + rank) (fixed random ranking of the experts), with alpha 0.06–0.09 per model so that, over many tokens, the most popular expert receives 2× the mean load. Three rounds:
Tokens
Shapes
Geomean
Total time
Worst / best
> 2% faster / slower
1
12
+8.8%
+7.7%
+3.3% / +19.0%
12 / 0
2
12
+44.3%
+39.1%
+20.8% / +97.1%
12 / 0
4
12
+28.1%
+22.9%
+8.7% / +68.9%
12 / 0
8
12
+11.1%
+6.9%
-0.8% / +41.1%
9 / 0
16–256
60
+1.9%
+2.6%
-2.7% / +9.9%
24 / 1
1–256 (decoding)
108
+10.5%
+3.9%
-2.7% / +97.1%
69 / 1
At decoding batch sizes this skew barely changes the routing (at 64 tokens 295 vs 295 active experts on average), and every token group stays within 1% of its uniform-routing speedup.
Per-shape skewed routing (108 rows)
Model
M
inter_dim
E
main → PR
Active experts
main µs
PR µs
Skewed
Uniform
Hunyuan3 FP8 per-tensor
1
192
192
BN64
8
17.64
15.52
+13.7%
+6.0%
Hunyuan3 FP8 per-tensor
2
192
192
BN64 → direct
16
30.10
20.26
+48.6%
+50.2%
Hunyuan3 FP8 per-tensor
4
192
192
BN64 → direct
31
40.77
31.06
+31.3%
+33.6%
Hunyuan3 FP8 per-tensor
8
192
192
BN64 → direct
57
59.32
52.44
+13.1%
+13.0%
Hunyuan3 FP8 per-tensor
16
192
192
BN64 → BN128
95
89.90
89.33
+0.6%
+0.9%
Hunyuan3 FP8 per-tensor
32
192
192
BN64 → BN128
142
128.76
129.16
-0.3%
-0.3%
Hunyuan3 FP8 per-tensor
64
192
192
BN64 → BN128
182
153.61
147.94
+3.8%
+3.9%
Hunyuan3 FP8 per-tensor
128
192
192
BN64 → BN128
192
186.22
176.01
+5.8%
+6.8%
Hunyuan3 FP8 per-tensor
256
192
192
CK → BN128
192
234.37
223.44
+4.9%
+5.5%
MiniMax-M3 FP8
1
384
128
BN64
4
20.95
20.04
+4.5%
+9.8%
MiniMax-M3 FP8
2
384
128
BN64 → direct
8
35.88
27.26
+31.6%
+33.9%
MiniMax-M3 FP8
4
384
128
BN64 → direct
16
53.02
44.13
+20.1%
+21.0%
MiniMax-M3 FP8
8
384
128
BN64 → direct
29
81.83
76.48
+7.0%
+7.5%
MiniMax-M3 FP8
16
384
128
CK → BN128
53
129.36
126.13
+2.6%
+2.5%
MiniMax-M3 FP8
32
384
128
CK
76
175.17
180.10
-2.7%
+0.4%
MiniMax-M3 FP8
64
384
128
CK
112
248.59
248.05
+0.2%
+0.2%
MiniMax-M3 FP8
128
384
128
CK
125
297.70
298.44
-0.2%
+0.1%
MiniMax-M3 FP8
256
384
128
CK
128
313.40
313.52
+0.0%
-0.3%
MiniMax-M3 FP8
1
384
129
BN64
5
20.99
20.22
+3.8%
+7.0%
MiniMax-M3 FP8
2
384
129
BN64 → direct
9
37.48
28.01
+33.8%
+35.7%
MiniMax-M3 FP8
4
384
129
BN64 → direct
18
56.16
51.67
+8.7%
+10.9%
MiniMax-M3 FP8
8
384
129
CK → direct
35
93.18
92.82
+0.4%
+0.2%
MiniMax-M3 FP8
16
384
129
CK → BN128
64
148.67
148.75
-0.1%
+0.8%
MiniMax-M3 FP8
32
384
129
CK
95
220.98
219.06
+0.9%
+0.0%
MiniMax-M3 FP8
64
384
129
CK
120
269.46
269.14
+0.1%
-0.3%
MiniMax-M3 FP8
128
384
129
CK
128
292.99
296.93
-1.3%
+0.2%
MiniMax-M3 FP8
256
384
129
CK
129
338.02
336.76
+0.4%
-0.2%
Qwen3.8-Flash-Next BF16
1
192
513
BN64
11
18.75
15.81
+18.6%
+15.6%
Qwen3.8-Flash-Next BF16
2
192
513
BN64 → direct
22
37.29
24.91
+49.7%
+49.7%
Qwen3.8-Flash-Next BF16
4
192
513
BN64 → direct
42
55.53
39.30
+41.3%
+39.2%
Qwen3.8-Flash-Next BF16
8
192
513
BN64 → direct
84
85.82
73.64
+16.5%
+15.5%
Qwen3.8-Flash-Next BF16
16
192
513
BN64
152
139.62
138.78
+0.6%
-0.6%
Qwen3.8-Flash-Next BF16
32
192
513
BN64
257
229.61
229.90
-0.1%
+0.0%
Qwen3.8-Flash-Next BF16
64
192
513
BN64
384
365.04
365.51
-0.1%
-0.2%
Qwen3.8-Flash-Next BF16
128
192
513
BN64 → BN128
473
471.26
452.77
+4.1%
+3.7%
Qwen3.8-Flash-Next BF16
256
192
513
CK → BN128
512
551.69
523.14
+5.5%
+5.3%
Qwen3.8-Flash-Next BF16
1
320
513
BN64
11
22.23
21.29
+4.4%
+5.5%
Qwen3.8-Flash-Next BF16
2
320
513
BN64 → direct
22
47.15
34.98
+34.8%
+35.1%
Qwen3.8-Flash-Next BF16
4
320
513
BN64 → direct
41
75.25
61.62
+22.1%
+22.5%
Qwen3.8-Flash-Next BF16
8
320
513
BN64 → direct
76
117.48
110.69
+6.1%
+6.6%
Qwen3.8-Flash-Next BF16
16
320
513
BN64 → BN128
140
193.28
192.35
+0.5%
+0.0%
Qwen3.8-Flash-Next BF16
32
320
513
CK → BN128
241
386.53
364.70
+6.0%
+10.0%
Qwen3.8-Flash-Next BF16
64
320
513
CK → BN128
366
582.54
560.26
+4.0%
+4.0%
Qwen3.8-Flash-Next BF16
128
320
513
CK → BN128
475
771.11
745.04
+3.5%
+3.9%
Qwen3.8-Flash-Next BF16
256
320
513
CK → BN128
511
860.15
849.43
+1.3%
+0.7%
Qwen3.8-Flash-Next BF16
1
640
513
BN64
11
36.68
35.52
+3.3%
+2.4%
Qwen3.8-Flash-Next BF16
2
640
513
BN64 → direct
22
76.74
63.51
+20.8%
+25.0%
Qwen3.8-Flash-Next BF16
4
640
513
BN64 → direct
42
128.91
116.74
+10.4%
+11.0%
Qwen3.8-Flash-Next BF16
8
640
513
BN64
83
228.51
229.21
-0.3%
-0.4%
Qwen3.8-Flash-Next BF16
16
640
513
CK → BN128
153
471.22
461.41
+2.1%
+0.3%
Qwen3.8-Flash-Next BF16
32
640
513
CK → BN128
261
814.37
765.02
+6.5%
+9.1%
Qwen3.8-Flash-Next BF16
64
640
513
CK → BN128
388
1248.73
1167.19
+7.0%
+8.6%
Qwen3.8-Flash-Next BF16
128
640
513
CK → BN128
477
1557.94
1453.71
+7.2%
+6.5%
Qwen3.8-Flash-Next BF16
256
640
513
CK → BN128
513
1684.42
1657.67
+1.6%
+5.8%
Qwen3.8-Flash-Next FP8
1
192
513
BN64
11
15.52
13.86
+12.0%
+9.1%
Qwen3.8-Flash-Next FP8
2
192
513
BN64 → direct
22
30.19
19.03
+58.6%
+60.5%
Qwen3.8-Flash-Next FP8
4
192
513
BN64 → direct
42
41.03
28.92
+41.9%
+42.1%
Qwen3.8-Flash-Next FP8
8
192
513
BN64 → direct
84
59.28
47.86
+23.9%
+25.1%
Qwen3.8-Flash-Next FP8
16
192
513
asm
152
90.29
90.67
-0.4%
+0.5%
Qwen3.8-Flash-Next FP8
32
192
513
BN64 → BN128
257
140.90
138.31
+1.9%
+1.8%
Qwen3.8-Flash-Next FP8
64
192
513
BN64 → BN128
384
202.07
197.74
+2.2%
+3.0%
Qwen3.8-Flash-Next FP8
128
192
513
asm
473
239.26
237.57
+0.7%
+0.2%
Qwen3.8-Flash-Next FP8
256
192
513
asm
512
295.52
286.86
+3.0%
-1.7%
Qwen3.8-Flash-Next FP8
1
320
513
BN64
11
18.09
17.19
+5.2%
+4.8%
Qwen3.8-Flash-Next FP8
2
320
513
BN64 → direct
22
36.30
24.88
+45.9%
+45.9%
Qwen3.8-Flash-Next FP8
4
320
513
BN64 → direct
41
51.33
40.68
+26.2%
+27.0%
Qwen3.8-Flash-Next FP8
8
320
513
asm → direct
76
72.75
70.69
+2.9%
+2.8%
Qwen3.8-Flash-Next FP8
16
320
513
BN64 → BN128
140
127.85
128.04
-0.1%
+2.7%
Qwen3.8-Flash-Next FP8
32
320
513
asm
241
214.72
212.13
+1.2%
+0.1%
Qwen3.8-Flash-Next FP8
64
320
513
CK
366
294.00
295.65
-0.6%
+0.0%
Qwen3.8-Flash-Next FP8
128
320
513
asm
475
397.63
399.89
-0.6%
+0.2%
Qwen3.8-Flash-Next FP8
256
320
513
asm
511
449.16
448.93
+0.1%
-1.0%
Qwen3.8-Flash-Next FP8
1
640
513
BN64
11
26.12
25.16
+3.8%
+3.8%
Qwen3.8-Flash-Next FP8
2
640
513
BN64 → direct
22
51.31
40.18
+27.7%
+27.1%
Qwen3.8-Flash-Next FP8
4
640
513
CK → direct
42
82.19
72.95
+12.7%
+13.1%
Qwen3.8-Flash-Next FP8
8
640
513
CK
83
133.97
135.09
-0.8%
-0.1%
Qwen3.8-Flash-Next FP8
16
640
513
CK
153
215.28
214.87
+0.2%
+0.0%
Qwen3.8-Flash-Next FP8
32
640
513
CK
261
390.11
375.29
+3.9%
-0.3%
Qwen3.8-Flash-Next FP8
64
640
513
CK
388
619.12
617.74
+0.2%
-1.2%
Qwen3.8-Flash-Next FP8
128
640
513
asm
477
800.62
791.77
+1.1%
+0.5%
Qwen3.8-Flash-Next FP8
256
640
513
asm
513
880.58
875.14
+0.6%
+1.3%
Qwen3.5-35B FP8
1
128
257
BN64
9
13.24
11.12
+19.1%
+13.3%
Qwen3.5-35B FP8
2
128
257
BN64 → direct
18
25.08
12.73
+97.0%
+91.4%
Qwen3.5-35B FP8
4
128
257
BN64 → direct
35
29.24
17.32
+68.8%
+66.3%
Qwen3.5-35B FP8
8
128
257
BN64 → direct
69
39.11
27.71
+41.1%
+40.4%
Qwen3.5-35B FP8
16
128
257
BN64
112
53.68
53.16
+1.0%
+0.2%
Qwen3.5-35B FP8
32
128
257
BN64 → BN128
179
65.89
65.80
+0.1%
+0.0%
Qwen3.5-35B FP8
64
128
257
BN64 → BN128
237
85.21
82.25
+3.6%
+2.9%
Qwen3.5-35B FP8
128
128
257
BN64 → BN128
252
101.69
96.23
+5.7%
+5.7%
Qwen3.5-35B FP8
256
128
257
CK → BN128
256
125.60
121.44
+3.4%
+3.2%
Qwen3.5-35B FP8
1
512
257
BN64
9
19.69
18.10
+8.8%
+8.1%
Qwen3.5-35B FP8
2
512
257
BN64 → direct
18
37.60
27.29
+37.8%
+34.8%
Qwen3.5-35B FP8
4
512
257
BN64 → direct
33
52.59
42.68
+23.2%
+21.0%
Qwen3.5-35B FP8
8
512
257
CK → direct
61
81.63
76.87
+6.2%
+5.1%
Qwen3.5-35B FP8
16
512
257
CK
107
123.26
122.62
+0.5%
-0.2%
Qwen3.5-35B FP8
32
512
257
CK
176
171.00
171.41
-0.2%
-0.1%
Qwen3.5-35B FP8
64
512
257
CK
228
212.57
211.58
+0.5%
+0.1%
Qwen3.5-35B FP8
128
512
257
CK
253
238.37
236.99
+0.6%
+0.0%
Qwen3.5-35B FP8
256
512
257
CK
257
252.51
251.91
+0.2%
-0.1%
Qwen3.5-397B FP8
1
128
513
BN64
11
17.62
15.98
+10.3%
+9.9%
Qwen3.5-397B FP8
2
128
513
BN64 → direct
21
33.70
21.18
+59.1%
+60.0%
Qwen3.5-397B FP8
4
128
513
BN64 → direct
43
45.37
31.95
+42.0%
+43.5%
Qwen3.5-397B FP8
8
128
513
BN64 → direct
87
65.88
53.14
+24.0%
+21.3%
Qwen3.5-397B FP8
16
128
513
BN64 → BN128
154
99.73
97.02
+2.8%
+2.5%
Qwen3.5-397B FP8
32
128
513
BN64 → BN128
261
152.53
147.93
+3.1%
+3.0%
Qwen3.5-397B FP8
64
128
513
BN64 → BN128
388
222.78
213.02
+4.6%
+5.1%
Qwen3.5-397B FP8
128
128
513
BN64 → BN128
478
305.60
278.05
+9.9%
+8.1%
Qwen3.5-397B FP8
256
128
513
CK → BN128
510
370.28
351.68
+5.3%
+4.9%
Odd token counts. 3 tokens use the 4-token row, 5–7 the 8-token row, 9–15 the 16-token row. Three rounds, speedup = main / PR − 1:
Tokens
Shapes
Geomean
Total time
Worst / best
> 2% faster / slower
3
12
+32.6%
+28.0%
+15.5% / +74.3%
12 / 0
5–7
36
+16.3%
+11.2%
-0.5% / +57.4%
30 / 0
9–15
84
+0.4%
+0.8%
-4.8% / +14.3%
12 / 9
3–7 tokens keep most of the gain. 9–15 tokens are neutral; 5 of the 9 slower runs are Hunyuan3, whose 16-token row moved to BN128 (-2.2% on average at 9–15).
Per-shape odd token counts (speedup)
Model
inter_dim
E
M=3
M=5
M=6
M=7
M=9
M=10
M=11
M=12
M=13
M=14
M=15
Hunyuan3 FP8 per-tensor
192
192
+31.7%
+25.0%
+17.9%
+13.8%
-4.8%
-2.8%
+1.6%
+0.4%
-3.8%
-3.3%
-2.6%
MiniMax-M3 FP8
384
128
+15.5%
+11.1%
+5.5%
+6.0%
+14.3%
+11.8%
+11.2%
-1.4%
-1.2%
-0.3%
+1.0%
MiniMax-M3 FP8
384
129
+23.6%
+16.3%
+12.4%
+6.1%
+1.8%
-0.2%
+0.3%
+0.7%
+1.3%
+1.3%
+0.7%
Qwen3.8-Flash-Next BF16
192
513
+44.6%
+24.6%
+22.2%
+18.9%
-0.3%
+0.2%
+0.4%
+0.1%
+0.4%
+0.4%
+0.3%
Qwen3.8-Flash-Next BF16
320
513
+21.7%
+14.5%
+11.5%
+8.9%
-0.3%
-2.5%
-0.7%
-1.9%
-1.5%
-0.5%
-2.2%
Qwen3.8-Flash-Next BF16
640
513
+16.9%
-0.2%
-0.3%
-0.1%
+5.7%
+5.0%
+3.1%
+1.9%
+3.5%
+0.8%
+4.7%
Qwen3.8-Flash-Next FP8
192
513
+50.6%
+29.8%
+26.4%
+25.3%
+0.1%
+0.1%
-0.1%
-0.5%
-0.1%
+0.0%
-0.5%
Qwen3.8-Flash-Next FP8
320
513
+28.2%
+28.2%
+15.6%
+9.0%
+0.7%
-2.3%
+0.7%
-1.4%
+0.3%
+1.4%
+0.3%
Qwen3.8-Flash-Next FP8
640
513
+19.0%
-0.1%
-0.1%
-0.5%
-0.1%
-0.5%
+0.0%
+0.2%
-1.1%
-0.3%
-0.8%
Qwen3.5-35B FP8
128
257
+74.3%
+57.4%
+49.9%
+40.9%
-1.0%
-0.6%
-0.9%
-0.7%
-0.5%
-0.7%
-0.7%
Qwen3.5-35B FP8
512
257
+34.4%
+17.3%
+9.7%
+5.9%
-1.2%
+0.0%
+0.0%
-0.7%
-1.2%
-0.7%
-0.6%
Qwen3.5-397B FP8
128
513
+41.9%
+32.7%
+30.4%
+24.3%
-2.2%
+3.4%
+2.2%
-1.9%
-1.5%
+2.7%
+3.4%
CI has no gfx942 performance check, so these manual runs are the only performance evidence for this PR.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Batch-1 configuration validation must reject tile dimensions other than (16, 16, 16) before approval.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Reject unsupported batch-1 tile dimensions
aiter/ops/flydsl/fused_moe_gfx942.py:92
This validation only checks BLOCK_N, but _run_batch1 hard-codes all three batch-1 tile dimensions (BLOCK_M=16, BLOCK_N=16, and BLOCK_K=16 is not passed through). A config such as 32_16_32_False:default:0 therefore passes unsupported_reason, is serialized as that tile, and executes the fixed 16×16×16 kernels. Reject any non-(16, 16, 16) tile for batch 1, just as the direct-decoding flag is constrained.
Thanks for the per-shape data. It shows that some configurations still regress against main:
Token counts that use a retuned row without being measured at tuning time. Hunyuan's 16-token row moved from BN64 to 16_128_64_False, and get_padded_M also routes 9–15 tokens to it. It is +0.9% at 16 tokens, but −4.8% at 9, −2.8% at 10, −3.8% at 13, −3.3% at 14 and −2.6% at 15 tokens. Other points slower by more than 2% in the same range: Qwen3.8-Flash-Next BF16 (inter_dim 320) at 10 and 15 tokens, Qwen3.8-Flash-Next FP8 (inter_dim 320) at 10 tokens, and Qwen3.5-397B at 9 tokens.
Prefill rows whose GEMM selection is unchanged but whose sort/reduce path changed. MiniMax-M3 M=1024/E=128 goes from 458.95 to 470.50 µs (−2.5%), and Qwen3.8-Flash-Next FP8 M=2048/inter_dim=192 goes from 503.24 to 520.26 µs (−3.3%). From 8192 tokens up, every Hunyuan and MiniMax-M3 prefill row is also 0.4–1.1% slower, and the gap grows with M (up to +177 µs and +220 µs at 131072 tokens).
Could you re-measure both performance and accuracy on gfx942 for all configurations that may be affected? That would cover every runtime token count each retuned decoding row serves (not only the tuned buckets) and all 71 prefill rows, with repeated runs for the rows above and the error/status reported next to each latency.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The low-level gfx942 routing and kernel changes require head-matched MI308X runtime correctness and performance validation that static review cannot provide.
Could you re-measure both performance and accuracy on gfx942 for all configurations that may be affected? That would cover every runtime token count each retuned decoding row serves (not only the tuned buckets) and all 71 prefill rows, with repeated runs for the rows above and the error/status reported next to each latency.
Done on 7b58f3a457 (on top of af98038f0), against main 2bafed4728 with each tree's own CSVs: MI308X, PTL enabled, FlyDSL 0.4.0, default rounding, --run_config latencies (10 warmups, 51 iterations, rotating inputs), main and PR measured back to back on the same GPU in alternating order. Every latency below is the median of three rounds, or of seven for the points that were more than 1% slower than main after three. err_ratio and status are the ones --run_config reports for that run.
Decoding: every served token count. A row tuned at M tokens also serves M/2+1 to M−1 tokens (get_padded_M), so I measured all 1572 token counts the 65 retuned rows serve. A retuned row is kept only if it is at least 1% faster than main on average over its token counts and no token count is more than 2% slower. Seven BN128 rows fail that and now keep main's BN64 selection, including every point you listed:
Row (tokens)
Before the revert: geomean
Worst
Reason
Hunyuan3 inter_dim 192, M=16 (9–16)
−1.9%
−5.8% (9)
5 token counts more than 2% slower (9, 10, 13, 14, 15)
Hunyuan3 inter_dim 192, M=32 (17–32)
−0.1%
−2.7% (21)
2 token counts more than 2% slower (20, 21)
Qwen3.8 BF16 inter_dim 320, M=16 (9–16)
+0.4%
−1.2% (10)
less than 1% faster on average
Qwen3.8 FP8 inter_dim 320, M=16 (9–16)
+0.3%
−1.2% (12)
less than 1% faster on average
Qwen3.5-35B inter_dim 128, M=32 (17–32)
−1.7%
−3.5% (18)
8 token counts more than 2% slower (18, 19, 20, 26, 27, 28, 29, 30)
Qwen3.5-35B inter_dim 128, M=64 (33–64)
+0.8%
−2.7% (43)
2 token counts more than 2% slower (43, 44)
Qwen3.5-397B inter_dim 128, M=16 (9–16)
+0.8%
−2.7% (9)
2 token counts more than 2% slower (9, 12)
Tokens (row M)
Rows
Token counts
Geomean
Total time
Worst / best
> 2% faster / slower
2 (2)
12
12
+42.8%
+38.1%
+22.8% / +90.0%
12 / 0
3–4 (4)
12
24
+30.4%
+25.1%
+9.9% / +80.5%
24 / 0
5–8 (8)
10
40
+18.4%
+15.2%
+0.1% / +58.0%
38 / 0
9–16 (16)
3
24
+3.1%
+2.8%
−0.8% / +14.1%
8 / 0
17–32 (32)
4
64
+4.2%
+5.4%
−1.3% / +15.9%
41 / 0
33–64 (64)
5
160
+4.3%
+5.9%
−0.6% / +9.9%
134 / 0
65–128 (128)
6
384
+4.8%
+5.5%
−0.9% / +10.6%
352 / 0
129–256 (256)
6
768
+6.2%
+5.3%
−0.1% / +14.1%
749 / 0
2–256, retuned rows
58
1476
+6.4%
+5.5%
−1.3% / +90.0%
1358 / 0
2–256, the 7 rows on main's selection
7
96
+0.0%
−0.1%
−5.5% / +2.7%
1 / 5
The seven reverted rows run the same kernel on both sides, so their spread is the measurement noise of a single token count.
Per row: all 65 retuned rows
Model
inter_dim
E
M
Tokens
main → PR
Rounds
Geomean
Worst (tokens)
Best
err_ratio main / PR
logits_diff main / PR
Status
Hunyuan3
192
192
2
2–2
BN64 → direct
3
+51.4%
+51.4% (2)
+51.4%
72.2% / 72.3%
0.0011 / 0.0011
ok
Hunyuan3
192
192
4
3–4
BN64 → direct
3
+31.4%
+30.2% (3)
+32.5%
72.9% / 73.1%
0.0011 / 0.0011
ok
Hunyuan3
192
192
8
5–8
BN64 → direct
3
+17.2%
+13.8% (8)
+23.2%
73.3% / 73.5%
0.0011 / 0.0011
ok
Hunyuan3
192
192
16
9–16
BN64 (main's)
3
−0.3%
−2.4% (11)
+2.7%
73.3% / 73.3%
0.0011 / 0.0011
ok
Hunyuan3
192
192
32
17–32
BN64 (main's)
3
+0.1%
−2.4% (24)
+2.0%
73.2% / 73.2%
0.0011 / 0.0011
ok
Hunyuan3
192
192
64
33–64
BN64 → BN128
3/7
+2.0%
−0.6% (41)
+4.8%
72.8% / 72.8%
0.0011 / 0.0011
ok
Hunyuan3
192
192
128
65–128
BN64 → BN128
3
+5.2%
+3.2% (65)
+7.6%
73.0% / 73.0%
0.0011 / 0.0011
ok
Hunyuan3
192
192
256
129–256
CK → BN128
3
+7.1%
+2.9% (237)
+12.7%
0.2% / 73.0%
0.0000 / 0.0011
ok
MiniMax-M3
384
128
2
2–2
BN64 → direct
3
+31.0%
+31.0% (2)
+31.0%
80.4% / 80.4%
0.0007 / 0.0007
ok
MiniMax-M3
384
128
4
3–4
BN64 → direct
3
+18.8%
+17.1% (3)
+20.5%
81.2% / 81.2%
0.0008 / 0.0008
ok
MiniMax-M3
384
128
8
5–8
BN64 → direct
3
+6.4%
+5.2% (7)
+8.1%
81.4% / 81.5%
0.0008 / 0.0008
ok
MiniMax-M3
384
128
16
9–16
CK → BN128
3/7
+4.8%
−0.8% (12)
+14.1%
2.8% / 81.9%
0.0000 / 0.0008
ok
MiniMax-M3
384
129
2
2–2
BN64 → direct
3
+33.9%
+33.9% (2)
+33.9%
82.7% / 82.8%
0.0009 / 0.0009
ok
MiniMax-M3
384
129
4
3–4
BN64 → direct
3
+17.9%
+12.3% (4)
+23.8%
82.3% / 82.3%
0.0009 / 0.0009
ok
MiniMax-M3
384
129
8
5–8
CK → direct
3
+8.8%
+0.1% (8)
+16.2%
3.9% / 82.2%
0.0000 / 0.0009
ok
MiniMax-M3
384
129
16
9–16
CK → BN128
3
+1.2%
+0.9% (10)
+1.5%
3.9% / 81.7%
0.0000 / 0.0008
ok
Qwen3.8 BF16
192
513
2
2–2
BN64 → direct
3
+47.4%
+47.4% (2)
+47.4%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
192
513
4
3–4
BN64 → direct
3
+41.2%
+38.4% (4)
+44.2%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
192
513
8
5–8
BN64 → direct
3
+19.5%
+14.5% (8)
+23.0%
0.1% / 0.1%
0.0000 / 0.0000
ok
Qwen3.8 BF16
192
513
128
65–128
BN64 → BN128
3/7
+2.6%
−0.9% (87)
+8.3%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
192
513
256
129–256
CK → BN128
3
+7.1%
+2.3% (199)
+10.9%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
320
513
2
2–2
BN64 → direct
3
+32.8%
+32.8% (2)
+32.8%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
320
513
4
3–4
BN64 → direct
3
+21.6%
+20.0% (3)
+23.3%
0.1% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
320
513
8
5–8
BN64 → direct
3
+10.2%
+6.3% (8)
+14.3%
0.0% / 0.1%
0.0000 / 0.0000
ok
Qwen3.8 BF16
320
513
16
9–16
BN64 (main's)
3
−0.3%
−0.5% (11)
−0.1%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
320
513
32
17–32
CK → BN128
3
+9.1%
+5.6% (28)
+15.9%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
320
513
64
33–64
CK → BN128
3
+6.8%
+2.0% (55)
+9.9%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
320
513
128
65–128
CK → BN128
3
+5.5%
+2.4% (105)
+8.4%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
320
513
256
129–256
CK → BN128
3
+3.4%
−0.1% (227)
+7.0%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
640
513
2
2–2
BN64 → direct
3
+22.8%
+22.8% (2)
+22.8%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 BF16
640
513
4
3–4
BN64 → direct
3
+13.2%
+9.9% (4)
+16.5%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 BF16
640
513
16
9–16
CK → BN128
3
+3.3%
+0.4% (14)
+7.8%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 BF16
640
513
32
17–32
CK → BN128
3
+4.9%
−0.2% (18)
+8.8%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 BF16
640
513
64
33–64
CK → BN128
3
+7.2%
+4.6% (61)
+9.5%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 BF16
640
513
128
65–128
CK → BN128
3
+6.6%
+2.6% (123)
+9.1%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 BF16
640
513
256
129–256
CK → BN128
3
+5.3%
+2.8% (218)
+7.4%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 FP8
192
513
2
2–2
BN64 → direct
3
+54.1%
+54.1% (2)
+54.1%
59.1% / 59.1%
0.0010 / 0.0010
ok
Qwen3.8 FP8
192
513
4
3–4
BN64 → direct
3
+46.8%
+41.0% (4)
+52.9%
58.8% / 59.0%
0.0010 / 0.0010
ok
Qwen3.8 FP8
192
513
8
5–8
BN64 → direct
3
+28.4%
+24.4% (8)
+33.0%
61.6% / 61.8%
0.0011 / 0.0011
ok
Qwen3.8 FP8
192
513
32
17–32
BN64 → BN128
3
+1.5%
−0.6% (18)
+3.3%
60.5% / 60.5%
0.0010 / 0.0010
ok
Qwen3.8 FP8
192
513
64
33–64
BN64 → BN128
3
+2.5%
+0.3% (34)
+3.7%
60.0% / 60.0%
0.0011 / 0.0011
ok
Qwen3.8 FP8
320
513
2
2–2
BN64 → direct
3
+45.8%
+45.8% (2)
+45.8%
64.8% / 64.8%
0.0010 / 0.0010
ok
Qwen3.8 FP8
320
513
4
3–4
BN64 → direct
3
+28.5%
+27.1% (4)
+30.0%
66.1% / 66.0%
0.0011 / 0.0011
ok
Qwen3.8 FP8
320
513
8
5–8
asm → direct
3
+14.6%
+1.6% (8)
+33.9%
19.4% / 66.4%
0.0001 / 0.0011
ok
Qwen3.8 FP8
320
513
16
9–16
BN64 (main's)
3
−1.1%
−5.5% (15)
+0.5%
65.7% / 65.8%
0.0011 / 0.0011
ok
Qwen3.8 FP8
640
513
2
2–2
BN64 → direct
3
+26.9%
+26.9% (2)
+26.9%
70.5% / 70.6%
0.0010 / 0.0010
ok
Qwen3.8 FP8
640
513
4
3–4
CK → direct
3
+16.0%
+10.2% (4)
+22.0%
0.2% / 71.4%
0.0000 / 0.0010
ok
Qwen3.5-35B
128
257
2
2–2
BN64 → direct
3
+90.0%
+90.0% (2)
+90.0%
47.8% / 47.9%
0.0012 / 0.0012
ok
Qwen3.5-35B
128
257
4
3–4
BN64 → direct
3
+73.3%
+66.3% (4)
+80.5%
51.3% / 51.5%
0.0010 / 0.0010
ok
Qwen3.5-35B
128
257
8
5–8
BN64 → direct
3
+47.0%
+38.8% (8)
+58.0%
52.0% / 52.0%
0.0009 / 0.0009
ok
Qwen3.5-35B
128
257
32
17–32
BN64 (main's)
3
+0.2%
−0.9% (29)
+1.5%
50.1% / 50.1%
0.0010 / 0.0010
ok
Qwen3.5-35B
128
257
64
33–64
BN64 (main's)
3
+0.3%
−1.2% (63)
+1.4%
49.9% / 49.8%
0.0010 / 0.0010
ok
Qwen3.5-35B
128
257
128
65–128
BN64 → BN128
3
+3.4%
+0.5% (72)
+5.6%
49.7% / 49.8%
0.0010 / 0.0010
ok
Qwen3.5-35B
128
257
256
129–256
CK → BN128
3
+8.4%
+2.4% (256)
+14.1%
0.7% / 49.6%
0.0000 / 0.0010
ok
Qwen3.5-35B
512
257
2
2–2
BN64 → direct
3
+32.7%
+32.7% (2)
+32.7%
69.1% / 69.0%
0.0011 / 0.0011
ok
Qwen3.5-35B
512
257
4
3–4
BN64 → direct
3
+26.7%
+19.5% (4)
+34.4%
69.5% / 69.6%
0.0011 / 0.0011
ok
Qwen3.5-35B
512
257
8
5–8
CK → direct
3
+9.9%
+3.2% (7)
+19.5%
0.1% / 68.4%
0.0000 / 0.0011
ok
Qwen3.5-397B
128
513
2
2–2
BN64 → direct
3
+56.7%
+56.7% (2)
+56.7%
67.7% / 67.6%
0.0011 / 0.0011
ok
Qwen3.5-397B
128
513
4
3–4
BN64 → direct
3
+40.8%
+39.9% (4)
+41.7%
67.1% / 66.8%
0.0010 / 0.0010
ok
Qwen3.5-397B
128
513
8
5–8
BN64 → direct
3
+27.3%
+24.3% (8)
+33.0%
66.2% / 66.1%
0.0010 / 0.0010
ok
Qwen3.5-397B
128
513
16
9–16
BN64 (main's)
3
−0.1%
−0.5% (16)
+0.3%
66.1% / 66.1%
0.0010 / 0.0010
ok
Qwen3.5-397B
128
513
32
17–32
BN64 → BN128
3/7
+1.4%
−1.3% (17)
+3.2%
65.6% / 65.5%
0.0010 / 0.0010
ok
Qwen3.5-397B
128
513
64
33–64
BN64 → BN128
3
+3.3%
+0.9% (36)
+5.2%
66.3% / 66.3%
0.0011 / 0.0011
ok
Qwen3.5-397B
128
513
128
65–128
BN64 → BN128
3
+5.3%
+2.7% (100)
+10.6%
66.3% / 66.3%
0.0010 / 0.0010
ok
Qwen3.5-397B
128
513
256
129–256
CK → BN128
3
+5.6%
+3.1% (255)
+9.6%
2.6% / 66.0%
0.0000 / 0.0010
ok
Per token count: speedup of every point (main / PR − 1, %)
Hunyuan3 inter_dim 192 E 192 M=2: 2:+51.4
Hunyuan3 inter_dim 192 E 192 M=4: 3:+30.2 4:+32.5
Hunyuan3 inter_dim 192 E 192 M=8: 5:+23.2 6:+18.0 7:+14.1 8:+13.8
Prefill. The sort/reduce changes did cost time. At 131072 tokens the aux sort is 41–68 µs slower than the plain sort (the removed invert_sorted_ids saved 19–32 µs), and Hunyuan3's per-tensor clear of the Gate/Up output adds about 130 µs. The prefill sort also cleared the whole [tokens, model_dim] output, which sorted_sum overwrites row by row anyway; main does the same. 7b58f3a457 drops that clear (accumulate=False), which saves 3 µs at 1024 tokens and 300–480 µs at 131072 tokens, more than the costs above. The per-tensor Gate/Up clear stays, since it keeps dropped routes out of the Down absmax. All 94 prefill rows, three rounds:
Prefill rows
Rows
Geomean
Total time
Worst / best
> 2% faster / slower
FlyDSL (changed path)
71
+1.0%
+1.3%
−0.3% / +3.6%
5 / 0
CK and ASM (unchanged code)
23
+0.2%
+0.0%
−0.6% / +2.4%
1 / 0
All
94
+0.8%
+1.1%
−0.6% / +3.6%
6 / 0
The rows you listed: MiniMax-M3 M=1024/E=128 is now +1.6%, Qwen3.8-Flash-Next FP8 M=2048/inter_dim=192 −0.3%, and every Hunyuan and MiniMax-M3 row from 8192 tokens up is 0.7% to 2.4% faster than main. MiniMax-M3 M=1024/E=129 measured −2.1% in the first three rounds; over eleven rounds on two GPUs it is −0.2%, and the per-kernel times put the difference in the Down GEMM, whose code is unchanged.
Per row: all 94 prefill rows
Model
M
inter_dim
E
Kernel
Rounds
main µs
PR µs
Speedup
err_ratio main / PR
logits_diff main / PR
Status
Hunyuan3
512
192
192
64_128_256_True
3
354.27
352.42
+0.5%
0.0% / 0.0%
0.0000 / 0.0000
ok
Hunyuan3
1024
192
192
64_128_256_True
3
384.15
378.68
+1.4%
0.0% / 0.0%
0.0000 / 0.0000
ok
Hunyuan3
2048
192
192
64_128_256_True
3
573.76
562.92
+1.9%
0.0% / 0.0%
0.0000 / 0.0000
ok
Hunyuan3
4096
192
192
64_128_256_True
3
869.80
863.09
+0.8%
0.0% / 0.0%
0.0000 / 0.0000
ok
Hunyuan3
8192
192
192
64_128_256_True
3
1486.57
1472.36
+1.0%
0.0% / 0.0%
0.0000 / 0.0000
ok
Hunyuan3
16384
192
192
64_128_256_True
3
2679.22
2660.10
+0.7%
0.0% / 0.0%
0.0000 / 0.0000
ok
Hunyuan3
32768
192
192
64_128_256_True
3
5092.76
5057.20
+0.7%
0.0% / 0.0%
0.0000 / 0.0000
ok
Hunyuan3
65536
192
192
64_128_256_True
3
10225.49
10144.39
+0.8%
0.0% / 0.0%
0.0000 / 0.0000
ok
Hunyuan3
131072
192
192
64_128_256_True
3
20709.73
20567.62
+0.7%
0.0% / 0.0%
0.0000 / 0.0000
ok
MiniMax-M3
512
384
128
CK
3
380.07
371.10
+2.4%
6.0% / 6.0%
0.0000 / 0.0000
ok
MiniMax-M3
1024
384
128
64_256_128_True
11
446.89
440.00
+1.6%
4.8% / 4.8%
0.0000 / 0.0000
ok
MiniMax-M3
2048
384
128
64_256_128_True
3
713.09
710.38
+0.4%
4.3% / 4.3%
0.0000 / 0.0000
ok
MiniMax-M3
4096
384
128
64_256_128_True
3
1074.46
1057.40
+1.6%
4.7% / 4.7%
0.0000 / 0.0000
ok
MiniMax-M3
8192
384
128
64_128_256_True
3
1868.83
1839.72
+1.6%
4.7% / 4.7%
0.0000 / 0.0000
ok
MiniMax-M3
16384
384
128
64_256_128_True
3
3279.14
3221.28
+1.8%
4.8% / 4.8%
0.0000 / 0.0000
ok
MiniMax-M3
32768
384
128
64_256_128_True
3
6289.28
6171.90
+1.9%
4.7% / 4.7%
0.0000 / 0.0000
ok
MiniMax-M3
65536
384
128
64_256_128_True
3
12160.27
11937.88
+1.9%
4.7% / 4.7%
0.0000 / 0.0000
ok
MiniMax-M3
131072
384
128
64_256_128_True
3
24251.69
23790.58
+1.9%
4.8% / 4.8%
0.0000 / 0.0000
ok
MiniMax-M3
512
384
129
64_256_128_True
3
423.04
421.54
+0.4%
4.3% / 4.3%
0.0000 / 0.0000
ok
MiniMax-M3
1024
384
129
64_256_128_True
11
462.95
463.84
−0.2%
4.6% / 4.6%
0.0000 / 0.0000
ok
MiniMax-M3
2048
384
129
64_256_128_True
3
892.31
883.21
+1.0%
4.3% / 4.3%
0.0000 / 0.0000
ok
MiniMax-M3
4096
384
129
64_128_256_True
3
1301.71
1291.79
+0.8%
4.6% / 4.6%
0.0000 / 0.0000
ok
MiniMax-M3
8192
384
129
64_128_256_True
3
2269.17
2239.18
+1.3%
4.5% / 4.5%
0.0000 / 0.0000
ok
MiniMax-M3
16384
384
129
64_256_128_True
3
4019.96
3956.61
+1.6%
4.5% / 4.5%
0.0000 / 0.0000
ok
MiniMax-M3
32768
384
129
64_256_128_True
3
7646.91
7547.09
+1.3%
4.6% / 4.6%
0.0000 / 0.0000
ok
MiniMax-M3
65536
384
129
64_256_128_True
3
14895.42
14671.70
+1.5%
4.6% / 4.6%
0.0000 / 0.0000
ok
MiniMax-M3
131072
384
129
64_256_128_True
3
29844.40
29151.30
+2.4%
4.6% / 4.6%
0.0000 / 0.0000
ok
Qwen3.8 BF16
512
192
513
CK
3
565.90
567.16
−0.2%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
1024
192
513
CK
3
630.17
629.02
+0.2%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
2048
192
513
64_128_128_True
3
751.89
748.12
+0.5%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
4096
192
513
64_128_128_True
3
1296.71
1289.21
+0.6%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
8192
192
513
64_128_128_True
3
2011.41
1990.46
+1.1%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
16384
192
513
64_128_128_True
3
3720.54
3682.46
+1.0%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
32768
192
513
64_128_128_True
3
6750.76
6710.82
+0.6%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
512
320
513
CK
3
887.03
886.21
+0.1%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
1024
320
513
CK
3
984.56
982.11
+0.2%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
2048
320
513
64_128_128_True
3
1187.10
1175.59
+1.0%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
4096
320
513
64_128_128_True
3
2075.03
2069.51
+0.3%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
8192
320
513
64_128_128_True
3
3268.75
3255.90
+0.4%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
16384
320
513
64_128_128_True
3
5832.21
5815.87
+0.3%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
32768
320
513
64_128_128_True
3
12182.53
12165.09
+0.1%
0.0% / 0.0%
0.0000 / 0.0000
ok
Qwen3.8 BF16
512
640
513
CK
6
1689.80
1686.64
+0.2%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 BF16
1024
640
513
CK
3
1918.31
1921.85
−0.2%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 BF16
2048
640
513
CK
3
2296.72
2302.93
−0.3%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 BF16
4096
640
513
CK
3
3763.93
3752.30
+0.3%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 BF16
8192
640
513
CK
3
6199.60
6202.24
+0.0%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 BF16
16384
640
513
CK
3
13404.53
13426.52
−0.2%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 BF16
32768
640
513
CK
3
25213.49
25228.18
−0.1%
0.2% / 0.2%
0.0000 / 0.0000
ok
Qwen3.8 FP8
512
192
513
asm
3
313.80
312.22
+0.5%
8.0% / 8.0%
0.0001 / 0.0001
ok
Qwen3.8 FP8
1024
192
513
asm
3
357.23
356.98
+0.1%
7.8% / 7.8%
0.0001 / 0.0001
ok
Qwen3.8 FP8
2048
192
513
64_128_256_True
3
479.31
480.73
−0.3%
1.3% / 1.3%
0.0000 / 0.0000
ok
Qwen3.8 FP8
4096
192
513
64_128_256_True
3
917.04
917.87
−0.1%
1.1% / 1.1%
0.0000 / 0.0000
ok
Qwen3.8 FP8
8192
192
513
64_128_256_True
3
1467.13
1457.65
+0.7%
1.3% / 1.3%
0.0000 / 0.0000
ok
Qwen3.8 FP8
16384
192
513
64_128_256_True
3
2712.16
2689.55
+0.8%
1.2% / 1.2%
0.0000 / 0.0000
ok
Qwen3.8 FP8
32768
192
513
64_128_256_True
3
5045.88
5011.20
+0.7%
1.2% / 1.2%
0.0000 / 0.0000
ok
Qwen3.8 FP8
512
320
513
asm
3
459.81
462.62
−0.6%
14.3% / 14.3%
0.0001 / 0.0001
ok
Qwen3.8 FP8
1024
320
513
asm
3
495.31
497.15
−0.4%
14.1% / 14.1%
0.0001 / 0.0001
ok
Qwen3.8 FP8
2048
320
513
64_128_256_True
3
754.34
752.66
+0.2%
1.7% / 1.7%
0.0000 / 0.0000
ok
Qwen3.8 FP8
4096
320
513
asm
3
1238.48
1239.66
−0.1%
13.9% / 13.9%
0.0001 / 0.0001
ok
Qwen3.8 FP8
8192
320
513
64_128_256_True
3
2068.84
2047.54
+1.0%
1.6% / 1.6%
0.0000 / 0.0000
ok
Qwen3.8 FP8
16384
320
513
64_128_256_True
3
3741.42
3720.37
+0.6%
1.6% / 1.6%
0.0000 / 0.0000
ok
Qwen3.8 FP8
32768
320
513
64_128_256_True
3
6953.90
6902.24
+0.7%
1.6% / 1.6%
0.0000 / 0.0000
ok
Qwen3.8 FP8
512
640
513
asm
3
870.28
864.26
+0.7%
53.8% / 53.8%
0.0004 / 0.0004
ok
Qwen3.8 FP8
1024
640
513
asm
3
995.54
988.36
+0.7%
53.9% / 53.9%
0.0004 / 0.0004
ok
Qwen3.8 FP8
2048
640
513
64_256_128_True
3
1290.02
1278.18
+0.9%
2.1% / 2.1%
0.0000 / 0.0000
ok
Qwen3.8 FP8
4096
640
513
64_256_128_True
3
2264.15
2258.07
+0.3%
1.8% / 1.8%
0.0000 / 0.0000
ok
Qwen3.8 FP8
8192
640
513
64_256_128_True
3
3455.67
3446.31
+0.3%
2.0% / 2.0%
0.0000 / 0.0000
ok
Qwen3.8 FP8
16384
640
513
64_256_128_True
3
6301.86
6274.98
+0.4%
1.9% / 1.9%
0.0000 / 0.0000
ok
Qwen3.8 FP8
32768
640
513
64_256_128_True
3
11399.53
11345.79
+0.5%
1.9% / 1.9%
0.0000 / 0.0000
ok
Qwen3.5-35B
512
128
257
asm
3
134.89
135.16
−0.2%
3.3% / 3.3%
0.0001 / 0.0001
ok
Qwen3.5-35B
1024
128
257
64_128_256_True
3
163.52
158.62
+3.1%
0.6% / 0.6%
0.0000 / 0.0000
ok
Qwen3.5-35B
2048
128
257
64_256_128_True
3
252.52
247.43
+2.1%
0.7% / 0.7%
0.0000 / 0.0000
ok
Qwen3.5-35B
4096
128
257
64_128_256_True
3
380.42
372.80
+2.0%
0.6% / 0.6%
0.0000 / 0.0000
ok
Qwen3.5-35B
8192
128
257
64_128_256_True
3
690.19
680.74
+1.4%
0.7% / 0.7%
0.0000 / 0.0000
ok
Qwen3.5-35B
16384
128
257
64_256_128_True
3
1230.75
1209.12
+1.8%
0.7% / 0.7%
0.0000 / 0.0000
ok
Qwen3.5-35B
32768
128
257
64_256_128_True
3
2385.62
2340.06
+1.9%
0.7% / 0.7%
0.0000 / 0.0000
ok
Qwen3.5-35B
65536
128
257
64_256_128_True
3
4610.54
4546.10
+1.4%
0.7% / 0.7%
0.0000 / 0.0000
ok
Qwen3.5-35B
131072
128
257
64_256_128_True
3
9434.55
9295.21
+1.5%
0.7% / 0.7%
0.0000 / 0.0000
ok
Qwen3.5-35B
512
512
257
CK
3
307.14
307.52
−0.1%
1.5% / 1.5%
0.0000 / 0.0000
ok
Qwen3.5-35B
1024
512
257
64_256_128_True
3
389.98
385.79
+1.1%
1.5% / 1.5%
0.0000 / 0.0000
ok
Qwen3.5-35B
2048
512
257
64_256_128_True
3
658.81
658.60
+0.0%
1.5% / 1.5%
0.0000 / 0.0000
ok
Qwen3.5-35B
4096
512
257
64_256_128_True
3
1015.85
1010.31
+0.5%
1.4% / 1.4%
0.0000 / 0.0000
ok
Qwen3.5-35B
8192
512
257
64_256_128_True
3
1724.92
1708.76
+0.9%
1.5% / 1.5%
0.0000 / 0.0000
ok
Qwen3.5-35B
16384
512
257
64_256_128_True
3
3123.82
3108.96
+0.5%
1.5% / 1.5%
0.0000 / 0.0000
ok
Qwen3.5-35B
32768
512
257
64_256_128_True
3
6051.12
6021.21
+0.5%
1.4% / 1.4%
0.0000 / 0.0000
ok
Qwen3.5-397B
512
128
513
asm
3
412.44
410.77
+0.4%
13.8% / 13.8%
0.0001 / 0.0001
ok
Qwen3.5-397B
1024
128
513
asm
3
446.22
445.95
+0.1%
13.2% / 13.2%
0.0001 / 0.0001
ok
Qwen3.5-397B
2048
128
513
64_128_256_True
3
567.40
567.64
+0.0%
1.6% / 1.6%
0.0000 / 0.0000
ok
Qwen3.5-397B
4096
128
513
64_256_128_True
3
1028.63
1018.31
+1.0%
1.6% / 1.6%
0.0000 / 0.0000
ok
Qwen3.5-397B
8192
128
513
64_256_128_True
3
1626.90
1609.94
+1.1%
1.7% / 1.7%
0.0000 / 0.0000
ok
Qwen3.5-397B
16384
128
513
64_256_128_True
3
3025.59
2987.76
+1.3%
1.7% / 1.7%
0.0000 / 0.0000
ok
Qwen3.5-397B
32768
128
513
64_256_128_True
3
5579.11
5525.54
+1.0%
1.7% / 1.7%
0.0000 / 0.0000
ok
Qwen3.5-397B
65536
128
513
64_256_128_True
3
11110.80
10969.20
+1.3%
1.8% / 1.8%
0.0000 / 0.0000
ok
Qwen3.5-397B
131072
128
513
64_256_128_True
3
22253.08
21474.98
+3.6%
1.8% / 1.8%
0.0000 / 0.0000
ok
Accuracy. Every decoding and prefill point reports status ok. err_ratio is the element-wise ratio outside atol/rtol against the --run_config reference, which quantizes the activations to FP8; its status also accepts logits_diff ≤ 0.01. FlyDSL decoding (BN64, BN128 and direct) reads BF16 activations even with FP8 weights, so its err_ratio is high on main as well (Hunyuan3 BN64: 73%). Where a row moves from CK or ASM, which quantize the activations like the reference, to FlyDSL decoding, err_ratio rises: 9 rows, Hunyuan3 inter_dim 192 E 192 M=256 0.2% → 73.0%; MiniMax-M3 inter_dim 384 E 128 M=16 2.8% → 81.9%; MiniMax-M3 inter_dim 384 E 129 M=8 3.9% → 82.2%; MiniMax-M3 inter_dim 384 E 129 M=16 3.9% → 81.7%; Qwen3.8 FP8 inter_dim 320 E 513 M=8 19.4% → 66.4%; Qwen3.8 FP8 inter_dim 640 E 513 M=4 0.2% → 71.4%; Qwen3.5-35B inter_dim 128 E 257 M=256 0.7% → 49.6%; Qwen3.5-35B inter_dim 512 E 257 M=8 0.1% → 68.4%; Qwen3.5-397B inter_dim 128 E 513 M=256 2.6% → 66.0%. logits_diff, recorded in a separate pass over all 1666 points, is at most 0.0012 on main and 0.0012 on the PR. The prefill rows keep main's kernels and their err_ratio matches main's. On the final tree, op_tests/test_flydsl_moe_gfx942.py, whose reference keeps BF16 activations for the decoding paths, passes with error 0 in both roundings (153 MoE rows, 32 sorted_sum rows), and op_tests/test_moe_2stage.py --no-legacy --csv-filter impl__flydsl_gfx942__ passes all 153 rows (largest logits_diff 0.0071).
* refactor: extract the legacy monolithic kernel into seperate files
* optimize down kernel and add qwen3.5 125b tuned csv
* apply review comments
* apply review comment
* opt batch2-8
* apply review comment
* apply review comments.
* apply review comments
* retune decoding csv: 1-8 tokens +24.3%
* avoid repeated expert IDs of a token
* use Opus aux sort for the sorted_sum route lookup to avoid call cost to `invert_sorted_ids`
* skip -1 pad-row routes in direct decoding; zero per-tensor Gate/Up output
Since #6916, routing gives rows past num_valid_rows expert -1 and weight 0
and expects sorting to drop them.
- Batch-1 and direct-decoding kernels read topk_ids without sorting; they now
skip -1 routes instead of reading weights and scales in front of expert 0.
- Per-tensor prefill zeroes the Gate/Up output: its Down absmax reads every
route row, including routes sorting dropped (merged repeated IDs, -1 ids).
Tests: pad_rows routing (-1 ids, zero weights). Weights and scales sit behind
a NaN expert and the Gate/Up output starts as inf, so reading expert -1 or a
skipped row fails the check. New edge cases: per-tensor prefill and B2 direct
decoding with pad rows.
* retune decoding csv on FlyDSL 0.4.0
Decoding rows (<= 256 tokens) start from main's selections. --e2e_tune ran with
TUNE_ONLY=flydslv2,cktile,flydsl,asm, AITER_FMOE_TUNE_EXCLUDE_KERNELS=fmoe_stage1
and AITER_FLYDSL_MOE_BF16_RTA_SIMPLIFIED=1 on MI308X with PTL enabled
(VECTOR,F8); each shape then keeps the lowest median of main's selection and
every eligible FlyDSL config over three interleaved rounds. Three picks that
are slower than main with the default rounding keep main's selection. 65 rows
change kernel versus main; rows above 256 tokens are main's.
* apply review comment
* skip the prefill output clear; batch 1 requires the 16_16_16 tile; 7 decoding rows keep main's selection
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part 3/3, follows #5795 and #5932 (both merged). Optimize MoE decoding (1–256 tokens) on MI308X/gfx942. Earlier revisions also covered gfx950 MXFP4/SiTUv2 decoding; that part is dropped and this PR is gfx942-only.
Changes
topk_idswithout sorting, on batched batch-1 kernels (grid over tokens × routes). It is tunable as16_16_16_False_True; the fifth tile field is optional, so existing config strings keep their meaning, andtest_config_stringscovers the new flag. Routes with expert ID −1 (padding rows oftopkwithnum_valid_rows, [ASM] [HIP] [FlyDSL] feat: pad-row routing count, custom all_gatherv, MoE quant-sort race fix #6916) are skipped.16_128_64computes 128 output columns per workgroup next to the BN6416_16_16, halving the workgroups and activation re-reads. Split-K always uses BK64, so no other BN64 tile is offered. Direct decoding is tuned only for 2–8 tokens, and batch 1 only as16_16_16.invert_sorted_idskernel. Routes the sort drops (repeated expert IDs of a token, −1 padding) are left out of the final sum, and per-tensor FP8 zeroes the Gate/Up output so they cannot raise the Down input absmax. The sort no longer clears the output, whichsorted_sumwrites in full.Performance
MI308X (gfx942)
MI308X/gfx942 (80 CUs, PTL enabled), ROCm 7.2.4, FlyDSL 0.4.0, default MoE sorting. Measured on
7b58f3a457; baseline is main2bafed4728(the merge base) with its shipped CSVs. The first three value columns use the default rounding (AITER_FLYDSL_MOE_BF16_RTA_SIMPLIFIEDunset), the RTA columnsAITER_FLYDSL_MOE_BF16_RTA_SIMPLIFIED=1on both trees. Values are the per-shape full-MoE GPU kernel-time sums that--run_configreports (10 warmups, 51 iterations, rotating inputs, IQR filtering); each is the geometric mean over shapes of the median of three alternating-order rounds. Speedup is baseline / PR3 − 1; these are not full-model throughput claims.Batch 1 keeps
16_16_16_False; its gain comes from the batch-1 kernel changes alone.Runtime recommendation: set
AITER_FLYDSL_MOE_BF16_RTA_SIMPLIFIED=1. The decoding rows are tuned with it, and with this PR it is 1.16% faster than the default rounding on the decoding rows (1.75% on all rows). Simplified RTA rounds BF16 halfway cases away from zero instead of to even and does not preserve NaN payloads;op_tests/test_flydsl_moe_gfx942.pypasses with error 0 either way.Rows above 256 tokens keep main's kernels. The 71 FlyDSL prefill rows are +1.09% (+0.89% with RTA), mostly because the sort no longer clears the output; the 23 CK and ASM prefill rows, whose code is unchanged, move by −0.07% (−0.10%), which is run-to-run noise.
Selection changes. 58 of 202 rows change kernel.
A row tuned at M tokens also serves M/2+1 to M−1 tokens (
get_padded_M). Every token count the retuned rows serve, 1476 points from 2 to 256 tokens, was measured as well: +6.4% on average, worst −1.3%, none more than 2% slower than main.Tuning. Main's tuned CSVs were the starting point and only decoding shapes (≤256 tokens, 108 of 202) were retuned. The whole-MoE stage (
--e2e_tune) ran with the environment of #5932 (TUNE_ONLY=flydslv2,cktile,flydsl,asm,AITER_FMOE_TUNE_EXCLUDE_KERNELS=fmoe_stage1, simplified BF16 RTA). Each decoding shape was then confirmed by a one-off script that re-timed main's selection and every eligible whole-MoE config over three interleaved rounds and kept the lowest median. Three picks that were more than 2% slower than main with the default rounding keep main's selection, and so do seven BN128 picks that were more than 2% slower than main at a token count they serve, or less than 1% faster on average.Reproduce (MI308X, gfx942, 80 CUs)
Run on this PR. For the baseline, run the
--run_configline on main (2bafed4728).Testing (this PR):