Repository navigation
[Triton/Gluon] [Config] [Bugfix] Add MiniMax-M3 MXFP4 shared-expert down_proj gemm_afp4wfp4 config - #5377
[Triton/Gluon] [Config] [Bugfix] Add MiniMax-M3 MXFP4 shared-expert down_proj gemm_afp4wfp4 config#5377rkarhila-amd wants to merge 1 commit into
Conversation
Use BK=256 for M_LEQ_128 and M_LEQ_256 so EAGLE3 verify batches at conc 32-64 do not emit NaNs on the short-K W4A4 down projection. Co-authored-by: Cursor <cursoragent@cursor.com>
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
PR title tags & labels: |
There was a problem hiding this comment.
🟡 Changes recommended
The affected exact shape and M buckets need automated regression coverage for finite and reference-matching output.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a gfx950 Triton configuration to avoid NaNs for MiniMax-M3’s MXFP4 shared-expert down projection.
Changes:
- Adds exact
N=6144, K=768dispatch. - Uses
BLOCK_SIZE_K=256forM≤128andM≤256. - Preserves default settings for other M buckets.
File summaries
| File | Description |
|---|---|
aiter/ops/triton/configs/gfx950/triton/gemm/gemm_afp4wfp4/GEMM-AFP4WFP4-N=6144-K=768.json |
Defines the shape-specific gfx950 tuning configuration. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| "M_LEQ_128": { | ||
| "BLOCK_SIZE_M": 128, | ||
| "BLOCK_SIZE_N": 64, | ||
| "BLOCK_SIZE_K": 256, |
|
Closing: the |
Summary
Add an exact-shape gfx950 Triton config for
gemm_afp4wfp4atN=6144, K=768so MiniMax-M3 MXFP4's non-fused shared-expertdown_projdoes not emit NaNs under EAGLE3 verify-batch sizes.Motivation
amd/MiniMax-M3-MXFP4(TP=4) uses a standalone shared-expert MLP. The W4A4down_projisM × 768 @ 768 × 6144and dispatchesaiter.ops.triton.gemm.basic.gemm_afp4wfp4. The default config for that shape usesBLOCK_SIZE_K=512in theM_LEQ_128andM_LEQ_256buckets. LogicalK=768is then tiled as512+256, and that K-tail path produces nondeterministic NaNs in the output (software W4A4 / A16W4 references stay finite and bit-exact).This shows up in EAGLE3 target-verify when
M ≈ concurrency × num_draft_tokenslands in(64, 256](e.g. conc=32 with 4–5 draft tokens →M=128or160). Gate-up (K=6144) and routed MoE were not the culprit.Changes
aiter/ops/triton/configs/gfx950/triton/gemm/gemm_afp4wfp4/GEMM-AFP4WFP4-N=6144-K=768.json.M_LEQ_128andM_LEQ_256, useBM=128, BN=64, BK=256, NUM_KSPLIT=1instead of the defaultBK=512.No kernel code changes. Config-only.
Accuracy (before / after)
Model:
amd/MiniMax-M3-MXFP4+ EAGLE3 draftInferact/MiniMax-M3-EAGLE3, TP=4, gfx950 (MI350X), non-fused shared expert (--disable-shared-experts-fusion),--moe-runner-backend aiter,--attention-backend triton.Workload: SGLang GSM8K
bench_sglang.py, 64 questions, 5-shot,max_new_tokens=1024.BK=512)down_projNaNsBK=256)Kernel replay on captured conc=32 verify activations (layer-3 shared
down_proj):BM=64, BN=64, BK=512: thousands of NaNs per run, nondeterministic across TP ranks.BM=128, BN=64, BK=256: 0 NaNs, bit-exact, ~5% faster on that microbench (~138.9 µs → ~132.2 µs).Triton version caveat
Tuning and end-to-end numbers were taken in the SGLang ROCm container
rocm/sgl-dev:v0.5.19-rocm720-mi35x-20260908, which currently ships Triton 3.7.0 (/opt/venv/.../triton, torch2.9.1+rocm7.2). It is unknown when the SGLang containers will move to a newer Triton. This config should be re-validated if/when that Triton pin changes; a later Triton may fix theBK=512K-tail bug or want a different tile.Reproduce Accuracy test
All steps below are inside
rocm/sgl-dev:v0.5.19-rocm720-mi35x-20260908on gfx950 (4× MI350X). CUDA graphs are left on (do not pass--disable-cuda-graph).(1) Install SGLang from PR #36866 and cherry-pick the MXFP4 SwiGLU plumbing fix
The MiniMax-M3 Quark W4A4 MoE path also needs sgl-project/sglang#34540 (
swiglu_limit+activation="swiglu"forwarding). Without that cherry-pick the model emits garbage at every concurrency and EAGLE3 accept length collapses to ~1.0.(2) Start the server
Wait until CUDA-graph capture finishes (
Capture target verify CUDA graph end/Engine startup timings).disable_cuda_graphmust beFalse. Healthy EAGLE3 accept length is ~3.2–3.5 (rate ~0.75–0.83), not ~1.0.(3) Run the GSM8K test
Expect Accuracy ≈ 0.969, Invalid = 0.000 at conc=32 with this config. Same command with
--parallel 64should stay in the mid-90s with 0% invalid.Testing
down_proj/gate_upfor EAGLE3-relevantM(including conc up to 128)gemm_afp4wfp4lookup picks this file byN,K)Documentation
Dependencies