Skip to content

[Triton/Gluon] [Config] [Bugfix] Add MiniMax-M3 MXFP4 shared-expert down_proj gemm_afp4wfp4 config - #5377

Closed
rkarhila-amd wants to merge 1 commit into
ROCm:mainfrom
rkarhila-amd:minimax-m3-mxfp4-shared-down-gemm-config
Closed

rkarhila-amd wants to merge 1 commit into
ROCm:mainfrom
rkarhila-amd:minimax-m3-mxfp4-shared-down-gemm-config

Conversation

@rkarhila-amd

@rkarhila-amd rkarhila-amd commented Sep 9, 2026 •

Copy link
Copy Markdown

Closing: this BK=512 / K=768 NaN bug is fixed in Triton 3.8.0. ROCm 10 SGLang images have moved to Triton 3.8.0 (verified in lmsysorg/sglang-rocm:v0.5.19-rocm10-mi35x-20260909: default gemm_afp4wfp4 for MiniMax-M3 shared down_proj N=6144,K=768 is finite and bit-exact vs BK=256). The exact-shape config is only needed on older Triton 3.7.0 containers (e.g. rocm/sgl-dev:v0.5.19-rocm720-mi35x-20260908).

Summary

Add an exact-shape gfx950 Triton config for gemm_afp4wfp4 at N=6144, K=768 so MiniMax-M3 MXFP4's non-fused shared-expert down_proj does not emit NaNs under EAGLE3 verify-batch sizes.

Motivation

amd/MiniMax-M3-MXFP4 (TP=4) uses a standalone shared-expert MLP. The W4A4 down_proj is M × 768 @ 768 × 6144 and dispatches aiter.ops.triton.gemm.basic.gemm_afp4wfp4. The default config for that shape uses BLOCK_SIZE_K=512 in the M_LEQ_128 and M_LEQ_256 buckets. Logical K=768 is then tiled as 512+256, and that K-tail path produces nondeterministic NaNs in the output (software W4A4 / A16W4 references stay finite and bit-exact).

This shows up in EAGLE3 target-verify when M ≈ concurrency × num_draft_tokens lands in (64, 256] (e.g. conc=32 with 4–5 draft tokens → M=128 or 160). Gate-up (K=6144) and routed MoE were not the culprit.

Changes

  • Add aiter/ops/triton/configs/gfx950/triton/gemm/gemm_afp4wfp4/GEMM-AFP4WFP4-N=6144-K=768.json.
  • For M_LEQ_128 and M_LEQ_256, use BM=128, BN=64, BK=256, NUM_KSPLIT=1 instead of the default BK=512.
  • Other M buckets are copied from the default table and were already finite on this shape.

No kernel code changes. Config-only.

Accuracy (before / after)

Model: amd/MiniMax-M3-MXFP4 + EAGLE3 draft Inferact/MiniMax-M3-EAGLE3, TP=4, gfx950 (MI350X), non-fused shared expert (--disable-shared-experts-fusion), --moe-runner-backend aiter, --attention-backend triton.

Workload: SGLang GSM8K bench_sglang.py, 64 questions, 5-shot, max_new_tokens=1024.

Setup conc Accuracy Invalid Notes
Before (default BK=512) 32 29.7% 28.1% 28/32 short generations were null/garbage; production down_proj NaNs
After (this config, BK=256) 32 96.9% 0.0% 0/32 null outputs; 80/80 replay runs finite and bit-exact vs software W4A4

Kernel replay on captured conc=32 verify activations (layer-3 shared down_proj):

  • Default BM=64, BN=64, BK=512: thousands of NaNs per run, nondeterministic across TP ranks.
  • Tuned BM=128, BN=64, BK=256: 0 NaNs, bit-exact, ~5% faster on that microbench (~138.9 µs → ~132.2 µs).

Triton version caveat

Tuning and end-to-end numbers were taken in the SGLang ROCm container rocm/sgl-dev:v0.5.19-rocm720-mi35x-20260908, which currently ships Triton 3.7.0 (/opt/venv/.../triton, torch 2.9.1+rocm7.2). It is unknown when the SGLang containers will move to a newer Triton. This config should be re-validated if/when that Triton pin changes; a later Triton may fix the BK=512 K-tail bug or want a different tile.

Reproduce Accuracy test

All steps below are inside rocm/sgl-dev:v0.5.19-rocm720-mi35x-20260908 on gfx950 (4× MI350X). CUDA graphs are left on (do not pass --disable-cuda-graph).

(1) Install SGLang from PR #36866 and cherry-pick the MXFP4 SwiGLU plumbing fix

The MiniMax-M3 Quark W4A4 MoE path also needs sgl-project/sglang#34540 (swiglu_limit + activation="swiglu" forwarding). Without that cherry-pick the model emits garbage at every concurrency and EAGLE3 accept length collapses to ~1.0.

# container already has AITER at /sgl-workspace/aiter
cd /sgl-workspace
git clone --depth 1 --branch amd/minimax-eagle-mxfp4-gfx950 \
  https://github.com/sgl-project/sglang.git /sglang-checkout
cd /sglang-checkout
git fetch --depth 50 origin amd/minimax-eagle-mxfp4-gfx950
# plumbing fix: https://github.com/sgl-project/sglang/pull/34540
git fetch https://github.com/sgl-project/sglang.git pull/34540/head
git cherry-pick --no-edit FETCH_HEAD
# If quark_w4a4_mxfp4_moe.py conflicts on AiterMoeQuantInfo, keep BOTH:
#   swiglu_limit=(self.moe_runner_config.gemm1_clamp_limit or self.moe_runner_config.swiglu_limit or 0.0)
#   fused_moe_kwargs=_fused_moe_kwargs
pip install -e /sglang-checkout/python --no-build-isolation --no-deps

# install this PR's GEMM config into the container AITER tree
cd /sgl-workspace/aiter
git fetch origin pull/5377/head
git checkout FETCH_HEAD -- \
  aiter/ops/triton/configs/gfx950/triton/gemm/gemm_afp4wfp4/GEMM-AFP4WFP4-N=6144-K=768.json

(2) Start the server

export CUDA_VISIBLE_DEVICES=0,1,2,3
export HF_HUB_CACHE=/hf-cache
export HF_HOME=/hf-cache
export PYTHONPATH=/sglang-checkout/python:${PYTHONPATH:-}
export SGLANG_USE_AITER=1
export SGLANG_SET_CPU_AFFINITY=0
export SGLANG_NUMA_BIND_V2=0
export SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1
export SGLANG_USE_AITER_MOE_GU_ITLV=0
export AITER_REBUILD=0
export PYTHONUNBUFFERED=1

python3 -m sglang.launch_server --disable-shared-experts-fusion \
  --model-path amd/MiniMax-M3-MXFP4 \
  --tp 4 \
  --trust-remote-code \
  --dtype bfloat16 \
  --quantization quark \
  --page-size 128 \
  --chunked-prefill-size 32768 \
  --mem-fraction-static 0.85 \
  --kv-cache-dtype fp8_e4m3 \
  --attention-backend triton \
  --moe-runner-backend aiter \
  --reasoning-parser minimax-m3 \
  --tool-call-parser minimax-m3 \
  --host 0.0.0.0 \
  --port 30047 \
  --watchdog-timeout 7200 \
  --disable-radix-cache \
  --disable-flashinfer-autotune \
  --log-level info \
  --speculative-draft-attention-backend triton \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path Inferact/MiniMax-M3-EAGLE3 \
  --speculative-draft-model-quantization unquant

Wait until CUDA-graph capture finishes (Capture target verify CUDA graph end / Engine startup timings). disable_cuda_graph must be False. Healthy EAGLE3 accept length is ~3.2–3.5 (rate ~0.75–0.83), not ~1.0.

(3) Run the GSM8K test

cd /sglang-checkout
export PYTHONPATH=/sglang-checkout/python HF_HUB_CACHE=/hf-cache HF_HOME=/hf-cache PYTHONUNBUFFERED=1
python3 benchmark/gsm8k/bench_sglang.py \
  --host 127.0.0.1 --port 30047 \
  --num-questions 64 --num-shots 5 \
  --parallel 32 --max-new-tokens 1024 \
  --enable-thinking \
  --tokenizer-path amd/MiniMax-M3-MXFP4 \
  --chat-template-kwargs '{"thinking_mode": "enabled"}'

Expect Accuracy ≈ 0.969, Invalid = 0.000 at conc=32 with this config. Same command with --parallel 64 should stay in the mid-90s with 0% invalid.

Testing

  • Software W4A4 / A16W4 reference replay on production activations (all four TP ranks)
  • M-bucket sweep of shared down_proj / gate_up for EAGLE3-relevant M (including conc up to 128)
  • GSM8K conc=32 before/after as in the table
  • Unit tests added/updated (config-only; existing gemm_afp4wfp4 lookup picks this file by N,K)
  • Tested on MI250X
  • Tested on MI300X
  • Tested on MI350X (gfx950)

Documentation

  • Docstrings updated
  • User guide updated

Dependencies

  • No new third-party dependencies added

Use BK=256 for M_LEQ_128 and M_LEQ_256 so EAGLE3 verify batches at conc 32-64 do not emit NaNs on the short-K W4A4 down projection.

Co-authored-by: Cursor <cursoragent@cursor.com>
@rkarhila-amd
rkarhila-amd requested a review from a team September 9, 2026 16:19
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
multigpu Aiter multi-GPU tests on the 8-GPU runner
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 5377 --add-label <label>

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to opt this PR out.

@github-actions github-actions Bot changed the title [Bugfix] Add MiniMax-M3 MXFP4 shared-expert down_proj gemm_afp4wfp4 config [Triton/Gluon] [Config] [Bugfix] Add MiniMax-M3 MXFP4 shared-expert down_proj gemm_afp4wfp4 config Sep 9, 2026
@zufayu
zufayu requested review from a team and azaidy September 10, 2026 02:38
@Boss2002n
Boss2002n requested a balanced review from Copilot September 10, 2026 03:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The affected exact shape and M buckets need automated regression coverage for finite and reference-matching output.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a gfx950 Triton configuration to avoid NaNs for MiniMax-M3’s MXFP4 shared-expert down projection.

Changes:

  • Adds exact N=6144, K=768 dispatch.
  • Uses BLOCK_SIZE_K=256 for M≤128 and M≤256.
  • Preserves default settings for other M buckets.
File summaries
File Description
aiter/ops/triton/configs/gfx950/triton/gemm/gemm_afp4wfp4/GEMM-AFP4WFP4-N=6144-K=768.json Defines the shape-specific gfx950 tuning configuration.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +50 to +53
"M_LEQ_128": {
"BLOCK_SIZE_M": 128,
"BLOCK_SIZE_N": 64,
"BLOCK_SIZE_K": 256,
@rkarhila-amd

Copy link
Copy Markdown
Author

Closing: the gemm_afp4wfp4 NaNs for MiniMax-M3 shared-expert down_proj (N=6144, K=768, default BK=512) are fixed in Triton 3.8.0. ROCm 10 SGLang images have moved to Triton 3.8.0 (lmsysorg/sglang-rocm:v0.5.19-rocm10-mi35x-20260909); a kernel replay there showed 0 NaNs and bit-exact match vs BK=256. This config remains a workaround only for Triton 3.7.0 containers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants