Skip to content

[Fix] [AMD] quark: do not declare the MiniMax-M3 TP8 MoE pad to CK-Tile - #49286

Open
tnguyeng wants to merge 1 commit into
sgl-project:mainfrom
tnguyeng:amd/fix-minimax-m3-tp8-cktile-pad
Open

tnguyeng wants to merge 1 commit into
sgl-project:mainfrom
tnguyeng:amd/fix-minimax-m3-tp8-cktile-pad

Conversation

@tnguyeng

@tnguyeng tnguyeng commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

MiniMax-M3 on TP8 falls into repetitive loops in normal mode (AIME25 0/30). Answers start fine, then repeat one line until the 32,768-token cap.

Modifications

  • python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py: when aiter_swiglu_oai_limit is set (gfx95, alpha 1.702, and a clamp, which is MiniMax-M3), pass intermediate_pad=0. Every other config still forwards layer.intermediate_pad.
  • test/registered/unit/layers/quantization/test_quark_config.py: DeepSeek-V4-Pro style SiLU with a clamp still forwards 128. The MiniMax SwiGLU-OAI config forwards 0.
  • Unchanged: the 384-to-512 weight pad and its loader, SGLANG_AITER_K3_OPT, deterministic mode, and every non-SwiGLU-OAI caller of this scheme (including DeepSeek-V4-Pro TP8, which also pads 384 to 512).

Accuracy Tests

Environment: 8x MI355X (gfx950), lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261105, SGLang f3532057e1, AITER 285579b62. TP 8, normal mode, SGLANG_AITER_K3_OPT unset, --attention-backend aiter. The candidate mounted intermediate_pad=0 into that image. MiniMax-M3 is the SwiGLU-OAI path this PR gates on, so those runs are this change. The control is the same image with the stock file.

AIME25, temperature 0.6, top-p 0.95, 32,768 tokens, one seed each:

Run Seed 0 Seed 1
Stock 0/30, 30/30 truncated not run
This change 24/30 (80%) 26/30 (86.67%)

GSM8K test split, all 1,319 questions, greedy, 2,048-token cap:

Run Correct Truncated at 2,048
Stock 372/1319 (28.20%) 1165/1319
This change 1264/1319 (95.83%) 12/1319

Unit test, on that image with this branch's tree:

python3 -m pytest test/registered/unit/layers/quantization/test_quark_config.py::TestQuarkMxfp4MoEAiterQuantInfo -q

2 passed.

Problems 1, 13, 14 and 27 still run to 32,768 tokens on AIME25. That tail is separate from the all-30 loop.

Speed Tests and Profiling

Same image and server as above. 512 forced output tokens, short prompt, ignore_eos. Output tokens per second:

Concurrency Stock This change SGLANG_AITER_K3_OPT=1
1 203 202 149
8 1189 987 1034
32 3559 3496 3314
64 6008 5913 5909
128 8323 9609 8419

One request matches stock. The K3 flag also stops the loop by not padding, and it costs about 27% at one request. Concurrency 8 on the candidate had a 0.78 s time to first token against about 0.16 s on the other points, so that one cell is a noisy sample. From 16 up the candidate stays with stock. No profiler trace in this change.

Checklist

  • Format: pre-commit (isort, ruff, ruff-format) passed on both files
  • Unit tests: TestQuarkMxfp4MoEAiterQuantInfo, 2 passed
  • Documentation: no new flag and no recipe change, so no doc edit
  • Accuracy: AIME25 and full GSM8K, stock against this change, table above
  • Speed: concurrency sweep, table above
  • Code style: comments state the constraint only

CI States

Latest PR Test (Base): ❌ Run #42604138360
Latest PR Test (Extra): ❌ Run #42604138341
Latest PR Test (AMD ROCm 10): ❌ Run #42604138434

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

CK-Tile SwiGLU stage 1 returns wrong decode output when told about the
zero-filled 384-to-512 pad. Passing intermediate_pad 0 on that path keeps
the padded weights and restores normal-mode accuracy at the stock speed.
@tnguyeng
tnguyeng force-pushed the amd/fix-minimax-m3-tp8-cktile-pad branch from e5a9419 to 0ba8db3 Compare November 9, 2026 10:50
@tnguyeng
tnguyeng marked this pull request as ready for review November 9, 2026 11:15

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@tnguyeng

tnguyeng commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/tag-and-rerun-ci

@chrikrah chrikrah left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tnguyeng From my perspective this can be merged. It narrows #38546's pad forwarding to SwiGLU-OAI only and still forwards your clamp from #34540. Dropping the declared pad there is exact: a zero gate column gives SwiGLU-OAI zero, whatever beta adds to up, and w2's pad rows are zero too.

# head 0ba8db3 (base e60f7e9), Python 3.12, macOS CPU, torch from PyPI, sglang from the exported tree
$ pytest -q test/registered/unit/layers/quantization/test_quark_config.py
59 passed, 34 subtests passed
# quark_w4a4_mxfp4_moe.py from e60f7e9, same tests
FAILED test_apply_drops_padding_for_swiglu_oai   (128 != 0)

mxfp4.py:1817 forwards intermediate_pad with a SwiGLU-OAI clamp for gpt-oss too. Does that path hit the same kernel, or only the separated gate/up layout?

@aarnetalman

Copy link
Copy Markdown
Contributor

Tested on 8x MI350X: amd/MiniMax-M3-MXFP4 at TP8, rocm/sgl-dev:v0.5.23-rocm724-mi35x-20261106 with this PR's quark_w4a4_mxfp4_moe.py change applied.

Accuracy: fixed. AIME25 (sgl-eval, temperature 0.6, top-p 0.95, max 32k tokens, thinking on), 16 seeds:

AIME25
TP8, without this PR 0.00% on every seed (every answer hits the token limit repeating one line)
TP8, with this PR 77.71% (seeds 0-7 77.50%, 8-15 77.92%)
TP4, without this PR, for reference 80.21%

--enable-deterministic-inference: the change also applies to the SwiGLU path that deterministic mode's MoE goes through, so I checked TP8 deterministic mode with it applied (with our open deterministic-MoE changes for MiniMax-M3, which deterministic mode on ROCm still needs, including ROCm/aiter#6583). Every exact check still passes: test_deterministic single / prefix / prefix with logprobs / radix cache, and a bitwise batch-size check (1,275 / 1,275 identical tokens and logprobs). GSM8K is 95.30%.

Speed (sglang.benchmark.serving, random dataset, normal mode; at concurrency 32, three rounds per setup in the order without / with / with / without; at concurrency 1, one round):

Without this PR With this PR Change
1k in / 1k out, concurrency 1 148.3 tok/s 147.7 tok/s -0.4%
1k in / 1k out, concurrency 32 2,996 tok/s (ITL p50 10.05 ms) 2,782 tok/s (ITL p50 10.88 ms) -7.1%
8k in / 1k out, concurrency 32 2,079 tok/s 1,992 tok/s -4.2%

The concurrency-32 rounds agree to within 0.4% per setup. Deterministic mode shows the same pattern in one round (-5.6% / -3.3%).

So single-request speed is unchanged, but there is a 4-7% throughput cost under load. A likely cause is that CK-Tile stage 1 now computes the zero-padded rows that it used to skip. It may be worth mentioning in the description, with a fix for the declared pad in stage 1 as a possible follow-up.

@tnguyeng

Copy link
Copy Markdown
Contributor Author

@tnguyeng From my perspective this can be merged. It narrows #38546's pad forwarding to SwiGLU-OAI only and still forwards your clamp from #34540. Dropping the declared pad there is exact: a zero gate column gives SwiGLU-OAI zero, whatever beta adds to up, and w2's pad rows are zero too.

# head 0ba8db3 (base e60f7e9), Python 3.12, macOS CPU, torch from PyPI, sglang from the exported tree
$ pytest -q test/registered/unit/layers/quantization/test_quark_config.py
59 passed, 34 subtests passed
# quark_w4a4_mxfp4_moe.py from e60f7e9, same tests
FAILED test_apply_drops_padding_for_swiglu_oai   (128 != 0)

mxfp4.py:1817 forwards intermediate_pad with a SwiGLU-OAI clamp for gpt-oss too. Does that path hit the same kernel, or only the separated gate/up layout?

Thanks @chrikrah for the review. The gpt-oss call in mxfp4.py:1817 does not hit the kernel this PR fixes. By default it uses the interleaved gate/up layout, and the separated layout is used only when SGLANG_USE_AITER_MOE_GU_ITLV=0

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants