Repository navigation
Conversation
CK-Tile SwiGLU stage 1 returns wrong decode output when told about the zero-filled 384-to-512 pad. Passing intermediate_pad 0 on that path keeps the padded weights and restores normal-mode accuracy at the stock speed.
e5a9419 to
0ba8db3
Compare
|
/tag-and-rerun-ci |
chrikrah
left a comment
There was a problem hiding this comment.
@tnguyeng From my perspective this can be merged. It narrows #38546's pad forwarding to SwiGLU-OAI only and still forwards your clamp from #34540. Dropping the declared pad there is exact: a zero gate column gives SwiGLU-OAI zero, whatever beta adds to up, and w2's pad rows are zero too.
# head 0ba8db3 (base e60f7e9), Python 3.12, macOS CPU, torch from PyPI, sglang from the exported tree
$ pytest -q test/registered/unit/layers/quantization/test_quark_config.py
59 passed, 34 subtests passed
# quark_w4a4_mxfp4_moe.py from e60f7e9, same tests
FAILED test_apply_drops_padding_for_swiglu_oai (128 != 0)
mxfp4.py:1817 forwards intermediate_pad with a SwiGLU-OAI clamp for gpt-oss too. Does that path hit the same kernel, or only the separated gate/up layout?
|
Tested on 8x MI350X: Accuracy: fixed. AIME25 (sgl-eval, temperature 0.6, top-p 0.95, max 32k tokens, thinking on), 16 seeds:
Speed (
The concurrency-32 rounds agree to within 0.4% per setup. Deterministic mode shows the same pattern in one round (-5.6% / -3.3%). So single-request speed is unchanged, but there is a 4-7% throughput cost under load. A likely cause is that CK-Tile stage 1 now computes the zero-padded rows that it used to skip. It may be worth mentioning in the description, with a fix for the declared pad in stage 1 as a possible follow-up. |
Thanks @chrikrah for the review. The gpt-oss call in |
Motivation
MiniMax-M3 on TP8 falls into repetitive loops in normal mode (AIME25 0/30). Answers start fine, then repeat one line until the 32,768-token cap.
intermediate_pad=0and keeps the padded weights.Modifications
python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py: whenaiter_swiglu_oai_limitis set (gfx95, alpha 1.702, and a clamp, which is MiniMax-M3), passintermediate_pad=0. Every other config still forwardslayer.intermediate_pad.test/registered/unit/layers/quantization/test_quark_config.py: DeepSeek-V4-Pro style SiLU with a clamp still forwards 128. The MiniMax SwiGLU-OAI config forwards 0.SGLANG_AITER_K3_OPT, deterministic mode, and every non-SwiGLU-OAI caller of this scheme (including DeepSeek-V4-Pro TP8, which also pads 384 to 512).Accuracy Tests
Environment: 8x MI355X (gfx950),
lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261105, SGLangf3532057e1, AITER285579b62. TP 8, normal mode,SGLANG_AITER_K3_OPTunset,--attention-backend aiter. The candidate mountedintermediate_pad=0into that image. MiniMax-M3 is the SwiGLU-OAI path this PR gates on, so those runs are this change. The control is the same image with the stock file.AIME25, temperature 0.6, top-p 0.95, 32,768 tokens, one seed each:
GSM8K test split, all 1,319 questions, greedy, 2,048-token cap:
Unit test, on that image with this branch's tree:
2 passed.
Problems 1, 13, 14 and 27 still run to 32,768 tokens on AIME25. That tail is separate from the all-30 loop.
Speed Tests and Profiling
Same image and server as above. 512 forced output tokens, short prompt,
ignore_eos. Output tokens per second:SGLANG_AITER_K3_OPT=1One request matches stock. The K3 flag also stops the loop by not padding, and it costs about 27% at one request. Concurrency 8 on the candidate had a 0.78 s time to first token against about 0.16 s on the other points, so that one cell is a noisy sample. From 16 up the candidate stays with stock. No profiler trace in this change.
Checklist
TestQuarkMxfp4MoEAiterQuantInfo, 2 passedCI States
Latest PR Test (Base): ❌ Run #42604138360
Latest PR Test (Extra): ❌ Run #42604138341
Latest PR Test (AMD ROCm 10): ❌ Run #42604138434
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: