Repository navigation
[AMD] Fix Quark W4A4 MXFP4 MoE clamped-SwiGLU on the AITER path - #34540
Conversation
|
/tag-and-rerun-ci |
|
/tag-run-ci-label |
Mirror PR sgl-project#34540 for bf16 unquant: rewrite silu+gemm1_clamp_limit to swiglu activation and pass swiglu_limit in AiterMoeQuantInfo so AITER MoE matches Triton accuracy with CUDA graphs enabled. Co-authored-by: Cursor <cursoragent@cursor.com>
Port PR sgl-project#34540 clamped-SwiGLU plumbing into Quark W4A4 MXFP4 MoE, mirror the AITER swiglu_limit path in Fp8 MoE, and teach Quark loading that MiniMax index_qkv_proj is q|k-only when index value is disabled. Co-authored-by: Cursor <cursoragent@cursor.com>
Clamped-SwiGLU plumbing for Quark MXFP4 MoE belongs in tnguyeng's PR sgl-project#34540; drop the duplicated quark_w4a4_mxfp4_moe.py hunks and unit test from this gfx950 branch. MXFP8 fp8.py and loader.py fixes remain. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Rebased onto current Two changes relative to the original commit:
I don't have push access to the head branch, so the rebased commit is below as a patch instead of a branch update. It applies cleanly on
0001-AMD-Fix-Quark-W4A4-MXFP4-MoE-clamped-SwiGLU-on-the-A.patchFrom 66c2125850ea2586a1870e8b71125787c86487d9 Mon Sep 17 00:00:00 2001
From: Tuan Nguyen Gia <tnguyeng@amd.com>
Date: Wed, 12 Aug 2026 01:46:03 -0500
Subject: [PATCH] [AMD] Fix Quark W4A4 MXFP4 MoE clamped-SwiGLU on the AITER
path
MiniMax-M3 (and other GPT-OSS-style clamped-SwiGLU MoEs) declare
activation="silu" together with gemm1_alpha/gemm1_clamp_limit and rely on
the AITER MoE runner to run the *clamped* SwiGLU kernel. The runner only
does so when quant_info.swiglu_limit > 0, and it selects the FP4 kernel /
q_dtype_a on activation == Swiglu.
Mxfp4MoEMethod (gpt-oss) already forwards both, but the Quark W4A4 MXFP4
MoE scheme used by MiniMax-M3 built AiterMoeQuantInfo without swiglu_limit
and left activation="silu". As a result the MXFP4 experts silently ran
plain SiLU on the non-clamped branch and produced garbage on every MoE
layer, regardless of attention/MoE backend or aiter version.
Mirror Mxfp4MoEMethod in QuarkW4A4MXFp4MoE:
- create_moe_runner: translate activation to "swiglu" for the AITER runner
when a clamp limit is set (gemm1_clamp_limit/swiglu_limit > 0).
- apply_weights: forward swiglu_limit (gemm1_clamp_limit or swiglu_limit)
onto AiterMoeQuantInfo.
Both changes are gated on a positive clamp limit, so plain-SwiGLU Quark
MXFP4 models (limit unset -> 0.0) keep activation="silu" and
swiglu_limit=0.0 and are byte-for-byte unchanged.
Add CPU-only unit tests covering the clamped vs non-clamped activation
translation and swiglu_limit forwarding.
Co-authored-by: Cursor <cursoragent@cursor.com>
---
.../quark/schemes/quark_w4a4_mxfp4_moe.py | 35 +++-
.../quantization/test_quark_w4a4_mxfp4_moe.py | 169 ++++++++++++++++++
2 files changed, 203 insertions(+), 1 deletion(-)
create mode 100644 test/registered/unit/layers/quantization/test_quark_w4a4_mxfp4_moe.py
diff --git a/python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py b/python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py
index 12243eb5c..4873d002e 100644
--- a/python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py
+++ b/python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py
@@ -4,6 +4,7 @@ from __future__ import annotations
import logging
import threading
+from dataclasses import replace
from typing import TYPE_CHECKING, Any
import torch
@@ -894,7 +895,24 @@ class QuarkW4A4MXFp4MoE(QuarkMoEScheme):
moe_runner_backend = MoeRunnerBackend.AITER
if moe_runner_backend.is_aiter():
- self.runner = MoeRunner(moe_runner_backend, moe_runner_config)
+ # The AITER MXFP4 per-1x32 kernel keys its q_dtype_a / kernel
+ # selection on activation == Swiglu. Clamped-SwiGLU checkpoints
+ # (e.g. MiniMax-M3) declare activation="silu" together with a
+ # gemm1_clamp_limit, so translate it to "swiglu" here, mirroring
+ # Mxfp4MoEMethod (gpt-oss). Without this the runner quantizes
+ # activations on the plain-SiLU branch and produces garbage.
+ # Plain-SwiGLU Quark MXFP4 models (no clamp) keep activation="silu"
+ # so their existing path is untouched. K3's SiTU is the one
+ # activation the kernel selects on its own -- leave it alone.
+ aiter_config = moe_runner_config
+ swiglu_limit = (
+ moe_runner_config.gemm1_clamp_limit
+ or moe_runner_config.swiglu_limit
+ or 0.0
+ )
+ if swiglu_limit > 0 and moe_runner_config.activation != "situ":
+ aiter_config = replace(moe_runner_config, activation="swiglu")
+ self.runner = MoeRunner(moe_runner_backend, aiter_config)
else:
# TODO(cwan): refactor other backends
pass
@@ -935,5 +953,20 @@ class QuarkW4A4MXFp4MoE(QuarkMoEScheme):
w2_scale=layer.w2_weight_scale,
expert_mask=layer.dispatcher.expert_mask_gpu,
fused_moe_kwargs=_fused_moe_kwargs,
+ # Forward the GPT-OSS-style clamped-SwiGLU limit so the AITER MoE
+ # runner enters the clamped-SwiGLU path (gate_mode + swiglu_limit)
+ # instead of plain SiLU. Mirrors Mxfp4MoEMethod; checkpoints without
+ # a clamp resolve to 0.0 and keep the previous behavior. K3 SiTU
+ # uses gemm1_clamp_limit as linear_beta, which the runner forwards
+ # on its own branch and must not be read as a swiglu limit.
+ swiglu_limit=(
+ 0.0
+ if self.moe_runner_config.activation == "situ"
+ else (
+ self.moe_runner_config.gemm1_clamp_limit
+ or self.moe_runner_config.swiglu_limit
+ or 0.0
+ )
+ ),
)
return self.runner.run(dispatch_output, quant_info)
diff --git a/test/registered/unit/layers/quantization/test_quark_w4a4_mxfp4_moe.py b/test/registered/unit/layers/quantization/test_quark_w4a4_mxfp4_moe.py
new file mode 100644
index 000000000..0890d27a0
--- /dev/null
+++ b/test/registered/unit/layers/quantization/test_quark_w4a4_mxfp4_moe.py
@@ -0,0 +1,169 @@
+"""Unit tests for the Quark W4A4 MXFP4 MoE clamped-SwiGLU plumbing.
+
+CPU-only, no model loading and no aiter/ROCm dependency: they exercise the
+pure-Python control flow in ``QuarkW4A4MXFp4MoE.create_moe_runner`` and
+``apply_weights`` that decides whether the AITER MoE runner is driven on the
+clamped-SwiGLU path (``activation="swiglu"`` + ``swiglu_limit``) or the plain
+SiLU path. See the regression these guard against: clamped-SwiGLU MXFP4
+checkpoints (e.g. MiniMax-M3) declare ``activation="silu"`` + a
+``gemm1_clamp_limit``, and without this plumbing the experts silently run plain
+SiLU and emit garbage.
+"""
+
+import sys
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+import sglang.srt.layers.moe.utils as moe_utils
+import sglang.srt.layers.quantization.quark.schemes.quark_w4a4_mxfp4_moe as quark_moe
+from sglang.srt.layers.moe import MoeRunnerBackend, MoeRunnerConfig
+from sglang.srt.layers.quantization.quark.schemes.quark_w4a4_mxfp4_moe import (
+ QuarkW4A4MXFp4MoE,
+)
+from sglang.test.ci.ci_register import register_cpu_ci
+
+register_cpu_ci(est_time=5, suite="base-a-test-cpu")
+
+
+def _make_scheme() -> QuarkW4A4MXFp4MoE:
+ # is_checkpoint_mxfp4_serialized=True skips the gfx95 hardware check.
+ return QuarkW4A4MXFp4MoE(
+ weight_config={"qscheme": "per_group"},
+ input_config={"qscheme": "per_group", "is_dynamic": True},
+ )
+
+
+def _force_aiter_runner(monkeypatch):
+ """Make ``create_moe_runner`` take the AITER branch and capture its config."""
+ captured = {}
+
+ def fake_moe_runner(backend, config):
+ captured["backend"] = backend
+ captured["config"] = config
+ return object()
+
+ monkeypatch.setattr(quark_moe, "MoeRunner", fake_moe_runner)
+ monkeypatch.setattr(
+ moe_utils, "get_moe_runner_backend", lambda: MoeRunnerBackend.AITER
+ )
+ return captured
+
+
+def _fake_layer():
+ # dtype uint8 so we never hit the float4_e2m1fn_x2 view branch (also guarded
+ # by delattr in the apply_weights tests below).
+ return SimpleNamespace(
+ w13_weight=torch.zeros(1, dtype=torch.uint8),
+ w2_weight=torch.zeros(1, dtype=torch.uint8),
+ w13_weight_scale=torch.zeros(1, dtype=torch.uint8),
+ w2_weight_scale=torch.zeros(1, dtype=torch.uint8),
+ dispatcher=SimpleNamespace(expert_mask_gpu=torch.zeros(1)),
+ )
+
+
+def _capturing_runner(store):
+ class _Runner:
+ def run(self, dispatch_output, quant_info):
+ store["quant_info"] = quant_info
+ return "ran"
+
+ return _Runner()
+
+
+def test_create_moe_runner_translates_activation_to_swiglu_when_clamped(monkeypatch):
+ captured = _force_aiter_runner(monkeypatch)
+ scheme = _make_scheme()
+
+ cfg = MoeRunnerConfig(activation="silu", gemm1_clamp_limit=7.0)
+ scheme.create_moe_runner(layer=None, moe_runner_config=cfg)
+
+ assert captured["config"].activation == "swiglu"
+ # The original config is left untouched (dataclasses.replace returns a copy).
+ assert cfg.activation == "silu"
+
+
+def test_create_moe_runner_keeps_silu_without_clamp(monkeypatch):
+ captured = _force_aiter_runner(monkeypatch)
+ scheme = _make_scheme()
+
+ cfg = MoeRunnerConfig(activation="silu") # no clamp
+ scheme.create_moe_runner(layer=None, moe_runner_config=cfg)
+
+ assert captured["config"].activation == "silu"
+ assert captured["config"] is cfg
+
+
+def test_create_moe_runner_leaves_situ_activation_alone(monkeypatch):
+ captured = _force_aiter_runner(monkeypatch)
+ scheme = _make_scheme()
+
+ # K3-style: activation="situ" with gemm1_clamp_limit carrying linear_beta,
+ # not a SwiGLU clamp. The AITER kernel selects SiTU itself.
+ cfg = MoeRunnerConfig(activation="situ", gemm1_clamp_limit=7.0)
+ scheme.create_moe_runner(layer=None, moe_runner_config=cfg)
+
+ assert captured["config"].activation == "situ"
+ assert captured["config"] is cfg
+
+
+def test_apply_weights_forwards_swiglu_limit_when_clamped(monkeypatch):
+ monkeypatch.delattr(torch, "float4_e2m1fn_x2", raising=False)
+ scheme = _make_scheme()
+ scheme.moe_runner_config = MoeRunnerConfig(
+ activation="swiglu", gemm1_clamp_limit=7.0
+ )
+ store = {}
+ scheme.runner = _capturing_runner(store)
+
+ result = scheme.apply_weights(_fake_layer(), dispatch_output=object())
+
+ assert result == "ran"
+ assert store["quant_info"].swiglu_limit == 7.0
+
+
+def test_apply_weights_defaults_swiglu_limit_zero_without_clamp(monkeypatch):
+ monkeypatch.delattr(torch, "float4_e2m1fn_x2", raising=False)
+ scheme = _make_scheme()
+ scheme.moe_runner_config = MoeRunnerConfig(activation="silu") # no clamp
+ store = {}
+ scheme.runner = _capturing_runner(store)
+
+ scheme.apply_weights(_fake_layer(), dispatch_output=object())
+
+ assert store["quant_info"].swiglu_limit == 0.0
+
+
+def test_apply_weights_zeroes_swiglu_limit_for_situ(monkeypatch):
+ monkeypatch.delattr(torch, "float4_e2m1fn_x2", raising=False)
+ scheme = _make_scheme()
+ scheme.moe_runner_config = MoeRunnerConfig(activation="situ", gemm1_clamp_limit=7.0)
+ store = {}
+ scheme.runner = _capturing_runner(store)
+
+ scheme.apply_weights(_fake_layer(), dispatch_output=object())
+
+ # Forwarding the limit here would make the runner clamp a SiTU linear_beta.
+ assert store["quant_info"].swiglu_limit == 0.0
+
+
+def test_apply_weights_leaves_fused_moe_kwargs_unset_off_gfx1250(monkeypatch):
+ monkeypatch.delattr(torch, "float4_e2m1fn_x2", raising=False)
+ # Pinned rather than read from the host: the gfx1250 branch imports aiter.
+ monkeypatch.setattr(quark_moe, "_is_gfx1250", False)
+ scheme = _make_scheme()
+ scheme.moe_runner_config = MoeRunnerConfig(
+ activation="swiglu", gemm1_clamp_limit=7.0
+ )
+ store = {}
+ scheme.runner = _capturing_runner(store)
+
+ scheme.apply_weights(_fake_layer(), dispatch_output=object())
+
+ assert store["quant_info"].fused_moe_kwargs is None
+ assert store["quant_info"].swiglu_limit == 7.0
+
+
+if __name__ == "__main__":
+ sys.exit(pytest.main([__file__, "-v"]))
--
2.53.0.windows.2 |
MiniMax-M3 (and other GPT-OSS-style clamped-SwiGLU MoEs) declare activation="silu" together with gemm1_alpha/gemm1_clamp_limit and rely on the AITER MoE runner to run the *clamped* SwiGLU kernel. The runner only does so when quant_info.swiglu_limit > 0, and it selects the FP4 kernel / q_dtype_a on activation == Swiglu. Mxfp4MoEMethod (gpt-oss) already forwards both, but the Quark W4A4 MXFP4 MoE scheme used by MiniMax-M3 built AiterMoeQuantInfo without swiglu_limit and left activation="silu". As a result the MXFP4 experts silently ran plain SiLU on the non-clamped branch and produced garbage on every MoE layer, regardless of attention/MoE backend or aiter version. Mirror Mxfp4MoEMethod in QuarkW4A4MXFp4MoE: - create_moe_runner: translate activation to "swiglu" for the AITER runner when a clamp limit is set (gemm1_clamp_limit/swiglu_limit > 0). - apply_weights: forward swiglu_limit (gemm1_clamp_limit or swiglu_limit) onto AiterMoeQuantInfo. Both changes are gated on a positive clamp limit, so plain-SwiGLU Quark MXFP4 models (limit unset -> 0.0) keep activation="silu" and swiglu_limit=0.0 and are byte-for-byte unchanged. Add CPU-only unit tests covering the clamped vs non-clamped activation translation and swiglu_limit forwarding. Co-authored-by: Cursor <cursoragent@cursor.com>
The scheme no longer rewrites silu to swiglu, so the tests check that combo and the forwarded swiglu_limit.
2f134f7 to
bdb56d2
Compare
Mirror PR sgl-project#34540 for bf16 unquant: rewrite silu+gemm1_clamp_limit to swiglu activation and pass swiglu_limit in AiterMoeQuantInfo so AITER MoE matches Triton accuracy with CUDA graphs enabled. Co-authored-by: Cursor <cursoragent@cursor.com>
Port PR sgl-project#34540 clamped-SwiGLU plumbing into Quark W4A4 MXFP4 MoE, mirror the AITER swiglu_limit path in Fp8 MoE, and teach Quark loading that MiniMax index_qkv_proj is q|k-only when index value is disabled. Co-authored-by: Cursor <cursoragent@cursor.com>
Clamped-SwiGLU plumbing for Quark MXFP4 MoE belongs in tnguyeng's PR test from this gfx950 branch. MXFP8 fp8.py and loader.py fixes remain. Co-authored-by: Cursor <cursoragent@cursor.com>
Mirror PR sgl-project#34540 for bf16 unquant: rewrite silu+gemm1_clamp_limit to swiglu activation and pass swiglu_limit in AiterMoeQuantInfo so AITER MoE matches Triton accuracy with CUDA graphs enabled. Co-authored-by: Cursor <cursoragent@cursor.com>
Port PR sgl-project#34540 clamped-SwiGLU plumbing into Quark W4A4 MXFP4 MoE, mirror the AITER swiglu_limit path in Fp8 MoE, and teach Quark loading that MiniMax index_qkv_proj is q|k-only when index value is disabled. Co-authored-by: Cursor <cursoragent@cursor.com>
Clamped-SwiGLU plumbing for Quark MXFP4 MoE belongs in tnguyeng's PR test from this gfx950 branch. MXFP8 fp8.py and loader.py fixes remain. Co-authored-by: Cursor <cursoragent@cursor.com>
Mirror PR sgl-project#34540 for bf16 unquant: rewrite silu+gemm1_clamp_limit to swiglu activation and pass swiglu_limit in AiterMoeQuantInfo so AITER MoE matches Triton accuracy with CUDA graphs enabled. Co-authored-by: Cursor <cursoragent@cursor.com>
Port PR sgl-project#34540 clamped-SwiGLU plumbing into Quark W4A4 MXFP4 MoE, mirror the AITER swiglu_limit path in Fp8 MoE, and teach Quark loading that MiniMax index_qkv_proj is q|k-only when index value is disabled. Co-authored-by: Cursor <cursoragent@cursor.com>
Clamped-SwiGLU plumbing for Quark MXFP4 MoE belongs in tnguyeng's PR test from this gfx950 branch. MXFP8 fp8.py and loader.py fixes remain. Co-authored-by: Cursor <cursoragent@cursor.com>
Accuracy on MiniMax-M3 MXFP4 (MI355X, TP4)Matched A/B on the same image, flags, GPUs, and port. The only change is the mounted source: this PR (
Server
Eval
|
|
/tag-and-rerun-ci |
…project#34540) Co-authored-by: Cursor <cursoragent@cursor.com>
Motivation
This PR fixes output quality for
amd/MiniMax-M3-MXFP4.The Quark W4A4 MXFP4 scheme omits
swiglu_limitwhen buildingAiterMoeQuantInfo, so MiniMax-M3's clamped-SiLU MoE (gemm1_alpha=1.702,gemm1_clamp_limit=7.0) never gets clamped in the AITER runner (layers/moe/moe_runner/aiter.py), producing garbage output.Modifications
python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py(mirrorsMxfp4MoEMethod):create_moe_runner: for the AITER backend, translateactivationto"swiglu"viadataclasses.replacewhen a clamp limit is set(
gemm1_clamp_limit/swiglu_limit > 0).apply_weights: forwardswiglu_limit = gemm1_clamp_limit or swiglu_limit or 0.0ontoAiterMoeQuantInfo.Both are gated on a positive clamp limit, so plain-SwiGLU Quark MXFP4 models (limit unset → 0.0) keep
activation="silu"andswiglu_limit=0.0and are unchanged.Runtime note: to match MiniMax-M3's
interleaved=Falsegate/up layout, launchwith
SGLANG_USE_AITER_MOE_GU_ITLV=0(SEPARATED gate_mode). This env isalready handled upstream in the AITER runner; this PR only makes the runner
reach that path.
Accuracy Tests
amd/MiniMax-M3-MXFP4, TP4, MI350x/MI355x.
BF16 baseline.
GSM8k test:
{
"gsm8k": {
"name": "gsm8k",
"alias": "gsm8k",
"sample_len": 1319,
"exact_match,strict-match": 0.9507202426080363,
"exact_match_stderr,strict-match": 0.00596215065581247,
"exact_match,flexible-extract": 0.9613343442001516,
"exact_match_stderr,flexible-extract": 0.005310583162098061
}
}
evalscope:10 times AIME25
{
"aime25/MiniMax-M3-MXFP4@aime25": 0.8367,
"aime25/mean_acc": 0.8367,
"aime25/default": 0.8367
}
Benchmarking and Profiling
The changed branch is taken only for clamped-SwiGLU checkpoints, and plain-SwiGLU Quark MXFP4 models run the
identical code path as before (
activation="silu",swiglu_limit=0.0), sothere is no hot-path change. Serving throughput of
amd/MiniMax-M3-MXFP4withthe fix applied (
SGLANG_USE_AITER,SGLANG_USE_AITER_MOE_GU_ITLV=0), TP4 onMI350x:
Config:
generated-shared-prefix, concurrency 4 (unbounded request rate),1024 in / 1024 out, 32 requests, 208 s.
Unit tests
test/registered/unit/layers/quantization/test_quark_w4a4_mxfp4_moe.py(CPU-only): asserts the activation translation happens iff a clamp limit is
set, and that
swiglu_limitis forwarded (clamped) / stays 0.0 (unclamped).Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ✅ Run #37529778610
Latest PR Test (Extra): ❌ Run #37529778027
Latest PR Test (AMD ROCm 10): ❌ Run #37529778873