Skip to content

[AMD] Fix Quark W4A4 MXFP4 MoE clamped-SwiGLU on the AITER path - #34540

Merged
BowenBao merged 3 commits into
sgl-project:mainfrom
tnguyeng:amd/fix-quark-mxfp4-moe-clamped-swiglu
Oct 7, 2026
Merged

BowenBao merged 3 commits into
sgl-project:mainfrom
tnguyeng:amd/fix-quark-mxfp4-moe-clamped-swiglu

Conversation

@tnguyeng

@tnguyeng tnguyeng commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

This PR fixes output quality for amd/MiniMax-M3-MXFP4.

The Quark W4A4 MXFP4 scheme omits swiglu_limit when building AiterMoeQuantInfo, so MiniMax-M3's clamped-SiLU MoE (gemm1_alpha=1.702, gemm1_clamp_limit=7.0) never gets clamped in the AITER runner (layers/moe/moe_runner/aiter.py), producing garbage output.

Modifications

python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py (mirrors Mxfp4MoEMethod):

  • create_moe_runner: for the AITER backend, translate activation to "swiglu" via dataclasses.replace when a clamp limit is set
    (gemm1_clamp_limit/swiglu_limit > 0).
  • apply_weights: forward swiglu_limit = gemm1_clamp_limit or swiglu_limit or 0.0 onto AiterMoeQuantInfo.

Both are gated on a positive clamp limit, so plain-SwiGLU Quark MXFP4 models (limit unset → 0.0) keep activation="silu" and swiglu_limit=0.0 and are unchanged.

Runtime note: to match MiniMax-M3's interleaved=False gate/up layout, launch
with SGLANG_USE_AITER_MOE_GU_ITLV=0 (SEPARATED gate_mode). This env is
already handled upstream in the AITER runner; this PR only makes the runner
reach that path.

Accuracy Tests

amd/MiniMax-M3-MXFP4, TP4, MI350x/MI355x.

  • Before this fix: MXFP4 output is incoherent (token-salad);
  • After this fix: coherent output; AIME25 ≈ 0.81–0.83 (n=10) vs 0.88 for the
    BF16 baseline.
    GSM8k test:
    {
    "gsm8k": {
    "name": "gsm8k",
    "alias": "gsm8k",
    "sample_len": 1319,
    "exact_match,strict-match": 0.9507202426080363,
    "exact_match_stderr,strict-match": 0.00596215065581247,
    "exact_match,flexible-extract": 0.9613343442001516,
    "exact_match_stderr,flexible-extract": 0.005310583162098061
    }
    }
    evalscope:10 times AIME25
    {
    "aime25/MiniMax-M3-MXFP4@aime25": 0.8367,
    "aime25/mean_acc": 0.8367,
    "aime25/default": 0.8367
    }

Benchmarking and Profiling

The changed branch is taken only for clamped-SwiGLU checkpoints, and plain-SwiGLU Quark MXFP4 models run the
identical code path as before (activation="silu", swiglu_limit=0.0), so
there is no hot-path change. Serving throughput of amd/MiniMax-M3-MXFP4 with
the fix applied (SGLANG_USE_AITER, SGLANG_USE_AITER_MOE_GU_ITLV=0), TP4 on
MI350x:
Config: generated-shared-prefix, concurrency 4 (unbounded request rate),
1024 in / 1024 out, 32 requests, 208 s.

Metric Value
Requests completed 32
Request throughput 0.15 req/s
Output throughput 307 tok/s
Total throughput 12,886 tok/s
Input throughput 12,579 tok/s
Effective concurrency 4.0
Latency (median / p99):
Metric Median
--- ---
E2E latency 25.5 s
TTFT 1.77 s
TPOT (per output token) 11.9 ms
ITL (inter-token) 11.6 ms

Unit tests

test/registered/unit/layers/quantization/test_quark_w4a4_mxfp4_moe.py
(CPU-only): asserts the activation translation happens iff a clamp limit is
set, and that swiglu_limit is forwarded (clamped) / stays 0.0 (unclamped).

Checklist

  • Format code with pre-commit (black 26.1.0, isort 7.0.0).
  • Add unit tests.
  • Provide accuracy/speed benchmark results (fill in exact numbers above).
  • Follow the SGLang code style guidance.

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #37529778610
Latest PR Test (Extra): ❌ Run #37529778027
Latest PR Test (AMD ROCm 10): ❌ Run #37529778873

@tnguyeng

tnguyeng commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor Author

/tag-and-rerun-ci

@tnguyeng

Copy link
Copy Markdown
Contributor Author

/tag-run-ci-label

rkarhila-amd added a commit to rkarhila-amd/sglang that referenced this pull request Aug 26, 2026
Mirror PR sgl-project#34540 for bf16 unquant: rewrite silu+gemm1_clamp_limit to
swiglu activation and pass swiglu_limit in AiterMoeQuantInfo so AITER
MoE matches Triton accuracy with CUDA graphs enabled.

Co-authored-by: Cursor <cursoragent@cursor.com>
rkarhila-amd added a commit to rkarhila-amd/sglang that referenced this pull request Aug 28, 2026
Port PR sgl-project#34540 clamped-SwiGLU plumbing into Quark W4A4 MXFP4 MoE, mirror
the AITER swiglu_limit path in Fp8 MoE, and teach Quark loading that
MiniMax index_qkv_proj is q|k-only when index value is disabled.

Co-authored-by: Cursor <cursoragent@cursor.com>
rkarhila-amd added a commit to rkarhila-amd/sglang that referenced this pull request Aug 28, 2026
Clamped-SwiGLU plumbing for Quark MXFP4 MoE belongs in tnguyeng's PR
sgl-project#34540; drop the duplicated quark_w4a4_mxfp4_moe.py hunks and unit
test from this gfx950 branch. MXFP8 fp8.py and loader.py fixes remain.

Co-authored-by: Cursor <cursoragent@cursor.com>
@vvagaytsev

Copy link
Copy Markdown
Contributor

Rebased onto current main (e7f744733). The bug is still live there: QuarkW4A4MXFp4MoE.create_moe_runner passes moe_runner_config through unchanged and apply_weights builds AiterMoeQuantInfo without swiglu_limit, so a clamped-SwiGLU MXFP4 checkpoint still runs on the plain-SiLU branch.

Two changes relative to the original commit:

  • Conflict resolution. [AMD] support gfx1250 on ROCM 10 #36871 added fused_moe_kwargs= to the same AiterMoeQuantInfo(...) call this PR adds swiglu_limit= to. Both kwargs are kept.
  • SiTU carve-out (new). Mxfp4MoEMethod on main skips the activation rewrite and forces swiglu_limit=0.0 when activation == "situ", because SiTU checkpoints reuse gemm1_clamp_limit as linear_beta and the runner forwards that on its own branch. Without the same guard here, a SiTU Quark MXFP4 checkpoint would have its linear_beta silently read as a SwiGLU clamp. The guard is applied in both create_moe_runner and apply_weights, and the registered CPU tests were extended to cover it, plus an assertion that fused_moe_kwargs stays None off gfx1250.

I don't have push access to the head branch, so the rebased commit is below as a patch instead of a branch update. It applies cleanly on e7f744733:

git fetch https://github.com/sgl-project/sglang.git main
git checkout -b quark-mxfp4-clamped-swiglu-rebased e7f744733
git am --keep-non-patch 0001-AMD-Fix-Quark-W4A4-MXFP4-MoE-clamped-SwiGLU-on-the-A.patch

ruff format and ruff check are clean on both files. I have not been able to run test/registered/unit/layers/quantization/test_quark_w4a4_mxfp4_moe.py locally (sglang does not import on Windows), so that still needs a CI run.

0001-AMD-Fix-Quark-W4A4-MXFP4-MoE-clamped-SwiGLU-on-the-A.patch
From 66c2125850ea2586a1870e8b71125787c86487d9 Mon Sep 17 00:00:00 2001
From: Tuan Nguyen Gia <tnguyeng@amd.com>
Date: Wed, 12 Aug 2026 01:46:03 -0500
Subject: [PATCH] [AMD] Fix Quark W4A4 MXFP4 MoE clamped-SwiGLU on the AITER
 path

MiniMax-M3 (and other GPT-OSS-style clamped-SwiGLU MoEs) declare
activation="silu" together with gemm1_alpha/gemm1_clamp_limit and rely on
the AITER MoE runner to run the *clamped* SwiGLU kernel. The runner only
does so when quant_info.swiglu_limit > 0, and it selects the FP4 kernel /
q_dtype_a on activation == Swiglu.

Mxfp4MoEMethod (gpt-oss) already forwards both, but the Quark W4A4 MXFP4
MoE scheme used by MiniMax-M3 built AiterMoeQuantInfo without swiglu_limit
and left activation="silu". As a result the MXFP4 experts silently ran
plain SiLU on the non-clamped branch and produced garbage on every MoE
layer, regardless of attention/MoE backend or aiter version.

Mirror Mxfp4MoEMethod in QuarkW4A4MXFp4MoE:
- create_moe_runner: translate activation to "swiglu" for the AITER runner
  when a clamp limit is set (gemm1_clamp_limit/swiglu_limit > 0).
- apply_weights: forward swiglu_limit (gemm1_clamp_limit or swiglu_limit)
  onto AiterMoeQuantInfo.

Both changes are gated on a positive clamp limit, so plain-SwiGLU Quark
MXFP4 models (limit unset -> 0.0) keep activation="silu" and
swiglu_limit=0.0 and are byte-for-byte unchanged.

Add CPU-only unit tests covering the clamped vs non-clamped activation
translation and swiglu_limit forwarding.

Co-authored-by: Cursor <cursoragent@cursor.com>
---
 .../quark/schemes/quark_w4a4_mxfp4_moe.py     |  35 +++-
 .../quantization/test_quark_w4a4_mxfp4_moe.py | 169 ++++++++++++++++++
 2 files changed, 203 insertions(+), 1 deletion(-)
 create mode 100644 test/registered/unit/layers/quantization/test_quark_w4a4_mxfp4_moe.py

diff --git a/python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py b/python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py
index 12243eb5c..4873d002e 100644
--- a/python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py
+++ b/python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py
@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import logging
 import threading
+from dataclasses import replace
 from typing import TYPE_CHECKING, Any
 
 import torch
@@ -894,7 +895,24 @@ class QuarkW4A4MXFp4MoE(QuarkMoEScheme):
             moe_runner_backend = MoeRunnerBackend.AITER
 
         if moe_runner_backend.is_aiter():
-            self.runner = MoeRunner(moe_runner_backend, moe_runner_config)
+            # The AITER MXFP4 per-1x32 kernel keys its q_dtype_a / kernel
+            # selection on activation == Swiglu. Clamped-SwiGLU checkpoints
+            # (e.g. MiniMax-M3) declare activation="silu" together with a
+            # gemm1_clamp_limit, so translate it to "swiglu" here, mirroring
+            # Mxfp4MoEMethod (gpt-oss). Without this the runner quantizes
+            # activations on the plain-SiLU branch and produces garbage.
+            # Plain-SwiGLU Quark MXFP4 models (no clamp) keep activation="silu"
+            # so their existing path is untouched. K3's SiTU is the one
+            # activation the kernel selects on its own -- leave it alone.
+            aiter_config = moe_runner_config
+            swiglu_limit = (
+                moe_runner_config.gemm1_clamp_limit
+                or moe_runner_config.swiglu_limit
+                or 0.0
+            )
+            if swiglu_limit > 0 and moe_runner_config.activation != "situ":
+                aiter_config = replace(moe_runner_config, activation="swiglu")
+            self.runner = MoeRunner(moe_runner_backend, aiter_config)
         else:
             # TODO(cwan): refactor other backends
             pass
@@ -935,5 +953,20 @@ class QuarkW4A4MXFp4MoE(QuarkMoEScheme):
             w2_scale=layer.w2_weight_scale,
             expert_mask=layer.dispatcher.expert_mask_gpu,
             fused_moe_kwargs=_fused_moe_kwargs,
+            # Forward the GPT-OSS-style clamped-SwiGLU limit so the AITER MoE
+            # runner enters the clamped-SwiGLU path (gate_mode + swiglu_limit)
+            # instead of plain SiLU. Mirrors Mxfp4MoEMethod; checkpoints without
+            # a clamp resolve to 0.0 and keep the previous behavior. K3 SiTU
+            # uses gemm1_clamp_limit as linear_beta, which the runner forwards
+            # on its own branch and must not be read as a swiglu limit.
+            swiglu_limit=(
+                0.0
+                if self.moe_runner_config.activation == "situ"
+                else (
+                    self.moe_runner_config.gemm1_clamp_limit
+                    or self.moe_runner_config.swiglu_limit
+                    or 0.0
+                )
+            ),
         )
         return self.runner.run(dispatch_output, quant_info)
diff --git a/test/registered/unit/layers/quantization/test_quark_w4a4_mxfp4_moe.py b/test/registered/unit/layers/quantization/test_quark_w4a4_mxfp4_moe.py
new file mode 100644
index 000000000..0890d27a0
--- /dev/null
+++ b/test/registered/unit/layers/quantization/test_quark_w4a4_mxfp4_moe.py
@@ -0,0 +1,169 @@
+"""Unit tests for the Quark W4A4 MXFP4 MoE clamped-SwiGLU plumbing.
+
+CPU-only, no model loading and no aiter/ROCm dependency: they exercise the
+pure-Python control flow in ``QuarkW4A4MXFp4MoE.create_moe_runner`` and
+``apply_weights`` that decides whether the AITER MoE runner is driven on the
+clamped-SwiGLU path (``activation="swiglu"`` + ``swiglu_limit``) or the plain
+SiLU path. See the regression these guard against: clamped-SwiGLU MXFP4
+checkpoints (e.g. MiniMax-M3) declare ``activation="silu"`` + a
+``gemm1_clamp_limit``, and without this plumbing the experts silently run plain
+SiLU and emit garbage.
+"""
+
+import sys
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+import sglang.srt.layers.moe.utils as moe_utils
+import sglang.srt.layers.quantization.quark.schemes.quark_w4a4_mxfp4_moe as quark_moe
+from sglang.srt.layers.moe import MoeRunnerBackend, MoeRunnerConfig
+from sglang.srt.layers.quantization.quark.schemes.quark_w4a4_mxfp4_moe import (
+    QuarkW4A4MXFp4MoE,
+)
+from sglang.test.ci.ci_register import register_cpu_ci
+
+register_cpu_ci(est_time=5, suite="base-a-test-cpu")
+
+
+def _make_scheme() -> QuarkW4A4MXFp4MoE:
+    # is_checkpoint_mxfp4_serialized=True skips the gfx95 hardware check.
+    return QuarkW4A4MXFp4MoE(
+        weight_config={"qscheme": "per_group"},
+        input_config={"qscheme": "per_group", "is_dynamic": True},
+    )
+
+
+def _force_aiter_runner(monkeypatch):
+    """Make ``create_moe_runner`` take the AITER branch and capture its config."""
+    captured = {}
+
+    def fake_moe_runner(backend, config):
+        captured["backend"] = backend
+        captured["config"] = config
+        return object()
+
+    monkeypatch.setattr(quark_moe, "MoeRunner", fake_moe_runner)
+    monkeypatch.setattr(
+        moe_utils, "get_moe_runner_backend", lambda: MoeRunnerBackend.AITER
+    )
+    return captured
+
+
+def _fake_layer():
+    # dtype uint8 so we never hit the float4_e2m1fn_x2 view branch (also guarded
+    # by delattr in the apply_weights tests below).
+    return SimpleNamespace(
+        w13_weight=torch.zeros(1, dtype=torch.uint8),
+        w2_weight=torch.zeros(1, dtype=torch.uint8),
+        w13_weight_scale=torch.zeros(1, dtype=torch.uint8),
+        w2_weight_scale=torch.zeros(1, dtype=torch.uint8),
+        dispatcher=SimpleNamespace(expert_mask_gpu=torch.zeros(1)),
+    )
+
+
+def _capturing_runner(store):
+    class _Runner:
+        def run(self, dispatch_output, quant_info):
+            store["quant_info"] = quant_info
+            return "ran"
+
+    return _Runner()
+
+
+def test_create_moe_runner_translates_activation_to_swiglu_when_clamped(monkeypatch):
+    captured = _force_aiter_runner(monkeypatch)
+    scheme = _make_scheme()
+
+    cfg = MoeRunnerConfig(activation="silu", gemm1_clamp_limit=7.0)
+    scheme.create_moe_runner(layer=None, moe_runner_config=cfg)
+
+    assert captured["config"].activation == "swiglu"
+    # The original config is left untouched (dataclasses.replace returns a copy).
+    assert cfg.activation == "silu"
+
+
+def test_create_moe_runner_keeps_silu_without_clamp(monkeypatch):
+    captured = _force_aiter_runner(monkeypatch)
+    scheme = _make_scheme()
+
+    cfg = MoeRunnerConfig(activation="silu")  # no clamp
+    scheme.create_moe_runner(layer=None, moe_runner_config=cfg)
+
+    assert captured["config"].activation == "silu"
+    assert captured["config"] is cfg
+
+
+def test_create_moe_runner_leaves_situ_activation_alone(monkeypatch):
+    captured = _force_aiter_runner(monkeypatch)
+    scheme = _make_scheme()
+
+    # K3-style: activation="situ" with gemm1_clamp_limit carrying linear_beta,
+    # not a SwiGLU clamp. The AITER kernel selects SiTU itself.
+    cfg = MoeRunnerConfig(activation="situ", gemm1_clamp_limit=7.0)
+    scheme.create_moe_runner(layer=None, moe_runner_config=cfg)
+
+    assert captured["config"].activation == "situ"
+    assert captured["config"] is cfg
+
+
+def test_apply_weights_forwards_swiglu_limit_when_clamped(monkeypatch):
+    monkeypatch.delattr(torch, "float4_e2m1fn_x2", raising=False)
+    scheme = _make_scheme()
+    scheme.moe_runner_config = MoeRunnerConfig(
+        activation="swiglu", gemm1_clamp_limit=7.0
+    )
+    store = {}
+    scheme.runner = _capturing_runner(store)
+
+    result = scheme.apply_weights(_fake_layer(), dispatch_output=object())
+
+    assert result == "ran"
+    assert store["quant_info"].swiglu_limit == 7.0
+
+
+def test_apply_weights_defaults_swiglu_limit_zero_without_clamp(monkeypatch):
+    monkeypatch.delattr(torch, "float4_e2m1fn_x2", raising=False)
+    scheme = _make_scheme()
+    scheme.moe_runner_config = MoeRunnerConfig(activation="silu")  # no clamp
+    store = {}
+    scheme.runner = _capturing_runner(store)
+
+    scheme.apply_weights(_fake_layer(), dispatch_output=object())
+
+    assert store["quant_info"].swiglu_limit == 0.0
+
+
+def test_apply_weights_zeroes_swiglu_limit_for_situ(monkeypatch):
+    monkeypatch.delattr(torch, "float4_e2m1fn_x2", raising=False)
+    scheme = _make_scheme()
+    scheme.moe_runner_config = MoeRunnerConfig(activation="situ", gemm1_clamp_limit=7.0)
+    store = {}
+    scheme.runner = _capturing_runner(store)
+
+    scheme.apply_weights(_fake_layer(), dispatch_output=object())
+
+    # Forwarding the limit here would make the runner clamp a SiTU linear_beta.
+    assert store["quant_info"].swiglu_limit == 0.0
+
+
+def test_apply_weights_leaves_fused_moe_kwargs_unset_off_gfx1250(monkeypatch):
+    monkeypatch.delattr(torch, "float4_e2m1fn_x2", raising=False)
+    # Pinned rather than read from the host: the gfx1250 branch imports aiter.
+    monkeypatch.setattr(quark_moe, "_is_gfx1250", False)
+    scheme = _make_scheme()
+    scheme.moe_runner_config = MoeRunnerConfig(
+        activation="swiglu", gemm1_clamp_limit=7.0
+    )
+    store = {}
+    scheme.runner = _capturing_runner(store)
+
+    scheme.apply_weights(_fake_layer(), dispatch_output=object())
+
+    assert store["quant_info"].fused_moe_kwargs is None
+    assert store["quant_info"].swiglu_limit == 7.0
+
+
+if __name__ == "__main__":
+    sys.exit(pytest.main([__file__, "-v"]))
-- 
2.53.0.windows.2

tnguyeng and others added 2 commits October 5, 2026 13:26
MiniMax-M3 (and other GPT-OSS-style clamped-SwiGLU MoEs) declare
activation="silu" together with gemm1_alpha/gemm1_clamp_limit and rely on
the AITER MoE runner to run the *clamped* SwiGLU kernel. The runner only
does so when quant_info.swiglu_limit > 0, and it selects the FP4 kernel /
q_dtype_a on activation == Swiglu.

Mxfp4MoEMethod (gpt-oss) already forwards both, but the Quark W4A4 MXFP4
MoE scheme used by MiniMax-M3 built AiterMoeQuantInfo without swiglu_limit
and left activation="silu". As a result the MXFP4 experts silently ran
plain SiLU on the non-clamped branch and produced garbage on every MoE
layer, regardless of attention/MoE backend or aiter version.

Mirror Mxfp4MoEMethod in QuarkW4A4MXFp4MoE:
- create_moe_runner: translate activation to "swiglu" for the AITER runner
  when a clamp limit is set (gemm1_clamp_limit/swiglu_limit > 0).
- apply_weights: forward swiglu_limit (gemm1_clamp_limit or swiglu_limit)
  onto AiterMoeQuantInfo.

Both changes are gated on a positive clamp limit, so plain-SwiGLU Quark
MXFP4 models (limit unset -> 0.0) keep activation="silu" and
swiglu_limit=0.0 and are byte-for-byte unchanged.

Add CPU-only unit tests covering the clamped vs non-clamped activation
translation and swiglu_limit forwarding.

Co-authored-by: Cursor <cursoragent@cursor.com>
The scheme no longer rewrites silu to swiglu, so the tests check that combo and the forwarded swiglu_limit.
@tnguyeng
tnguyeng force-pushed the amd/fix-quark-mxfp4-moe-clamped-swiglu branch from 2f134f7 to bdb56d2 Compare October 5, 2026 14:10
@tnguyeng
tnguyeng requested a review from HaiShaw as a code owner October 5, 2026 14:10
rkarhila-amd added a commit to rkarhila-amd/sglang that referenced this pull request Oct 6, 2026
Mirror PR sgl-project#34540 for bf16 unquant: rewrite silu+gemm1_clamp_limit to
swiglu activation and pass swiglu_limit in AiterMoeQuantInfo so AITER
MoE matches Triton accuracy with CUDA graphs enabled.

Co-authored-by: Cursor <cursoragent@cursor.com>
rkarhila-amd added a commit to rkarhila-amd/sglang that referenced this pull request Oct 6, 2026
Port PR sgl-project#34540 clamped-SwiGLU plumbing into Quark W4A4 MXFP4 MoE, mirror
the AITER swiglu_limit path in Fp8 MoE, and teach Quark loading that
MiniMax index_qkv_proj is q|k-only when index value is disabled.

Co-authored-by: Cursor <cursoragent@cursor.com>
rkarhila-amd added a commit to rkarhila-amd/sglang that referenced this pull request Oct 6, 2026
Clamped-SwiGLU plumbing for Quark MXFP4 MoE belongs in tnguyeng's PR
test from this gfx950 branch. MXFP8 fp8.py and loader.py fixes remain.

Co-authored-by: Cursor <cursoragent@cursor.com>
rkarhila-amd added a commit to rkarhila-amd/sglang that referenced this pull request Oct 6, 2026
Mirror PR sgl-project#34540 for bf16 unquant: rewrite silu+gemm1_clamp_limit to
swiglu activation and pass swiglu_limit in AiterMoeQuantInfo so AITER
MoE matches Triton accuracy with CUDA graphs enabled.

Co-authored-by: Cursor <cursoragent@cursor.com>
rkarhila-amd added a commit to rkarhila-amd/sglang that referenced this pull request Oct 6, 2026
Port PR sgl-project#34540 clamped-SwiGLU plumbing into Quark W4A4 MXFP4 MoE, mirror
the AITER swiglu_limit path in Fp8 MoE, and teach Quark loading that
MiniMax index_qkv_proj is q|k-only when index value is disabled.

Co-authored-by: Cursor <cursoragent@cursor.com>
rkarhila-amd added a commit to rkarhila-amd/sglang that referenced this pull request Oct 6, 2026
Clamped-SwiGLU plumbing for Quark MXFP4 MoE belongs in tnguyeng's PR
test from this gfx950 branch. MXFP8 fp8.py and loader.py fixes remain.

Co-authored-by: Cursor <cursoragent@cursor.com>
rkarhila-amd added a commit to rkarhila-amd/sglang that referenced this pull request Oct 6, 2026
Mirror PR sgl-project#34540 for bf16 unquant: rewrite silu+gemm1_clamp_limit to
swiglu activation and pass swiglu_limit in AiterMoeQuantInfo so AITER
MoE matches Triton accuracy with CUDA graphs enabled.

Co-authored-by: Cursor <cursoragent@cursor.com>
rkarhila-amd added a commit to rkarhila-amd/sglang that referenced this pull request Oct 6, 2026
Port PR sgl-project#34540 clamped-SwiGLU plumbing into Quark W4A4 MXFP4 MoE, mirror
the AITER swiglu_limit path in Fp8 MoE, and teach Quark loading that
MiniMax index_qkv_proj is q|k-only when index value is disabled.

Co-authored-by: Cursor <cursoragent@cursor.com>
rkarhila-amd added a commit to rkarhila-amd/sglang that referenced this pull request Oct 6, 2026
Clamped-SwiGLU plumbing for Quark MXFP4 MoE belongs in tnguyeng's PR
test from this gfx950 branch. MXFP8 fp8.py and loader.py fixes remain.

Co-authored-by: Cursor <cursoragent@cursor.com>
@tnguyeng

tnguyeng commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Accuracy on MiniMax-M3 MXFP4 (MI355X, TP4)

Matched A/B on the same image, flags, GPUs, and port. The only change is the mounted source: this PR (bdb56d2ca0) versus upstream/main (f048d5aa4b).
The configured SwiGLU clamp is applied, and accuracy matches main. The gaps are inside the measurement noise.

Task This PR upstream/main Difference
GSM8K strict-match (5-shot, n=1319) 95.07% ± 0.60 94.84% ± 0.61 +0.23 pt
GSM8K flexible-extract (5-shot, n=1319) 95.00% ± 0.60 94.77% ± 0.61 +0.23 pt
MMLU generative (5-shot, n=14042) 83.07% ± 0.30 82.58% ± 0.31 +0.49 pt
niah_single_2 @ 65,536 (n=500) 98.2% 98.0% +0.2 pt
niah_single_2 @ 122,880 (n=500) 94.2% ± 1.05 93.0% ± 1.14 +1.2 pt
GSM8K and the 122,880 needle sit inside the combined standard error (0.85 pt and 1.55 pt). The 65,536 needle differs by 1 example out of 500. MMLU is 0.49 pt higher, and its combined standard error is 0.43 pt.

Server

  • Image: lmsysorg/sglang-rocm:v0.5.20-rocm10-mi35x-20260928
  • Model: amd/MiniMax-M3-MXFP4
  • Hardware: MI355X, HIP_VISIBLE_DEVICES=0,1,2,3, --tp 4, port 32115
  • --quantization quark --dtype bfloat16 --context-length 131072 --chunked-prefill-size 32768 --mem-fraction-static 0.85 --kv-cache-dtype fp8_e4m3 --attention-backend aiter --moe-runner-backend aiter --enable-aiter-allreduce-fusion --reasoning-parser auto
  • Env: SGLANG_USE_AITER=1, SGLANG_USE_AITER_MOE_GU_ITLV=0, SGLANG_USE_AITER_AR=1, ROCM_QUICK_REDUCE_QUANTIZATION=NONE, SGLANG_MINIMAX_M3_INDEX_TOPK_FREQ=2
  • TunableOp off
  • This PR: local python/ mounted into the container. aiter_swiglu_oai_limit is present in quark_w4a4_mxfp4_moe.py.
  • Upstream: same mount after git checkout upstream/main (f048d5aa4b), so that symbol is absent.

Eval

lm_eval with local-chat-completions, num_concurrent=8, tokenized_requests=false, seed 1234.

  • GSM8K: --num_fewshot 5 --fewshot_as_multiturn --apply_chat_template, max_gen_toks=16384
  • MMLU: task mmlu_generative, --num_fewshot 5 --apply_chat_template, max_gen_toks=2048
  • NIAH: niah_single_2, one length per run, think_end_token=</mm:think>, max_gen_toks=4096. The long run uses max_seq_lengths:[122880] so the prompt plus 4096 generated tokens fits in the 131072 context.

@tnguyeng

tnguyeng commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

/tag-and-rerun-ci

@BowenBao BowenBao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@BowenBao BowenBao added the run-ci CI: run the baseline test suite on this PR label Oct 6, 2026
@BowenBao
BowenBao enabled auto-merge (squash) October 6, 2026 16:39
@BowenBao
BowenBao merged commit c0df52b into sgl-project:main Oct 7, 2026
163 of 185 checks passed
cursor Bot pushed a commit to cipheraxat/sglang that referenced this pull request Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants