Skip to content

fix(rocm): shuffle block-FP8 MoE experts for AITER only when the runner is AITER - #41636

Open
xiaohong42 wants to merge 2 commits into
sgl-project:mainfrom
xiaohong42:gfx942/moe-shuffle-follows-runner
Open

xiaohong42 wants to merge 2 commits into
sgl-project:mainfrom
xiaohong42:gfx942/moe-shuffle-follows-runner

Conversation

@xiaohong42

@xiaohong42 xiaohong42 commented Sep 29, 2026 •

Copy link
Copy Markdown

Motivation

Fp8MoEMethod.process_weights_after_loading_block_quant shuffles the experts into AITER's (16, 16) layout whenever SGLANG_USE_AITER is set, both in the gfx94x (FNUZ) branch and in the non-FNUZ branch. Which runner serves the layer is decided separately, in create_moe_runner: AITER by default when the A2A backend supports it, Triton when --moe-runner-backend triton is passed or when the A2A backend does not support AITER. apply() calls AITER's kernels only for an AITER runner, and the Triton runner reads the experts in checkpoint order.

So with AITER on and the Triton runner, every expert GEMM runs on shuffled weights and nothing reports it. On gfx942 (MI308X) 16372 of 16384 MoE output elements come out wrong, with a mean relative error of 1.4.

The MXFP8 -> block and FP4 -> FP8 branches of the same function already shuffle only when self.runner.runner_backend.is_aiter(), with a comment saying why; the two plain branches did not follow.

Affected: ROCm deployments with SGLANG_USE_AITER=1 whose block-FP8 MoE layers run on the Triton runner, i.e. an explicit --moe-runner-backend triton (miles' DeepSeek-V4 recipe on gfx942 is one) or an A2A backend without AITER support. The default AITER runner is unaffected.

Modifications

  1. fix(rocm): shuffle gfx94x block-FP8 MoE experts only for the AITER runner: computes runner_is_aiter exactly as the sibling branches do and requires it for the shuffle in the FNUZ branch.
  2. fix(rocm): shuffle non-FNUZ block-FP8 MoE experts only for the AITER runner: the same for the non-FNUZ branch (gfx950 and later). The elif _use_aiter: stays, so a non-AITER runner on ROCm never falls through to the CPU or CUDA branches below it. This one is only unit-tested, because the branch is not reachable on gfx94x and no gfx950 machine was available; it can be dropped on its own.

A method without a runner is treated as not AITER, as the MXFP8 branch already does. FusedMoE.__init__ creates the runner before any weight is loaded, so this does not arise in normal loading.

Accuracy Tests

test main this PR
test/registered/unit/layers/quantization/test_fp8_moe_aiter_shuffle_follows_runner.py (CPU): with AITER on, the Triton runner and a method without a runner get the same bytes as with AITER off, in the FNUZ and non-FNUZ branches; the AITER runner still gets shuffled experts the four Triton / no-runner cases fail (their experts are shuffled) 4/4 pass
test/registered/amd/test_fp8_moe_triton_runner_with_aiter.py (gfx94x): the real Triton fused MoE over experts finalized with AITER on vs with AITER off; as a control, the same kernel fed AITER's layout fails: 16372/16384 output elements mismatched pass: bit-identical; the control is off by a mean relative error of 1.4

The GPU test ran on MI308X (gfx942) against this branch. The unit tests under test/registered/unit/layers/quantization and test/registered/unit/layers/moe show no new failures against main.

Reproduce:

python -m pytest -q test/registered/unit/layers/quantization/test_fp8_moe_aiter_shuffle_follows_runner.py
python -m pytest -q test/registered/amd/test_fp8_moe_triton_runner_with_aiter.py   # gfx94x with AITER

Speed Tests and Profiling

No serving hot-path change: the condition is evaluated once, at finalization. Triton-runner configurations skip a shuffle they could not use.

Checklist

  • Formatted with the repo's pre-commit hooks (isort 7.0.0, ruff 0.15.1, ruff-format, codespell)
  • Unit tests added (see Accuracy Tests)
  • Documentation: not needed (no user-facing option)
  • Accuracy results above; speed see above

Relation to other PRs


CI States

Latest PR Test (Base): ❌ Run #36527216586
Latest PR Test (Extra): ❌ Run #36527216345
Latest PR Test (AMD ROCm 10): ❌ Run #36527216680

xiaohong42 and others added 2 commits September 29, 2026 05:28
…nner

On gfx94x the plain block-FP8 MoE finalization shuffles the experts into
AITER's (16, 16) layout whenever SGLANG_USE_AITER is set, whichever runner
will read them. The runner is chosen separately: an explicit
--moe-runner-backend triton, or the automatic choice when the A2A backend
does not support AITER, gives the Triton runner, which reads the shuffled
experts as if they were in checkpoint order. The MoE output is then wrong
almost everywhere (99.9% of the elements in the GPU test below, a mean
relative error of 1.4), and nothing reports it.

Fp8MoEMethod.apply already routes to AITER's kernels only when
self.runner.runner_backend.is_aiter(), and the MXFP8 -> block and FP4 -> FP8
branches of this finalization already shuffle only then. The plain FNUZ
branch now uses the same condition, so the AITER runner keeps its shuffled
experts and every other runner gets them as loaded.

Tests:
- test/registered/unit/layers/quantization/test_fp8_moe_aiter_shuffle_follows_runner.py
  (CPU): with AITER on, the Triton runner (and a method without a runner,
  which the MXFP8 branch also treats as not AITER) gets the same bytes as
  with AITER off, and the AITER runner still gets shuffled experts. The
  Triton cases fail on main.
- test/registered/amd/test_fp8_moe_triton_runner_with_aiter.py (gfx94x): the
  real Triton fused MoE over experts finalized with AITER on equals the one
  with AITER off, bit for bit, while fed AITER's layout it is off by a mean
  relative error above 0.5. On main it fails with 99.9% of the output
  elements mismatched.

Co-authored-by: Cursor <cursoragent@cursor.com>
…runner

The non-FNUZ branch of the same finalization (gfx950 and later ROCm
targets) has the same condition as the gfx94x one fixed in the previous
commit: with SGLANG_USE_AITER set it shuffles the experts into AITER's
(16, 16) layout for any runner, so the Triton runner reads them in the wrong
order. It now shuffles only when the runner is AITER. The branch is still
taken whenever AITER is on, so a non-AITER runner on ROCm never falls
through to the CPU or CUDA paths below it.

This branch is not reachable on gfx94x hardware, so it is covered by the CPU
test only: with AITER on, the Triton runner (and a method without a runner)
keeps the weights exactly as loaded, and the AITER runner gets them
shuffled. The Triton cases fail without this commit.

Co-authored-by: Cursor <cursoragent@cursor.com>
xiaohong42 added a commit to xiaohong42/sglang that referenced this pull request Sep 29, 2026
These tests expect the experts shuffled whenever AITER is on. The shuffle is
only for AITER's own kernels, and sgl-project#41636 makes the
finalization skip it for any other runner, which a method built without
create_moe_runner has none of. The MoE methods here now carry the AITER runner
that AITER-on configurations pick, so the tests pass with and without that
change; nothing else about them changes.

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant