Repository navigation
fix(rocm): shuffle block-FP8 MoE experts for AITER only when the runner is AITER - #41636
Open
xiaohong42 wants to merge 2 commits into
Open
xiaohong42 wants to merge 2 commits into
xiaohong42 wants to merge 2 commits into
Conversation
…nner On gfx94x the plain block-FP8 MoE finalization shuffles the experts into AITER's (16, 16) layout whenever SGLANG_USE_AITER is set, whichever runner will read them. The runner is chosen separately: an explicit --moe-runner-backend triton, or the automatic choice when the A2A backend does not support AITER, gives the Triton runner, which reads the shuffled experts as if they were in checkpoint order. The MoE output is then wrong almost everywhere (99.9% of the elements in the GPU test below, a mean relative error of 1.4), and nothing reports it. Fp8MoEMethod.apply already routes to AITER's kernels only when self.runner.runner_backend.is_aiter(), and the MXFP8 -> block and FP4 -> FP8 branches of this finalization already shuffle only then. The plain FNUZ branch now uses the same condition, so the AITER runner keeps its shuffled experts and every other runner gets them as loaded. Tests: - test/registered/unit/layers/quantization/test_fp8_moe_aiter_shuffle_follows_runner.py (CPU): with AITER on, the Triton runner (and a method without a runner, which the MXFP8 branch also treats as not AITER) gets the same bytes as with AITER off, and the AITER runner still gets shuffled experts. The Triton cases fail on main. - test/registered/amd/test_fp8_moe_triton_runner_with_aiter.py (gfx94x): the real Triton fused MoE over experts finalized with AITER on equals the one with AITER off, bit for bit, while fed AITER's layout it is off by a mean relative error above 0.5. On main it fails with 99.9% of the output elements mismatched. Co-authored-by: Cursor <cursoragent@cursor.com>
…runner The non-FNUZ branch of the same finalization (gfx950 and later ROCm targets) has the same condition as the gfx94x one fixed in the previous commit: with SGLANG_USE_AITER set it shuffles the experts into AITER's (16, 16) layout for any runner, so the Triton runner reads them in the wrong order. It now shuffles only when the runner is AITER. The branch is still taken whenever AITER is on, so a non-AITER runner on ROCm never falls through to the CPU or CUDA paths below it. This branch is not reachable on gfx94x hardware, so it is covered by the CPU test only: with AITER on, the Triton runner (and a method without a runner) keeps the weights exactly as loaded, and the AITER runner gets them shuffled. The Triton cases fail without this commit. Co-authored-by: Cursor <cursoragent@cursor.com>
xiaohong42
requested review from
Alisehen,
AniZpZ,
BBuf,
Edwardf0t1,
FlamingoPg,
HaiShaw,
OrangeRedeng,
b8zhong,
ch-wan and
mmangkad
as code owners
September 29, 2026 05:39
xiaohong42
added a commit
to xiaohong42/sglang
that referenced
this pull request
Sep 29, 2026
These tests expect the experts shuffled whenever AITER is on. The shuffle is only for AITER's own kernels, and sgl-project#41636 makes the finalization skip it for any other runner, which a method built without create_moe_runner has none of. The MoE methods here now carry the AITER runner that AITER-on configurations pick, so the tests pass with and without that change; nothing else about them changes. Co-authored-by: Cursor <cursoragent@cursor.com>
This was referenced Oct 10, 2026
4 of 5 tasks
4 of 5 tasks
This was referenced Oct 22, 2026
This was referenced Oct 31, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Fp8MoEMethod.process_weights_after_loading_block_quantshuffles the experts into AITER's (16, 16) layout wheneverSGLANG_USE_AITERis set, both in the gfx94x (FNUZ) branch and in the non-FNUZ branch. Which runner serves the layer is decided separately, increate_moe_runner: AITER by default when the A2A backend supports it, Triton when--moe-runner-backend tritonis passed or when the A2A backend does not support AITER.apply()calls AITER's kernels only for an AITER runner, and the Triton runner reads the experts in checkpoint order.So with AITER on and the Triton runner, every expert GEMM runs on shuffled weights and nothing reports it. On gfx942 (MI308X) 16372 of 16384 MoE output elements come out wrong, with a mean relative error of 1.4.
The MXFP8 -> block and FP4 -> FP8 branches of the same function already shuffle only when
self.runner.runner_backend.is_aiter(), with a comment saying why; the two plain branches did not follow.Affected: ROCm deployments with
SGLANG_USE_AITER=1whose block-FP8 MoE layers run on the Triton runner, i.e. an explicit--moe-runner-backend triton(miles' DeepSeek-V4 recipe on gfx942 is one) or an A2A backend without AITER support. The default AITER runner is unaffected.Modifications
fix(rocm): shuffle gfx94x block-FP8 MoE experts only for the AITER runner: computesrunner_is_aiterexactly as the sibling branches do and requires it for the shuffle in the FNUZ branch.fix(rocm): shuffle non-FNUZ block-FP8 MoE experts only for the AITER runner: the same for the non-FNUZ branch (gfx950 and later). Theelif _use_aiter:stays, so a non-AITER runner on ROCm never falls through to the CPU or CUDA branches below it. This one is only unit-tested, because the branch is not reachable on gfx94x and no gfx950 machine was available; it can be dropped on its own.A method without a runner is treated as not AITER, as the MXFP8 branch already does.
FusedMoE.__init__creates the runner before any weight is loaded, so this does not arise in normal loading.Accuracy Tests
maintest/registered/unit/layers/quantization/test_fp8_moe_aiter_shuffle_follows_runner.py(CPU): with AITER on, the Triton runner and a method without a runner get the same bytes as with AITER off, in the FNUZ and non-FNUZ branches; the AITER runner still gets shuffled expertstest/registered/amd/test_fp8_moe_triton_runner_with_aiter.py(gfx94x): the real Triton fused MoE over experts finalized with AITER on vs with AITER off; as a control, the same kernel fed AITER's layoutThe GPU test ran on MI308X (gfx942) against this branch. The unit tests under
test/registered/unit/layers/quantizationandtest/registered/unit/layers/moeshow no new failures againstmain.Reproduce:
python -m pytest -q test/registered/unit/layers/quantization/test_fp8_moe_aiter_shuffle_follows_runner.py python -m pytest -q test/registered/amd/test_fp8_moe_triton_runner_with_aiter.py # gfx94x with AITERSpeed Tests and Profiling
No serving hot-path change: the condition is evaluated once, at finalization. Triton-runner configurations skip a shuffle they could not use.
Checklist
Relation to other PRs
ifline, so whichever merges second needs a one-line rebase: keep fix(rocm): make in-place reloads of gfx94x block-FP8 weights and FP8 KV-cache scales work #41359's body under this PR's condition. fix(rocm): make in-place reloads of gfx94x block-FP8 weights and FP8 KV-cache scales work #41359's tests give the shuffled cases an AITER runner, so they pass with or without this change.sglang-miles, the branch miles runs on; this is themaincounterpart, plus the non-FNUZ branch.CI States
Latest PR Test (Base): ❌ Run #36527216586
Latest PR Test (Extra): ❌ Run #36527216345
Latest PR Test (AMD ROCm 10): ❌ Run #36527216680