Skip to content

[Fix] [AMD] MoE: Preserve CT FP8 Triton layout and reduction dimensions - #45486

Merged
BBuf merged 1 commit into
sgl-project:mainfrom
lunarwhite:lunarwhite/ct-fp8-runner-layout
Oct 30, 2026
Merged

BBuf merged 1 commit into
sgl-project:mainfrom
lunarwhite:lunarwhite/ct-fp8-runner-layout

Conversation

@lunarwhite

@lunarwhite lunarwhite commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

With SGLANG_USE_AITER=1 and an explicit Triton MoE runner, compressed-tensors CHANNEL FP8 layers receive AITER-shuffled weights and lose 128 reduction elements in the non-block FP8 kernel (1024 → 896 in the regression). Both cause incorrect outputs.

Modifications

  • Shuffle CT CHANNEL FP8 weights only for an actual AITER runner.
  • Match the non-block FP8 kernel's padding calculation to its existing caller when AITER is enabled.
  • Add an AMD CI-registered runtime quantization test under test/registered/unit/layers/quantization/, covering checkpoint loading, postprocessing, and numerical Triton/AITER MoE comparisons with 1- and 33-token inputs.

Related: #41636 addresses the shuffle mechanism in Fp8MoEMethod block-FP8; this PR fixes the separate compressed-tensors CHANNEL path and the non-block kernel dimension calculation.

Accuracy Tests

Tested on MI355X (gfx950), Python 3.10.12, PyTorch 2.9.1+rocm7.2.0.git7e1940d4, HIP 7.2.26015-fc0010cf6a, using the official image:

lmsysorg/sglang-rocm@sha256:8cb29fc407c2e46bdb1866c5edb91f109ac1acbbcb6905089beaa6c79b0813a2

Base: 9db58ab547516435b544ae050dd6c2f91b52bcf3
Head: 7d7d8b79eec61eda1884c54311b2580bd6159cff

Layer controls use identical seeded weights, routing and test source, SGLANG_MOE_PADDING=0, isolated processes/caches, and verified import paths/hashes.

Variant Explicit Triton, 1 / 33 tokens AITER, 1 / 33 tokens
Base, AITER disabled pass / pass not run
Base, AITER enabled fail / fail pass / pass
Only shuffle guard fail / fail (cosine 0.7757 / 0.8207) pass / pass
Only kernel padding fix fail / fail pass / pass
Both fixes pass / pass pass / pass

Both final test methods pass without errors/skips. The independent reference uses dequantized checkpoint weights and identical routing; fixed thresholds are cosine > 0.99 and relative L2 < 0.1. Controls fail numerically, establishing that both fixes are required.

A separate Quark per-tensor dynamic regression covers another shared-kernel consumer: base/AITER off passes, base/AITER on fails both numerical subtests, and patched/AITER on passes (no errors/skips).

Reproduce the GPU-tested revisions on MI35x

Run inside the image above, from outside the source tree:

git clone --branch lunarwhite/ct-fp8-runner-layout https://github.com/lunarwhite/sglang.git /tmp/ct-sglang
git -C /tmp/ct-sglang checkout --detach 7d7d8b79eec61eda1884c54311b2580bd6159cff
cd /tmp
export SGLANG_USE_AITER=1 SGLANG_MOE_PADDING=0 SGLANG_TEST_MAX_RETRY=0
export SGLANG_CACHE_DIR=/tmp/ct-sglang-head-cache
PYTHONPATH=/tmp/ct-sglang/python python /tmp/ct-sglang/test/registered/unit/layers/quantization/test_compressed_tensors_fp8_moe_layout.py

To reproduce the controls with the same test file (each command is a separate process):

git -C /tmp/ct-sglang worktree add --detach /tmp/ct-base 9db58ab547516435b544ae050dd6c2f91b52bcf3
git -C /tmp/ct-sglang worktree add --detach /tmp/ct-kernel-only 9db58ab547516435b544ae050dd6c2f91b52bcf3
git -C /tmp/ct-kernel-only restore --source=7d7d8b79eec61eda1884c54311b2580bd6159cff -- python/sglang/kernels/ops/moe/fused_moe_triton_kernels.py
TEST_FILE=/tmp/ct-sglang/test/registered/unit/layers/quantization/test_compressed_tensors_fp8_moe_layout.py
# Pass: AITER-disabled control, one test method.
SGLANG_USE_AITER=0 SGLANG_CACHE_DIR=/tmp/ct-control-cache PYTHONPATH=/tmp/ct-base/python python "$TEST_FILE" TestCompressedTensorsFp8MoELayout.test_explicit_triton
# Expected numerical failure in Triton, with AITER passing:
SGLANG_CACHE_DIR=/tmp/ct-base-cache PYTHONPATH=/tmp/ct-base/python python "$TEST_FILE"
SGLANG_CACHE_DIR=/tmp/ct-kernel-cache PYTHONPATH=/tmp/ct-kernel-only/python python "$TEST_FILE"

The separate AITER-disabled + padding-enabled configuration is outside this validation.

Real-checkpoint validation

RedHatAI/Qwen3-30B-A3B-FP8-dynamic, revision 10a439027a4fd228adc03a7bffb1aac10a179938, was evaluated on the same MI355X image and base/head sources. All four arms observed actual CT CHANNEL execution in all 48 layers and verified the loaded source hashes. TP1, eager execution, AITER enabled, padding disabled; 200 GSM8K test examples, five demonstrations from the train split, greedy decoding, maximum 512 generated tokens.

Configuration Correct / 200
Base, AITER runner 180
Base, AITER runner repeat 179
Base, explicit Triton runner 0
Patched, explicit Triton runner 181

All 800 predictions were independently rescored and prompt/dataset hashes verified. Every base Triton output hit the 512-token cap. This is a configuration-specific sanity check, not full-dataset accuracy or statistical equivalence; default AITER is the known-good reference.

Speed Tests and Profiling

A separate fixed-output probe (four prompts, 64 tokens each, three timed repeats after warmup) measured median output rates of 103.88 tok/s for base Triton and 104.75 tok/s for patched Triton; AITER reference runs measured 99.71 and 103.64 tok/s. This bounded probe is not a production throughput or speedup claim.

Checklist

Disclosure: Found by Hyperloom session analysis, reviewed and validated by human.


CI States

Latest PR Test (Base): ✅ Run #40641071798
Latest PR Test (Extra): ❌ Run #40641072021
Latest PR Test (AMD ROCm 10): ❌ Run #40641071972

@lunarwhite lunarwhite changed the title [Fix] [AMD] MOE: Preserve CT FP8 Triton layout and reduction dimensions [Fix] [AMD] MoE: Preserve CT FP8 Triton layout and reduction dimensions Oct 21, 2026

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lunarwhite The runner guard and padding fix look consistent with their callers. Could you move test_compressed_tensors_fp8_moe_layout.py to test/registered/unit/layers/quantization/? It tests checkpoint loading, postprocessing and FusedMoE runner integration, so it belongs with runtime quantization tests under the current kernel layout. Please keep the real Triton/AITER numerical comparison and AMD CI registration; this coverage is useful.

With AITER enabled, explicit Triton CT CHANNEL FP8 layers receive
shuffled weights and the non-block FP8 kernel drops 128 reduction
elements. Select the shuffle from the resolved runner and match kernel
padding to the existing caller contract.

Add real weight-loading and numerical MoE tests for Triton and AITER.
MI355X controls show either partial fix still fails while both fixes
pass the unchanged accuracy thresholds.

Signed-off-by: Yuedong Wu <dwcn22@outlook.com>
@lunarwhite
lunarwhite force-pushed the lunarwhite/ct-fp8-runner-layout branch from f4b5bd7 to f663086 Compare October 28, 2026 05:58
@lunarwhite

Copy link
Copy Markdown
Contributor Author

@BBuf Thanks for your review! I've moved the file as per your ask. Please take a relook when you're around.

@haishuok0525

Copy link
Copy Markdown
Contributor

/tag-and-rerun-ci amd-extra

@github-actions github-actions Bot added run-ci CI: run the baseline test suite on this PR run-ci-amd-extra labels Oct 29, 2026

@BBuf BBuf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the runtime quantization test is now in the right place and retains the real Triton/AITER numerical coverage. The runner-specific shuffle guard and padding fix look good. Please wait for the relevant AMD CI before merging.

@BBuf
BBuf merged commit 196ee6b into sgl-project:main Oct 30, 2026
266 of 295 checks passed
@lunarwhite
lunarwhite deleted the lunarwhite/ct-fp8-runner-layout branch October 30, 2026 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jit-kernel run-ci CI: run the baseline test suite on this PR run-ci-amd-extra

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants