Repository navigation
[Fix] [AMD] MoE: Preserve CT FP8 Triton layout and reduction dimensions - #45486
Conversation
BBuf
left a comment
There was a problem hiding this comment.
@lunarwhite The runner guard and padding fix look consistent with their callers. Could you move test_compressed_tensors_fp8_moe_layout.py to test/registered/unit/layers/quantization/? It tests checkpoint loading, postprocessing and FusedMoE runner integration, so it belongs with runtime quantization tests under the current kernel layout. Please keep the real Triton/AITER numerical comparison and AMD CI registration; this coverage is useful.
With AITER enabled, explicit Triton CT CHANNEL FP8 layers receive shuffled weights and the non-block FP8 kernel drops 128 reduction elements. Select the shuffle from the resolved runner and match kernel padding to the existing caller contract. Add real weight-loading and numerical MoE tests for Triton and AITER. MI355X controls show either partial fix still fails while both fixes pass the unchanged accuracy thresholds. Signed-off-by: Yuedong Wu <dwcn22@outlook.com>
f4b5bd7 to
f663086
Compare
|
@BBuf Thanks for your review! I've moved the file as per your ask. Please take a relook when you're around. |
|
/tag-and-rerun-ci amd-extra |
BBuf
left a comment
There was a problem hiding this comment.
Thanks, the runtime quantization test is now in the right place and retains the real Triton/AITER numerical coverage. The runner-specific shuffle guard and padding fix look good. Please wait for the relevant AMD CI before merging.
…ns (sgl-project#45486) Signed-off-by: Yuedong Wu <dwcn22@outlook.com>
Motivation
With
SGLANG_USE_AITER=1and an explicit Triton MoE runner, compressed-tensors CHANNEL FP8 layers receive AITER-shuffled weights and lose 128 reduction elements in the non-block FP8 kernel (1024 → 896 in the regression). Both cause incorrect outputs.Modifications
test/registered/unit/layers/quantization/, covering checkpoint loading, postprocessing, and numerical Triton/AITER MoE comparisons with 1- and 33-token inputs.Related: #41636 addresses the shuffle mechanism in
Fp8MoEMethodblock-FP8; this PR fixes the separate compressed-tensors CHANNEL path and the non-block kernel dimension calculation.Accuracy Tests
Tested on MI355X (gfx950), Python 3.10.12, PyTorch
2.9.1+rocm7.2.0.git7e1940d4, HIP7.2.26015-fc0010cf6a, using the official image:Base:
9db58ab547516435b544ae050dd6c2f91b52bcf3Head:
7d7d8b79eec61eda1884c54311b2580bd6159cffLayer controls use identical seeded weights, routing and test source,
SGLANG_MOE_PADDING=0, isolated processes/caches, and verified import paths/hashes.Both final test methods pass without errors/skips. The independent reference uses dequantized checkpoint weights and identical routing; fixed thresholds are cosine > 0.99 and relative L2 < 0.1. Controls fail numerically, establishing that both fixes are required.
A separate Quark per-tensor dynamic regression covers another shared-kernel consumer: base/AITER off passes, base/AITER on fails both numerical subtests, and patched/AITER on passes (no errors/skips).
Reproduce the GPU-tested revisions on MI35x
Run inside the image above, from outside the source tree:
To reproduce the controls with the same test file (each command is a separate process):
The separate AITER-disabled + padding-enabled configuration is outside this validation.
Real-checkpoint validation
RedHatAI/Qwen3-30B-A3B-FP8-dynamic, revision10a439027a4fd228adc03a7bffb1aac10a179938, was evaluated on the same MI355X image and base/head sources. All four arms observed actual CT CHANNEL execution in all 48 layers and verified the loaded source hashes. TP1, eager execution, AITER enabled, padding disabled; 200 GSM8K test examples, five demonstrations from the train split, greedy decoding, maximum 512 generated tokens.All 800 predictions were independently rescored and prompt/dataset hashes verified. Every base Triton output hit the 512-token cap. This is a configuration-specific sanity check, not full-dataset accuracy or statistical equivalence; default AITER is the known-good reference.
Speed Tests and Profiling
A separate fixed-output probe (four prompts, 64 tokens each, three timed repeats after warmup) measured median output rates of 103.88 tok/s for base Triton and 104.75 tok/s for patched Triton; AITER reference runs measured 99.71 and 103.64 tok/s. This bounded probe is not a production throughput or speedup claim.
Checklist
Disclosure: Found by Hyperloom session analysis, reviewed and validated by human.
CI States
Latest PR Test (Base): ✅ Run #40641071798
Latest PR Test (Extra): ❌ Run #40641072021
Latest PR Test (AMD ROCm 10): ❌ Run #40641071972