Repository navigation
[AMD] [GLM-5.3-Flash Day 0] Enable FP8 and Quark MXFP4 MoE on gfx950 - #38546
Merged
hnyls2002 merged 25 commits intoSep 22, 2026
Merged
Conversation
Raiden-Makoto
requested review from
1am9trash,
Alisehen,
AniZpZ,
BBuf,
Edwardf0t1,
FlamingoPg,
Fridge003,
HaiShaw,
OrangeRedeng,
Ying1123,
b8zhong,
ch-wan,
hubertlu-tw,
ispobock,
kkHuang-amd,
merrymercy,
mmangkad and
yichiche
as code owners
September 8, 2026 17:26
Raiden-Makoto
marked this pull request as draft
September 8, 2026 18:04
Raiden-Makoto
marked this pull request as ready for review
September 8, 2026 19:28
Raiden-Makoto
force-pushed
the
RM/glm53-moe-enablement-reopen
branch
from
September 8, 2026 19:37
c01b955 to
ef3955c
Compare
Route the official uniform-MXFP4 expert bank through AITER with clamped SwiGLU, explicit separated layout ownership, and checkpoint-driven MoE-to-mHC correctness coverage.
Make the physical separated gate/up layout explicit for AITER and validate source FP8 routed, shared, and MTP experts alongside Quark MXFP4.
Raiden-Makoto
force-pushed
the
RM/glm53-moe-enablement-reopen
branch
from
September 8, 2026 19:42
ef3955c to
73ed73c
Compare
Jacob0226
reviewed
Sep 16, 2026
1am9trash
pushed a commit
that referenced
this pull request
Sep 17, 2026
Squashed from #38546, which carries the same change against main and remains open there for review. Co-Authored-By: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
hnyls2002
reviewed
Sep 22, 2026
Collaborator
There was a problem hiding this comment.
what does this test guard?
Collaborator
There was a problem hiding this comment.
The gfx950 e2e builds AiterMoeQuantInfo by hand, so it never exercises
maybe_get_hip_aiter_quant_info or QuarkW4A4MXFp4MoE.apply_weights: deleting
either assembly leaves it green. Those two checks move to the fp8 and quark
unit files that already own those methods.
The other four cases went: the runner setdefault is covered end to end by
test_top1_and_top8_match_dequantized_oracle with real numerics, the preshuffle
guard tests a state create_moe_runner cannot produce, and the packed-shape case
re-asserts pre-existing MXFP4 math with intermediate_pad zero throughout.
…back The block-FP8 probe ran before should_ignore_layer, so a layer named in the checkpoint's exclude list could be quantized and excluded_fp8_config was never consulted. Every other quantization config in the tree checks the skip list first; compressed_tensors even gates fused expansion on it. _get_block_fp8_config also skipped the output_tensors/bias refusal that _get_scheme_from_config and get_moe_scheme both make, so a per_block config with a quantized bias loaded through FP8 with those fields ignored instead of raising NotImplementedError. Reverting the Optional split removes the partially-specified ValueError, which sat on the path every Quark checkpoint takes and could not fire before this branch. With the global fallback restored no shard is ever None, and the probe reads the same config the linear and MoE schemes do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Requested in review. The manual suite needed GLM53_FLASH_FP8_PATH and GLM53_FLASH_MXFP4_PATH to do anything, so it never ran for anyone else; the GSM8K results in the PR body already record what was checked against the real checkpoints. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…files The gfx950 e2e builds AiterMoeQuantInfo by hand, so it never exercises maybe_get_hip_aiter_quant_info or QuarkW4A4MXFp4MoE.apply_weights: deleting either assembly leaves it green. Those two checks move to the fp8 and quark unit files that already own those methods. The other four cases went: the runner setdefault is covered end to end by test_top1_and_top8_match_dequantized_oracle with real numerics, the preshuffle guard tests a state create_moe_runner cannot produce, and the packed-shape case re-asserts pre-existing MXFP4 math with intermediate_pad zero throughout. The fp8 case now builds Fp8MoEMethod through its real constructor instead of object.__new__, matching the file it moved into. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hnyls2002
approved these changes
Sep 22, 2026
kevin-mii
added a commit
to zcnrex/sglang
that referenced
this pull request
Sep 23, 2026
Conflict in moe_runner/aiter.py: main (sgl-project#38546) switched the swiglu_limit gate_mode assignment to extra.setdefault so a gate_mode passed through fused_moe_kwargs wins; this branch changed the condition so gfx95 honors config.gate_up_interleaved. Kept both: setdefault with this branch's condition, and condensed the two comments into one two-line comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hdt98
added a commit
to hdt98/sglang
that referenced
this pull request
Sep 25, 2026
Upstream sgl-project#38546/sgl-project#39317 rewrote quark.py and utils.py. Re-layer the parts of this PR that upstream does not provide: - _find_moe_config + num_routed_experts: per-expert config resolution for a fused MoE module, threaded into get_moe_scheme so the fnmatch sweep runs once per layer instead of twice (review comment 3). - _get_block_fp8_config: keep the activation-grouping checks so an incompatible input_tensors spec is rejected, not silently reinterpreted. - should_ignore_layer: materialize `ignore` once; it is typed Iterable and scanned several times, so a one-shot iterator tested empty after pass one. Adopt upstream's exclusion contract (one excluded expert excludes the fused module, sgl-project#39317) and update the affected test to assert it.
3 of 4 tasks
This was referenced Sep 26, 2026
This was referenced Oct 31, 2026
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replacement for #38037, which GitHub closed when its deleted support base was merged into main.
Replacement for #37629, which was accidentally squash-merged and then reverted by #37880. This reapplies the same reviewed change on the current support-branch head.
GLM-5.3-Flash ships two gfx950 MoE checkpoint paths that share one model architecture but use different quantization contracts:
zai-org/GLM-5.3-Flash: routed, shared, and MTP experts use block FP8[128,128]with dynamic FP8 activations.amd/GLM-5.3-Flash-Quark-MXFP4: routed and shared experts in layers 3-44 use OCP MXFP4 1×32; MTP layer 45 remains block FP8.Both paths load gate and up projections into separate contiguous halves and require clamped SwiGLU (
swiglu_limit=10.0). The support branch preshuffles the weights but lets AITER infer gate/up semantics from the global interleave setting. Quark also drops clamp/padding metadata and does not make preshuffle ownership explicit.GateMode.SEPARATEDswiglu_limit=10.0Scope
Five production files:
fp8.py: preserve separated gate/up layout and forward block-FP8 clamp, scale, and padding metadata.quark_w4a4_mxfp4_moe.py: own AITER preshuffle and forward MXFP4 clamp/layout/padding metadata.aiter.py: preserve an explicitly selected gate mode.quark.py: map per-layer Quark configuration names and dispatch serialized block-FP8 linear/MoE layers through the existing FP8 methods.glm5_next.py: map Flash checkpoint names, including block-FP8weight_scaleto runtimeweight_scale_inv.Five test files cover CPU contracts, model-name/scale mapping, synthetic gfx950 numerics, and both real checkpoints. DSA, k-pool, MLA, mHC backend selection, CUDA graphs, EP/A2A, and scheduler policy are unchanged.
PR boundary: #39317 exclusively owns direct fused-module and per-expert Quark
excludehandling inquark/utils.py. This PR no longer modifies that file ortest_quark_utils.py. Fullamd/GLM-5.3-Flash-Quark-MXFP4serving requires both changes; they no longer duplicate code.Test plan
Combined main-based integration (
RM/glm53-day0-main-integration@22f233a853) on 4x MI355X/gfx950 usingrocm/sgl-dev:v0.5.19-rocm724-mi35x-20260910:amd/GLM-5.3-Flash-Quark-MXFP4@b5688f2549: 20/20 GSM8K (100.00%), 100.00% stop rate, zero truncations/errors.zai-org/GLM-5.3-Flash@03eb536628block FP8: 19/20 GSM8K (95.00%), 100.00% stop rate, zero truncations/errors.17 + 25(42).Recovery validation:
Checkpoint exclusion behavior is intentionally tested in #39317, not duplicated here.
CI States
Latest PR Test (Base): ⏳ Run #35683916123
Latest PR Test (Extra): ❌ Run #35683915919
Latest PR Test (AMD ROCm 10): ⏳ Run #35683916078