Skip to content

[AMD] [GLM-5.3-Flash Day 0] Enable FP8 and Quark MXFP4 MoE on gfx950 - #38546

Merged
hnyls2002 merged 25 commits into
sgl-project:mainfrom
Raiden-Makoto:RM/glm53-moe-enablement-reopen
Sep 22, 2026
Merged

hnyls2002 merged 25 commits into
sgl-project:mainfrom
Raiden-Makoto:RM/glm53-moe-enablement-reopen

Conversation

@Raiden-Makoto

@Raiden-Makoto Raiden-Makoto commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Replacement for #38037, which GitHub closed when its deleted support base was merged into main.

Replacement for #37629, which was accidentally squash-merged and then reverted by #37880. This reapplies the same reviewed change on the current support-branch head.

GLM-5.3-Flash ships two gfx950 MoE checkpoint paths that share one model architecture but use different quantization contracts:

  • zai-org/GLM-5.3-Flash: routed, shared, and MTP experts use block FP8 [128,128] with dynamic FP8 activations.
  • amd/GLM-5.3-Flash-Quark-MXFP4: routed and shared experts in layers 3-44 use OCP MXFP4 1×32; MTP layer 45 remains block FP8.

Both paths load gate and up projections into separate contiguous halves and require clamped SwiGLU (swiglu_limit=10.0). The support branch preshuffles the weights but lets AITER infer gate/up semantics from the global interleave setting. Quark also drops clamp/padding metadata and does not make preshuffle ownership explicit.

Contract Before With this PR
block-FP8 layout tile-only preshuffle, no physical layout metadata records and forwards GateMode.SEPARATED
Quark MXFP4 layout global interleave default can override checkpoint ordering pins separated mode; AITER preserves explicit quant-method mode
SwiGLU FP8 forwards the limit; Quark omits it both forward swiglu_limit=10.0
preshuffle Quark can preshuffle without owning the selected runner requires an owned AITER runner
unsupported Quark runner fails later with an unset runner fails during construction
padding Quark padding is not forwarded hidden/intermediate padding is recorded and forwarded

Scope

Five production files:

  • fp8.py: preserve separated gate/up layout and forward block-FP8 clamp, scale, and padding metadata.
  • quark_w4a4_mxfp4_moe.py: own AITER preshuffle and forward MXFP4 clamp/layout/padding metadata.
  • aiter.py: preserve an explicitly selected gate mode.
  • quark.py: map per-layer Quark configuration names and dispatch serialized block-FP8 linear/MoE layers through the existing FP8 methods.
  • glm5_next.py: map Flash checkpoint names, including block-FP8 weight_scale to runtime weight_scale_inv.

Five test files cover CPU contracts, model-name/scale mapping, synthetic gfx950 numerics, and both real checkpoints. DSA, k-pool, MLA, mHC backend selection, CUDA graphs, EP/A2A, and scheduler policy are unchanged.

PR boundary: #39317 exclusively owns direct fused-module and per-expert Quark exclude handling in quark/utils.py. This PR no longer modifies that file or test_quark_utils.py. Full amd/GLM-5.3-Flash-Quark-MXFP4 serving requires both changes; they no longer duplicate code.

Test plan

Combined main-based integration (RM/glm53-day0-main-integration @ 22f233a853) on 4x MI355X/gfx950 using rocm/sgl-dev:v0.5.19-rocm724-mi35x-20260910:

  • amd/GLM-5.3-Flash-Quark-MXFP4@b5688f2549: 20/20 GSM8K (100.00%), 100.00% stop rate, zero truncations/errors.
  • zai-org/GLM-5.3-Flash@03eb536628 block FP8: 19/20 GSM8K (95.00%), 100.00% stop rate, zero truncations/errors.
  • Both checkpoints loaded all 62 shards, completed decode graph capture, and returned a correct stopped generation for 17 + 25 (42).

Recovery validation:

  • scoped pre-commit passed;
  • Quark mixed-precision configuration: 18 tests passed;
  • GLM block-FP8 scale-name mapping: 1 test passed;
  • existing MoE CPU, gfx950 synthetic, and checkpoint contracts remain covered by this PR.

Checkpoint exclusion behavior is intentionally tested in #39317, not duplicated here.


CI States

Latest PR Test (Base): ⏳ Run #35683916123
Latest PR Test (Extra): ❌ Run #35683915919
Latest PR Test (AMD ROCm 10): ⏳ Run #35683916078

Route the official uniform-MXFP4 expert bank through AITER with clamped SwiGLU, explicit separated layout ownership, and checkpoint-driven MoE-to-mHC correctness coverage.
Make the physical separated gate/up layout explicit for AITER and validate source FP8 routed, shared, and MTP experts alongside Quark MXFP4.
@Raiden-Makoto
Raiden-Makoto force-pushed the RM/glm53-moe-enablement-reopen branch from ef3955c to 73ed73c Compare September 8, 2026 19:42
@1am9trash 1am9trash added amd run-ci CI: run the baseline test suite on this PR labels Sep 9, 2026
Comment thread python/sglang/srt/models/glm5_next.py Resolved

@Jacob0226 Jacob0226 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

1am9trash pushed a commit that referenced this pull request Sep 17, 2026
Squashed from #38546, which carries the same change
against main and remains open there for review.

Co-Authored-By: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this test.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deleted

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remote this test

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deleted

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this test guard?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The gfx950 e2e builds AiterMoeQuantInfo by hand, so it never exercises
maybe_get_hip_aiter_quant_info or QuarkW4A4MXFp4MoE.apply_weights: deleting
either assembly leaves it green. Those two checks move to the fp8 and quark
unit files that already own those methods.

The other four cases went: the runner setdefault is covered end to end by
test_top1_and_top8_match_dequantized_oracle with real numerics, the preshuffle
guard tests a state create_moe_runner cannot produce, and the packed-shape case
re-asserts pre-existing MXFP4 math with intermediate_pad zero throughout.

kevin-mii and others added 4 commits September 21, 2026 19:56
…back

The block-FP8 probe ran before should_ignore_layer, so a layer named in the
checkpoint's exclude list could be quantized and excluded_fp8_config was never
consulted. Every other quantization config in the tree checks the skip list
first; compressed_tensors even gates fused expansion on it.

_get_block_fp8_config also skipped the output_tensors/bias refusal that
_get_scheme_from_config and get_moe_scheme both make, so a per_block config
with a quantized bias loaded through FP8 with those fields ignored instead of
raising NotImplementedError.

Reverting the Optional split removes the partially-specified ValueError, which
sat on the path every Quark checkpoint takes and could not fire before this
branch. With the global fallback restored no shard is ever None, and the probe
reads the same config the linear and MoE schemes do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Requested in review. The manual suite needed GLM53_FLASH_FP8_PATH and
GLM53_FLASH_MXFP4_PATH to do anything, so it never ran for anyone else; the
GSM8K results in the PR body already record what was checked against the real
checkpoints.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…files

The gfx950 e2e builds AiterMoeQuantInfo by hand, so it never exercises
maybe_get_hip_aiter_quant_info or QuarkW4A4MXFp4MoE.apply_weights: deleting
either assembly leaves it green. Those two checks move to the fp8 and quark
unit files that already own those methods.

The other four cases went: the runner setdefault is covered end to end by
test_top1_and_top8_match_dequantized_oracle with real numerics, the preshuffle
guard tests a state create_moe_runner cannot produce, and the packed-shape case
re-asserts pre-existing MXFP4 math with intermediate_pad zero throughout.

The fp8 case now builds Fp8MoEMethod through its real constructor instead of
object.__new__, matching the file it moved into.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@hnyls2002
hnyls2002 merged commit b44e248 into sgl-project:main Sep 22, 2026
93 of 127 checks passed
kevin-mii added a commit to zcnrex/sglang that referenced this pull request Sep 23, 2026
Conflict in moe_runner/aiter.py: main (sgl-project#38546) switched the swiglu_limit
gate_mode assignment to extra.setdefault so a gate_mode passed through
fused_moe_kwargs wins; this branch changed the condition so gfx95 honors
config.gate_up_interleaved. Kept both: setdefault with this branch's
condition, and condensed the two comments into one two-line comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Raiden-Makoto
Raiden-Makoto deleted the RM/glm53-moe-enablement-reopen branch September 24, 2026 23:42
hdt98 added a commit to hdt98/sglang that referenced this pull request Sep 25, 2026
Upstream sgl-project#38546/sgl-project#39317 rewrote quark.py and utils.py. Re-layer the parts of
this PR that upstream does not provide:

- _find_moe_config + num_routed_experts: per-expert config resolution for a
  fused MoE module, threaded into get_moe_scheme so the fnmatch sweep runs
  once per layer instead of twice (review comment 3).
- _get_block_fp8_config: keep the activation-grouping checks so an
  incompatible input_tensors spec is rejected, not silently reinterpreted.
- should_ignore_layer: materialize `ignore` once; it is typed Iterable and
  scanned several times, so a one-shot iterator tested empty after pass one.

Adopt upstream's exclusion contract (one excluded expert excludes the fused
module, sgl-project#39317) and update the affected test to assert it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants