Skip to content

[AMD] [GLM-5.3-Flash Day 0] Route mHC through AITER on gfx950 - #38545

Merged
hnyls2002 merged 14 commits into
sgl-project:mainfrom
Raiden-Makoto:RM/glm53-enablement
Sep 22, 2026
Merged

hnyls2002 merged 14 commits into
sgl-project:mainfrom
Raiden-Makoto:RM/glm53-enablement

Conversation

@Raiden-Makoto

@Raiden-Makoto Raiden-Makoto commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Replacement for #37626, which GitHub closed when its deleted support base was merged into main.

GLM-5.3-Flash uses mHC in all 45 decoder layers (hc_mult=4, hidden_size=4096). The support branch already carries the flattened mHC state contract and MHCLayerCommunicator, but hc_pre and hc_post have no AITER dispatch: enabling SGLANG_USE_AITER=1 still leaves them on TileLang when its flags are set, or on the Torch fallback otherwise.

This is a Day-0 correctness blocker on gfx950, not a performance-only path. Every attention and FFN sublayer crosses an mHC pre/post boundary before DSA, KDA, or MoE can produce a model output.

Condition Before With this PR
HIP + gfx95 + SGLANG_USE_AITER=1 no AITER mHC route aiter.ops.mhc.mhc_pre / mhc_post
AITER symbol missing no probe one warning, latch disabled, Torch fallback
AITER runtime failure request failure one warning, latch disabled, Torch fallback
HIP + DeepGEMM mHC flag reaches NVIDIA-only prenorm path rejected by the backend gate
non-HIP TileLang/DeepGEMM existing optimized route unchanged

The AITER imports stay lazy because this module is imported during model-registry discovery. No AITER symbol is required until the gfx950 runtime actually selects this backend.

Scope

python/sglang/kernels/ops/layernorm/mhc.py is the only production file changed.

  • Add the HIP + gfx95 + AITER dispatch and failure latch.
  • Gate TileLang mHC pre/post and DeepGEMM prenorm to their non-HIP route.
  • Preserve the existing Torch, TileLang, gfx1250, communicator, model, and tensor contracts.
  • Add one gfx950 test file (test/registered/kernels/ops/layernorm/test_mhc_aiter_hip.py): gate selects AITER, HIP with AITER off stays on Torch and never loads TileLang, import/runtime failures latch to Torch, and mhc_pre/mhc_post match the Torch oracle.

The fused mhc_post_pre boundary is intentionally untouched. It is a later launch-fusion optimization; this PR enables the standalone mhc_post -> mhc_pre sequence required for correctness. DSA, k-pool, MLA, MoE, CUDA graphs, and full-model serving are outside this PR.

Test plan

Combined main-based integration (RM/glm53-day0-main-integration tested at 9d0d62b744; current integration head 4e2f84ff3c; TP8/TP4 validation predates the above-cap guard, whose below-cap path is unchanged, pinned AITER d9e5ef7ce0, fresh caches): TP8 GSM8K 97.50% in 87.0s with 0.00% errors; TP4 GSM8K 96.97% in 320.2s with 0.00% errors. Decode CUDA graphs were enabled in both runs.

Recovery base: main @ 30e7a3072d.
Recovery validation on MI355X: scoped pre-commit passed; 6 CPU, 3 gfx950, and 3 checkpoint tests passed at the time. The test set was since cut to one gfx950 file with 4 cases; the CPU dispatch file and the checkpoint-driven manual test were removed (see below).

Docker: rocm/sgl-dev:v0.5.18-rocm720-mi35x-20260901.
Base: xinyuan/glm-5.3-flash-support @ 545bd6f839.
The same test files were run in both arms; mhc.py is the only variable.

Arm Result
Baseline CPU dispatch: 1 passed, 5 errors; every AITER-specific gfx950/checkpoint case fails because _try_aiter_mhc_pre / _try_aiter_mhc_post do not exist. The checkpoint-only tensor audit passes.
This PR CPU dispatch: 6 passed; isolated gfx950: 3 passed; checkpoint-driven: 3 passed. Not re-executed after the test cut: the surviving gfx950 file runs in stage-b-test-1-gpu-small-amd-mi35x.

The gfx950 file covers production hidden_size=4096, hc_mult=4, token counts 1/8/17/32/64/128, 2 and 20 Sinkhorn iterations, norm fused/unfused, plus the gate, the HIP-never-TileLang path, and the failure latch, all unmocked on device.

The checkpoint arm (one-time, test file removed) used zai-org/GLM-5.3-Flash and amd/GLM-5.3-Flash-Quark-MXFP4@b5688f2549 without loading either model. It verified all 270 mHC tensors are complete and byte-identical, ran every real attention and FFN parameter set across layers 0-44 through AITER against the Torch oracle, and ran the complete two-sublayer flow for every layer. The file was dropped because no sglang diff can turn it red; the evidence is recorded here.

Check Detail
Registration register_amd_ci(suite="stage-b-test-1-gpu-small-amd-mi35x") only; nothing registers on CPU or CUDA
pre-commit all hooks pass on all four files, including registered-test CI validation
Accuracy not measured — GLM-5.3-Flash still depends on the remaining Day-0 enablement PRs before server-level accuracy is meaningful
Speed not measured — this PR establishes standalone mHC correctness and does not add post/pre fusion

CI States

Latest PR Test (Base): ⏳ Run #35686411262
Latest PR Test (Extra): ❌ Run #35686410991
Latest PR Test (AMD ROCm 10): ⏳ Run #35686410943

@Raiden-Makoto
Raiden-Makoto marked this pull request as draft September 8, 2026 18:04
@Raiden-Makoto
Raiden-Makoto marked this pull request as ready for review September 8, 2026 19:28
@1am9trash 1am9trash added amd run-ci CI: run the baseline test suite on this PR labels Sep 9, 2026
@Raiden-Makoto

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@Raiden-Makoto

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@Jacob0226 Jacob0226 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Production behavior changes are gated to the gfx95 + SGLANG_USE_AITER

1am9trash pushed a commit that referenced this pull request Sep 17, 2026
Squashed from #38545, which carries the same change
against main and remains open there for review.

The one conflict in this integration lands here: amd_glm5.3_flash adds
`get_platform` to the import block that #38545 extends with
`is_gfx95_supported`, `is_hip` and `get_bool_env_var`. Resolved as the
union; all five names are used in the merged file.

Co-Authored-By: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@kevin-mii

Copy link
Copy Markdown
Collaborator

base-b: 19/20 jobs green on f37aca3, no failures. Last shard (base-b-test-1-gpu-large (3)) is still queued on runner availability: https://github.com/sgl-project/sglang/actions/runs/35673091512

Noting it here since a cleanup commit will follow and move the head SHA.

Production (mhc.py):
- Drop _AITER_MHC_IMPORT_WARNED: an import failure sets the runtime latch,
  which turns the gate off, so the wrapper is never re-entered.
- Drop is_hip() from the gate: is_gfx95_supported() is already False on every
  non-HIP build.
- Read SGLANG_USE_AITER through envs, as the env-var convention requires for
  new call sites.
- Give the post wrapper the pre wrapper's shape: the import and the kernel
  call each get their own try, so an unavailable symbol is not reported as a
  kernel failure.
- Drop the noqa: BLE001 markers; the ruff hook only selects F401, F821, UP037.

Tests: one file, test/registered/kernels/ops/layernorm/test_mhc_aiter_hip.py,
registered on the mi35x lane, beside the CUDA-registered test_mhc_kernels.py.
Nothing launches a server, so e2e/ was the wrong tree. Four cases, each
guarding a failure nothing else in CI would catch:
- the gate resolves True on real gfx950, else production silently serves Torch;
- with AITER off, HIP stays on Torch and never loads TileLang -- the
  TileLang/DeepGEMM flags default on, so the HIP gates are load-bearing on
  every AMD run and no AMD-registered test loads an mHC model;
- an import or kernel failure latches the route off instead of failing the
  request;
- mhc_pre/mhc_post match the Torch oracle at GLM-5.3-Flash dimensions, with
  the fused norm as a parameter.

Removed: the CPU dispatch file (its two HIP-only cases moved here and run
unmocked; the other four exercised code this PR does not touch), the
two-sublayer flow case (same kernel through one more wrapper, plus replay
determinism, a vendor property), and the checkpoint-driven manual test (its
tensor-identity case compares two HuggingFace checkpoints; no sglang diff can
turn it red, and the evidence stays in the PR body).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@kevin-mii

Copy link
Copy Markdown
Collaborator

Pushed a squashed cleanup commit (15fe7b8): trims the AITER dispatch wrappers in mhc.py and collapses the tests into one gfx950 file, test/registered/kernels/ops/layernorm/test_mhc_aiter_hip.py, registered on stage-b-test-1-gpu-small-amd-mi35x. Details in the commit message and the updated PR body.

Three of its four cases (gate, HIP-never-loads-TileLang, failure latch) are new to hardware, so I dispatched the mi35x stage directly on this head rather than wait for stage A:

https://github.com/sgl-project/sglang/actions/runs/35686604741

Will follow up here with the result.

@kevin-mii

Copy link
Copy Markdown
Collaborator

The dispatched mi35x run is green: https://github.com/sgl-project/sglang/actions/runs/35686604741

stage-b-test-1-gpu-small-amd-mi35x (rocm10, linux-mi35x-gpu-1) picked up test/registered/kernels/ops/layernorm/test_mhc_aiter_hip.py at its new path and ran all four cases (Ran 4 tests, 11s wall against an est_time of 120). That covers the three cases that were new to hardware: gate selects AITER on gfx950, HIP with AITER off stays on Torch without loading TileLang, and import/runtime failures latch to Torch.

The PR's own pr-test-amd run on this head still has to reach the same job after stage A; this was an early answer, not a replacement.

@hnyls2002
hnyls2002 merged commit 095e451 into sgl-project:main Sep 22, 2026
108 of 133 checks passed
@Raiden-Makoto
Raiden-Makoto deleted the RM/glm53-enablement branch September 24, 2026 23:42
hdt98 added a commit to hdt98/sglang that referenced this pull request Sep 25, 2026
Reset quark.py to upstream sgl-project#39779's per-expert fused config resolution and
keep only the block-FP8 activation-grouping check. Fused shared experts are
CUDA-only, so the removed shared-expert precision matching loses nothing on
ROCm.

Drop tests whose targets are now upstream code or duplicate upstream
coverage: k-pool cache layout and backend dispatch (sgl-project#39341), TileLang
zero-RoPE (sgl-project#38547), transform_index widths (sgl-project#39340), mHC HIP dispatch
(sgl-project#38545), the NextN name mapper, write-plan multi-pool, negative-slot
scatter, and the comment-only AITER runner. Keep one focused test per
production change this branch owns.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd jit-kernel run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants