Repository navigation
[AMD] [GLM-5.3-Flash Day 0] Route mHC through AITER on gfx950 - #38545
Conversation
61e1b8d to
584c734
Compare
Route GLM-5.3-Flash mHC through AITER with safe Torch fallback and add isolated production-dimension and checkpoint-driven coverage.
584c734 to
95dce2a
Compare
|
/rerun-failed-ci |
|
/rerun-failed-ci |
Jacob0226
left a comment
There was a problem hiding this comment.
LGTM. Production behavior changes are gated to the gfx95 + SGLANG_USE_AITER
Squashed from #38545, which carries the same change against main and remains open there for review. The one conflict in this integration lands here: amd_glm5.3_flash adds `get_platform` to the import block that #38545 extends with `is_gfx95_supported`, `is_hip` and `get_bool_env_var`. Resolved as the union; all five names are used in the merged file. Co-Authored-By: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
# Conflicts: # python/sglang/kernels/ops/layernorm/mhc.py
# Conflicts: # python/sglang/kernels/ops/layernorm/mhc.py
|
base-b: 19/20 jobs green on f37aca3, no failures. Last shard ( Noting it here since a cleanup commit will follow and move the head SHA. |
Production (mhc.py): - Drop _AITER_MHC_IMPORT_WARNED: an import failure sets the runtime latch, which turns the gate off, so the wrapper is never re-entered. - Drop is_hip() from the gate: is_gfx95_supported() is already False on every non-HIP build. - Read SGLANG_USE_AITER through envs, as the env-var convention requires for new call sites. - Give the post wrapper the pre wrapper's shape: the import and the kernel call each get their own try, so an unavailable symbol is not reported as a kernel failure. - Drop the noqa: BLE001 markers; the ruff hook only selects F401, F821, UP037. Tests: one file, test/registered/kernels/ops/layernorm/test_mhc_aiter_hip.py, registered on the mi35x lane, beside the CUDA-registered test_mhc_kernels.py. Nothing launches a server, so e2e/ was the wrong tree. Four cases, each guarding a failure nothing else in CI would catch: - the gate resolves True on real gfx950, else production silently serves Torch; - with AITER off, HIP stays on Torch and never loads TileLang -- the TileLang/DeepGEMM flags default on, so the HIP gates are load-bearing on every AMD run and no AMD-registered test loads an mHC model; - an import or kernel failure latches the route off instead of failing the request; - mhc_pre/mhc_post match the Torch oracle at GLM-5.3-Flash dimensions, with the fused norm as a parameter. Removed: the CPU dispatch file (its two HIP-only cases moved here and run unmocked; the other four exercised code this PR does not touch), the two-sublayer flow case (same kernel through one more wrapper, plus replay determinism, a vendor property), and the checkpoint-driven manual test (its tensor-identity case compares two HuggingFace checkpoints; no sglang diff can turn it red, and the evidence stays in the PR body). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
b06f77d to
15fe7b8
Compare
|
Pushed a squashed cleanup commit ( Three of its four cases (gate, HIP-never-loads-TileLang, failure latch) are new to hardware, so I dispatched the mi35x stage directly on this head rather than wait for stage A: https://github.com/sgl-project/sglang/actions/runs/35686604741 Will follow up here with the result. |
|
The dispatched mi35x run is green: https://github.com/sgl-project/sglang/actions/runs/35686604741
The PR's own |
Reset quark.py to upstream sgl-project#39779's per-expert fused config resolution and keep only the block-FP8 activation-grouping check. Fused shared experts are CUDA-only, so the removed shared-expert precision matching loses nothing on ROCm. Drop tests whose targets are now upstream code or duplicate upstream coverage: k-pool cache layout and backend dispatch (sgl-project#39341), TileLang zero-RoPE (sgl-project#38547), transform_index widths (sgl-project#39340), mHC HIP dispatch (sgl-project#38545), the NextN name mapper, write-plan multi-pool, negative-slot scatter, and the comment-only AITER runner. Keep one focused test per production change this branch owns.
Summary
Replacement for #37626, which GitHub closed when its deleted support base was merged into main.
GLM-5.3-Flash uses mHC in all 45 decoder layers (
hc_mult=4,hidden_size=4096). The support branch already carries the flattened mHC state contract andMHCLayerCommunicator, buthc_preandhc_posthave no AITER dispatch: enablingSGLANG_USE_AITER=1still leaves them on TileLang when its flags are set, or on the Torch fallback otherwise.This is a Day-0 correctness blocker on gfx950, not a performance-only path. Every attention and FFN sublayer crosses an mHC pre/post boundary before DSA, KDA, or MoE can produce a model output.
SGLANG_USE_AITER=1aiter.ops.mhc.mhc_pre/mhc_postThe AITER imports stay lazy because this module is imported during model-registry discovery. No AITER symbol is required until the gfx950 runtime actually selects this backend.
Scope
python/sglang/kernels/ops/layernorm/mhc.pyis the only production file changed.test/registered/kernels/ops/layernorm/test_mhc_aiter_hip.py): gate selects AITER, HIP with AITER off stays on Torch and never loads TileLang, import/runtime failures latch to Torch, andmhc_pre/mhc_postmatch the Torch oracle.The fused
mhc_post_preboundary is intentionally untouched. It is a later launch-fusion optimization; this PR enables the standalonemhc_post -> mhc_presequence required for correctness. DSA, k-pool, MLA, MoE, CUDA graphs, and full-model serving are outside this PR.Test plan
Combined main-based integration (
RM/glm53-day0-main-integrationtested at9d0d62b744; current integration head4e2f84ff3c; TP8/TP4 validation predates the above-cap guard, whose below-cap path is unchanged, pinned AITERd9e5ef7ce0, fresh caches): TP8 GSM8K 97.50% in 87.0s with 0.00% errors; TP4 GSM8K 96.97% in 320.2s with 0.00% errors. Decode CUDA graphs were enabled in both runs.Recovery base:
main@30e7a3072d.Recovery validation on MI355X: scoped pre-commit passed; 6 CPU, 3 gfx950, and 3 checkpoint tests passed at the time. The test set was since cut to one gfx950 file with 4 cases; the CPU dispatch file and the checkpoint-driven manual test were removed (see below).
Docker:
rocm/sgl-dev:v0.5.18-rocm720-mi35x-20260901.Base:
xinyuan/glm-5.3-flash-support@545bd6f839.The same test files were run in both arms;
mhc.pyis the only variable._try_aiter_mhc_pre/_try_aiter_mhc_postdo not exist. The checkpoint-only tensor audit passes.stage-b-test-1-gpu-small-amd-mi35x.The gfx950 file covers production
hidden_size=4096,hc_mult=4, token counts 1/8/17/32/64/128, 2 and 20 Sinkhorn iterations, norm fused/unfused, plus the gate, the HIP-never-TileLang path, and the failure latch, all unmocked on device.The checkpoint arm (one-time, test file removed) used
zai-org/GLM-5.3-Flashandamd/GLM-5.3-Flash-Quark-MXFP4@b5688f2549without loading either model. It verified all 270 mHC tensors are complete and byte-identical, ran every real attention and FFN parameter set across layers 0-44 through AITER against the Torch oracle, and ran the complete two-sublayer flow for every layer. The file was dropped because no sglang diff can turn it red; the evidence is recorded here.register_amd_ci(suite="stage-b-test-1-gpu-small-amd-mi35x")only; nothing registers on CPU or CUDApre-commitCI States
Latest PR Test (Base): ⏳ Run #35686411262
Latest PR Test (Extra): ❌ Run #35686410991
Latest PR Test (AMD ROCm 10): ⏳ Run #35686410943