Repository navigation
Run GLM-5.3-Flash MI300X/MI325X block-FP8 linears on aiter's CK GEMM / GLM-5.3-Flash MI300X/MI325X 的 block-FP8 线性层改用 aiter CK GEMM - #5117
Open
fengshaoyi-amd wants to merge 2 commits into
Conversation
fengshaoyi-amd
requested review from
1am9trash,
billishyahao,
chunfangamd,
seungrokj and
zijiecode
as code owners
November 17, 2026 10:08
Contributor
|
Thanks for the contribution!
中文感谢你的贡献!
|
fengshaoyi-amd
added a commit
to fengshaoyi-amd/InferenceX
that referenced
this pull request
Nov 17, 2026
fengshaoyi-amd
force-pushed
the
glm53-gfx942-blockscale-ck
branch
from
November 17, 2026 10:11
90c58fb to
e45582e
Compare
Contributor
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
…le-ck Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # inferencex-e2e/perf-changelog.yaml
Collaborator
Author
|
/use 44072959411 |
Contributor
|
@fengshaoyi-amd staged run 44072959411.
This run remains available across future |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On gfx942, SGLang runs dense block-FP8 linears through aiter's Triton
gemm_a8w8_blockscale. In the image's aiter (e7d2453f2), four of GLM-5.3-Flash's six linear shapes at TP4 have no gfx942 Triton config. They run the 128x128 default tile, 6-14x slower than aiter's CK blockscale GEMM at decode token counts (MI325X, M = 8: 205 vs 15 us for N=1024 K=4096).This PR sets
SGLANG_AITER_GFX942_BLOCKSCALE_USE_CK=1(sgl-project/sglang#41071) in the GLM-5.3-Flash MI300X and MI325X AgentX recipes, which routes those linears to CK. It is interim: once aiter ships Triton configs for these shapes and the image picks them up, the default path catches up.Risk: sgl-project/sglang#41071 kept CK opt-in because Qwen3-32B-FP8 lost about 2 GSM8K points with it, for a reason it could not find. GLM-5.3-Flash shows no such loss (below). MI300X was not measured; it runs the same gfx942 kernels from the same image.
Results on MI325X, outside InferenceX CI: the MI325X recipe's c1 and c8 overrides, TP4 EP4, AgentX trace replay at InferenceX's golden acceptance length, 900 s at c1 and 1200 s at c8. The runs without the flag were one day earlier on the same node.
GSM8K, target model only (no MTP): lm-evaluation-harness
gsm8k, 5-shot as multi-turn chat, 1319 questions, greedy. 0.9719 (1282/1319) without the flag, 0.9742 (1285/1319) with it, the same under strict-match and flexible-extract. Paired by question, 3 questions are solved only without it and 6 only with it (exact McNemar p = 0.51).中文
在 gfx942 上,SGLang 默认用 aiter 的 Triton
gemm_a8w8_blockscale计算稠密 block-FP8 线性层。镜像自带的 aiter(e7d2453f2)里,GLM-5.3-Flash 在 TP4 下的 6 个线性层形状中有 4 个没有 gfx942 的 Triton 配置,只能用 128x128 的默认 tile。在 decode 规模下,它比 aiter 的 CK blockscale GEMM 慢 6–14 倍(MI325X,M = 8,N=1024 K=4096:205 us 对 15 us)。本 PR 在 GLM-5.3-Flash 的 MI300X 和 MI325X AgentX 配方中设置
SGLANG_AITER_GFX942_BLOCKSCALE_USE_CK=1(sgl-project/sglang#41071),让这些线性层改走 CK。这是过渡方案:等 aiter 为这些形状加上 Triton 配置、镜像也更新之后,默认路径就能追上。风险:sgl-project/sglang#41071 让 CK 保持可选,是因为 Qwen3-32B-FP8 开启后 GSM8K 下降约 2 分,原因未查明。GLM-5.3-Flash 没有出现这种下降:只跑目标模型时,GSM8K 从 0.9719 变为 0.9742,逐题配对检验 p = 0.51。MI300X 未实测,它和 MI325X 用同一镜像、同一套 gfx942 kernel。
MI325X 结果(在 InferenceX CI 之外测得,c1 / c8):P50 TPOT 从 19.9 / 25.8 ms 降到 15.8 / 20.5 ms,每卡输出吞吐从 13.4 / 41.9 升到 15.8 / 49.3 tok/s。
AI model disclosure