Skip to content

Run GLM-5.3-Flash MI300X/MI325X block-FP8 linears on aiter's CK GEMM / GLM-5.3-Flash MI300X/MI325X 的 block-FP8 线性层改用 aiter CK GEMM - #5117

Open
fengshaoyi-amd wants to merge 2 commits into
SemiAnalysisAI:mainfrom
fengshaoyi-amd:glm53-gfx942-blockscale-ck
Open

fengshaoyi-amd wants to merge 2 commits into
SemiAnalysisAI:mainfrom
fengshaoyi-amd:glm53-gfx942-blockscale-ck

Conversation

@fengshaoyi-amd

@fengshaoyi-amd fengshaoyi-amd commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

On gfx942, SGLang runs dense block-FP8 linears through aiter's Triton gemm_a8w8_blockscale. In the image's aiter (e7d2453f2), four of GLM-5.3-Flash's six linear shapes at TP4 have no gfx942 Triton config. They run the 128x128 default tile, 6-14x slower than aiter's CK blockscale GEMM at decode token counts (MI325X, M = 8: 205 vs 15 us for N=1024 K=4096).

This PR sets SGLANG_AITER_GFX942_BLOCKSCALE_USE_CK=1 (sgl-project/sglang#41071) in the GLM-5.3-Flash MI300X and MI325X AgentX recipes, which routes those linears to CK. It is interim: once aiter ships Triton configs for these shapes and the image picks them up, the default path catches up.

Risk: sgl-project/sglang#41071 kept CK opt-in because Qwen3-32B-FP8 lost about 2 GSM8K points with it, for a reason it could not find. GLM-5.3-Flash shows no such loss (below). MI300X was not measured; it runs the same gfx942 kernels from the same image.

Results on MI325X, outside InferenceX CI: the MI325X recipe's c1 and c8 overrides, TP4 EP4, AgentX trace replay at InferenceX's golden acceptance length, 900 s at c1 and 1200 s at c8. The runs without the flag were one day earlier on the same node.

concurrency 1 concurrency 8
P50 TPOT (ms) 19.9 -> 15.8 (0.80x) 25.8 -> 20.5 (0.79x)
Output tok/s per GPU 13.4 -> 15.8 41.9 -> 49.3
P50 interactivity (tok/s/user) 38.8 -> 50.6 34.5 -> 42.0
P50 TTFT (s) 1.10 -> 0.79 0.59 -> 0.58

GSM8K, target model only (no MTP): lm-evaluation-harness gsm8k, 5-shot as multi-turn chat, 1319 questions, greedy. 0.9719 (1282/1319) without the flag, 0.9742 (1285/1319) with it, the same under strict-match and flexible-extract. Paired by question, 3 questions are solved only without it and 6 only with it (exact McNemar p = 0.51).

中文

在 gfx942 上,SGLang 默认用 aiter 的 Triton gemm_a8w8_blockscale 计算稠密 block-FP8 线性层。镜像自带的 aiter(e7d2453f2)里,GLM-5.3-Flash 在 TP4 下的 6 个线性层形状中有 4 个没有 gfx942 的 Triton 配置,只能用 128x128 的默认 tile。在 decode 规模下,它比 aiter 的 CK blockscale GEMM 慢 6–14 倍(MI325X,M = 8,N=1024 K=4096:205 us 对 15 us)。

本 PR 在 GLM-5.3-Flash 的 MI300X 和 MI325X AgentX 配方中设置 SGLANG_AITER_GFX942_BLOCKSCALE_USE_CK=1(sgl-project/sglang#41071),让这些线性层改走 CK。这是过渡方案:等 aiter 为这些形状加上 Triton 配置、镜像也更新之后,默认路径就能追上。

风险:sgl-project/sglang#41071 让 CK 保持可选,是因为 Qwen3-32B-FP8 开启后 GSM8K 下降约 2 分,原因未查明。GLM-5.3-Flash 没有出现这种下降:只跑目标模型时,GSM8K 从 0.9719 变为 0.9742,逐题配对检验 p = 0.51。MI300X 未实测,它和 MI325X 用同一镜像、同一套 gfx942 kernel。

MI325X 结果(在 InferenceX CI 之外测得,c1 / c8):P50 TPOT 从 19.9 / 25.8 ms 降到 15.8 / 20.5 ms,每卡输出吞吐从 13.4 / 41.9 升到 15.8 / 49.3 tok/s。

AI model disclosure
  • Claude Opus 5.5, in Cursor: investigation, the MI325X measurements, the recipe and changelog change, and this description.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

fengshaoyi-amd added a commit to fengshaoyi-amd/InferenceX that referenced this pull request Nov 17, 2026
@fengshaoyi-amd fengshaoyi-amd changed the title Run GLM-5.3-Flash MI300X/MI325X block-FP8 linears on aiter's CK GEMM / GLM-5.3-Flash MI300X/MI325X 的 block-FP8 线性层改用 aiter CK GEMM Run GLM-5.3-Flash MI300X/MI325X block-FP8 linears on aiter's CK GEMM Nov 17, 2026
@fengshaoyi-amd
fengshaoyi-amd force-pushed the glm53-gfx942-blockscale-ck branch from 90c58fb to e45582e Compare November 17, 2026 10:11
@fengshaoyi-amd fengshaoyi-amd changed the title Run GLM-5.3-Flash MI300X/MI325X block-FP8 linears on aiter's CK GEMM Run GLM-5.3-Flash MI300X/MI325X block-FP8 linears on aiter's CK GEMM / GLM-5.3-Flash MI300X/MI325X 的 block-FP8 线性层改用 aiter CK GEMM Nov 17, 2026
@Fangzhou-Ai Fangzhou-Ai added the full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures label Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Dispatched trusted sweep run for approved external revision e45582e2f46f1ee95f6884a3812596b0cf488167.
New commits are not trusted automatically; remove and re-add the primary sweep label to approve a new SHA.

已为获批的外部提交 e45582e2f46f1ee95f6884a3812596b0cf488167 调度trusted sweep run。
后续新提交不会自动获得信任;如需批准新的 SHA,请移除并重新添加主扫描标签。

…le-ck

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@fengshaoyi-amd

Copy link
Copy Markdown
Collaborator Author

/use 44072959411

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

@fengshaoyi-amd staged run 44072959411.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants