Skip to content

Add GLM-5.3-Flash FP8 MI325X AgentX with HiCache / 新增 GLM-5.3-Flash FP8 MI325X AgentX(含 HiCache) - #4722

Merged
Oseltamivir merged 19 commits into
mainfrom
feat/glm53flash-fp8-mi325x-sglang
Nov 5, 2026
Merged

Oseltamivir merged 19 commits into
mainfrom
feat/glm53flash-fp8-mi325x-sglang

Conversation

@functionstackx

@functionstackx functionstackx commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Adds the GLM-5.3-Flash native FP8 AgentX curve on MI325X as one config key, glm5.3flash-fp8-mi325x-sglang-agentic-mtp. The curve runs on the official lmsysorg/sglang:v0.5.22-rocm10-mi30x image, pinned by digest sha256:5658cd0d…. This PR merges the base sweep (this PR) with the HiCache comparison from #4735 into a single curve: GPU-resident KV at low concurrency and SGLang HiCache (CPU DRAM KV offload) at high concurrency. Dominated points are dropped.

arm conc KV expected tput/GPU (tok/s) expected p90 E2EL (s) expected median interactivity (tok/s/user) source
base 1, 2, 4, 8 GPU 1746, 2050, 3224, 6776 64.4, 58.1, 75.0, 44.1 55.8, 52.2, 48.0, 39.6 run 41308067217 (v0.5.21 image)
base 12 GPU 8428 66.1 38.2 run 41352867682 cache-off arm
HiCache 16, 24, 32 GPU + CPU DRAM 9963, 14064, 16101 73.0, 95.6, 126.2 34.7, 26.1, 22.6 run 41352867682 cache-on arm
  • Expected Pareto coverage: 5 points on the throughput vs p90 E2EL frontier (c8, c12, c16, c24, c32, computed with infx.workflows.pareto_coverage). All 8 points are on the throughput vs interactivity frontier.
  • Dropped points: cache-off c16, c24 and c32 are dominated by HiCache, which gives +3%, +5% and +12% throughput at lower p90. Cache-on c12 ties cache-off c12 (8478 vs 8428 tok/s/GPU, p90 66.4 vs 66.1 s), so I kept the variant that uses no CPU DRAM.
  • All 8 points are estimates until this sweep finishes:
    • Both source runs set AIPERF_APPLY_CHAT_TEMPLATE=true. This PR removes it so that every GLM-5.3-Flash SKU measures with the same client.
    • c1–c8 were measured on v0.5.21. Where both images were measured (cache-off c12 and c16), moving from v0.5.21 to v0.5.22 changed throughput by ≤1%, interactivity by +4–10% and p90 E2EL by +3–9%.
    • Each chosen point otherwise uses the exact recipe flags it was measured with.

Recipe

CPU DRAM for KV offload

  • Measured allocation: the HiCache arm allocates 988–1045 GB per node. That is below the MODELS.md proportional cap of 1208.6 GB (2,305,165 MiB available on the node × 4/8 GPUs).
  • What bounds it: usage is bounded by the default hicache-ratio of 2.0, not by the 0.60 fraction. The ceiling from the fraction alone would be 4 × 319.8 GiB = 1373 GB.
  • Source of the figures: the server logs of run 41352867682. Each of the four ranks allocates the following (1e9-byte GB):
conc KV host pool Mamba host DSA indexer host per rank per node
16 194.43 60.52 6.27 261.22 1044.9 GB
24 187.56 60.52 6.04 254.12 1016.5 GB
32 180.70 60.52 5.82 247.04 988.2 GB
  • Host tier size: exactly 2× the device KV pool. At c32 that is 14,705,408 host tokens vs 7,352,576 device tokens.
  • Recorded value: dram-utilization: 0.87 records 1051 GB (2,305,165 MiB × 0.87 × 4/8). That covers the largest measured allocation (c16) instead of the policy maximum.

Exceptions and harness-only settings

  • No SGLang cookbook entry (recorded exception): MI325X has no SGLang cookbook entry for GLM-5.3-Flash, so this recipe keeps the existing Add GLM-5.3-Flash FP8 MI325X AgentX with HiCache / 新增 GLM-5.3-Flash FP8 MI325X AgentX(含 HiCache) #4722/[DNM, Experimental] GLM-5.3-Flash MI325X FP8 HiCache and concurrency / [DNM, Experimental] GLM-5.3-Flash MI325X FP8 HiCache与并发优化 #4735 MI325X settings. The closest published cell is MI355X FP8 Low Latency. These settings match it: TP4/EP4, quantization fp8, page-size 64, mem-fraction-static 0.82, chunked/max prefill 8192, TileLang DSA, linear-attn-verify triton, AITER MoE, EAGLE 5/1/6 with attention mode prefill and a2a none, the glm45/glm47 parsers, the watchdog/dist timeouts, SGLANG_USE_AITER=1, AITER_ONLINE_TUNE=0 and SGLANG_DSA_FUSE_TOPK=0.
  • Not in the MI355X cell (MI325X/gfx942-specific):
    • BF16 KV (the cookbook's BF16 + TileLang overlay)
    • dsa-topk-backend torch and linear-attn-backend triton
    • SGLANG_OPT_USE_TOPK_V2=false and SGLANG_OPT_FUSE_MHC_POST_PRE=0
    • the 2×CONC admission/graph caps
    • the HiCache flags above
  • Checkpoint staging (stage-hf-checkpoint.sh) is harness-only:
    • Why it is needed: the MI325X cluster has no pre-staged GLM-5.3-Flash checkpoint and no models inventory entry.
    • What it does: under a per-repository lock, the worker setup downloads pinned revision 46a050bec13002ad29ee795c16e61cf4fa3c4a9c into the cluster's shared Hugging Face cache, with a 400 GB disk budget and no deletion. Serving then stays offline on that snapshot.
    • Tests: infx/tests/srt_slurm/test_checkpoint_staging.py exercises the script against a fake Hub.
    • Duplication: H100 Add GLM-5.3-Flash FP8 H100 SGLang AgentX with HiCache / 新增 GLM-5.3-Flash FP8 H100 SGLang AgentX(含 HiCache) #4717 carries the same two files byte for byte. Whichever PR merges second drops its copy.
  • Other harness-only settings:
    • HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE, PYTHONNOUSERSITE and SGLANG_TIMEOUT_KEEP_ALIVE
    • enable-metrics
    • served-model-name, revision and tokenizer-path
    • the AIPERF_* client timeouts and metric prefix
    • health_check
  • Accuracy: in the source runs GSM8K scored 97.42% (cache-off c16, run 41308067217) and 97.27% (cache-on c32, run 41352867682). This sweep's evaluation runs at HiCache c32.
  • Request errors in the source runs: InvalidInferenceResultError records fall almost entirely in the warmup window (1–4 per point remain in the profiled window).

Validation: the full sweep for this head is pending and will be linked here.

中文

本 PR 以单一配置键 glm5.3flash-fp8-mi325x-sglang-agentic-mtp 添加 MI325X 上 GLM-5.3-Flash 原生 FP8 AgentX 曲线,使用以 digest 固定的官方 lmsysorg/sglang:v0.5.22-rocm10-mi30x 镜像。本 PR 将自身的基础扫描与 #4735 的 HiCache 对比合并为一条曲线:低并发(1、2、4、8、12)使用 GPU 内 KV,高并发(16、24、32)使用 SGLang HiCache(CPU DRAM KV offload),并去掉被支配的点。预期 p90 E2EL 前沿为 5 个点,交互性前沿为全部 8 个点。

在本次扫描完成前,8 个预期点均为估计值:两次源运行都设置了 AIPERF_APPLY_CHAT_TEMPLATE=true,本 PR 已移除该设置,使所有 GLM-5.3-Flash SKU 采用相同的客户端;c1–c8 测于 v0.5.21,在两个镜像均有测量的并发点上,v0.5.21→v0.5.22 吞吐变化 ≤1%,交互性 +4–10%,p90 E2EL +3–9%。

HiCache 实测每节点主机内存分配为 988–1045 GB,低于 MODELS.md 按 GPU 比例计算的上限 1208.6 GB;用量由默认 hicache-ratio 2.0 决定(主机层恰为设备 KV 池的 2 倍),而非 0.60 fraction(仅按 fraction 的上限为 1373 GB)。dram-utilization: 0.87 记录 1051 GB,覆盖实测最大值(c16,4 × 261.22 GB)。

MI325X 没有 GLM-5.3-Flash 的 SGLang cookbook 条目,作为已记录的例外保留 #4722/#4735 的 MI325X 配置;与最接近的 MI355X FP8 cookbook 配置的对照见英文部分。stage-hf-checkpoint.sh 仅属测试框架:MI325X 集群没有预先部署的 checkpoint,worker setup 在仓库级文件锁下将固定 revision 下载到共享缓存(400 GB 磁盘预算,不删除文件),服务保持离线;H100 #4717 含有完全相同的文件,后合并者删除重复。draft 按原样以出厂精度提供,SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0,ROCM_QUICK_REDUCE_QUANTIZATION=NONE,无引擎补丁。

AI model disclosure

@functionstackx
functionstackx marked this pull request as ready for review November 1, 2026 11:37
@functionstackx functionstackx added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 1, 2026
@claude

claude Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @functionstackx's task in 45s —— View job


Review: PR #4722

No blocking issues. There is one 🟡 warning, posted inline: glm5.3flash is a new model prefix, but there is no row for it in inferencex-e2e/docs/MODELS.md. It is missing on this branch and on the base branch (#4708).

Checks that passed:

  • Changelog: the new entry is at the end of perf-changelog.yaml and covers glm5.3flash-fp8-mi325x-sglang-agentic-mtp.
  • Image: the config uses a public lmsysorg/sglang image pinned to a digest.
  • Expert parallelism and chat template: the SGLang recipe sets expert-parallel-size explicitly, and AIPERF_APPLY_CHAT_TEMPLATE=true is set for the MTP run.
  • Checkpoint download: MI325X has no staged copy, so the model resolves to hf:zai-org/GLM-5.3-Flash. The download and engine load both pin the same revision.
  • Staging script: stage-hf-checkpoint.sh holds a per-repository lock while it checks disk space and downloads.

I couldn't run pytest on the new staging test because the command wasn't permitted in this environment.

glm5.3flash-fp8-mi325x-sglang-agentic-mtp:
image: lmsysorg/sglang:v0.5.21-rocm10-mi30x@sha256:4420a7e492fce623285c635b9a6a7e09ccf77a3c812006494e4414655d46662d
model: zai-org/GLM-5.3-Flash
model-prefix: glm5.3flash

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 WARNING: New model prefix glm5.3flash is not in inferencex-e2e/docs/MODELS.md. It is not on this branch or on the base branch (#4708). MODELS.md currently lists only glm5.3.

Why it matters: MODELS.md is the source of truth for supported model prefixes. Without an entry, this is the first active master config for a model the docs don't list. That breaks the model and deprecation validation that relies on that file.

Fix: Add a GLM-5.3-Flash row to the model table in inferencex-e2e/docs/MODELS.md, next to the glm5.3 row at line 130. Give it the prefix glm5.3flash, the date and PR link, and the scenario Agentic coding (MTP only, per the deprecation policy). You can also add it in #4708 if that PR should own the model introduction.

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

@Oseltamivir

Copy link
Copy Markdown
Collaborator

/use 41308067217

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

@Oseltamivir staging run 41308067217. Existing staged runs will be preserved; a completion link will be posted here.

Match the image of the MI325X HiCache comparison (#4735) so the cache-off
line of both runs shares one image and the runs form one curve.
@Oseltamivir Oseltamivir added full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) and removed full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) labels Nov 5, 2026
@Oseltamivir
Oseltamivir changed the base branch from feat/glm53flash-golden-al to main November 5, 2026 11:25
@Oseltamivir Oseltamivir changed the title [DNM, Experimental] Add native GLM-5.3-Flash FP8 MI325X / [DNM, Experimental] 添加原生 GLM-5.3-Flash FP8 MI325X Add GLM-5.3-Flash FP8 MI325X AgentX with HiCache / 新增 GLM-5.3-Flash FP8 MI325X AgentX(含 HiCache) Nov 5, 2026
@Oseltamivir Oseltamivir added full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) and removed full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) labels Nov 5, 2026
@Oseltamivir

Copy link
Copy Markdown
Collaborator

/use 42022933758

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

@Oseltamivir staged run 42022933758.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

…age size 256

SGLang v0.5.22 forces page size 256 for DeepSeek DSA, so the recipe now
states the value that runs.
@Oseltamivir
Oseltamivir merged commit 29424e3 into main Nov 5, 2026
5 checks passed
@Oseltamivir
Oseltamivir deleted the feat/glm53flash-fp8-mi325x-sglang branch November 5, 2026 16:18
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category new duplicate skipped unmapped failed
benchmark 8 0 8 0 0
eval 1 1 1 0 0
trace 8 0 0 0 0
power 8 0 0 0 0

benchmarkFiles 9, rawBenchmarkRows 16, coveredByPerJobFiles 8, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 1319, gpuMetricSamplesNew 165344, powerPublicationPoints 8, changelogEntriesWritten 1

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants