Repository navigation
Add GLM-5.3-Flash FP8 MI325X AgentX with HiCache / 新增 GLM-5.3-Flash FP8 MI325X AgentX(含 HiCache) - #4722
Conversation
|
Claude finished @functionstackx's task in 45s —— View job Review: PR #4722No blocking issues. There is one 🟡 warning, posted inline: Checks that passed:
I couldn't run |
| glm5.3flash-fp8-mi325x-sglang-agentic-mtp: | ||
| image: lmsysorg/sglang:v0.5.21-rocm10-mi30x@sha256:4420a7e492fce623285c635b9a6a7e09ccf77a3c812006494e4414655d46662d | ||
| model: zai-org/GLM-5.3-Flash | ||
| model-prefix: glm5.3flash |
There was a problem hiding this comment.
🟡 WARNING: New model prefix glm5.3flash is not in inferencex-e2e/docs/MODELS.md. It is not on this branch or on the base branch (#4708). MODELS.md currently lists only glm5.3.
Why it matters: MODELS.md is the source of truth for supported model prefixes. Without an entry, this is the first active master config for a model the docs don't list. That breaks the model and deprecation validation that relies on that file.
Fix: Add a GLM-5.3-Flash row to the model table in inferencex-e2e/docs/MODELS.md, next to the glm5.3 row at line 130. Give it the prefix glm5.3flash, the date and PR link, and the scenario Agentic coding (MTP only, per the deprecation policy). You can also add it in #4708 if that PR should own the model introduction.
|
Thanks for the contribution!
中文感谢你的贡献!
|
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42022933758 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42022933758 |
|
/use 41308067217 |
|
@Oseltamivir staging run 41308067217. Existing staged runs will be preserved; a completion link will be posted here. |
Match the image of the MI325X HiCache comparison (#4735) so the cache-off line of both runs shares one image and the runs form one curve.
|
/use 42022933758 |
|
@Oseltamivir staged run 42022933758. This run remains available across future |
…age size 256 SGLang v0.5.22 forces page size 256 for DeepSeek DSA, so the recipe now states the value that runs.
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 9, rawBenchmarkRows 16, coveredByPerJobFiles 8, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 1319, gpuMetricSamplesNew 165344, powerPublicationPoints 8, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Adds the GLM-5.3-Flash native FP8 AgentX curve on MI325X as one config key,
glm5.3flash-fp8-mi325x-sglang-agentic-mtp. The curve runs on the officiallmsysorg/sglang:v0.5.22-rocm10-mi30ximage, pinned by digestsha256:5658cd0d…. This PR merges the base sweep (this PR) with the HiCache comparison from #4735 into a single curve: GPU-resident KV at low concurrency and SGLang HiCache (CPU DRAM KV offload) at high concurrency. Dominated points are dropped.infx.workflows.pareto_coverage). All 8 points are on the throughput vs interactivity frontier.AIPERF_APPLY_CHAT_TEMPLATE=true. This PR removes it so that every GLM-5.3-Flash SKU measures with the same client.Recipe
glm-5.3-flash__mtp__native__default-defaulton main (MTP5 = 3.60).max-running-requestsandcuda-graph-max-bs-decodeare 2×CONC. These come from the existing Add GLM-5.3-Flash FP8 MI325X AgentX with HiCache / 新增 GLM-5.3-Flash FP8 MI325X AgentX(含 HiCache) #4722/[DNM, Experimental] GLM-5.3-Flash MI325X FP8 HiCache and concurrency / [DNM, Experimental] GLM-5.3-Flash MI325X FP8 HiCache与并发优化 #4735 recipes and never limit admission.--enable-hierarchical-cache --hicache-host-memory-fraction 0.60 --hicache-io-backend direct --hicache-mem-layout page_first_direct, from [DNM, Experimental] GLM-5.3-Flash MI325X FP8 HiCache and concurrency / [DNM, Experimental] GLM-5.3-Flash MI325X FP8 HiCache与并发优化 #4735.SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0andSGLANG_GLM_NEXTN_MOE_PTPC=0. The NextN draft runs at its shipped precision through the image's default handling.ROCM_QUICK_REDUCE_QUANTIZATION=NONEdisables it.SGLANG_OPT_FUSE_MHC_POST_PRE=0selects the upstream unfused mHC path because the fused path failed HIP compilation on gfx942.CPU DRAM for KV offload
hicache-ratioof 2.0, not by the 0.60 fraction. The ceiling from the fraction alone would be 4 × 319.8 GiB = 1373 GB.dram-utilization: 0.87records 1051 GB (2,305,165 MiB × 0.87 × 4/8). That covers the largest measured allocation (c16) instead of the policy maximum.Exceptions and harness-only settings
SGLANG_USE_AITER=1,AITER_ONLINE_TUNE=0andSGLANG_DSA_FUSE_TOPK=0.dsa-topk-backend torchandlinear-attn-backend tritonSGLANG_OPT_USE_TOPK_V2=falseandSGLANG_OPT_FUSE_MHC_POST_PRE=0stage-hf-checkpoint.sh) is harness-only:modelsinventory entry.46a050bec13002ad29ee795c16e61cf4fa3c4a9cinto the cluster's shared Hugging Face cache, with a 400 GB disk budget and no deletion. Serving then stays offline on that snapshot.infx/tests/srt_slurm/test_checkpoint_staging.pyexercises the script against a fake Hub.HF_HUB_OFFLINE,TRANSFORMERS_OFFLINE,PYTHONNOUSERSITEandSGLANG_TIMEOUT_KEEP_ALIVEenable-metricsAIPERF_*client timeouts and metric prefixhealth_checkInvalidInferenceResultErrorrecords fall almost entirely in the warmup window (1–4 per point remain in the profiled window).Validation: the full sweep for this head is pending and will be linked here.
中文
本 PR 以单一配置键
glm5.3flash-fp8-mi325x-sglang-agentic-mtp添加 MI325X 上 GLM-5.3-Flash 原生 FP8 AgentX 曲线,使用以 digest 固定的官方lmsysorg/sglang:v0.5.22-rocm10-mi30x镜像。本 PR 将自身的基础扫描与 #4735 的 HiCache 对比合并为一条曲线:低并发(1、2、4、8、12)使用 GPU 内 KV,高并发(16、24、32)使用 SGLang HiCache(CPU DRAM KV offload),并去掉被支配的点。预期 p90 E2EL 前沿为 5 个点,交互性前沿为全部 8 个点。在本次扫描完成前,8 个预期点均为估计值:两次源运行都设置了
AIPERF_APPLY_CHAT_TEMPLATE=true,本 PR 已移除该设置,使所有 GLM-5.3-Flash SKU 采用相同的客户端;c1–c8 测于 v0.5.21,在两个镜像均有测量的并发点上,v0.5.21→v0.5.22 吞吐变化 ≤1%,交互性 +4–10%,p90 E2EL +3–9%。HiCache 实测每节点主机内存分配为 988–1045 GB,低于 MODELS.md 按 GPU 比例计算的上限 1208.6 GB;用量由默认
hicache-ratio2.0 决定(主机层恰为设备 KV 池的 2 倍),而非 0.60 fraction(仅按 fraction 的上限为 1373 GB)。dram-utilization: 0.87记录 1051 GB,覆盖实测最大值(c16,4 × 261.22 GB)。MI325X 没有 GLM-5.3-Flash 的 SGLang cookbook 条目,作为已记录的例外保留 #4722/#4735 的 MI325X 配置;与最接近的 MI355X FP8 cookbook 配置的对照见英文部分。
stage-hf-checkpoint.sh仅属测试框架:MI325X 集群没有预先部署的 checkpoint,worker setup 在仓库级文件锁下将固定 revision 下载到共享缓存(400 GB 磁盘预算,不删除文件),服务保持离线;H100 #4717 含有完全相同的文件,后合并者删除重复。draft 按原样以出厂精度提供,SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0,ROCM_QUICK_REDUCE_QUANTIZATION=NONE,无引擎补丁。AI model disclosure
claude-opus-5-5), working as a per-SKU agent, consolidated Add GLM-5.3-Flash FP8 MI325X AgentX with HiCache / 新增 GLM-5.3-Flash FP8 MI325X AgentX(含 HiCache) #4722 and [DNM, Experimental] GLM-5.3-Flash MI325X FP8 HiCache and concurrency / [DNM, Experimental] GLM-5.3-Flash MI325X FP8 HiCache与并发优化 #4735 into this PR: point selection, recipe/config/changelog, the DRAM accounting and this description.