Repository navigation
[NVIDIA][AgentX] GB200 GLM-5.2 NVFP4 Dynamo+SGLang: PP8-prefill disagg and TP4/TP8 agg points / GB200 GLM-5.2 NVFP4 Dynamo+SGLang:PP8 prefill 分离式与 TP4/TP8 聚合配置点 - #4398
Conversation
…TP8 agg points Move both GB200 GLM-5.2 NVFP4 Dynamo + SGLang AgentX keys to lmsysorg/sglang:nightly-dev-20261020-ab580b2e (digest pinned), running the bundled ai-dynamo 1.6.0.dev20261019. - agg: replace TP8 c1-c16 with TP4 c1 and TP8 c1 (MTP 5, fp8 KV, HiCache 100 GB/rank). - disagg: replace the TP4 1P1D, DEP8-prefill 1P4D/1P6D and DEP8/DEP16 2P1D points with TP1/PP8 prefill workers feeding TP4 decode (1P6D c36, 1P4D c48), DEP8 decode (1P2D c64, 1P2D c96, 1P1D c96) and DEP16 decode (2P1D c224, 3P1D c336). - Remove the -agentic-mtp and -agentic-mtp-agg keys and the four recipes only they and the replaced points used. - Append one perf-changelog entry covering both keys. 将 GB200 GLM-5.2 NVFP4 Dynamo + SGLang AgentX 的两个 key 迁移到 lmsysorg/sglang:nightly-dev-20261020-ab580b2e(固定 digest),使用镜像自带的 ai-dynamo 1.6.0.dev20261019。 - 聚合:将 TP8 c1-c16 替换为 TP4 c1 与 TP8 c1(MTP 5 步、fp8 KV、 每 rank 100 GB HiCache)。 - 分离式:将 TP4 1P1D、DEP8 prefill 的 1P4D/1P6D 以及 DEP8/DEP16 2P1D 配置点替换为 TP1/PP8 prefill worker,分别接 TP4 decode(1P6D c36、 1P4D c48)、DEP8 decode(1P2D c64、1P2D c96、1P1D c96)与 DEP16 decode (2P1D c224、3P1D c336)。 - 移除 -agentic-mtp 与 -agentic-mtp-agg 两个 key,以及仅被它们和被替换配置点 使用的四个 recipe。 - 追加一条覆盖两个 key 的 perf-changelog 记录。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献!
|
设置 #4398 的 changelog pr-link。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. It reworks every GB200 GLM-5.2 AgentX point on a new nightly image and adds a new PP8-prefill topology, so it would still be worth having a code owner look at it.
What was reviewed:
- The two remaining master keys against the new
agg-variants.yamlanddisagg-variants.yamloverrides. I checked the worker counts, the newppfields, the decodeepvalues and the image digest and router version, which match across both keys. - The deleted keys and recipes. The only remaining
disagg-mtp-variants.yamlreferences belong to other models' recipe dirs, not this one. - The appended perf-changelog entry. Its
pr-linkis still thepull/XXXplaceholder and needs the real PR number before merge.
Extended reasoning...
The change rewrites the GB200 GLM-5.2 NVFP4 Dynamo+SGLang AgentX recipes and master entries (about 500 lines added and 730 removed across 7 files), deletes two config keys and three recipes, and appends one changelog entry. It touches no security-sensitive surface; it is benchmark configuration only. It is a defer rather than an approval because nvidia-master.yaml and the gb200 recipe tree are owned by CODEOWNERS. The change also moves to an unreleased nightly image and introduces a new PP8-prefill layout whose performance tuning is a judgement call for a domain owner.
This review covers commit a74de28, which is no longer the latest commit on this pull request; later commits are not covered by it.
…agg recipe Remove UCX_TLS and UCX_CUDA_IPC_ENABLE_MNNVL from the prefill and decode workers so NIXL uses its default UCX transport selection, and drop the matching sentence from the changelog entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] I've started babysitting the pipeline of this PR. I'll keep monitoring it until it passes. If a job fails, I may rerun it when the cause is infrastructure, or push a fix or a merge of the latest |
…-5.2 disagg recipe" Without UCX_TLS, NIXL falls back to the InfiniBand devices in UCX_NET_DEVICES for P->D KV transfer, and the 1P1D DEP8 c96 point hit NIXL_ERR_REMOTE_DISCONNECT on enough requests to exceed the 10% error threshold. Restore the cuda_ipc MNNVL transport on both roles. This reverts commit a2a1ad6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=39765601214 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=39765601214 |
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#46005 has been merged. This PR's AgentX configs ( This PR has no authorized |
|
/use 39765601214 |
|
@nvpohanh staged run 39765601214. This run remains available across future |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Passing sweep on in-PR commit): PASS — Run Sweep 39765601214 ran on the PR head ✅ Check 2 (Evals pass): PASS — ✅ Check 3 (Upstream recipe merged and complete): PASS — sgl-project/sglang#46005 is ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist template): PASS — all 18 items of the current ✅ Check 6 (Upstream image / engine-first): PASS — both entries are ✅ Check 7 (No deprecated models/scenarios): PASS — GLM-5.2 agentic coding is active in ✅ Check 8 (No architecture hacks): PASS — no ✅ Check 9 (Spec-decode via chat template): PASS — the AgentX AIPerf replay uses the chat endpoint with ✅ Check 10 (No engine patches): PASS — the diff changes only the recipe YAMLs, ✅ Check 11 (Golden simulated AL): PASS — ➖ Check 12 (Append-only): N/A — the new ✅ Check 13 (Draft precision): PASS — the draft is the built-in NextN layer 78. ✅ Check 14 (Pareto coverage): PASS — one curve, GLM-5.2 / agentic-coding / gb200 dynamo-sglang / fp4 / run 39765601214. Recomputed from ✅ Check 15 (PR description): PASS — stated topologies, node counts, mem-fraction, running-request, HiCache, PP partition, image, and Dynamo version claims match the recipes and master config at ✅ Check 16 (NVIDIA overview): PASS — GLM-5.2 has an SGLang cell in the Framework Recipes table of the NVIDIA overview. ➖ Check 17 (Metrics publication): N/A — both affected keys are Assessed commit: |
Moves both GB200 GLM-5.2 NVFP4 Dynamo + SGLang AgentX keys to
lmsysorg/sglang:nightly-dev-20261020-ab580b2e(multi-arch digest pinned), running the ai-dynamo1.6.0.dev20261019bundled in the image (dynamo.install: false). The disaggregated points move to TP1/PP8 prefill workers that span two nodes each (sgl-project/sglang#39378).glm5.2-fp4-gb200-dynamo-sglang-agentic-aggglm5.2-fp4-gb200-dynamo-sglang-agentic-disaggSGLANG_SIMULATE_ACC_*orSGLANG_NVFP4_CKPT_FP8_NEXTN_MOE.10,10,10,10,10,10,10,8because the last PP stage also holds the LM head and MTP draft layer. Prefill uses 8k chunks; HiCache is 160 GB/rank with 64 running requests except at 1P1D c96.dram-utilizationstays 0.80; 4 x 160 GB plus the DSA-indexer host pool fit in the 721 GB per-node budget.UCX_TLS=cuda_copy,cuda_ipc,sm,self,tcpandUCX_CUDA_IPC_ENABLE_MNNVL=y, so P->D KV moves over NVLink. DEP decode workers setexpert-parallel-sizeexplicitly to match the master config'sep.glm5.2-fp4-gb200-dynamo-sglang-agentic-mtpand-agentic-mtp-aggkeys and the four recipes (agg.yaml,agg-mtp-variants.yaml,disagg-mtp-variants.yaml,disagg-dep8-mtp-variants.yaml) that only the removed points used.中文
将 GB200 GLM-5.2 NVFP4 Dynamo + SGLang AgentX 的两个 config key 迁移到
lmsysorg/sglang:nightly-dev-20261020-ab580b2e(固定 multi-arch digest),直接使用镜像自带的 ai-dynamo1.6.0.dev20261019(dynamo.install: false)。分离式配置点改为 TP1/PP8 prefill worker,每个 worker 跨两个节点(sgl-project/sglang#39378)。glm5.2-fp4-gb200-dynamo-sglang-agentic-agg:glm5.2-fp4-gb200-dynamo-sglang-agentic-disagg:SGLANG_SIMULATE_ACC_*或SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE。10,10,10,10,10,10,10,8,因为最后一个 PP stage 还承载 LM head 与 MTP draft 层。prefill 使用 8k chunk;除 1P1D c96 外,HiCache 为每 rank 160 GB、64 个运行请求。dram-utilization保持 0.80;4 x 160 GB 加上 DSA indexer 主机内存池可以放入每节点 721 GB 的预算。UCX_TLS=cuda_copy,cuda_ipc,sm,self,tcp与UCX_CUDA_IPC_ENABLE_MNNVL=y,使 P->D KV 经 NVLink 传输。DEP decode worker 显式设置expert-parallel-size,与 master config 的ep一致。glm5.2-fp4-gb200-dynamo-sglang-agentic-mtp与-agentic-mtp-agg两个 key,以及仅被这些配置点使用的四个 recipe(agg.yaml、agg-mtp-variants.yaml、disagg-mtp-variants.yaml、disagg-dep8-mtp-variants.yaml)。AI model disclosure
claude-opus-5-5), via Claude Code.🤖 Generated with Claude Code