Skip to content

[NVIDIA][AgentX] GB200 GLM-5.2 NVFP4 Dynamo+SGLang: PP8-prefill disagg and TP4/TP8 agg points / GB200 GLM-5.2 NVFP4 Dynamo+SGLang:PP8 prefill 分离式与 TP4/TP8 聚合配置点 - #4398

Merged
Oseltamivir merged 6 commits into
mainfrom
pohanh/glm52-gb200-dynamo-sglang-pp8-1022
Oct 23, 2026
Merged

Oseltamivir merged 6 commits into
mainfrom
pohanh/glm52-gb200-dynamo-sglang-pp8-1022

Conversation

@nvpohanh

@nvpohanh nvpohanh commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Moves both GB200 GLM-5.2 NVFP4 Dynamo + SGLang AgentX keys to lmsysorg/sglang:nightly-dev-20261020-ab580b2e (multi-arch digest pinned), running the ai-dynamo 1.6.0.dev20261019 bundled in the image (dynamo.install: false). The disaggregated points move to TP1/PP8 prefill workers that span two nodes each (sgl-project/sglang#39378).

Config key Topology Conc Nodes Key settings
glm5.2-fp4-gb200-dynamo-sglang-agentic-agg TP4 1 1 cutedsl BF16 GEMM, 4 running requests and decode graphs, mem-fraction 0.85
TP8 1 2 flashinfer_trtllm FP4 GEMM, 2 running requests and decode graphs, mem-fraction 0.8
glm5.2-fp4-gb200-dynamo-sglang-agentic-disagg 1x PP8 prefill + 6x TP4 decode 36 8 decode radix cache, 24 requests per decode worker, decode mem-fraction 0.94
1x PP8 prefill + 4x TP4 decode 48 6 as c36, prefill mem-fraction 0.83
1x PP8 prefill + 2x DEP8 decode 64 6 32 requests per decode rank
1x PP8 prefill + 2x DEP8 decode 96 6 as c64, prefill mem-fraction 0.83
1x PP8 prefill + 1x DEP8 decode 96 4 prefill HiCache 100 GB/rank, 16 prefill requests, prefill mem-fraction 0.83
2x PP8 prefill + 1x DEP16 decode 224 8 32 requests per decode rank, decode mem-fraction 0.94
3x PP8 prefill + 1x DEP16 decode 336 10 as c224
  • Every point uses EAGLE MTP on the checkpoint's built-in NextN head (5 steps, top-k 1, 6 draft tokens) on every serving role, fp8 KV cache, and HiCache write_back DRAM KV offload. Synthetic acceptance follows the golden GLM-5.2 thinking-on AL (3.61 at 5 steps). No recipe sets SGLANG_SIMULATE_ACC_* or SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE.
  • Prefill splits the 78 layers 10,10,10,10,10,10,10,8 because the last PP stage also holds the LM head and MTP draft layer. Prefill uses 8k chunks; HiCache is 160 GB/rank with 64 running requests except at 1P1D c96. dram-utilization stays 0.80; 4 x 160 GB plus the DSA-indexer host pool fit in the 721 GB per-node budget.
  • KV transfer uses NIXL. Both roles set UCX_TLS=cuda_copy,cuda_ipc,sm,self,tcp and UCX_CUDA_IPC_ENABLE_MNNVL=y, so P->D KV moves over NVLink. DEP decode workers set expert-parallel-size explicitly to match the master config's ep.
  • Routing matches the GB300 keys: prefill workers advertise a 3600 s session-affinity TTL to the prefill router; decode workers advertise an active-request weight of 8192, so decode routing balances request counts. Tachometer and the Dynamo OTel collector are disabled.
  • Removes the aggregated TP8 c1-c16 points, the TP4 1P1D c10/c12 points, the DEP8-prefill 1P4D c48 and 1P6D c45 points, and the DEP8/DEP16 2P1D c128 point. Deletes the glm5.2-fp4-gb200-dynamo-sglang-agentic-mtp and -agentic-mtp-agg keys and the four recipes (agg.yaml, agg-mtp-variants.yaml, disagg-mtp-variants.yaml, disagg-dep8-mtp-variants.yaml) that only the removed points used.
中文

将 GB200 GLM-5.2 NVFP4 Dynamo + SGLang AgentX 的两个 config key 迁移到 lmsysorg/sglang:nightly-dev-20261020-ab580b2e(固定 multi-arch digest),直接使用镜像自带的 ai-dynamo 1.6.0.dev20261019(dynamo.install: false)。分离式配置点改为 TP1/PP8 prefill worker,每个 worker 跨两个节点(sgl-project/sglang#39378)。

  • glm5.2-fp4-gb200-dynamo-sglang-agentic-agg:
    • TP4 c1(1 个节点):cutedsl BF16 GEMM,4 个运行请求与 decode graph,mem-fraction 0.85。
    • TP8 c1(2 个节点):flashinfer_trtllm FP4 GEMM,2 个运行请求与 decode graph,mem-fraction 0.8。
  • glm5.2-fp4-gb200-dynamo-sglang-agentic-disagg:
    • 1P6D c36(1 个 PP8 prefill + 6 个 TP4 decode,8 个节点)与 1P4D c48(4 个 TP4 decode,6 个节点,prefill mem-fraction 0.83):decode 启用 radix cache,每个 decode worker 24 个请求,decode mem-fraction 0.94。
    • 1P2D c64 与 1P2D c96(1 个 PP8 prefill + 2 个 DEP8 decode,6 个节点;c96 的 prefill mem-fraction 为 0.83)。
    • 1P1D c96(1 个 PP8 prefill + 1 个 DEP8 decode,4 个节点):prefill HiCache 每 rank 100 GB,16 个 prefill 请求,prefill mem-fraction 0.83。
    • 2P1D c224(8 个节点)与 3P1D c336(10 个节点):PP8 prefill + 1 个 DEP16 decode,decode mem-fraction 0.94。
    • DEP decode 每个 rank 32 个请求。
  • 所有配置点在各个服务角色上均使用基于 checkpoint 内置 NextN head 的 EAGLE MTP(5 步、top-k 1、6 个 draft token),fp8 KV cache,以及 HiCache write_back DRAM KV offload。模拟接受长度采用 GLM-5.2 thinking-on 的 golden AL(5 步为 3.61)。recipe 中未设置 SGLANG_SIMULATE_ACC_* 或 SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE。
  • prefill 将 78 层划分为 10,10,10,10,10,10,10,8,因为最后一个 PP stage 还承载 LM head 与 MTP draft 层。prefill 使用 8k chunk;除 1P1D c96 外,HiCache 为每 rank 160 GB、64 个运行请求。dram-utilization 保持 0.80;4 x 160 GB 加上 DSA indexer 主机内存池可以放入每节点 721 GB 的预算。
  • KV 传输使用 NIXL。两个角色都设置 UCX_TLS=cuda_copy,cuda_ipc,sm,self,tcp 与 UCX_CUDA_IPC_ENABLE_MNNVL=y,使 P->D KV 经 NVLink 传输。DEP decode worker 显式设置 expert-parallel-size,与 master config 的 ep 一致。
  • 路由与 GB300 key 相同:prefill worker 向 prefill router 声明 3600 秒 session affinity TTL;decode worker 向 decode router 声明 active-request 权重 8192,使 decode 路由按请求数均衡。关闭 Tachometer 与 Dynamo OTel collector。
  • 移除聚合 TP8 c1-c16、TP4 1P1D c10/c12、DEP8 prefill 的 1P4D c48 与 1P6D c45、DEP8/DEP16 2P1D c128 配置点;删除 glm5.2-fp4-gb200-dynamo-sglang-agentic-mtp 与 -agentic-mtp-agg 两个 key,以及仅被这些配置点使用的四个 recipe(agg.yaml、agg-mtp-variants.yaml、disagg-mtp-variants.yaml、disagg-dep8-mtp-variants.yaml)。
AI model disclosure
  • Model/version: Claude Opus 5.5 (claude-opus-5-5), via Claude Code.
  • Role: drove the automated configuration search whose measured settings these recipes reproduce, ported those settings into the recipes and master-config entries, wrote the changelog entry and PR text, and ran the local changelog, matrix and test checks.

🤖 Generated with Claude Code

…TP8 agg points

Move both GB200 GLM-5.2 NVFP4 Dynamo + SGLang AgentX keys to
lmsysorg/sglang:nightly-dev-20261020-ab580b2e (digest pinned), running the
bundled ai-dynamo 1.6.0.dev20261019.

- agg: replace TP8 c1-c16 with TP4 c1 and TP8 c1 (MTP 5, fp8 KV,
  HiCache 100 GB/rank).
- disagg: replace the TP4 1P1D, DEP8-prefill 1P4D/1P6D and DEP8/DEP16 2P1D
  points with TP1/PP8 prefill workers feeding TP4 decode (1P6D c36,
  1P4D c48), DEP8 decode (1P2D c64, 1P2D c96, 1P1D c96) and DEP16 decode
  (2P1D c224, 3P1D c336).
- Remove the -agentic-mtp and -agentic-mtp-agg keys and the four recipes
  only they and the replaced points used.
- Append one perf-changelog entry covering both keys.

将 GB200 GLM-5.2 NVFP4 Dynamo + SGLang AgentX 的两个 key 迁移到
lmsysorg/sglang:nightly-dev-20261020-ab580b2e(固定 digest),使用镜像自带的
ai-dynamo 1.6.0.dev20261019。

- 聚合:将 TP8 c1-c16 替换为 TP4 c1 与 TP8 c1(MTP 5 步、fp8 KV、
  每 rank 100 GB HiCache)。
- 分离式:将 TP4 1P1D、DEP8 prefill 的 1P4D/1P6D 以及 DEP8/DEP16 2P1D
  配置点替换为 TP1/PP8 prefill worker,分别接 TP4 decode(1P6D c36、
  1P4D c48)、DEP8 decode(1P2D c64、1P2D c96、1P1D c96)与 DEP16 decode
  (2P1D c224、3P1D c336)。
- 移除 -agentic-mtp 与 -agentic-mtp-agg 两个 key,以及仅被它们和被替换配置点
  使用的四个 recipe。
- 追加一条覆盖两个 key 的 perf-changelog 记录。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

设置 #4398 的 changelog pr-link。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and didn't find any bugs. It reworks every GB200 GLM-5.2 AgentX point on a new nightly image and adds a new PP8-prefill topology, so it would still be worth having a code owner look at it.

What was reviewed:

  • The two remaining master keys against the new agg-variants.yaml and disagg-variants.yaml overrides. I checked the worker counts, the new pp fields, the decode ep values and the image digest and router version, which match across both keys.
  • The deleted keys and recipes. The only remaining disagg-mtp-variants.yaml references belong to other models' recipe dirs, not this one.
  • The appended perf-changelog entry. Its pr-link is still the pull/XXX placeholder and needs the real PR number before merge.
Extended reasoning...

The change rewrites the GB200 GLM-5.2 NVFP4 Dynamo+SGLang AgentX recipes and master entries (about 500 lines added and 730 removed across 7 files), deletes two config keys and three recipes, and appends one changelog entry. It touches no security-sensitive surface; it is benchmark configuration only. It is a defer rather than an approval because nvidia-master.yaml and the gb200 recipe tree are owned by CODEOWNERS. The change also moves to an unreleased nightly image and introduces a new PP8-prefill layout whose performance tuning is a judgement call for a domain owner.

This review covers commit a74de28, which is no longer the latest commit on this pull request; later commits are not covered by it.

…agg recipe

Remove UCX_TLS and UCX_CUDA_IPC_ENABLE_MNNVL from the prefill and decode
workers so NIXL uses its default UCX transport selection, and drop the
matching sentence from the changelog entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh nvpohanh added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 22, 2026
@nvpohanh

Copy link
Copy Markdown
Collaborator Author

[by Claude Code] I've started babysitting the pipeline of this PR. I'll keep monitoring it until it passes. If a job fails, I may rerun it when the cause is infrastructure, or push a fix or a merge of the latest main when the cause is in this PR.

…-5.2 disagg recipe"

Without UCX_TLS, NIXL falls back to the InfiniBand devices in
UCX_NET_DEVICES for P->D KV transfer, and the 1P1D DEP8 c96 point hit
NIXL_ERR_REMOTE_DISCONNECT on enough requests to exceed the 10% error
threshold. Restore the cuda_ipc MNNVL transport on both roles.

This reverts commit a2a1ad6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

@nvpohanh

Copy link
Copy Markdown
Collaborator Author

[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs (glm5.2-fp4-gb200-dynamo-sglang-agentic-agg, glm5.2-fp4-gb200-dynamo-sglang-agentic-disagg) will happen in sgl-project/sglang#46005, in the Agentic Long-Context Deployment section of the GLM-5.2 cookbook page.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (4d4e4c739) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

@nvpohanh

Copy link
Copy Markdown
Collaborator Author

[by Claude Code] sgl-project/sglang#46005 has been merged. This PR's AgentX configs (glm5.2-fp4-gb200-dynamo-sglang-agentic-agg, glm5.2-fp4-gb200-dynamo-sglang-agentic-disagg) are now in the Agentic Long-Context Deployment section of the SGLang GLM-5.2 cookbook: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.2#agentic-long-context-deployment

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (4d4e4c739) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

@nvpohanh

Copy link
Copy Markdown
Collaborator Author

/use 39765601214

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

@nvpohanh staged run 39765601214.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

@nvpohanh

Copy link
Copy Markdown
Collaborator Author

As a PR reviewer and CODEOWNER, I have reviewed this and have:

  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
  • Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

  • Scope: two framework: dynamo-sglang keys on cluster:gb200-nv, both on nvidia/GLM-5.2-NVFP4, scenario agentic-coding, MTP (EAGLE 5 steps / topk 1 / 6 draft tokens), image lmsysorg/sglang:nightly-dev-20261020-ab580b2e@sha256:0af411bb320a95bb5e1c58b4c20c9852920a19aa9f12f148f69731b0dca3b576 (SGLang ab580b2e060059c0f8387b598a9f648dcec13f93, bundled ai-dynamo 1.6.0.dev20261019, dynamo.install: false).
    • glm5.2-fp4-gb200-dynamo-sglang-agentic-agg (nvidia-master.yaml L7223): TP4 c1, TP8 (2 nodes) c1.
    • glm5.2-fp4-gb200-dynamo-sglang-agentic-disagg (L7264): TP1/PP8 prefill (2 nodes per worker) with 1P6D TP4 c36, 1P4D TP4 c48, 1P2D DEP8 c64 and c96, 1P1D DEP8 c96, 2P1D DEP16 c224, 3P1D DEP16 c336.
    • The PR also removes the -agentic-mtp and -agentic-mtp-agg keys and their recipes.
  • PR validation: Run Sweep 39765601214 (pull_request, full-sweep-fail-fast) ran on 4d4e4c7392c088d16ba83b0b9db165b7b752a99e, the current PR head, and completed success on attempt 2. It is pinned with /use 39765601214. Its results_bmk, bmk_agentic_* and eval_results_all artifacts are unexpired (expire 2027-01-20).
    • All 9 multi-node agentic / benchmarks succeeded in attempt 1 with 0 error-dropped requests: agg TP4 c1 (canary) and TP8 c1; disagg 1P6D c36, 1P4D c48, 1P2D c64, 1P2D c96, 1P1D c96, 2P1D c224 and 3P1D c336.
    • Attempt 1 had one infrastructure failure: the 1P6D c36 eval (job) lost one of six identical TP4 decode workers on node watchtower-navy-cn11 at startup to a sticky CUDA error: unspecified launch failure in the target FP4 MoE autotuner. Fail-fast cancelled three other evals. Attempt 2 reran only those four evals, and all passed. The same decode configuration passed its benchmark in attempt 1.
    • The skipped lanes (single-node, fixed-sequence, non-agentic eval) have empty matrices because the PR only changes these two multi-node agentic keys.
  • Evals: all 7 multi-node agentic eval / jobs succeeded. gsm8k em_strict, n=1319, infrastructure_success: true, from eval_results_all:
    • 1P1D DEP8 c96 0.9644 (job), 1P2D DEP8 c64 0.9697 (job), 1P2D DEP8 c96 0.9689 (job), 1P6D TP4 c36 0.9674 (job), 1P4D TP4 c48 0.9697 (job), 2P1D DEP16 c224 0.9712 (job), 3P1D DEP16 c336 0.9682 (job).
    • All clear the default 0.90 threshold in infx/evals/thresholds.yaml, which has no glm5.2 override. They also clear the 0.94 glm5 value.
    • All evals ran on the same image (build commit ab580b2e) with real EAGLE verification: no eval worker has SGLANG_SIMULATE_ACC_*, and the logged accept length is about 4.5 to 5.6.
    • The two agg points have no eval in this lane; the disagg evals cover the same checkpoint, image and MTP settings.
  • Chat template: the AgentX replay drives the Dynamo frontend with AIPerf endpoint type chat and streaming; the frontend logs show only /v1/chat/completions. Requests send chat_template_kwargs: {"enable_thinking": true}, and THINKING_MODE is thinking_on. The checkpoint ships chat_template.jinja, and nothing overrides it.
  • Synthetic acceptance (golden AL): the InferenceX launcher (infx/srt_slurm/synthetic_acceptance.py) sets SGLANG_SIMULATE_ACC_LEN=3.61, SGLANG_SIMULATE_ACC_METHOD=match-expected and SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token on every prefill, decode and agg worker of all 9 benchmarks (job logs and worker environment dumps). The recipes set none of them.
  • Draft precision: the draft is the built-in NextN/MTP layer model.layers.78 of nvidia/GLM-5.2-NVFP4 (num_hidden_layers: 78, num_nextn_predict_layers: 1); the logs show speculative_draft_model_path=/model, the target checkpoint.
    • Shipped: BF16. hf_quant_config.json (NVFP4) lists model.layers.78* in exclude_modules. The layer-78 safetensors headers hold 791 tensors: 790 BF16 and the F32 mlp.gate.e_score_correction_bias, with no scale tensors. The run used snapshot aec724e8, whose weights and configs match Hugging Face main (53e0691e).
    • Pinned image default: draft quantization is not set, so SGLang inherits the target's modelopt_fp4 (serving_hook.py L654-669). The NextN model (GlmMoeDsaForCausalLMNextN, a DeepseekV3ForCausalLMNextN subclass) then sets quant_config = None for modelopt_fp4 (deepseek_nextn.py L66-79). At this commit that message is logger.debug, so it does not appear in the logs.
      • With --speculative-moe-runner-backend flashinfer_trtllm, the unquantized draft MoE uses UnquantizedFusedMoEMethod and the BF16 kernel trtllm_bf16_moe (unquant.py, flashinfer_trtllm.py).
    • Effective: BF16. Every worker logs _speculative_draft_quantization_explicitly_set=False, dtype=auto, json_model_override_args={} and speculative_draft_kv_cache_dtype=None. The KV cache is FP8 for target and draft (--kv-cache-dtype fp8_e4m3, matching the checkpoint's kv_cache_quant_algo: FP8).
      • Draft weight memory per rank matches BF16 experts: 20.84 GiB on the PP8 last stage (TP1, whole layer; BF16 would be about 20.3 GiB, NVFP4 experts about 7 GiB), 5.37 GiB at TP4, 4.64 GiB at DEP8, 3.46 GiB at DEP16 and 2.80 GiB at TP8 agg.
    • SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not set: it is absent from the recipes, from all 89 logged worker environments (prefill, decode and agg; benchmark and eval) and from the job logs. It defaults to EnvBool(False) (environ.py L1095). SGLANG_GLM_NEXTN_MOE_PTPC and --speculative-draft-model-quantization are also absent.
  • Models and architecture: GLM-5.2 is active for the agentic coding scenario in MODELS.md as of 2026-10-23. The diff has no --hf-overrides, json-model-override-args or other architecture changes; the logs show json_model_override_args={}.
  • Dynamo TRT-LLM metrics: N/A. Both keys are dynamo-sglang; no Dynamo TRT-LLM serving path is affected.
  • Image and engine order: upstream lmsysorg/sglang image, pinned by digest. These entries are dynamo-sglang, an open-source engine, so the engine-first ordering is satisfied by this submission.
  • Recipe publication (merged): every point is in the SGLang cookbook via sgl-project/sglang#46005, MERGED (mergedAt: 2026-10-23T08:04:02Z, merge commit 02d24451).
    • Published page: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.2#agentic-long-context-deployment (hardware GB200, checkpoint nvidia/GLM-5.2-NVFP4, KV offload HiCache, Dynamo router).
    • Data: docs/src/snippets/agentx/zai-org/glm-5.2.jsx has the 8 GB200 cells (agg TP4, agg TP8, 1P6D, 1P4D, 1P2D c64/c96, 1P1D, 2P1D, 3P1D). They mirror this PR's head 4d4e4c73 and show it as in progress.
    • I rebuilt the cookbook commands for all 9 points and diffed them against the recipe base plus overrides and the logged worker command lines and environments in run 39765601214. Every production-useful setting matches, for the agg worker and for the prefill, decode and Dynamo frontend/nginx roles:
      • Image digest; TP/PP/EP/DP and DP attention; modelopt_fp4 and fp8_e4m3 KV; MTP 5/1/6.
      • Backends: moe-runner, fp4-gemm, bf16-gemm, speculative MoE runner and A2A.
      • HiCache size, write_back, direct and page_first_direct; chunked prefill; CUDA graph batch size; max running requests; mem-fraction-static; radix-cache flags.
      • SGLANG_PP_LAYER_PARTITION=10,10,10,10,10,10,10,8 and NIXL KV transfer with UCX_TLS=cuda_copy,cuda_ipc,sm,self,tcp and UCX_CUDA_IPC_ENABLE_MNNVL=y.
      • MC_FORCE_MNNVL, NCCL_MNNVL_ENABLE, NCCL_CUMEM_ENABLE, NVSHMEM_REMOTE_TRANSPORT, SGLANG_MOONCAKE_CUSTOM_MEM_POOL, SGLANG_USE_MESSAGE_QUEUE_BROADCASTER, SGLANG_CLIP_MAX_NEW_TOKENS_ESTIMATION and the disaggregation timeouts.
      • Router: --router-mode kv, replica sync, the three --active-*-threshold None flags, session-affinity TTL 3600, decode active-request weight 8192, frontend counts and nginx session affinity on X-Dynamo-Session-ID.
    • The only differences are placeholders: model path, ports, IP addresses and node ranks.
    • The GB200 lane's install-torchao.sh (pip install torchao) appears in the cookbook as a setup step with its reason.
    • Excluded as InferenceX-only:
      • SGLANG_SIMULATE_ACC_*, the synthetic acceptance control. The cookbook discloses it with AL 3.61.
      • --enable-metrics, --enable-cache-report, --enable-request-time-stats-logging and MC_TE_METRIC. These are Prometheus and logging output for AIPerf result collection; they do not change scheduling or kernels.
      • --dump-config-to, --nccl-port, DYN_SYSTEM_PORT, DYN_RESPONSE_PLANE, HEAD_NODE_IP, ETCD_ENDPOINTS and NATS_SERVER: srt-slurm process placement, ports and diagnostics.
      • PYTHONUNBUFFERED, PIP_BREAK_SYSTEM_PACKAGES, DYN_LOG, DYN_SKIP_SGLANG_LOG_FORMATTING, SGLANG_CUDA_COREDUMP_BEFORE_CRASH, SGLANG_PYSPY_DUMP_BEFORE_CRASH, SGLANG_TORCH_PROFILER_DIR and NCCL_DEBUG: logging, crash-dump and package-install toggles.
      • Cluster node environment the recipe does not set: NCCL_IB_*, NCCL_SOCKET_IFNAME, NCCL_NET_GDR_C2C, UCX_NET_DEVICES and OMPI_MCA_*. These are NIC names and MPI settings of this cluster.
      • The tachometer and OTel collector toggles (both off), the AIPerf client settings, the srt-slurm health_check, and the Slurm mem/cpus-per-task settings.
  • No engine patches: the diff (5 GB200 GLM-5.2 recipes, nvidia-master.yaml, perf-changelog.yaml) has no .patch, git apply, inline source rewrites, sed on site-packages or engine wheel installs, so no waiver is needed. The lane's install-torchao.sh only runs pip install torchao (torchao 0.18.0, a third-party package that SGLang ab580b2e does not use for serving); it does not modify SGLang, Dynamo or FlashInfer.
  • append-only: N/A. This PR's perf-changelog.yaml entry has no append-only key.
  • Pareto coverage: one affected curve, GLM-5.2 / agentic-coding / gb200_dynamo-sglang / fp4 / run 39765601214. Agg and disagg map to the same app series, so all 9 points form one input.
    • Data: results_bmk (artifact 12451183833), which matches all nine bmk_agentic_* artifacts. Metrics: per_gpu.total_tput_tps vs latency.e2el.p90 (and .p75). Counted with infx/workflows/pareto_coverage.py from main.
    • P90: 7 frontier points of 9 measured; P75: 7 of 9. Frontier (conc, P90 E2EL s, tput/GPU): 1P6D c36 (9.87, 8390), 1P4D c48 (12.53, 14033), 1P2D c64 (16.91, 17885), 1P2D c96 (19.66, 22692), 1P1D c96 (22.69, 32057), 2P1D c224 (24.31, 37666), 3P1D c336 (27.33, 42272). The agg TP8 c1 and TP4 c1 points are dominated.
    • Dashboard: https://inferencex.semianalysis.com/inference?unofficialRun=39765601214

Signed: nvpohanh

@github-actions

Copy link
Copy Markdown
Contributor

✅✅✅ Verdict: PASS ✅✅✅

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @nvpohanh is a listed owner of inferencex-e2e/configs/nvidia-master.yaml and the srt-slurm-recipes/**/gb200-*/ recipes; perf-changelog.yaml falls only under the catch-all.

✅ Check 1 (Passing sweep on in-PR commit): PASS — Run Sweep 39765601214 ran on the PR head 4d4e4c73 (attempt 2 success); all 9 multi-node agentic / and 7 multi-node agentic eval / check-runs are success. The single-node/fixed-seq lanes are skipped with empty matrices because the PR touches only multi-node agentic keys.

✅ Check 2 (Evals pass): PASS — eval_results_all gsm8k em_strict 0.9644–0.9719 (n=1319, infrastructure_success: true) across all 7 disagg points, above the 0.90 default (and the 0.94 glm5 bar), on image lmsysorg/sglang:nightly-dev-20261020-ab580b2e@sha256:0af411bb… as in the PR config.

✅ Check 3 (Upstream recipe merged and complete): PASS — sgl-project/sglang#46005 is MERGED (2026-10-23T08:04:02Z); glm-5.2.jsx at 02d24451 has all 8 GB200 cells (agg TP4/TP8; disagg 1P6D, 1P4D, 1P2D c64/c96, 1P1D, 2P1D, 3P1D). Their prefill/decode/agg flags, env, PP8 partition, NIXL/UCX settings, MoE/A2A backends, and Dynamo frontend/nginx counts match the recipes; the exclusions are documented as InferenceX-only.

✅ Check 4 (Reuse command): PASS — /use 39765601214 posted by nvpohanh (COLLABORATOR) on its own line, naming the Check 1 run.

✅ Check 5 (Latest checklist template): PASS — all 18 items of the current PR_REVIEW_CHECKLIST.md are present and checked.

✅ Check 6 (Upstream image / engine-first): PASS — both entries are framework: dynamo-sglang on the upstream lmsysorg/sglang image, pinned by digest; an open-source engine entry is exempt from the ordering rule.

✅ Check 7 (No deprecated models/scenarios): PASS — GLM-5.2 agentic coding is active in MODELS.md as of 2026-10-23.

✅ Check 8 (No architecture hacks): PASS — no --hf-overrides, json-model-override-args, or model-file edits in the diff.

✅ Check 9 (Spec-decode via chat template): PASS — the AgentX AIPerf replay uses the chat endpoint with chat_template_kwargs: {"enable_thinking": true} set in AIPERF_EXTRA_INPUTS of both recipes.

✅ Check 10 (No engine patches): PASS — the diff changes only the recipe YAMLs, nvidia-master.yaml, and perf-changelog.yaml; it adds no patches, inline source rewrites, or engine wheel installs.

✅ Check 11 (Golden simulated AL): PASS — synthetic_acceptance.py derives AL from speculative-num-steps: 5 → glm5.2_mtp.yaml thinking_on 5 = 3.61. The c36 job log shows SGLANG_SIMULATE_ACC_LEN=3.61, match-expected, and real-draft-token on every prefill and decode worker.

➖ Check 12 (Append-only): N/A — the new perf-changelog.yaml entry has no append-only key.

✅ Check 13 (Draft precision): PASS — the draft is the built-in NextN layer 78. hf_quant_config.json at aec724e8 excludes model.layers.78* from NVFP4 (it ships BF16). At ab580b2e, DeepseekModelNextN sets quant_config=None for modelopt_fp4, and the FP8 cast requires SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE (default EnvBool(False)), which is unset in the recipes and the worker env dumps; there is no draft-quantization flag. Effective: BF16, with FP8 KV shared with the target.

✅ Check 14 (Pareto coverage): PASS — one curve, GLM-5.2 / agentic-coding / gb200 dynamo-sglang / fp4 / run 39765601214. Recomputed from results_bmk with pareto_coverage.py: P90 7/9 frontier points, P75 7/9 (agg TP4 c1 and TP8 c1 dominated).

✅ Check 15 (PR description): PASS — stated topologies, node counts, mem-fraction, running-request, HiCache, PP partition, image, and Dynamo version claims match the recipes and master config at 4d4e4c73.

✅ Check 16 (NVIDIA overview): PASS — GLM-5.2 has an SGLang cell in the Framework Recipes table of the NVIDIA overview.

➖ Check 17 (Metrics publication): N/A — both affected keys are dynamo-sglang; no Dynamo TRT-LLM serving path is affected.

Assessed commit: 4d4e4c7392c088d16ba83b0b9db165b7b752a99e.

@Oseltamivir
Oseltamivir merged commit 4416f5e into main Oct 23, 2026
28 checks passed
@Oseltamivir
Oseltamivir deleted the pohanh/glm52-gb200-dynamo-sglang-pp8-1022 branch October 23, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants