Repository navigation
[NVIDIA][AgentX] GB300 DeepSeek-V4-Pro FP4 Dynamo+SGLang: PP8-prefill disagg and TP4/TP8 agg DSpark points / [NVIDIA][AgentX] GB300 DeepSeek-V4-Pro FP4 Dynamo+SGLang:PP8 prefill 分离式与 TP4/TP8 聚合 DSpark 配置点 - #4401
Conversation
Move dsv4-fp4-gb300-dynamo-sglang-agentic-agg and -disagg to lmsysorg/sglang:nightly-dev-20261019-3c560089 with its bundled Dynamo 1.6.0.dev20261018. Disagg uses TP1 x PP8 prefill workers with a 2x HiCache host tier and W4A4 MegaMoE for the target model on prefill and decode; the DSpark draft MoE stays at the image default (W4A8). Points: 1P+DEP16 c80/c160/c200 (KV router, token-balanced decode), 1P+DEP16 c300, 2P+DEP16 c720, 2P+DEP12 c800 and 2P+DEP8 c880 (round-robin frontend, request-balanced decode), with DSpark block size 6/6/5/5/4. Agg runs TP8 over two nodes at c1 and TP4 on one node at c1 and c4. 将 dsv4-fp4-gb300-dynamo-sglang-agentic-agg 与 -disagg 更新到 lmsysorg/sglang:nightly-dev-20261019-3c560089 及其自带的 Dynamo 1.6.0.dev20261018。 分离式配置使用 TP1 x PP8 预填充 worker,并保留 2 倍 HiCache 主机内存层; 目标模型在预填充与解码端均使用 W4A4 MegaMoE,DSpark 草稿 MoE 保持镜像默认 精度(W4A8)。测试点:1P+DEP16 c80/c160/c200(KV 路由,按 token 均衡解码), 1P+DEP16 c300、2P+DEP16 c720、2P+DEP12 c800 与 2P+DEP8 c880(前端轮询, 按请求数均衡解码),DSpark block size 分别为 6/6/5/5/4。 聚合式配置在两节点上以 TP8 运行 c1,在单节点上以 TP4 运行 c1 与 c4。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献!
|
|
|
||
| override_tp8: | ||
| name: "agg-gb300-tp8-mtp-lowlatency" | ||
| # TP8 over two nodes for the single-stream point. |
There was a problem hiding this comment.
🟡 nit (optional): New header comments that describe a topology break AGENTS.md's mandatory "No header comments" rule. sweep:^\s*# (TP8 over|TP4 on one|Low concurrency|One prefill worker|TP1 x PP8|DSpark block size 6|Prefill keeps a HiCache) Examples: "# TP8 over two nodes for the single-stream point." at agg-variants.yaml:117. Also "# Low concurrency: KV-aware decode routing, ..." at disagg-variants.yaml:233. Fix: delete them, or cut each down to the line-level reason a value was chosen.
Why this was flagged
AGENTS.md:64 says "Never add broad, overarching comments at the top of a file or above a recipe, override, or section. Do not describe what the file, topology, or config is." The diff adds comments that only restate the topology or config. In agg-variants.yaml there is one above override_tp8_c1 at :117 and one above override_tp4 at :128. In disagg-variants.yaml there is one above the prefill role at :60 and one above override_1p1d_pp8_dep16_kv at :233. In nvidia-master.yaml there are search-space and scenario headers at :6613, :6647 and :6650. Nothing breaks at runtime. The result is that the recipes and master config drift from a convention the repo marks as mandatory, which reviewers and later agents copy from the nearest pattern.
Verification: nit. The trigger is always present: the diff adds comments above overrides and sections that only describe the topology. AGENTS.md:63-64 says: "Do not describe what the file, topology, or config is." agg-variants.yaml:117 has # TP8 over two nodes for the single-stream point. directly above override_tp8_c1:. Nothing at runtime breaks.
There was a problem hiding this comment.
[by Claude Code] Addressed in cd647ee: removed the header comments that only restated topology or config from agg-variants.yaml, disagg-variants.yaml, and nvidia-master.yaml. The remaining comments give the line-level reason for a specific value.
Remove header comments that only restate topology or config, per the AGENTS.md code-comment rules. No recipe or matrix value changes. 按照 AGENTS.md 的代码注释规则,删除仅复述拓扑或配置的标题性注释。 配方与矩阵取值均无变化。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] I've started babysitting the pipeline of this PR. I'll keep monitoring it until it passes. If a job fails, I may rerun it when the cause is infrastructure, or push a fix or a merge of the latest |
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=39763988786 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=39763988786 |
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#46005 has been merged. This PR's AgentX configs ( This PR has no authorized |
|
/use 39763988786 |
|
@nvpohanh staged run 39763988786. This run remains available across future |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:Assessed head
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Sweep on in-PR commit): PASS — run 39763988786 (pull_request, head ✅ Check 2 (Evals): PASS — ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#46005 is MERGED (2026-10-23T08:04:02Z, ✅ Check 4 (Reuse command): PASS — COLLABORATOR ✅ Check 5 (Latest checklist): PASS — all items of the current ✅ Check 6 (Upstream image / engine order): PASS — both entries use upstream ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (Architecture hacks): PASS — the pinned recipes have no ✅ Check 9 (Chat template): PASS — the agentic replay client drives the Dynamo frontend's chat-completions endpoint, and both recipes set ✅ Check 10 (Engine patches): PASS — the diff and pinned recipes contain no patches, preambles, setup scripts or package installs ( ✅ Check 11 (Agentic golden AL): PASS — ➖ Check 12 (Append-only): N/A — the new changelog entry does not use ✅ Check 13 (Draft precision): PASS — the draft is the DSpark head bundled in the target checkpoint, with no draft quantization, dtype or path override, and ✅ Check 14 (Pareto coverage): PASS — curve dsv4 / agentic-coding / GB300 dynamo-sglang / fp4 / run 39763988786 / P90 E2EL vs ✅ Check 15 (PR description): PASS — the topologies, node counts (7/7/9/8/7), DSpark blocks, golden AL values, mem fractions, per-rank decode limits, routing modes, image, Dynamo version and removed points match the pinned recipes and master config. ✅ Check 16 (NVIDIA overview): PASS — the NVIDIA overview Framework Recipes table's DeepSeek-V4 row has a populated SGLang cell. It links to the SGLang DeepSeek-V4 cookbook, which explicitly covers ➖ Check 17 (Metrics publication): N/A — the Dynamo TRT-LLM metrics controls do not apply to these Assessed commit: |
Moves both GB300 DeepSeek-V4-Pro-0813 FP4 Dynamo + SGLang AgentX keys to
lmsysorg/sglang:nightly-dev-20261019-3c560089(multi-arch digest pinned), running the ai-dynamo1.6.0.dev20261018bundled in the image (dynamo.install: false). The disaggregated points move from DEP8 prefill workers with a Mooncake store to TP1/PP8 prefill workers that span two nodes each and keep a HiCache host tier.dsv4-fp4-gb300-dynamo-sglang-agentic-aggdsv4-fp4-gb300-dynamo-sglang-agentic-disaggDisaggregated node counts include one node shared by etcd, NATS and the AgentX client.
deepseek-ai/DeepSeek-V4-Pro-0813(1 step, top-k 1, block size as above and matched on prefill and decode). Synthetic acceptance follows the committeddsv4-pro-0813-dsparkthinking-on golden AL (3.77, 3.61 and 3.36 at block 6, 5 and 4), injected at launch. No recipe setsSGLANG_SIMULATE_ACC_*orSGLANG_NVFP4_CKPT_FP8_NEXTN_MOE.enable-w4a4-mxfp4-megamoe);no-speculative-enable-w4a4-mxfp4-megamoekeeps the DSpark draft MoE at the image default (W4A8), so the draft is served as shipped. Both roles enable the FP4 indexer.8,8,8,8,8,8,7,6, uses 16k chunks, 256 running requests and mem-fraction 0.88, with HiCache write_back athicache-ratio: 2(page_first_direct, direct I/O). That host tier is about 186 GB per PP rank and about 707 GB per prefill node, inside thedram-utilization: 0.80share of the 920,000 MiB gb300-nv node budget.中文
将 GB300 DeepSeek-V4-Pro-0813 FP4 Dynamo + SGLang AgentX 的两个 config key 迁移到
lmsysorg/sglang:nightly-dev-20261019-3c560089(固定 multi-arch digest),直接使用镜像自带的 ai-dynamo1.6.0.dev20261018(dynamo.install: false)。分离式配置点从带 Mooncake store 的 DEP8 prefill worker 改为 TP1/PP8 prefill worker,每个 worker 跨两个节点,并保留 HiCache 主机内存层。dsv4-fp4-gb300-dynamo-sglang-agentic-agg:dsv4-fp4-gb300-dynamo-sglang-agentic-disagg:deepseek-ai/DeepSeek-V4-Pro-0813自带的 DSpark draft(1 步、top-k 1,block size 如上,prefill 与 decode 一致)。模拟接受长度采用已提交的dsv4-pro-0813-dsparkthinking-on golden AL(block 6、5、4 分别为 3.77、3.61、3.36),在启动时注入。recipe 中未设置SGLANG_SIMULATE_ACC_*或SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE。enable-w4a4-mxfp4-megamoe);no-speculative-enable-w4a4-mxfp4-megamoe使 DSpark draft MoE 保持镜像默认精度(W4A8),即按发布形态提供 draft。两个角色均启用 FP4 indexer。8,8,8,8,8,8,7,6,使用 16k chunk、256 个运行请求、mem-fraction 0.88,并以hicache-ratio: 2(page_first_direct,direct I/O)启用 HiCache write_back。该主机内存层每个 PP rank 约 186 GB,每个 prefill 节点约 707 GB,在 gb300-nv 每节点 920,000 MiB 预算的dram-utilization: 0.80份额之内。AI model disclosure
claude-opus-5-5), via Claude Code.🤖 Generated with Claude Code