Skip to content

[NVIDIA][AgentX] GB300 DeepSeek-V4-Pro FP4 Dynamo+SGLang: PP8-prefill disagg and TP4/TP8 agg DSpark points / [NVIDIA][AgentX] GB300 DeepSeek-V4-Pro FP4 Dynamo+SGLang:PP8 prefill 分离式与 TP4/TP8 聚合 DSpark 配置点 - #4401

Merged
Oseltamivir merged 3 commits into
mainfrom
pohanh/dsv4-gb300-dynamo-sglang-pp8-dspark-1022
Oct 23, 2026

Conversation

@nvpohanh

Copy link
Copy Markdown
Collaborator

Moves both GB300 DeepSeek-V4-Pro-0813 FP4 Dynamo + SGLang AgentX keys to lmsysorg/sglang:nightly-dev-20261019-3c560089 (multi-arch digest pinned), running the ai-dynamo 1.6.0.dev20261018 bundled in the image (dynamo.install: false). The disaggregated points move from DEP8 prefill workers with a Mooncake store to TP1/PP8 prefill workers that span two nodes each and keep a HiCache host tier.

Config key Topology Conc Nodes Key settings
dsv4-fp4-gb300-dynamo-sglang-agentic-agg TP8 1 2 DSpark block 6, mem-fraction 0.94
TP4 1, 4 1 DSpark block 6, fastokens frontend tokenizer, mem-fraction 0.92
dsv4-fp4-gb300-dynamo-sglang-agentic-disagg 1x PP8 prefill + 1x DEP16 decode 80, 160, 200 7 DSpark block 6; KV-router frontend, decode balances DP ranks by tokens
1x PP8 prefill + 1x DEP16 decode 300 7 DSpark block 6; round-robin frontend, decode balances DP ranks by request count
2x PP8 prefill + 1x DEP16 decode 720 9 DSpark block 5; routing as c300
2x PP8 prefill + 1x DEP12 decode 800 8 DSpark block 5; routing as c300
2x PP8 prefill + 1x DEP8 decode 880 7 DSpark block 4; routing as c300

Disaggregated node counts include one node shared by etcd, NATS and the AgentX client.

  • Every point uses DSpark with the draft that ships in deepseek-ai/DeepSeek-V4-Pro-0813 (1 step, top-k 1, block size as above and matched on prefill and decode). Synthetic acceptance follows the committed dsv4-pro-0813-dspark thinking-on golden AL (3.77, 3.61 and 3.36 at block 6, 5 and 4), injected at launch. No recipe sets SGLANG_SIMULATE_ACC_* or SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE.
  • Disaggregated prefill and decode run the target MoE with W4A4 MegaMoE (enable-w4a4-mxfp4-megamoe); no-speculative-enable-w4a4-mxfp4-megamoe keeps the DSpark draft MoE at the image default (W4A8), so the draft is served as shipped. Both roles enable the FP4 indexer.
  • Prefill splits the 61 layers 8,8,8,8,8,8,7,6, uses 16k chunks, 256 running requests and mem-fraction 0.88, with HiCache write_back at hicache-ratio: 2 (page_first_direct, direct I/O). That host tier is about 186 GB per PP rank and about 707 GB per prefill node, inside the dram-utilization: 0.80 share of the 920,000 MiB gb300-nv node budget.
  • Decode allows 3072 running requests (192 per rank at DEP16, 256 at DEP12, 384 at DEP8) with decode CUDA graphs up to batch 256 per rank, mem-fraction 0.88 and SWA full-token ratio 0.02.
  • KV transfer uses Mooncake over MNNVL. Prefill workers advertise KV routing with a 3600 s session-affinity TTL to the prefill router. Five Dynamo frontends sit behind nginx session affinity. Tachometer and the Dynamo OTel collector are disabled.
  • Removes the previous DEP8-prefill points (1P1D c120/c480, 2P1D c960, 3P1D c1440, 4P1D c2400) and their Mooncake store services, and the previous agg TP8 c4 point.
中文

将 GB300 DeepSeek-V4-Pro-0813 FP4 Dynamo + SGLang AgentX 的两个 config key 迁移到 lmsysorg/sglang:nightly-dev-20261019-3c560089(固定 multi-arch digest),直接使用镜像自带的 ai-dynamo 1.6.0.dev20261018(dynamo.install: false)。分离式配置点从带 Mooncake store 的 DEP8 prefill worker 改为 TP1/PP8 prefill worker,每个 worker 跨两个节点,并保留 HiCache 主机内存层。

  • dsv4-fp4-gb300-dynamo-sglang-agentic-agg:
    • TP8 c1(2 个节点):DSpark block 6,mem-fraction 0.94。
    • TP4 c1 与 c4(1 个节点):DSpark block 6,前端使用 fastokens tokenizer,mem-fraction 0.92。
  • dsv4-fp4-gb300-dynamo-sglang-agentic-disagg:
    • 1 个 PP8 prefill + 1 个 DEP16 decode,c80/c160/c200(7 个节点):DSpark block 6;前端使用 KV router,decode 按 token 数在 DP rank 间均衡。
    • 1 个 PP8 prefill + 1 个 DEP16 decode,c300(7 个节点):DSpark block 6;前端轮询,decode 按请求数在 DP rank 间均衡。
    • 2 个 PP8 prefill + 1 个 DEP16 decode,c720(9 个节点):DSpark block 5;路由同 c300。
    • 2 个 PP8 prefill + 1 个 DEP12 decode,c800(8 个节点):DSpark block 5;路由同 c300。
    • 2 个 PP8 prefill + 1 个 DEP8 decode,c880(7 个节点):DSpark block 4;路由同 c300。
    • 分离式节点数包含 etcd、NATS 与 AgentX 客户端共用的一个节点。
  • 所有配置点均使用 deepseek-ai/DeepSeek-V4-Pro-0813 自带的 DSpark draft(1 步、top-k 1,block size 如上,prefill 与 decode 一致)。模拟接受长度采用已提交的 dsv4-pro-0813-dspark thinking-on golden AL(block 6、5、4 分别为 3.77、3.61、3.36),在启动时注入。recipe 中未设置 SGLANG_SIMULATE_ACC_* 或 SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE。
  • 分离式 prefill 与 decode 的目标模型 MoE 使用 W4A4 MegaMoE(enable-w4a4-mxfp4-megamoe);no-speculative-enable-w4a4-mxfp4-megamoe 使 DSpark draft MoE 保持镜像默认精度(W4A8),即按发布形态提供 draft。两个角色均启用 FP4 indexer。
  • prefill 将 61 层划分为 8,8,8,8,8,8,7,6,使用 16k chunk、256 个运行请求、mem-fraction 0.88,并以 hicache-ratio: 2(page_first_direct,direct I/O)启用 HiCache write_back。该主机内存层每个 PP rank 约 186 GB,每个 prefill 节点约 707 GB,在 gb300-nv 每节点 920,000 MiB 预算的 dram-utilization: 0.80 份额之内。
  • decode 允许 3072 个运行请求(DEP16 每 rank 192 个,DEP12 每 rank 256 个,DEP8 每 rank 384 个),decode CUDA graph 每 rank 最大 batch 256,mem-fraction 0.88,SWA full-token ratio 0.02。
  • KV 传输使用基于 MNNVL 的 Mooncake。prefill worker 向 prefill router 声明 KV 路由及 3600 秒 session affinity TTL。5 个 Dynamo frontend 位于 nginx session affinity 之后。关闭 Tachometer 与 Dynamo OTel collector。
  • 移除此前的 DEP8 prefill 配置点(1P1D c120/c480、2P1D c960、3P1D c1440、4P1D c2400)及其 Mooncake store 服务,以及此前的聚合 TP8 c4 配置点。
AI model disclosure
  • Model/version: Claude Opus 5.5 (claude-opus-5-5), via Claude Code.
  • Role: drove the automated configuration search whose measured settings these recipes reproduce, ported those settings into the recipes and master-config entries, wrote the changelog entry and PR text, and ran the local recipe generation, matrix and test checks.

🤖 Generated with Claude Code

Move dsv4-fp4-gb300-dynamo-sglang-agentic-agg and -disagg to
lmsysorg/sglang:nightly-dev-20261019-3c560089 with its bundled Dynamo
1.6.0.dev20261018.

Disagg uses TP1 x PP8 prefill workers with a 2x HiCache host tier and
W4A4 MegaMoE for the target model on prefill and decode; the DSpark draft
MoE stays at the image default (W4A8). Points: 1P+DEP16 c80/c160/c200
(KV router, token-balanced decode), 1P+DEP16 c300, 2P+DEP16 c720,
2P+DEP12 c800 and 2P+DEP8 c880 (round-robin frontend, request-balanced
decode), with DSpark block size 6/6/5/5/4.

Agg runs TP8 over two nodes at c1 and TP4 on one node at c1 and c4.

将 dsv4-fp4-gb300-dynamo-sglang-agentic-agg 与 -disagg 更新到
lmsysorg/sglang:nightly-dev-20261019-3c560089 及其自带的 Dynamo
1.6.0.dev20261018。

分离式配置使用 TP1 x PP8 预填充 worker,并保留 2 倍 HiCache 主机内存层;
目标模型在预填充与解码端均使用 W4A4 MegaMoE,DSpark 草稿 MoE 保持镜像默认
精度(W4A8)。测试点:1P+DEP16 c80/c160/c200(KV 路由,按 token 均衡解码),
1P+DEP16 c300、2P+DEP16 c720、2P+DEP12 c800 与 2P+DEP8 c880(前端轮询,
按请求数均衡解码),DSpark block size 分别为 6/6/5/5/4。

聚合式配置在两节点上以 TP8 运行 c1,在单节点上以 TP4 运行 c1 与 c4。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.


override_tp8:
name: "agg-gb300-tp8-mtp-lowlatency"
# TP8 over two nodes for the single-stream point.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): New header comments that describe a topology break AGENTS.md's mandatory "No header comments" rule. sweep:^\s*# (TP8 over|TP4 on one|Low concurrency|One prefill worker|TP1 x PP8|DSpark block size 6|Prefill keeps a HiCache) Examples: "# TP8 over two nodes for the single-stream point." at agg-variants.yaml:117. Also "# Low concurrency: KV-aware decode routing, ..." at disagg-variants.yaml:233. Fix: delete them, or cut each down to the line-level reason a value was chosen.

Why this was flagged

AGENTS.md:64 says "Never add broad, overarching comments at the top of a file or above a recipe, override, or section. Do not describe what the file, topology, or config is." The diff adds comments that only restate the topology or config. In agg-variants.yaml there is one above override_tp8_c1 at :117 and one above override_tp4 at :128. In disagg-variants.yaml there is one above the prefill role at :60 and one above override_1p1d_pp8_dep16_kv at :233. In nvidia-master.yaml there are search-space and scenario headers at :6613, :6647 and :6650. Nothing breaks at runtime. The result is that the recipes and master config drift from a convention the repo marks as mandatory, which reviewers and later agents copy from the nearest pattern.

Verification: nit. The trigger is always present: the diff adds comments above overrides and sections that only describe the topology. AGENTS.md:63-64 says: "Do not describe what the file, topology, or config is." agg-variants.yaml:117 has # TP8 over two nodes for the single-stream point. directly above override_tp8_c1:. Nothing at runtime breaks.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] Addressed in cd647ee: removed the header comments that only restated topology or config from agg-variants.yaml, disagg-variants.yaml, and nvidia-master.yaml. The remaining comments give the line-level reason for a specific value.

Remove header comments that only restate topology or config, per the
AGENTS.md code-comment rules. No recipe or matrix value changes.

按照 AGENTS.md 的代码注释规则,删除仅复述拓扑或配置的标题性注释。
配方与矩阵取值均无变化。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh

Copy link
Copy Markdown
Collaborator Author

[by Claude Code] I've started babysitting the pipeline of this PR. I'll keep monitoring it until it passes. If a job fails, I may rerun it when the cause is infrastructure, or push a fix or a merge of the latest main when the cause is in this PR.

@nvpohanh nvpohanh added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@nvpohanh

Copy link
Copy Markdown
Collaborator Author

[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs (dsv4-fp4-gb300-dynamo-sglang-agentic-agg, dsv4-fp4-gb300-dynamo-sglang-agentic-disagg) will happen in sgl-project/sglang#46005, in the Agentic Long-Context Deployment section of the DeepSeek-V4 cookbook page.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (cd647ee00) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

@nvpohanh

Copy link
Copy Markdown
Collaborator Author

[by Claude Code] sgl-project/sglang#46005 has been merged. This PR's AgentX configs (dsv4-fp4-gb300-dynamo-sglang-agentic-agg, dsv4-fp4-gb300-dynamo-sglang-agentic-disagg) are now in the Agentic Long-Context Deployment section of the SGLang DeepSeek-V4 cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4#agentic-long-context-deployment

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (cd647ee00) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

@nvpohanh

Copy link
Copy Markdown
Collaborator Author

/use 39763988786

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

@nvpohanh staged run 39763988786.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

@nvpohanh

nvpohanh commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator Author

As a PR reviewer and CODEOWNER, I have reviewed this and have:

  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
  • Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

Assessed head cd647ee0072122bbef9f53ed6fe502eb504813ff. Pinned run 39763988786 (run-sweep.yml, pull_request, head cd647ee0, conclusion success). Attempt 1 had one c880 benchmark failure: the prefill PP pipeline stalled and the NCCL watchdog aborted it after 600 s. Fail-fast then cancelled c300. Attempt 2 reran only those two jobs, and both succeeded. Reuse: /use 39763988786.

  • Scope: both configs are modified: dsv4-fp4-gb300-dynamo-sglang-agentic-agg (TP8 over 2 nodes c1; TP4 c1 and c4) and dsv4-fp4-gb300-dynamo-sglang-agentic-disagg (TP1×PP8 prefill with HiCache: 1P+DEP16 c80/c160/c200/c300, 2P+DEP16 c720, 2P+DEP12 c800, 2P+DEP8 c880). Both use framework: dynamo-sglang on cluster:gb300-nv with image lmsysorg/sglang:nightly-dev-20261019-3c560089@sha256:c1f4d9fc…48c0, an OCI multi-arch index for amd64 and arm64. The fingerprints show SGLANG_BUILD_COMMIT=3c560089e1818a5217a73f2168273485a5350341.

  • CODEOWNER: nvpohanh is listed for inferencex-e2e/configs/nvidia-master.yaml and for /inferencex-e2e/benchmarks/*_node/srt-slurm-recipes/**/gb300-*/. perf-changelog.yaml is covered by the catch-all only.

  • Code quality: the diff touches 2 recipe files and 2 master-config entries, and appends one changelog entry. The pr-link: …/pull/XXX placeholder follows CONTRIBUTING.md; merge_with_reuse fills it in at merge. CI, PR infx diff and check-changelog passed. The bot review found nothing blocking, and its single nit (descriptive header comments) is fixed in cd647ee0.

  • PR validation: all 10 multi-node agentic / benchmark jobs succeeded on the latest attempt, and none was skipped. Generic single-node */ jobs are skipped because this is a multi-node agentic sweep. Every point has a bmk_agentic_* artifact with records_error_dropped of 0, except c800 (9 of 96,115 profiled requests) and c880 (12 of 94,632). All of those are InvalidInferenceResultError.

  • Evals: the 5 multi-node agentic eval / jobs (disagg c200/c300/c720/c800/c880) succeeded on the same image. In eval_results_all/agg_eval_all.json, gsm8k em_strict is c200 0.9735, c300 0.9719, c720 0.9735, c800 0.9750 and c880 0.9727. The dsv4 threshold in infx/evals/thresholds.yaml is 0.91.

  • Chat template: the replay client sends POST /v1/chat/completions to the Dynamo frontends (465,515 requests in the frontend logs). Every point's frontend_0.out logs Detected DeepSeek V4 model, using native Rust formatter model_type=Some("deepseek_v4"). AIPERF_EXTRA_INPUTS sets chat_template_kwargs {"reasoning_effort":"high","thinking":true}.

  • Draft precision: the draft is the DSpark head bundled in deepseek-ai/DeepSeek-V4-Pro-0813 (mtp.*), loaded from the target checkpoint. Worker log: DSpark draft weights are bundled in the target checkpoint; defaulting --speculative-draft-model-path to --model-path (/model).

    • Shipped: HF main is 72e1d323, last modified 2026-08-13. config.json has quantization_config fp8/e4m3/ue8m0 with 128×128 blocks, expert_dtype: fp4 and dspark_block_size: 5. The safetensors headers of shards 64–66 hold all 7,009 mtp.* tensors:
      • attention, shared experts and main_proj are F8_E4M3 with F8_E8M0 scales;
      • routed experts are FP4: 3,456 I8-packed weights with F8_E8M0 scales;
      • norms and other weights are BF16;
      • attn_sink, the gate bias and hc_* are F32.
    • Effective: the recipes set no draft quantization, draft dtype or draft path. Logged server args on every worker of all 10 benchmark and 5 eval jobs (15 server-log artifacts):
      • speculative_draft_model_quantization=None, speculative_draft_kv_cache_dtype=None, quantization=None, dtype=auto, json_model_override_args={};
      • every draft load logs type=DeepseekV4ForCausalLMDSpark, quant=fp8, fmt=e4m3.
    • Disagg W4A4 target, W4A8 draft: prefill and decode set enable-w4a4-mxfp4-megamoe: true for the target and no-speculative-enable-w4a4-mxfp4-megamoe: true. The logs show enable_w4a4_mxfp4_megamoe=True and speculative_enable_w4a4_mxfp4_megamoe=False.
      • In SGLang 3c560089, an unset speculative flag makes the draft inherit the target's W4A4 (moe/utils.py#L582, #L609-L615). False keeps the draft's MegaMoE MMA at the default fp8xfp4, rather than mxf4xmxf4 (mega_moe.py#L60).
      • Draft routed experts therefore run FP4 weights × FP8 activations, which is DeepSeek's reference act_quant + fp4_gemm and the image default without any W4A4 setting.
    • Agg: TP4/TP8 set neither W4A4 flag (enable_w4a4_mxfp4_megamoe=False, speculative_enable_w4a4_mxfp4_megamoe=None) and use the flashinfer_mxfp4 default MXFP8 activations. Draft attention dequantizes FP8 wo_a to BF16 by default (deepseek_v4_dspark.py#L113-L121). The KV cache is the DeepSeek-V4 default for target and draft.
    • SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE: not enabled. Its default is EnvBool(False) (environ.py#L1090). It is absent from:
      • the PR diff and both recipes, at the top level and in all overrides;
      • every rendered config.yaml;
      • all 33 fingerprint_*.json runtime environments across the 15 artifacts, which include image ENV such as SGLANG_BUILD_COMMIT.
    • Draft length: speculative-dspark-block-size is 6 for c1–c300, 5 for c720/c800 and 4 for c880, each matched on prefill and decode with speculative-num-draft-tokens = block + 1. All are supported runtime choices; the logs show using gamma=6/4 … draft config block_size=5, and gamma 5 matches the shipped value.
  • Golden AL: dsv4-pro-0813-dspark.yaml thinking_on gives block 6 = 3.77, 5 = 3.61 and 4 = 3.36. infx/srt_slurm/synthetic_acceptance.py injects these values; the recipes do not set them. Every benchmark worker gets SGLANG_SIMULATE_ACC_METHOD=match-expected and SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token, with SGLANG_SIMULATE_ACC_LEN as follows:

    • 3.77 at c1 (TP4 and TP8), c4, c80, c160, c200 and c300;
    • 3.61 at c720 and c800;
    • 3.36 at c880.
      Eval-only jobs run real acceptance, with no SGLANG_SIMULATE_ACC_*.
  • MODELS.md: dsv4 Agentic coding is active, and its support row lists the native DSpark heads of deepseek-ai/DeepSeek-V4-Pro-0813. Nothing in this PR is deprecated as of 2026-10-23.

  • Architecture: there is no --hf-overrides, json_model_override_args is {}, and nothing skips layers, experts or the indexer. SGLANG_PP_LAYER_PARTITION=8,8,8,8,8,8,7,6 only places all 61 layers across PP ranks. The FP4 indexer and target W4A4 MegaMoE are lower-precision target computations, and the evals cover them.

  • Dynamo TRT-LLM metrics: N/A. This is Dynamo + SGLang, not TRT-LLM. The recipes use the image's bundled Dynamo 1.6.0.dev20261018 (dynamo.install: false).

  • Image / engine order: upstream lmsysorg/sglang, run as shipped. dynamo-sglang is an open-source engine deployment, so the ordering rule does not apply.

  • Upstream recipe: merged SGLang cookbook PR [Doc] cookbook: Refresh SemiAnalysis AgentX launches with seven open InferenceX PRs sgl-project/sglang#46005 (MERGED 2026-10-23T08:04:02Z, merge commit 02d24451). It is published at https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4#agentic-long-context-deployment. The source is deepseek-v4.jsx, identical on main and at the merge commit. The relevant cells all have prov.sha cd647ee0, the same image digest, and the golden AL per point:

    • dsv4-fp4-gb300-dynamo-sglang-agentic-agg#0 (TP8, 2 nodes) and agg#1 (TP4, c1/c4);
    • disagg#0 (1P1D PP8 + DEP16, c80/c160/c200 with the KV router; c300 patched to --router-mode round-robin and --load-balance-method total_requests);
    • disagg#2 (2P1D DEP16 c720), disagg#3 (DEP12 c800) and disagg#4 (DEP8 c880).
    • Coverage: I decoded each cell with the page's own decoder and diffed it against the effective recipe (base + override) for every point and role. Every serving flag and production env var in the recipe appears with the same value, and none differs. This covers:
      • etcd, NATS and nginx session affinity, and the 5 Dynamo frontends with their router args;
      • the prefill worker (2 nodes per worker): PP8, SGLANG_PP_LAYER_PARTITION, HiCache hicache-ratio 2 write_back direct page_first_direct, chunked-prefill-size 16384, max-running-requests 256, W4A4 flags, FP4 indexer, MegaMoE and Mooncake MNNVL env, and DSpark;
      • the decode worker: DEP16/12/8, max-running-requests 3072, cuda-graph-max-bs-decode 256, load-balance-method, and the MegaMoE env;
      • the agg worker and the fastokens frontend env.
    • Excluded as InferenceX-only:
      • Slurm, srun and health_check options;
      • PIP_BREAK_SYSTEM_PACKAGES and PYTHONUNBUFFERED;
      • logging-only settings: DYN_LOG, DYN_SKIP_SGLANG_LOG_FORMATTING, SGLANG_LOG_FORWARD_ITERS, SGLANG_LOG_MS and gc-warning-threshold-secs (which only sets when a slow-GC warning is logged; gc-threshold is covered);
      • enable-metrics and enable-cache-report, which drive benchmark server-metric scraping and cached-token reporting without changing scheduling;
      • AIPERF_* and AGENTIC_WARMUP_GRACE_PERIOD;
      • SGLANG_SIMULATE_ACC_*, which the cookbook discloses as synthetic-acceptance;
      • dram-utilization and the concurrency values;
      • the /model mount path, which the cookbook replaces with the HF ID.
  • NVIDIA overview (Check 16): the listing check should use the Framework Recipes table of https://docs.nvidia.com/llm-inference-quick-start-recipes/overview, not the Dynamo Disaggregated Recipes table.

    • Which link covers this PR: the Framework Recipes table's DeepSeek-V4 row has a populated SGLang cell (✅). It links to https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4, the published page that holds the exact commands for this PR: the "Agentic Long-Context Deployment" section with the dsv4-fp4-gb300-dynamo-sglang-agentic-agg / -disagg cells from [Doc] cookbook: Refresh SemiAnalysis AgentX launches with seven open InferenceX PRs sgl-project/sglang#46005.
    • Why the -Pro row's SGLang cell is empty: the SGLang cookbook documents DeepSeek-V4-Pro, including deepseek-ai/DeepSeek-V4-Pro-0813, and DeepSeek-V4-Flash together on that single DeepSeek-V4 page, rather than on a separate DeepSeek-V4-Pro page. So the overview's DeepSeek-V4 SGLang link is the SGLang listing for this model.
    • Why the earlier verdict's link doesn't apply: the Dynamo table's DeepSeek-V4-Pro cell links to ai-dynamo's own Dynamo recipe, which is a separate document. It is not the SGLang listing this PR relies on.
  • No engine patches: there are no .patch files, git apply, sed/heredoc rewrites, monkey-patching, setup scripts or package installs; dynamo.install: false. No waiver is needed.

  • Append-only: N/A. No changelog entry uses append-only: true.

  • Pareto coverage: there is one affected curve: dsv4 / agentic-coding / GB300 dynamo-sglang / fp4 / run 39763988786 / P90. Both config keys map to this series, and no other GB300 dynamo-sglang dsv4 config is on main. I took throughput.per_gpu.total_tput_tps against latency.e2el.p90 from the 10 bmk_agentic_* artifacts and ran infx.workflows.pareto_coverage. Result: 7 frontier points of 10, 0 invalid:

    • (15.30 s, 21,896) 1P1D c80;
    • (17.54, 38,601) c160;
    • (18.80, 47,593) c200;
    • (21.41, 67,823) c300;
    • (27.00, 111,603) 2P1D DEP16 c720;
    • (33.04, 128,718) DEP12 c800;
    • (43.14, 143,775) DEP8 c880.

    TP4 c4 (15.59, 7,333), TP8 c1 (16.80, 2,545) and TP4 c1 (18.28, 4,841) are dominated. Dashboard: https://inferencex.semianalysis.com/inference?unofficialRun=39763988786

Signed: nvpohanh

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

✅✅✅ Verdict: PASS ✅✅✅

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — nvpohanh is a named owner of inferencex-e2e/configs/nvidia-master.yaml and /inferencex-e2e/benchmarks/*_node/srt-slurm-recipes/**/gb300-*/ (both recipe files); perf-changelog.yaml falls only under the catch-all.

✅ Check 1 (Sweep on in-PR commit): PASS — run 39763988786 (pull_request, head cd647ee0 = PR tip, attempt 2, success): all 10 multi-node agentic / and 5 multi-node agentic eval / check-runs on cd647ee0 executed with success; the generic single-node */ and eval / jobs are skipped because this sweep is multi-node agentic only.

✅ Check 2 (Evals): PASS — eval_results_all/agg_eval_all.json gives gsm8k em_strict 0.9735 / 0.9719 / 0.9735 / 0.9750 / 0.9727 (disagg c200 / c300 / c720 / c800 / c880, n=1319 each), above the dsv4 threshold of 0.91, on the PR's lmsysorg/sglang:nightly-dev-20261019-3c560089@sha256:c1f4d9fc… image.

✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#46005 is MERGED (2026-10-23T08:04:02Z, 02d24451). At the merge commit, deepseek-v4.jsx has cells agg#0/#1 and disagg#0/#2/#3/#4 with prov.sha cd647ee0 and the same image digest. They carry the etcd/NATS/nginx and 5-frontend commands (KV router, plus the c300+ round-robin/total_requests patch), the PP8 prefill (SGLANG_PP_LAYER_PARTITION, HiCache ratio 2 write_back, W4A4 plus --no-speculative-enable-w4a4-mxfp4-megamoe), the DEP16/12/8 decode, the TP4 fastokens agg env, and DSpark 6/5/4. The documented exclusions are harness, logging or synthetic-acceptance settings only.

✅ Check 4 (Reuse command): PASS — COLLABORATOR nvpohanh posted /use 39763988786 on its own line (comment).

✅ Check 5 (Latest checklist): PASS — all items of the current PR_REVIEW_CHECKLIST.md template, including the merged-upstream sub-item and the Pareto item, are present and checked.

✅ Check 6 (Upstream image / engine order): PASS — both entries use upstream lmsysorg/sglang:nightly-dev-20261019-3c560089 (digest-pinned) with framework: dynamo-sglang, an open-source engine entry, so the ordering rule does not apply.

✅ Check 7 (Deprecated models): PASS — dsv4 Agentic coding is active in MODELS.md as of 2026-10-23, and its support row covers the native DSpark heads of deepseek-ai/DeepSeek-V4-Pro-0813.

✅ Check 8 (Architecture hacks): PASS — the pinned recipes have no --hf-overrides, override args or layer/expert/indexer skipping; SGLANG_PP_LAYER_PARTITION=8,8,8,8,8,8,7,6 only places the 61 layers across PP ranks.

✅ Check 9 (Chat template): PASS — the agentic replay client drives the Dynamo frontend's chat-completions endpoint, and both recipes set AIPERF_EXTRA_INPUTS chat_template_kwargs {"reasoning_effort":"high","thinking":true}.

✅ Check 10 (Engine patches): PASS — the diff and pinned recipes contain no patches, preambles, setup scripts or package installs (dynamo.install: false).

✅ Check 11 (Agentic golden AL): PASS — infx/srt_slurm/synthetic_acceptance.py injects SGLANG_SIMULATE_ACC_LEN (with match-expected, real-draft-token) from the generation role's speculative-dspark-block-size, giving 3.77 for block 6 (agg c1/c4, disagg c80–c300), 3.61 for block 5 (c720/c800) and 3.36 for block 4 (c880). These match dsv4-pro-0813-dspark.yaml thinking_on, and the recipes set no AL themselves.

➖ Check 12 (Append-only): N/A — the new changelog entry does not use append-only: true.

✅ Check 13 (Draft precision): PASS — the draft is the DSpark head bundled in the target checkpoint, with no draft quantization, dtype or path override, and SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is unset (EnvBool(False), environ.py#L1090). On disagg, no-speculative-enable-w4a4-mxfp4-megamoe makes _draft_w4a4_mxfp4_megamoe_override keep the draft MegaMoE MMA at the default fp8xfp4. On agg, flashinfer_mxfp4 is the image's auto default for DSV4 on Blackwell (arg_groups/model_overrides/deepseek_v4.py).

✅ Check 14 (Pareto coverage): PASS — curve dsv4 / agentic-coding / GB300 dynamo-sglang / fp4 / run 39763988786 / P90 E2EL vs total_tput_tps, from the 10 bmk_agentic_* artifacts on the same image: infx.workflows.pareto_coverage gives 7/5 frontier points (c80, c160, c200, c300, c720, c800, c880) with 0 invalid. The agg TP4 c1/c4 and TP8 c1 points are dominated.

✅ Check 15 (PR description): PASS — the topologies, node counts (7/7/9/8/7), DSpark blocks, golden AL values, mem fractions, per-rank decode limits, routing modes, image, Dynamo version and removed points match the pinned recipes and master config.

✅ Check 16 (NVIDIA overview): PASS — the NVIDIA overview Framework Recipes table's DeepSeek-V4 row has a populated SGLang cell. It links to the SGLang DeepSeek-V4 cookbook, which explicitly covers deepseek-ai/DeepSeek-V4-Pro-0813 (Pro Official 0813 with bundled DSpark) and holds this PR's GB300 cells.

➖ Check 17 (Metrics publication): N/A — the Dynamo TRT-LLM metrics controls do not apply to these dynamo-sglang workers, which set enable-metrics: true on every role.

Assessed commit: cd647ee0072122bbef9f53ed6fe502eb504813ff.

@Oseltamivir
Oseltamivir merged commit 788f218 into main Oct 23, 2026
28 checks passed
@Oseltamivir
Oseltamivir deleted the pohanh/dsv4-gb300-dynamo-sglang-pp8-dspark-1022 branch October 23, 2026 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants