Skip to content

[Klaud Cold] Update DeepSeek-V4.1-Flash FP4 GB300 vLLM AgentX DSpark image to nightly-00b6c4c5 / [Klaud Cold] 将 DeepSeek-V4.1-Flash FP4 GB300 vLLM AgentX DSpark 镜像更新至 nightly-00b6c4c5 - #4558

Merged
adibarra merged 24 commits into
mainfrom
klaud/auto-0585c424a63a787d-7b81b812f72a4c1b
Nov 6, 2026
Merged

adibarra merged 24 commits into
mainfrom
klaud/auto-0585c424a63a787d-7b81b812f72a4c1b

Conversation

@Klaud-Cold

@Klaud-Cold Klaud-Cold commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Goal: Update vLLM image from vllm/vllm-openai:nightly-dev-arm64-cu130-ac9126e58aa7 to vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b.
Baseline: 2026-10-02 · vllm/vllm-openai:nightly-dev-arm64-cu130-ac9126e58aa7
Mean latency · Sources: API 1, API 2, API 3, API 4

Point Total tok/s/GPU ↑ Output tok/s/GPU ↑ TTFT ms ↓ TPOT ms ↓
AgentX c1 TP4 EP1 26c006 7,786.02 46.72 190.09 1.63
AgentX c4 TP4 EP1 015548 9,853.54 67.92 138.72 1.81
AgentX c8 TP2 EP2 adf0de 30,707.77 221.93 186.76 2.97
AgentX c8 TP4 EP1 529367 16,766.26 116.11 124.26 2.04
AgentX c16 TP2 EP2 080255 57,472.58 364.34 162.41 3.33
AgentX c16 TP4 EP1 9faa1f 29,327.4 189.85 139.44 2.4
AgentX c32 TP2 EP2 8f5bc0 120,679.53 692.92 206.41 4.52
AgentX c64 TP2 EP2 d6aedf 219,542.57 1,541.95 249.22 7.3
AgentX c128 TP2 EP2 a2c3ad 304,874.85 2,440.16 365.16 12.46
AgentX c192 TP2 EP2 164f25 415,091.09 3,108.26 459.37 17.39
Eval Score ↑ Samples
gsm8k/em_strict · c16 97.27% 1,319
gsm8k/em_strict · c192 97.04% 1,319

Stop: Close this PR at any time to stop this candidate now; Klaud cancels its runs and deletes its branch.

中文

**目标:**将 vLLM 镜像从 vllm/vllm-openai:nightly-dev-arm64-cu130-ac9126e58aa7 更新至 vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b。
**基线:**2026-10-02 · vllm/vllm-openai:nightly-dev-arm64-cu130-ac9126e58aa7
平均延迟 · 来源: API 1, API 2, API 3, API 4;数值及异常说明见上表。
**停止:**随时关闭此 PR 即可立即停止该候选:Klaud 会取消其运行并删除其分支。

…-00b6c4c5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

Initial attempt · Running · Run 40426160663 / attempt 1 · 2026-10-27 00:25 UTC
vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4 · 9800514def3f · Mean latency
Change: Image moves from vLLM main nightly ac9126e5 (2026-10-01, release-v2 #7614) to main nightly 00b6c4c5 (2026-10-26, release-v2 #9153, CUDA 13.0.2), digest-pinned; recipe unchanged. Source diff vllm-project/vllm@ac9126e...00b6c4c keeps every used flag (deepseek_v41 parsers, engram, FLASHINFER_MLA_SPARSE_DSV41/FLASHMLA_MEGA_ATTN_DSV41 mxfp4 indexer, deep_gemm_mega_moe, dspark sampling, V2 runner, Rust frontend). Unverified: decoder_replay_cudagraph_capture_sizes appears in compilation.py only at 00b6c4c5.

Point Output tok/s/GPU ↑ TTFT ms ↓ TPOT ms ↓
AgentX c1 TP4 EP1 f96de5 N/A N/A N/A
AgentX c8 TP2 EP2 c83f24 N/A N/A N/A

Note: All rows: pending.
Note: All rows: request errors unavailable.
Note: All rows: Δ N/A: no matched baseline.

Next: Wait for the smoke run (c1 TP4 and c8 TP2 throughput, c16 TP4 and c192 TP2 evals) on gb300-nv.

中文

初次尝试 · 运行中 · Run 40426160663 / attempt 1 · 2026-10-27 00:25 UTC
vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4 · 9800514def3f · 平均延迟
**变更:**镜像从 vLLM main nightly ac9126e5(2026-10-01,release-v2 #7614)更新为 main nightly 00b6c4c5(2026-10-26,release-v2 #9153,CUDA 13.0.2),按 digest 固定;配方未改动。源码对比 vllm-project/vllm@ac9126e...00b6c4c 保留所有使用的参数(deepseek_v41 解析器、engram、FLASHINFER_MLA_SPARSE_DSV41/FLASHMLA_MEGA_ATTN_DSV41 mxfp4 indexer、deep_gemm_mega_moe、dspark 采样、V2 runner、Rust 前端)。未核实:decoder_replay_cudagraph_capture_sizes 仅在 00b6c4c5 的 compilation.py 中出现。实测数值及异常说明见上表。
**下一步:**等待 gb300-nv 上的 smoke 运行(c1 TP4 与 c8 TP2 吞吐,c16 TP4 与 c192 TP2 评测)完成。

@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

parked · Session 1 is waiting on 40426160663. Recovery reruns preempted runs and starts the next session once they end.

中文

parked · 第 1 个会话正在等待 40426160663。恢复流程会重跑被抢占的运行,并在运行结束后启动下一个会话。

@github-actions github-actions Bot added the klaud-parked Klaud parked until its queued or running benchmark runs end label Oct 27, 2026
@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

resumed · The session ended before finishing; session 2 continues it in a new run of its planner.

中文

resumed · 会话未完成即已结束;第 2 个会话将在其规划运行的新运行中继续。

@github-actions github-actions Bot added klaud-in-progress Klaud is working on this candidate or will resume it and removed klaud-parked Klaud parked until its queued or running benchmark runs end labels Oct 27, 2026
@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

Initial attempt · Passed · Run 40426160663 / attempt 3 · 2026-10-27 04:49 UTC
vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4 · 9800514def3f · Mean latency
Change: Image moves from vLLM main nightly ac9126e5 (2026-10-01, release-v2 #7614) to main nightly 00b6c4c5 (2026-10-26, release-v2 #9153, CUDA 13.0.2), digest-pinned; recipe unchanged. Source diff vllm-project/vllm@ac9126e...00b6c4c keeps every used flag (deepseek_v41 parsers, engram, FLASHINFER_MLA_SPARSE_DSV41/FLASHMLA_MEGA_ATTN_DSV41 mxfp4 indexer, deep_gemm_mega_moe, dspark sampling, V2 runner, Rust frontend). Unverified: decoder_replay_cudagraph_capture_sizes appears in compilation.py only at 00b6c4c5.

Point Output tok/s/GPU ↑ TTFT ms ↓ TPOT ms ↓
AgentX c1 TP4 EP1 f96de5 46.76 (N/A) 235.43 (N/A) 1.61 (N/A)
AgentX c8 TP2 EP2 c83f24 223.47 (N/A) 151.04 (N/A) 2.77 (N/A)

Note: All rows: Δ N/A: no matched baseline.
Note: AgentX c8 TP2 EP2 c83f24: 2 request errors.

Eval Score ↑ Samples
gsm8k/em_strict · c16 97.35% (+0.08 pp) 1,319 each
gsm8k/em_strict · c192 97.42% (+0.38 pp) 1,319 each

Next: Smoke passed on attempt 3 after two preemptions (c8 had 2 empty-content turns, no server error); append the perf-changelog entry and dispatch the full-sweep-fail-fast final sweep.

中文

初次尝试 · 已通过 · Run 40426160663 / attempt 3 · 2026-10-27 04:49 UTC
vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4 · 9800514def3f · 平均延迟
**变更:**镜像从 vLLM main nightly ac9126e5(2026-10-01,release-v2 #7614)更新为 main nightly 00b6c4c5(2026-10-26,release-v2 #9153,CUDA 13.0.2),按 digest 固定;配方未改动。源码对比 vllm-project/vllm@ac9126e...00b6c4c 保留所有使用的参数(deepseek_v41 解析器、engram、FLASHINFER_MLA_SPARSE_DSV41/FLASHMLA_MEGA_ATTN_DSV41 mxfp4 indexer、deep_gemm_mega_moe、dspark 采样、V2 runner、Rust 前端)。未核实:decoder_replay_cudagraph_capture_sizes 仅在 00b6c4c5 的 compilation.py 中出现。实测数值及异常说明见上表。
**下一步:**smoke 在两次抢占后于第 3 次尝试通过(c8 有 2 个空内容轮次,无服务端错误);追加 perf-changelog 条目并启动 full-sweep-fail-fast 最终扫描。

@Klaud-Cold Klaud-Cold added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 27, 2026
@Klaud-Cold

Klaud-Cold commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator Author

Final full sweep · Passed · Run 40451966589 / attempt 1 · 2026-10-27 07:21 UTC
vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4 · 6609c4983964 · Mean latency
Change: Validate the complete updated-image family at this head.

Point Output tok/s/GPU ↑ TTFT ms ↓ TPOT ms ↓
AgentX c1 TP4 EP1 26c006 46.77 (+0.1%) 236.94 (+24.6%) 1.6 (-1.8%)
AgentX c4 TP4 EP1 015548 66.97 (-1.4%) 207.68 (+49.7%) 1.79 (-1.1%)
AgentX c8 TP2 EP2 adf0de 222.87 (+0.4%) 147.82 (-20.9%) 2.8 (-5.7%)
AgentX c8 TP4 EP1 529367 117.7 (+1.4%) 136.24 (+9.6%) 1.9 (-6.9%)
AgentX c16 TP2 EP2 080255 365.44 (+0.3%) 165.89 (+2.1%) 3.3 (-0.9%)
AgentX c16 TP4 EP1 9faa1f 192.36 (+1.3%) 153.23 (+9.9%) 2.34 (-2.5%)
AgentX c32 TP2 EP2 8f5bc0 695.42 (+0.4%) 212.12 (+2.8%) 4.47 (-1.1%)
AgentX c64 TP2 EP2 d6aedf 1,543.78 (+0.1%) 238.5 (-4.3%) 7.47 (+2.3%)
AgentX c128 TP2 EP2 a2c3ad 2,506.06 (+2.7%) 295.56 (-19.1%) 11.38 (-8.7%)
AgentX c192 TP2 EP2 164f25 3,092.8 (-0.5%) 417.22 (-9.2%) 17.46 (+0.4%)

Note: AgentX c8 TP2 EP2 adf0de, AgentX c16 TP4 EP1 9faa1f, AgentX c32 TP2 EP2 8f5bc0: 1 request error.
Note: AgentX c128 TP2 EP2 a2c3ad: 4 request errors.
Note: AgentX c192 TP2 EP2 164f25: 2 request errors.

Eval Score ↑ Samples
gsm8k/em_strict · c16 97.27% (+0.00 pp) 1,319 each
gsm8k/em_strict · c192 97.27% (+0.23 pp) 1,319 each

Next: Mark ready for maintainer review.

中文

最终完整 sweep · 已通过 · Run 40451966589 / attempt 1 · 2026-10-27 07:21 UTC
vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4 · 6609c4983964 · 平均延迟
**变更:**验证此提交更新镜像后的完整配置族。实测数值及异常说明见上表。
**下一步:**标记为就绪,等待维护者审查。

@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

parked · Session 2 is waiting on 40451966589. Recovery reruns preempted runs and starts the next session once they end.

中文

parked · 第 2 个会话正在等待 40451966589。恢复流程会重跑被抢占的运行,并在运行结束后启动下一个会话。

@github-actions github-actions Bot added klaud-parked Klaud parked until its queued or running benchmark runs end and removed klaud-in-progress Klaud is working on this candidate or will resume it labels Oct 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@Klaud-Cold
Klaud-Cold marked this pull request as ready for review October 27, 2026 07:21
@Klaud-Cold
Klaud-Cold requested a review from a team October 27, 2026 07:21
@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

validated · Repairs: unknown · Runs: 40426160663, 40451922676, 40451966589
All owned runs ended. Full sweep verified; ready for review.

中文

validated · 修复次数:未知 · 运行:40426160663, 40451922676, 40451966589
所有自有运行均已结束。完整 sweep 已验证;已就绪,等待审查。

@github-actions github-actions Bot added klaud-finished Klaud verified the full sweep; ready for review and removed klaud-parked Klaud parked until its queued or running benchmark runs end labels Oct 27, 2026
@claude

claude Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @Klaud-Cold's task in 17s —— View job


LGTM - no blocking issues found

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

@adibarra

adibarra commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

/use 40451966589

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

@adibarra staged run 40451966589.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

Resolve perf-changelog.yaml by taking main's file and re-appending this PR's entry.

中文:合并 origin/main;perf-changelog.yaml 采用 main 版本,并在末尾重新追加本 PR 的条目。
@adibarra
adibarra merged commit f7e72d4 into main Nov 6, 2026
29 checks passed
@adibarra
adibarra deleted the klaud/auto-0585c424a63a787d-7b81b812f72a4c1b branch November 6, 2026 16:27
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category new duplicate skipped unmapped failed
benchmark 10 0 0 0 0
eval 2 2 2 0 0
trace 10 0 0 0 0
power 10 0 0 0 0

benchmarkFiles 10, rawBenchmarkRows 10, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 2638, gpuMetricSamplesNew 147650, powerPublicationPoints 10, changelogEntriesWritten 1

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

hjjq added a commit that referenced this pull request Nov 6, 2026
…eep]

Keep this PR's GB300 image (vLLM main a2bcb565) over #4558's nightly-00b6c4c5, which is 862 commits older.
Resolve the perf-changelog conflict with main's entries kept and this PR's entry last, and name
nightly-00b6c4c5 as the image this entry replaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) klaud-finished Klaud verified the full sweep; ready for review

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants