Skip to content

[Klaud Cold] Update MiniMax-M3 FP4 GB300 Dynamo-vLLM AgentX MTP disagg image to nightly-00b6c4c5 / [Klaud Cold] 将 MiniMax-M3 FP4 GB300 Dynamo-vLLM AgentX MTP disagg 镜像更新至 nightly-00b6c4c5 - #4557

Closed
Klaud-Cold wants to merge 1 commit into
mainfrom
klaud/auto-6308768623a5422d-903ece6b136fadb9
Closed

Klaud-Cold wants to merge 1 commit into
mainfrom
klaud/auto-6308768623a5422d-903ece6b136fadb9

Conversation

@Klaud-Cold

@Klaud-Cold Klaud-Cold commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Goal: Update vLLM image from vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 to vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4.
Baseline: 2026-08-21 · vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7
Mean latency · Sources: API 1, API 2, API 3, API 4

Point Total tok/s/GPU ↑ Output tok/s/GPU ↑ TTFT ms ↓ TPOT ms ↓
AgentX c1 PTP4x1/DTP4x1 b6fdfc 2,821.03 19.22 419.42 2.85
AgentX c20 PTP2x1/DTP4x1 553930 23,913.77 151.32 504.92 4.11
AgentX c24 PTP1x1/DTP4x3 8d4b12 14,070.32 83.58 779.31 3.27
AgentX c24 PTP2x1/DTP4x1 720f96 29,109.65 176.1 913.2 4.56
AgentX c48 PTP2x1/DTP2x3 b9ce69 38,508.87 254.63 1,770.09 5.67
AgentX c120 PTP2x2/DTP2x5 4463b4 43,977.29 346.08 2,529.72 7.55
Eval Score ↑ Samples
gsm8k/em_strict · c24 97.12% 1,319
gsm8k/em_strict · c24 96.97% 1,319
gsm8k/em_strict · c48 97.27% 1,319
gsm8k/em_strict · c120 97.19% 1,319

Stop: Close this PR at any time to stop this candidate now; Klaud cancels its runs and deletes its branch.

中文

**目标:**将 vLLM 镜像从 vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7 更新至 vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4。
**基线:**2026-08-21 · vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7
平均延迟 · 来源: API 1, API 2, API 3, API 4;数值及异常说明见上表。
**停止:**随时关闭此 PR 即可立即停止该候选:Klaud 会取消其运行并删除其分支。

…to nightly-00b6c4c5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@Klaud-Cold

Klaud-Cold commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator Author

Initial attempt · Failed · Run 40426135146 / attempt 1 · 2026-10-27 10:39 UTC
vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4 · 9800514def3f · Mean latency
Change: Measure ref cac6148, an image-only bump to vLLM nightly 00b6c4c5; all 5 smoke benchmarks passed with zero request errors, but the c20 PTP2x1/DTP4x1 eval failed at startup on nodes im-gb300-r03-c005/c006 when Dynamo raised 'Invalid TCP bind address' because DYN_TCP_RPC_HOST, DYN_EVENT_PLANE_HOST and DYN_TCP_RESPONSE_STREAM_HOST were inherited from the node environment as the interface name enP22s22f0np0 (neither srtctl nor the recipe sets them, and every passing job on r01 nodes resolved an IP), and fail-fast cancelled the other 8 evals.

Point Output tok/s/GPU ↑ TTFT ms ↓ TPOT ms ↓
AgentX c1 PTP4x1/DTP4x1 7172d7 20.06 (N/A) 454.47 (N/A) 2.43 (N/A)
AgentX c24 PTP1x1/DTP4x3 4cb506 84.82 (N/A) 777.41 (N/A) 2.8 (N/A)
AgentX c20 PTP2x1/DTP4x1 bcf27d 152.2 (N/A) 816.96 (N/A) 3.75 (N/A)
AgentX c48 PTP2x1/DTP2x3 40a647 263.66 (N/A) 1,209.14 (N/A) 5.33 (N/A)
AgentX c120 PTP2x2/DTP2x5 c8c4ff 360.04 (N/A) 1,981.02 (N/A) 6.92 (N/A)

Note: All rows: Δ N/A: no matched baseline.

Next: Rerun the failed and cancelled eval jobs on the same run and head as infrastructure retry 1, since the failure comes from node environment rather than the image or recipe.

中文

初次尝试 · 失败 · Run 40426135146 / attempt 1 · 2026-10-27 10:39 UTC
vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4 · 9800514def3f · 平均延迟
**变更:**测量 ref cac6148,仅将镜像更新至 vLLM nightly 00b6c4c5;全部 5 个冒烟基准测试通过且无请求错误,但 c20 PTP2x1/DTP4x1 评测在节点 im-gb300-r03-c005/c006 启动时失败:Dynamo 报出 'Invalid TCP bind address',原因是 DYN_TCP_RPC_HOST、DYN_EVENT_PLANE_HOST 和 DYN_TCP_RESPONSE_STREAM_HOST 从节点环境继承为网卡名 enP22s22f0np0(srtctl 和配方均未设置这些变量,且所有在 r01 节点上通过的任务都解析到了 IP),fail-fast 随后取消了其余 8 个评测。实测数值及异常说明见上表。
**下一步:**由于失败源于节点环境而非镜像或配方,在同一运行和同一 head 上重跑失败及被取消的评测任务,作为基础设施重试 1。

@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

parked · Session 1 is waiting on 40426135146. Recovery reruns preempted runs and starts the next session once they end.

中文

parked · 第 1 个会话正在等待 40426135146。恢复流程会重跑被抢占的运行,并在运行结束后启动下一个会话。

@github-actions github-actions Bot added the klaud-parked Klaud parked until its queued or running benchmark runs end label Oct 27, 2026
@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

resumed · The session ended before finishing; session 2 continues it in a new run of its planner.

中文

resumed · 会话未完成即已结束;第 2 个会话将在其规划运行的新运行中继续。

@github-actions github-actions Bot added klaud-in-progress Klaud is working on this candidate or will resume it and removed klaud-parked Klaud parked until its queued or running benchmark runs end labels Oct 27, 2026
@Klaud-Cold

Klaud-Cold commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator Author

Infrastructure retry 1/2 · Failed · Run 40426135146 / attempt 2 · 2026-10-27 11:14 UTC
vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4 · 9800514def3f · Mean latency
Change: Infrastructure retry 1 on the same head and image reran the 9 evals, but the c48 PTP2x1/DTP2x3 eval again failed at startup on r03 nodes im-gb300-r03-c003..c006: the prefill worker exited with code 134 after Dynamo 1.4.0.dev20260730 (pinned by the recipe, unchanged by this PR) raised "Invalid TCP bind address", because the r03 node environment sets DYN_TCP_RPC_HOST, DYN_EVENT_PLANE_HOST and DYN_TCP_RESPONSE_STREAM_HOST to the interface name enP22s22f0np0, while passing r01 jobs do not set them; fail-fast cancelled the other 8 evals.

Point Output tok/s/GPU ↑ TTFT ms ↓ TPOT ms ↓
AgentX c1 PTP4x1/DTP4x1 7172d7 20.06 (N/A) 454.47 (N/A) 2.43 (N/A)
AgentX c24 PTP1x1/DTP4x3 4cb506 84.82 (N/A) 777.41 (N/A) 2.8 (N/A)
AgentX c20 PTP2x1/DTP4x1 bcf27d 152.2 (N/A) 816.96 (N/A) 3.75 (N/A)
AgentX c48 PTP2x1/DTP2x3 40a647 263.66 (N/A) 1,209.14 (N/A) 5.33 (N/A)
AgentX c120 PTP2x2/DTP2x5 c8c4ff 360.04 (N/A) 1,981.02 (N/A) 6.92 (N/A)

Note: All rows: Δ N/A: no matched baseline.

Next: Stop as inconclusive: the same node-environment failure on r03 nodes repeated twice with no eval passing, it is independent of the image and cannot be fixed within this recipe, so the image remains neither validated nor shown incompatible.

中文

基础设施重试 1/2 · 失败 · Run 40426135146 / attempt 2 · 2026-10-27 11:14 UTC
vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b@sha256:76a9614ed4ed20694914c87e82a0344bd73e1c6e081459a2467175ab903657e4 · 9800514def3f · 平均延迟
**变更:**在同一 head 和镜像上的基础设施重试 1 重跑了 9 个评测,但 c48 PTP2x1/DTP2x3 评测再次在 r03 节点 im-gb300-r03-c003..c006 启动时失败:Dynamo 1.4.0.dev20260730(由配方固定,本 PR 未改动)报出 "Invalid TCP bind address" 后 prefill 进程以 134 退出,原因是 r03 节点环境把 DYN_TCP_RPC_HOST、DYN_EVENT_PLANE_HOST 和 DYN_TCP_RESPONSE_STREAM_HOST 设为网卡名 enP22s22f0np0,而通过的 r01 任务未设置这些变量;fail-fast 随后取消了其余 8 个评测。实测数值及异常说明见上表。
**下一步:**以 inconclusive 结束:r03 节点上同一节点环境故障已重复两次且没有评测通过,该故障与镜像无关,也无法在本配方范围内修复,因此该镜像既未通过验证,也未被证明不兼容。

@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

parked · Session 2 is waiting on 40426135146. Recovery reruns preempted runs and starts the next session once they end.

中文

parked · 第 2 个会话正在等待 40426135146。恢复流程会重跑被抢占的运行,并在运行结束后启动下一个会话。

@github-actions github-actions Bot added klaud-parked Klaud parked until its queued or running benchmark runs end and removed klaud-in-progress Klaud is working on this candidate or will resume it labels Oct 27, 2026
@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

resumed · The session ended before finishing; session 3 continues it in a new run of its planner.

中文

resumed · 会话未完成即已结束;第 3 个会话将在其规划运行的新运行中继续。

@github-actions github-actions Bot added klaud-in-progress Klaud is working on this candidate or will resume it and removed klaud-parked Klaud parked until its queued or running benchmark runs end labels Oct 27, 2026
@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

inconclusive · Cleanup pending. Stop and confirm owned runs before closing.

中文

inconclusive · 清理待完成。先停止并确认自有运行结束,再关闭 PR。

@Klaud-Cold Klaud-Cold closed this Oct 27, 2026
@Klaud-Cold
Klaud-Cold deleted the klaud/auto-6308768623a5422d-903ece6b136fadb9 branch October 27, 2026 11:14
@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

inconclusive · Repairs: 0 · Runs: 40426135146
Reason: Stop as inconclusive: the same node-environment failure on r03 nodes repeated twice with no eval passing, it is independent of the image and cannot be fixed within this recipe, so the image remains neither validated nor shown incompatible.
All owned runs ended. PR closed; branch deleted for retry.

中文

inconclusive · 修复次数:0 · 运行:40426135146
**原因:**以 inconclusive 结束:r03 节点上同一节点环境故障已重复两次且没有评测通过,该故障与镜像无关,也无法在本配方范围内修复,因此该镜像既未通过验证,也未被证明不兼容。
所有自有运行均已结束。PR 已关闭;分支已删除,可重新尝试。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

klaud-error Klaud failed or could not decide; needs a look

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant