Skip to content

[PowerX] collect GLM-5.2 GB200 disaggregated power / [PowerX] 采集 GLM-5.2 GB200 分离部署功耗 - #4553

Merged
edwingao28 merged 17 commits into
mainfrom
fix/glm52-gb200-power-coverage
Nov 5, 2026
Merged

edwingao28 merged 17 commits into
mainfrom
fix/glm52-gb200-power-coverage

Conversation

@edwingao28

@edwingao28 edwingao28 commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Seven GLM-5.2 GB200 Dynamo-SGLang disaggregated AgentX variants had no measured power. They now collect GPU power and require valid AgentX power. The two unchanged aggregated variants rerun in the same sweep, so the AgentX curve keeps all nine points; serving settings and evals are unchanged.

Testing: sweep 41812471632: 9/9 points power valid, gsm8k 9/9 pass; the dashboard curve keeps its 9 points. Disaggregated prefill is TP1/PP8.

Topology Conc GPUs (P+D) GPU W total Output tok/s/GPU J/output token gsm8k
Agg TP4 1 4 1719 37.2 11.57 0.969
Agg TP8 1 8 2711 19.2 17.63 0.967
1P x 6D TP4 36 8+24 14422 66.2 9.07 0.970
1P x 4D TP4 48 8+16 13522 136.9 6.19 0.971
1P x 2D DEP8 64 8+16 13841 189.7 4.58 0.970
1P x 1D DEP8 96 8+8 9686 506.8 2.40 0.970
1P x 2D DEP8 96 8+16 14859 263.0 3.54 0.969
2P x 1D DEP16 224 16+16 18023 523.2 2.16 0.968
3P x 1D DEP16 336 24+16 22075 768.3 1.80 0.966
PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
  • If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
  • Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@edwingao28
edwingao28 changed the base branch from fix/qwen38-gb300-power-coverage to main October 29, 2026 11:29
@edwingao28 edwingao28 added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 29, 2026
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

@edwingao28

Copy link
Copy Markdown
Collaborator Author

/use 40865828393

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

@edwingao28 staging run 40865828393 failed. Inspect the staging workflow.

中文:将 main 合并到 GLM-5.2 GB200 功耗覆盖分支。
@edwingao28 edwingao28 added full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) and removed full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) labels Nov 3, 2026
@edwingao28 edwingao28 added full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) and removed full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) labels Nov 3, 2026
@edwingao28

Copy link
Copy Markdown
Collaborator Author

/use 41692447507

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

@edwingao28 staged run 41692447507.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

@edwingao28 edwingao28 added full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) and removed full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) labels Nov 4, 2026
@edwingao28

Copy link
Copy Markdown
Collaborator Author

/use 41812471632

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

@edwingao28 staged run 41812471632.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

@edwingao28
edwingao28 marked this pull request as ready for review November 4, 2026 23:36
@edwingao28
edwingao28 requested a review from a team November 4, 2026 23:36
@claude

claude Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @edwingao28's task in 22s —— View job


LGTM - no blocking issues found

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

@edwingao28
edwingao28 merged commit 932ad8a into main Nov 5, 2026
30 checks passed
@edwingao28
edwingao28 deleted the fix/glm52-gb200-power-coverage branch November 5, 2026 22:47
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category new duplicate skipped unmapped failed
benchmark 9 0 9 0 0
eval 9 9 0 0 0
trace 9 0 0 0 0
power 51 0 0 0 0

benchmarkFiles 10, rawBenchmarkRows 18, coveredByPerJobFiles 9, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 11871, gpuMetricSamplesNew 1122688, powerPublicationPoints 9, changelogEntriesWritten 1

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant