fix(power): DCGM CPU power is the CPU rail; publish rail + SysIO breakdown, prefer ACPI in auto

[FrankD412]

Summary

DCGM-mode CPU power has been reporting the CPU rail, not the socket envelope — about half of what ACPI-mode reports for the same hardware (~53 W vs ~100 W per socket on GB200 reference runs). This PR makes that explicit in the data, adds the SysIO rail DCGM also exposes, and stops auto mode from picking DCGM over the fuller ACPI source.

Follows #453 (one row per socket, rail names in one module). Independent of #475 (Pareto/HTML report), which already renders the warnings this PR adds.

Why (verified against NVIDIA/DCGM source)

DCGM has no CPU power backend of its own. The sysmon module walks /sys/class/hwmon/*/device/power1_oem_info, matches the label prefix, and reads power1_average from that channel — i.e. it is a second consumer of the same ACPI power_meter interface our ACPI reader uses. Pinned to master @ 64df9f8:

DCGM field	reads hwmon label	our rail
1130 DCGM_FI_DEV_CPU_POWER_WATTS (def, getter)	CPU Power Socket N → power1_average (L49-53, L165-168)	cpu_rail
1132 DCGM_FI_DEV_SYSIO_POWER_UTIL_CURRENT (def)	SysIO Power Socket N → power1_average	soc
1131 DCGM_FI_DEV_CPU_POWER_LIMIT_WATTS	Grace Power Socket N → power1_cap only (L67-71, L183-186)	(envelope's cap, not its draw)

No DCGM field reports the Grace Power Socket N envelope's average. Real samples (per-socket means over the whole run):

run	source	channel	mean W
3054237	ACPI	CPU Power (cpu_rail)	52.9
3054237	ACPI	Grace Power (envelope, power_w)	100.6
2998956	ACPI	CPU Power / Grace Power	55.4 / 104.0
430661	DCGM	field 1130 (power_w)	56.2

What changes

Vocabulary — cpu_rails.DCGM_FIELD_RAIL_KINDS = {1130: cpu_rail, 1132: soc} (+ DCGM_PRIMARY_FIELD_ID, field names, hwmon labels) in the one module that owns rail names. cpu_sample.dcgm_rail_readings() is the one place field ids become rails: 1130 → the dcgm primary (power_w, unchanged) and the cpu_rail component; 1132 → soc. Host reader and scrape parser both call it.

Producers

• Python host collector (srtctl.core.cpu_power): watches 1130 + 1132; wide CSV gets cpu_rail_w = 1130 and soc_w = 1132; node metadata gains a power_fields table (field id, name, rail, hwmon label, column).
• Rust cpu-power-exporter: watches [1130, 1132], falling back to [1130] alone (WARN, with DCGM's reason) if this libdcgm refuses the pair for CPU entities — 1130 alone is the floor every earlier exporter had. /metrics emits one cpu_power_dcgm_watts{socket,field_id,source} sample per field, 1130 first so a legacy collector still meets it first. --source auto is now ACPI-first (was DCGM-first, unlike the Python collector), and the Python host collector (srtctl.core.cpu_power) walks the same ladder. ACPI is not trusted on discovery alone: every sensor is read, and if all read zero the probe is retried once after 1 s before falling back to DCGM at WARN — a node whose power_meter channels never report would otherwise publish zeros that integrate to 0 J and pass for a measurement. --source acpi bails on all-zero instead of falling back.
• cpu_parser: reads the field_id label; an unlabelled sample (older exporter binary) is 1130, so old binaries keep working and now also get cpu_rail_w filled.

Report — CpuSamples.sources; a DCGM-sourced run gets the warning CPU power source is DCGM field 1130 = CPU rail only (not the socket envelope; roughly half of ACPI-mode CPU power): not comparable with ACPI-mode runs.

Docs — cpu-power-telemetry.md (field→rail table, auto order, column semantics), config-reference.md, dcgm-4.7-runtime-support.md, CpuPowerExporterConfig docstring; stale "rails blank for DCGM" comments corrected.

Data correctness

• power_w / total_power_w semantics are unchanged for every mode: 1130 stays the DCGM-mode socket power (SysIO is never summed in), ACPI is untouched. Summing 1130+1132 into power_w was considered and rejected — the ACPI envelope is ~100 W vs cpu_rail+soc ≈ 59 W, so the sum would still be wrong while looking less wrong.
• Real-sample check: power_energy_report --json-out for 430661 (DCGM) and 3054237 (ACPI) is identical to the pre-change output except for the one new warning on 430661.
• A new-shape scrape body for 430661's first sample parses to socket 0: power_w=50.149, rails={cpu_rail: 50.149, soc: 6.2}, total_power_w=96.914 — the same total the run recorded.

Compatibility:

srtctl	exporter binary	result
new	new	power_w=1130, cpu_rail_w=1130, soc_w=1132
new	old	power_w=1130, cpu_rail_w=1130, soc_w blank
old	new	as before (old parser ignores field_id, dedups by socket)

Testing

• uv run pytest on the power suites: 482 passed (cpu_rails / cpu_sample / cpu_parser / cpu_power / collector / contract / samples / energy report / validator / artifacts / telemetry). New cases: field-labelled body, body without 1130 (no socket published), garbage field_id, rail-only warning present for DCGM and absent for ACPI, host reader watching both fields with a socket missing 1132.
• cargo test -p cpu-power-exporter: 18 passed (render_dcgm_metrics ordering/labels, POWER_FIELDS pin, field-set candidates all start with 1130 and end at [1130], ACPI probe dead / live / retry-rescues-a-late-first-sample). cargo fmt --check clean.
• Postprocess: 64 pass, including the two new skip-logging cases.
• Host collector ladder (tests/test_cpu_power.py, 8 new): DCGM falls back to 1130 when 1132 is refused (rejected field group cleaned up, WARN, soc kind not expected, power_fields[].watched in metadata); fails when every set is refused; ACPI probe dead / live / retry-rescues; auto steps dead ACPI → DCGM with the count logged; live ACPI never constructs DCGM; explicit acpi fails on dead sensors.
• make lint: ruff clean; ty diagnostic count unchanged (38, pre-existing).

Also in this PR

• postprocess_stage._build_power_energy_report logging — the skip reason was DEBUG-only, so a run with power CSVs and no power_energy_report.json left no trace in the sweep log. Auditing the sample runs, two such gaps (3090377: benchmark died before aiperf wrote profile_export_aiperf.json; 648928: window narrower than the sample spacing) were only diagnosable by re-running the report offline. When any samples.csv exists the skip is now INFO with the reason; telemetry-off stays DEBUG. Two tests pin both paths.

What a missing reading looks like

A component rail that cannot be read is blank, never 0, for every cause: the exporter/collector fell back to 1130 alone (blank on every row, one WARN at startup), DCGM returned a non-OK status for that (socket, field) on that scrape, or the value was zero/non-finite (dropped — 0 from these files means "not measured"). A socket missing its primary (ACPI total or field 1130) yields no row at all rather than a blank power_w. Rails never substitute for the primary and never join total_power_w, so a missing soc_w changes nothing else. Documented in docs/cpu-power-telemetry.md → Output Format.

Not verified on hardware

Everything above is verified against DCGM source, real sample CSVs, and unit tests. Nothing in this PR has run on a live Grace node. Specifically unconfirmed:

• Field 1132 returns non-zero for CPU entities (if it reads blank, soc_w is left empty; nothing else depends on it).
• Field 1133 (Module Power Socket N) is not the Grace envelope — it has a usage getter in DCGM, so the "no DCGM field reports the envelope" claim is only strictly true with 1133 ruled out. It is deliberately excluded from the table until measured (cpu_rails.py).
• The 2×N entity-major dcgmEntitiesGetLatestValues buffer, and the [1130,1132] → [1130] field-set fallback, exercise a real libdcgm. Only render_dcgm_metrics and the candidate table are unit-testable.
• ACPI-first auto on a node whose hwmon actually carries the envelope.

One visit to a GB200 node settles all four: dcgmi dmon -e 1130,1131,1132,1133 -i cpu:0 -c 3 alongside cat /sys/class/hwmon/*/device/power1_oem_info. Output will be pasted here.

The 430661 explanation (ACPI discovery found nothing, so the old DCGM-first auto landed on DCGM) is inferred from the flow, not observed — its exporter log only shows the DCGM connection. #524 (legacy hwmon*/device/name registration) is the likely actual cause and is the right fix for discovery; this PR makes the fallback honest and preferred-against, #524 removes the reason for it.

Notes for review

• The field_id label on the existing metric (rather than a new metric family) was chosen for wire compactness; an old collector sees two samples per socket and relies on 1130 being emitted first, which the exporter guarantees and a unit test pins. The new parser itself is order-independent (cpu_parser.py says so), so the ordering exists only for old binaries.
• The ACPI liveness probe cannot rescue a node via DCGM in practice — DCGM's sysmon reads the same power1_average files, so it sees the same zero. Its value is refusing to publish zero pollution and naming the problem in the exporter log.
• DCGM_CPU_RAIL_ONLY_WARNING is appended per concurrency point; a rollup-level warning when a combined report mixes ACPI and DCGM runs belongs in feat(power-report): interactive Pareto / power-over-time HTML report (AIP-1388) #475's build_combined_report and is tracked on AIP-1397.

Tracking: AIP-1397 (https://linear.app/nvidia/issue/AIP-1397)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 93.46405% with 10 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@641a07f). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/srtctl/core/cpu_power.py	90.00%	10 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #481   +/-   ##
=======================================
  Coverage        ?   84.67%           
=======================================
  Files           ?      156           
  Lines           ?    22628           
  Branches        ?        0           
=======================================
  Hits            ?    19161           
  Misses          ?     3467           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[FrankD412]

Review — verified the DCGM claim against source, ran the suites

Verdict: looks right; two things to tighten before un-drafting, one open question.

Verified

• The central claim holds. I read DcgmSystemMonitor.cpp @ 64df9f8 independently: GetCpuSocketIdFromContents maps "CPU Power Socket " → usage file (m_cpuSocketToPowerUsagePath, served by GetCurrentCPUPowerUsage → 1130), "SysIO Power Socket " → m_sysioSocketToPowerUsagePath (1132), and "Grace Power Socket " → DCGM_POWER_CAP_FILE only (power1_cap, served by GetCurrentPowerCap → 1131). There is also a "Module Power Socket " → GetCurrentModulePowerUsage path (field 1133 DCGM_FI_DEV_MODULE_POWER_UTIL_CURRENT) — see the open question below.
• Sample data agrees. Across the 5 lyris_pareto_2 ACPI runs + 3054237 + 3074937: Grace Power ≈ 98–109 W/socket, CPU Power ≈ 51–56 W, SysIO ≈ 6.1–6.6 W. 430661 (DCGM, CPU0:cpuPowerUsageW) ≈ 51/58 W — squarely the CPU-rail band. So the report's 791 W vs 1,448 W for the same P1×4+D6×4 topology is exactly this.
• Tests: 428 passed across the cpu/power suites in a clean worktree. ruff check on the PR's files: clean (the 50 findings in src tests are all pre-existing on upstream/main, none in touched files). Couldn't run cargo test here (macOS 27 SDK / tapi linker breakage on this box, unrelated to the PR).
• power_w semantics unchanged for both modes; rejecting 1130+1132 as a fake envelope is the right call.

Tighten

1. Legacy-collector compatibility depends on scrape ordering that the parser doesn't actually assume. The PR text says an old collector "relies on 1130 being emitted first" — true for the old parser, and the Rust test pins it. But the new _parse_dcgm uses setdefault per (socket, field_id), so it's order-independent; fine. Worth stating in cpu_parser.py that the ordering guarantee exists only for old binaries, so nobody later "cleans up" the emission order thinking it's free.
2. Warning text is per-run but the comparability problem is per-rollup. DCGM_CPU_RAIL_ONLY_WARNING fires on a DCGM run in isolation, where it's informational; the real hazard is a combined report mixing ACPI and DCGM runs (the gb300 rollup does exactly this: 430661 alongside 3090373). Suggest also emitting a report-level note when sources differ across included runs — cheap to add in build_combined_report once feat(power-report): interactive Pareto / power-over-time HTML report (AIP-1388) #475 lands, or as a follow-up; noting it here so it isn't lost.

Open question

• Field 1133 (Module Power Socket N). DCGM does have a usage getter for it. On GB200/GB300 the "module" is Grace+Blackwell superchip power, so it's almost certainly not the Grace envelope either and would double-count GPUs — but the PR's "no DCGM field reports the envelope" statement is only strictly true if 1133 is confirmed not to be it. A dcgmi dmon -e 1130,1131,1132,1133 -i cpu:0 -c 3 on a live node alongside cat /sys/class/hwmon/*/device/power1_oem_info would settle both this and the "1132 non-zero" item you already flagged, and is worth pasting into the PR.

Nits

• cpu_rails.DCGM_FIELD_NAMES / DCGM_FIELD_HWMON_LABELS / DCGM_FIELD_RAIL_KINDS are three parallel dicts keyed by field id; a single dict[int, DcgmField(name, label, kind)] would make it impossible for them to drift. Not blocking.
• CpuSamples.sources: frozenset[str] defaults to empty for fixtures, which means a fixture-built DCGM sample never triggers the warning — fine, but a one-line comment in the test that asserts the warning saying it sets source= explicitly for that reason would help the next reader.

[FrankD412]

Thanks — addressed in e283ec9.

Done

• Tighten 1 (ordering): comment added in cpu_parser._parse_dcgm stating the new parser keys on (socket, field_id) and is order-independent; the exporter's "1130 first per socket" order is kept solely for older srtctl parsers and shouldn't be cleaned up while old collectors may scrape new binaries. test_cpu_rails pins DCGM_POWER_FIELD_IDS == (1130, 1132) with that rationale.
• Nit (parallel dicts): collapsed into one DcgmPowerField(field_id, name, hwmon_label, kind) record per field — DCGM_POWER_FIELDS / DCGM_FIELD_BY_ID; the DCGM_FIELD_RAIL_KINDS / DCGM_POWER_FIELD_IDS views consumers use are derived from it. Metadata power_fields now iterates the records.
• Nit (fixture sources): comment on the warning test explaining it loads through the CSV because CpuSamples.sources is populated from the source column and direct fixtures leave it empty.
• Codecov line: the entity_id not in self._cpu_ids guard in DcgmCpuPowerReader.read_watts is now exercised (a value for an un-enumerated entity is dropped).
• Open question (1133): you're right that "no DCGM field reports the envelope" was only strictly true with 1133 ruled out. I can't run the live check from here, so the branch now says exactly what is known: cpu_rails documents 1131 (cap) and 1133 (Module Power Socket N, scope on GB200/GB300 unverified) as deliberately excluded, test_cpu_rails asserts neither is in the table, and the docs table lists 1133 as unverified/excluded with the dcgmi dmon -e 1130,1131,1132,1133 -i cpu:0 -c 3 + cat /sys/class/hwmon/*/device/power1_oem_info recipe. Will paste the live output here once someone has a node.

Deferred

• Tighten 2 (rollup-level mixed-source warning): agreed it's the real hazard (430661 next to 3090373 in the gb300 rollup). build_combined_report lives in feat(power-report): interactive Pareto / power-over-time HTML report (AIP-1388) #475, so this goes there once it lands — tracked as a checkbox on AIP-1397 rather than in this PR.

Verification unchanged: report JSON for 430661 and 3054237 identical to the pre-change output except the warning; 132 power tests + 14 cargo tests pass, ruff clean, ty count unchanged.

[edwingao28]

the pr looks great i left some comments^

[edwingao28]

Could we add a test around init_metrics_state? This changes the default from DCGM first to ACPI first, but the Rust tests only cover the ACPI probe finds a lisensor and still pass if auto tried DCGM first or --source acpi fell back instead of returning an error

[FrankD412]

Ah I see -- I wasn't honoring the contract that was established. If a specific backend is specified, then we throw an error instead of trying a fallback. This should be fixed now.

[edwingao28]

Here should we check the socket totals before choosing ACPI? If the Grace total is stuck at 0 W but the CPU rail is live, this probe could still accepts ACPI and we record 0 W for that socket. The Python probe_live seem has the same issue

[FrankD412]

So I think the issue here is that even if this returns 0, we go with it because DCGM can't provide anything more if even ACPI (which is a primary source) doesn't report it. Some breakdowns from DCGM show that it doesn't reference the Grace total power and instead only reports the power cap but never the actual average power. Here's a breakdown from Claude:

From NVIDIA/DCGM modules/sysmon/DcgmSystemMonitor.cpp @ 64df9f8:

hwmon label	DCGM opens	stored in	getter → field
CPU Power Socket N	power1_average (L49–53, L165–168)	m_cpuSocketToPowerUsagePath	GetCurrentCPUPowerUsage → 1130 DCGM_FI_DEV_CPU_POWER_WATTS
SysIO Power Socket N	power1_average (L61–65, L169–172)	m_sysioSocketToPowerUsagePath	GetCurrentSysIOPowerUsage → 1132 DCGM_FI_DEV_SYSIO_POWER_UTIL_CURRENT
Module Power Socket N	power1_average (L55–59, L173–176)	m_moduleSocketToPowerUsagePath	GetCurrentModulePowerUsage → 1133 DCGM_FI_DEV_MODULE_POWER_UTIL_CURRENT (scope on GB200/GB300 unverified; excluded from our table)
Grace Power Socket N (the socket envelope)	power1_cap only (L44 GRACE_CAP_SOCKET_BEGINNING, L67–71, L183–187)	m_socketToPowerCapPath — no *UsagePath sibling (.h L124–127)	GetCurrentPowerCap (L289–295) → 1131 DCGM_FI_DEV_CPU_POWER_LIMIT_WATTS

Any other label is dropped at L73–77 (log_debug("Ignoring file contents: ...")). No DCGM field reads the Grace device's power1_average, so the socket envelope's draw is never collected — 1131 is its configured limit.

[FrankD412]

@edwingao28 -- thanks for the review, mind taking another look?

[github-actions]

src/srtctl diff: +417 -81