Repository navigation
UCS/TOPO: Mark device class as incomplete on NVML errors - #11949
Conversation
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
|
Note: The CUDA path itself (NVML unavailable → class marked incomplete) has no automated coverage; it is only reachable on hardware without NVML, so the gtest coverage of the UCS API is probably the practical limit here. Worth confirming with the author whether a mock/CI job exercises it. Note: |
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
|
Related trade-off worth confirming with the author (not a separate comment): when NVML is unavailable but no GPUs are actually hidden, all ranks now fall back to ordinal 0 for Coverage: The UCS API contract (invalid class, incomplete marking, class change, state extract/restore isolation) is well covered by the new |
|
🤖 CI Triage Agent — TL;DR: The build never got past Full analysisSummary: Root cause: Pure CI infrastructure/authentication failure, not a code regression:
Implicated commit: unknown — not a source regression (no repo content was checked out) File: none (failure is in the Azure Pipelines Suggested fix:
Related: none found (search for this signature in openucx/ucx returned no matching issues/PRs)
|
|
🤖 Starting review — findings will be posted here when done. |
| nvml_status = uct_cuda_init_devices_nvml(visible_gpu_bus_ids, | ||
| num_visible_gpus); | ||
| if (nvml_status != UCS_OK) { | ||
| topo_status = ucs_topo_device_class_mark_incomplete( |
There was a problem hiding this comment.
the ACC class is also used by rocm/ze/gaudi, so on a node without an nvidia driver (no cuda devices and no nvml) we disable ordinals for their gpus even though nothing is hidden. can we mark only when num_visible_gpus > 0?
| ucs_status_t status; | ||
|
|
||
| /* Init visible devices first using the CUDA driver */ | ||
| num_visible_gpus = uct_cuda_init_devices_cu(visible_gpu_bus_ids); |
There was a problem hiding this comment.
uct_cuda_init_devices_cu() also skips devices on error and keeps going (it does continue when cuDeviceGet() or uct_cuda_get_sys_dev_and_bus_id() fails), which leaves the ACC inventory incomplete without marking it. Remaining GPUs then shift down by one, which is the same rank-inconsistent skew this PR prevents on the NVML side. Should that path mark incomplete as well?
There was a problem hiding this comment.
we should be good as if only uct_cuda_init_devices_cu() was to skip a device and keep going, nvml would display it anyways and we would still be able to run algo as nvml would show them. afaiu cuda ordinals would not be skewed in this case.
|
🤖 CI Triage Agent — TL;DR: The Coverity Full analysisSummary: Root cause: The build itself was clean (551/551 compilation units,
but is assigned without the lock in Implicated commit: a361f7a "UCS/TOPO: Mark device class as incomplete on NVML errors" by Guy Ealey Morag (follow-up [REDACTED:Hex High Entropy String] "UCS/TOPO: Fix PR comments"), branch File: src/ucs/sys/topo/base/topo.c:1545 (supporting sites: :1133, :1486–1490, :1511) Suggested fix: Make the initialization consistent with the other accesses so Coverity's lock-consistency heuristic is satisfied. Simplest, in void ucs_topo_init()
{
ucs_spinlock_init(&ucs_topo_global_ctx.lock, 0);
kh_init_inplace(bus_to_sys_dev, &ucs_topo_global_ctx.bus_to_sys_dev_hash);
ucs_spin_lock(&ucs_topo_global_ctx.lock);
ucs_topo_global_ctx.num_devices = 0;
ucs_topo_global_ctx.device_class_incomplete_mask = 0;
ucs_spin_unlock(&ucs_topo_global_ctx.lock);
ucs_list_add_tail(...);
}The lock/unlock pair costs nothing at init time and removes the finding. If you prefer not to lock during init, add an explicit suppression immediately above line 1545 ( Related: #11949 (this PR); introduced on top of #11869 "UCS/TOPO: Build device inventory"
|
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
What?
Mark the
ACCdevice class as incomplete if any nvml error occurs during initialization.Why?
If any part of the nvml initialization flow fails, it means that the device ordinal calculation may results in unexpected result.
Adding a marker for incomplete classes allows returning
INVALIDwhen asking for a device class ordinal of an incomplete class, leading to legacy behavior instead of usage of a possibly wrong ordinal.