Skip to content

UCS/TOPO: Mark device class as incomplete on NVML errors - #11949

Merged
tvegas1 merged 4 commits into
openucx:masterfrom
guy-ealey-morag:no-ordinal-on-nvml-error
Sep 20, 2026
Merged

tvegas1 merged 4 commits into
openucx:masterfrom
guy-ealey-morag:no-ordinal-on-nvml-error

Conversation

@guy-ealey-morag

Copy link
Copy Markdown
Contributor

What?

Mark the ACC device class as incomplete if any nvml error occurs during initialization.

Why?

If any part of the nvml initialization flow fails, it means that the device ordinal calculation may results in unexpected result.
Adding a marker for incomplete classes allows returning INVALID when asking for a device class ordinal of an incomplete class, leading to legacy behavior instead of usage of a possibly wrong ordinal.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/cuda/base/cuda_util.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread test/gtest/ucs/test_topo.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Note: The CUDA path itself (NVML unavailable → class marked incomplete) has no automated coverage; it is only reachable on hardware without NVML, so the gtest coverage of the UCS API is probably the practical limit here. Worth confirming with the author whether a mock/CI job exercises it.

Note: ucs_topo_build_groups_inner() also consumes device_class for the ACC inventory and does not consult the new mask. It looks intentional (group building is still a TODO), but it may be worth a follow-up once groups are used.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/cuda/base/cuda_util.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Related trade-off worth confirming with the author (not a separate comment): when NVML is unavailable but no GPUs are actually hidden, all ranks now fall back to ordinal 0 for SINGLE_NET_DEVICE and multi-lane tie-breaking, i.e. the feature is effectively disabled rather than using the visible-GPU ordinals. That looks intentional ("don't guess"), but it is a behavior change for NVML-less setups.

Coverage: The UCS API contract (invalid class, incomplete marking, class change, state extract/restore isolation) is well covered by the new device_bdf_ordinal_incomplete_class test. The CUDA-side trigger remains uncovered, as already noted in the earlier review round.

@svc-ucx

svc-ucx commented Sep 15, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Build build_x86_64 ubuntu2604) · commit e7af9dc8

TL;DR: The build never got past git fetch — the checkout step failed 15 times with "Invalid username or token / Authentication failed for 'https://github.com/openucx/ucx/'", an Azure agent credential problem on swx-rdmz-instinct02-13, not anything in PR #11949. Fix by repairing/rotating the GitHub service-connection credential and removing the interfering gh auth git-credential helper on that agent.

Full analysis

Summary: Checkout openucx/ucx@refs/pull/11949/merge to s failed on all 5 RetryHelper attempts (3 fetch retries each) with git exit code 128; no compilation or test ever ran.

Root cause: Pure CI infrastructure/authentication failure, not a code regression:

  • Every fetch attempt logs gh auth git-credential: "erase" operation not supported immediately followed by remote: Invalid username or token. Password authentication is not supported for Git operations. and fatal: Authentication failed.
  • The gh auth git-credential line shows a machine-level credential.helper pointing at the GitHub CLI is configured on the agent host/container. That helper is supplying a stale/invalid credential for github.com, so the fetch is authenticating with a bad user/token instead of the pipeline-injected http.extraheader bearer credential. Note the log also shows git config --get-all http.extraheader and --get-regexp .*extraheader returning no output — the Azure-supplied auth header was not present in the repo config, leaving the broken CLI helper as the only credential source.
  • The failure is deterministic and identical across all 15 attempts and occurs before any source is on disk (git reset --hard HEAD → "unknown revision"), which rules out a code, build-script, or timeout cause. The whole job lasted ~2m15s with no gaps in activity, so this is not a hang or wall-clock kill.
  • Commit [REDACTED:Hex High Entropy String] / branch no-ordinal-on-nvml-error is not implicated — the merge ref 507b3539... was never fetched.

Implicated commit: unknown — not a source regression (no repo content was checked out)

File: none (failure is in the Azure Pipelines Get sources task on agent swx-rdmz-instinct02-13, container ubuntu26.04/builder:mofed-26.04-0.7.6.0)

Suggested fix:

  1. Rotate/re-authorize the GitHub service connection used by the UCX PR pipeline — the token it presents is being rejected as invalid. Do not paste the token value anywhere; regenerate it in GitHub and update the Azure DevOps service connection.
  2. On agent swx-rdmz-instinct02-13 (and the builder image if it's baked in there), remove the GitHub-CLI credential helper that hijacks github.com auth, e.g. git config --system --unset-all credential.helper / git config --global --unset-all credential.helper, or scope it away from github.com. Also clear any stale gh auth state (~/.config/gh/hosts.yml) for the swx-azure-svc / swx-azure-svc_azpcontainer user.
  3. As a workaround to confirm the diagnosis, set persistCredentials: true and/or add -c credential.helper= to the checkout, or drain this agent and re-run the PR — if it passes on a different agent, the problem is host-local.
  4. Longer term, add a pipeline pre-checkout step that asserts no unexpected credential.helper is configured, so this fails fast with a clear message instead of 2 minutes of retries.

Related: none found (search for this signature in openucx/ucx returned no matching issues/PRs)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 56a433c3-d888-48c8-a1b0-6ed7d97530df in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

nvml_status = uct_cuda_init_devices_nvml(visible_gpu_bus_ids,
num_visible_gpus);
if (nvml_status != UCS_OK) {
topo_status = ucs_topo_device_class_mark_incomplete(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the ACC class is also used by rocm/ze/gaudi, so on a node without an nvidia driver (no cuda devices and no nvml) we disable ordinals for their gpus even though nothing is hidden. can we mark only when num_visible_gpus > 0?

ucs_status_t status;

/* Init visible devices first using the CUDA driver */
num_visible_gpus = uct_cuda_init_devices_cu(visible_gpu_bus_ids);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uct_cuda_init_devices_cu() also skips devices on error and keeps going (it does continue when cuDeviceGet() or uct_cuda_get_sys_dev_and_bus_id() fails), which leaves the ACC inventory incomplete without marking it. Remaining GPUs then shift down by one, which is the same rank-inconsistent skew this PR prevents on the NVML side. Should that path mark incomplete as well?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should be good as if only uct_cuda_init_devices_cu() was to skip a device and keep going, nvml would display it anyways and we would still be able to run algo as nvml would show them. afaiu cuda ordinals would not be skewed in this case.

@svc-ucx

svc-ucx commented Sep 16, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Coverity coverity release on coverity_rh7) · commit 6bc729dc

TL;DR: The Coverity release job failed because static analysis found 1 new defect — a MISSING_LOCK data-race report on ucs_topo_global_ctx.device_class_incomplete_mask written unlocked in ucs_topo_init() (topo.c:1545), a field newly introduced by this PR. Fix by taking ucs_topo_global_ctx.lock around the initialization (or adding a /* coverity[missing_lock] */ suppression, since init is single-threaded).

Full analysis

Summary: coverity release on coverity_rh7 — cov-analyze completed successfully but cov-format-errors reported nerrors=1, so the stage exited non-zero: ##[error]Coverity found 1 issues: /src/ucs/sys/topo/base/topo.c:1545 Type: Data race condition (MISSING_LOCK).

Root cause: The build itself was clean (551/551 compilation units, BUILD SUCCESS, cov-build "completed successfully"). The failure is purely the Coverity gate. PR #11949 adds a new global field ucs_topo_global_ctx.device_class_incomplete_mask, which is consistently mutated under the spinlock in the three other places Coverity found:

  • ucs_topo_device_class_mark_incomplete() — lock at topo.c:1132, access at :1133
  • ucs_topo_extract_state() — lock at :1482, access at :1486–1490
  • ucs_topo_restore_state() — lock at :1504, access at :1511

but is assigned without the lock in ucs_topo_init() at topo.c:1545. Coverity's heuristic ("accessed with lock held 3 out of 4 times") therefore flags the init write as a race. Functionally this is a false positive — ucs_topo_init() runs before any other thread can touch the context, and the spinlock is only created one line earlier at :1542 — but it is a new finding attributable to this PR, so the gate fails. (Note num_devices on the adjacent line :1544 escapes the report only because it is also read unlocked elsewhere, e.g. :1474.)

Implicated commit: a361f7a "UCS/TOPO: Mark device class as incomplete on NVML errors" by Guy Ealey Morag (follow-up [REDACTED:Hex High Entropy String] "UCS/TOPO: Fix PR comments"), branch no-ordinal-on-nvml-error, PR #11949

File: src/ucs/sys/topo/base/topo.c:1545 (supporting sites: :1133, :1486–1490, :1511)

Suggested fix: Make the initialization consistent with the other accesses so Coverity's lock-consistency heuristic is satisfied. Simplest, in ucs_topo_init():

void ucs_topo_init()
{
    ucs_spinlock_init(&ucs_topo_global_ctx.lock, 0);
    kh_init_inplace(bus_to_sys_dev, &ucs_topo_global_ctx.bus_to_sys_dev_hash);

    ucs_spin_lock(&ucs_topo_global_ctx.lock);
    ucs_topo_global_ctx.num_devices                  = 0;
    ucs_topo_global_ctx.device_class_incomplete_mask = 0;
    ucs_spin_unlock(&ucs_topo_global_ctx.lock);

    ucs_list_add_tail(...);
}

The lock/unlock pair costs nothing at init time and removes the finding. If you prefer not to lock during init, add an explicit suppression immediately above line 1545 (/* coverity[missing_lock] */) with a comment noting that ucs_topo_init() is single-threaded. As a secondary cleanup worth doing in the same PR, ucs_topo_extract_state() reads ucs_topo_global_ctx.num_devices at :1474–1475 outside the lock before re-reading it under the lock at :1485 — move the devices_size computation inside the critical section (or size the allocation for the max) to avoid a genuine TOCTOU on the copy length at :1484.

Related: #11949 (this PR); introduced on top of #11869 "UCS/TOPO: Build device inventory"

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id a8d1f16d-c981-4246-9252-c61879dd11e5 in the triage console for the audit trail.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/cuda/base/cuda_util.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread src/uct/cuda/base/cuda_util.c Resolved
@tvegas1
tvegas1 merged commit cef0c5b into openucx:master Sep 20, 2026
162 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants