Skip to content

UCS/TOPO: Simplify system device access under topo lock - #12045

Merged
tvegas1 merged 5 commits into
openucx:masterfrom
guy-ealey-morag:topo-cleanup
Nov 10, 2026
Merged

tvegas1 merged 5 commits into
openucx:masterfrom
guy-ealey-morag:topo-cleanup

Conversation

@guy-ealey-morag

@guy-ealey-morag guy-ealey-morag commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

What?

Look up and validate system devices through one helper under the topology lock, and fix two bugs this exposed:

  • A self-deadlock when a sysfs path lookup failed with debug logging enabled: the debug message called ucs_topo_sys_device_get_name(), which re-took the non-recursive topology spinlock.
  • An out-of-bounds read of devices[] in ucs_topo_get_distance_sysfs() for a device ID that was never registered. Such a device now gets the default distance, as an unknown device does.

Why?

Each getter and setter repeated the same unknown and out-of-range checks, with different error messages and exit paths, and some paths skipped the range check entirely.

How?

Helpers that expect the lock to be held have a _nolock suffix and assert it in debug builds. New gtests cover the deadlock (it hangs without the fix) and the fallback for an unregistered device.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 7, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Static_check Static checks) · commit 35a2f193

TL;DR: The Static checks stage failed because cppcheck emits 6 uninitvar errors (CWE-457) for the new UCS_TOPO_SYS_DEVICE_GET_FIELD macro in topo.c — cppcheck can't follow the statement-expression and thinks _get_device is dereferenced uninitialized in ucs_typeof(_get_device->_field). Initialize _get_device = NULL at declaration (and/or derive the type from the struct rather than the pointer) to silence it.

Full analysis

Summary: Azure Pipelines job "Static_check Static checks" failed at the cstools reports step: static checkers found errors — 6 distinct cppcheck uninitvar errors (reported twice each, 12 lines total) all pointing at variable _get_device in src/ucs/sys/topo/base/topo.c.

Root cause: The build itself succeeded (BUILD SUCCESS, all CCLD/CXXLD steps clean). The failure is purely from the static-analysis gate. PR #12045 introduced a new statement-expression macro:

#define UCS_TOPO_SYS_DEVICE_GET_FIELD(_sys_dev, _field, _default) \
    ({ \
        ucs_topo_sys_device_info_t *_get_device;            /* <-- uninitialized */ \
        ucs_typeof(_get_device->_field) _get_value = (_default); \
        ...

_get_device is declared without an initializer, and the very next line uses _get_device->_field inside ucs_typeof(...). ucs_typeof is __typeof__, an unevaluated compile-time construct — so the code is actually correct at runtime — but cppcheck does not model __typeof__ as unevaluated and treats the -> as a real dereference of an uninitialized pointer, hence error[uninitvar]: Uninitialized variable: _get_device.

The six reported lines are exactly the six call sites of the macro, each expanding the same flaw:

  • topo.c:1130 — ucs_topo_sys_device_get_name (field name)
  • topo.c:1186 — ucs_topo_sys_device_get_flags (field flags)
  • topo.c:1278 — ucs_topo_sys_device_get_numa_node (field numa_node)
  • topo.c:1284 — ucs_topo_sys_device_get_pci_id (field pci_id)
  • topo.c:1428 — ucs_topo_device_has_sibling (field sibling_sys_dev)
  • topo.c:1436 — ucs_topo_sys_device_get_user_value (field user_value)

Each appears twice because the file is analyzed under two configurations/passes. This is a true-positive-shaped false positive: not a runtime bug, but it does block the gate.

Implicated commit: 35a2f193 — "UCS/TOPO: Simplify system device access under topo lock" by Guy Ealey Morag (the HEAD commit of this PR; it is the commit that added the macro).

File: src/ucs/sys/topo/base/topo.c:361-373 (macro definition; line 363 is the uninitialized declaration, line 364 the flagged dereference)

Suggested fix: Avoid the uninitialized-pointer dereference in the typeof. Cleanest option — take the type from the struct type directly, so no pointer variable is involved in ucs_typeof:

#define UCS_TOPO_SYS_DEVICE_GET_FIELD(_sys_dev, _field, _default) \
    ({ \
        ucs_topo_sys_device_info_t *_get_device = NULL; \
        ucs_typeof(((ucs_topo_sys_device_info_t*)0)->_field) _get_value = \
                (_default); \
        \
        ucs_spin_lock(&ucs_topo_global_ctx.lock); \
        _get_device = ucs_topo_get_sys_device_nolock(_sys_dev); \
        if (_get_device != NULL) { \
            _get_value = _get_device->_field; \
        } \
        ucs_spin_unlock(&ucs_topo_global_ctx.lock); \
        _get_value; \
    })

Simply adding = NULL on line 363 is the minimal change and will very likely clear the warning on its own (cppcheck's uninitvar tracks the declaration, not the NULL-ness). Re-run the Static checks stage to confirm cs.err is empty.

Two secondary cleanup notes, not the cause of this failure:

  • The same job also logged ##[error]Path does not exist: .../s/static_check from PublishPipelineArtifact. This is a cascading artifact-upload failure caused by the earlier step aborting before the static_check directory was populated, plus a container-vs-host path mismatch (the step ran against /__w/1/s inside the container while publish looks at [REDACTED:Base64 High Entropy String]). It will resolve once the cppcheck errors are fixed, but the path discrepancy is worth a separate look.
  • Note the cppcheck paths are reported as /src/ucs/... (leading slash) due to --strip-path-prefix /__w/1/s; the real file is src/ucs/sys/topo/base/topo.c.

Related: PR #12045 (the PR under test). Prior commits in the same topo refactor series by the same author: 318d8625 (#11990), 752278ab (#11976), cef0c5b3 (#11949), f421391e (#11869) — none of which previously tripped cppcheck, confirming the macro is newly introduced regression in the static gate.

🛡️ This comment had 1 potential secret(s) redacted (Base64 High Entropy String). See request_id 01fdcc24-2c1b-42dd-bc39-4a454837dcbb in the triage console for the audit trail.

Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 8, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Build build_x86_64 rhel82) · commit f281b45b

TL;DR: The rhel82 x86_64 build failed to compile src/ucs/sys/topo/base/topo.c — GCC's -Werror=maybe-uninitialized flags device as possibly uninitialized at line 661 because the new ucs_topo_get_sys_device_checked_nolock() helper only writes *device_p on some paths and GCC can't prove the error path returns. Fix by initializing device = NULL at declaration (and/or always assigning *device_p = NULL at the top of the helper).

Full analysis

Summary: make step "Build on x86_64" (rhel82, gcc, -O3 -Wall -Werror) failed with error: 'device' may be used uninitialized in this function [-Werror=maybe-uninitialized] in ucs_topo_sys_dev_to_sysfs_path, aborting the build with make[3]: *** [Makefile:1979: sys/topo/base/libucs_la-topo.lo] Error 1.

Root cause: Compile error, not a test/flake. ucs_topo_sys_dev_to_sysfs_path() declares ucs_topo_sys_device_info_t *device; uninitialized (topo.c:653) and relies on ucs_topo_get_sys_device_checked_nolock(sys_dev, &device) to populate it. In that helper (topo.c:340-356) the early-return branch if (sys_dev == UCS_SYS_DEVICE_ID_UNKNOWN) returns UCS_ERR_INVALID_PARAM without ever writing *device_p. At -O3 GCC inlines/partially-clones the function (note the second diagnostic against ucs_topo_is_pci_bridge.part.10) and can no longer prove that device is always written before the dereference at topo.c:661 (&device->bus_id), so it emits -Wmaybe-uninitialized, which -Werror turns fatal. This is new code introduced by the PR under test — the indirect out-parameter pattern replaced a direct ucs_topo_get_sys_device_nolock() call whose return value was assigned unconditionally.

Implicated commit: f281b45b — "UCS/TOPO: Simplify system device access under topo lock" by Guy Ealey Morag (the PR #12045 head commit)

File: src/ucs/sys/topo/base/topo.c:653 (declaration) / :661 (use); helper at src/ucs/sys/topo/base/topo.c:340-356

Suggested fix: Two complementary changes, either of which silences the warning; doing both is safest:

  1. Initialize the local at the call site:
static ucs_status_t
ucs_topo_sys_dev_to_sysfs_path(ucs_sys_device_t sys_dev, char *path, size_t max)
{
    ucs_topo_sys_device_info_t *device = NULL;   /* was uninitialized */
    ucs_status_t status;
  1. Make the helper always define its out-parameter, so every caller is safe and GCC's analysis is unambiguous:
static ucs_status_t
ucs_topo_get_sys_device_checked_nolock(ucs_sys_device_t sys_dev,
                                       ucs_topo_sys_device_info_t **device_p)
{
    *device_p = NULL;                            /* add this first line */

    if (sys_dev == UCS_SYS_DEVICE_ID_UNKNOWN) {
        ucs_error("system device is unknown");
        return UCS_ERR_INVALID_PARAM;
    }
    ...
}

Also audit the other callers of ucs_topo_get_sys_device_checked_nolock() added in this commit for the same uninitialized-local pattern, since only -O3 exposes it (lower-optimization CI jobs may pass while rhel82 fails). Note the helper is also missing the ucs_assert(ucs_spinlock_is_held(...)) that ucs_topo_get_sys_device_nolock() has — worth adding for consistency with its "Must be called with the topology lock held" contract.

Related: PR #12045 (the failing PR); prior topo work #11869, #11990

@tvegas1 tvegas1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, need to rebase after #12126 which changed sibling to a set of siblings (bitmap)

Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: No test exercises ucs_topo_get_distance() with a registered-but-stale/out-of-range sys_dev, which is exactly the behavior that changed; test_topo.get_distance only covers UCS_SYS_DEVICE_ID_UNKNOWN. If the new error return is intentional, a focused test asserting UCS_ERR_INVALID_PARAM for num_devices as sys_dev (like the existing pci_id_invalid_sys_dev test) would pin it down.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Rename the lock-held sysfs path helpers to _nolock, and add gtests for the debug-log deadlock and for an unregistered device in get_distance.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread test/gtest/ucs/test_topo.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: no test covers the deadlock path through ucs_topo_is_pci_bridge_nolock() (sibling matching with a device whose sysfs path is missing); the sibling tests all require real HCA/GPU/DMA PCIe devices and are skipped otherwise.

Residual coverage gap: get_distance_no_sysfs_entry fails by hanging rather than failing an assertion if the deadlock regresses, so a regression shows up as a CI timeout.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

tvegas1
tvegas1 previously approved these changes Nov 6, 2026
@tvegas1
tvegas1 enabled auto-merge (squash) November 6, 2026 12:03
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Comment thread test/gtest/ucs/test_topo.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

src/ucs/sys/topo/base/topo.c

minor: using ucs_typeof(ucs_topo_global_ctx.devices[0]._field) for the field type is unnecessary — ucs_field_type() already does this and does not go through the global array:

        ucs_field_type(ucs_topo_sys_device_info_t, _field) \
                _get_value = (_default); \

test/gtest/ucs/test_topo.cc

pls add a short comment that debug level is what reaches the failing sysfs-path log; otherwise this scoped_log_level(UCS_LOG_LEVEL_DEBUG) line looks removable and the deadlock coverage is silently lost.

@svc-ucx

svc-ucx commented Nov 6, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests new on worker 2) · commit ae90c49d

TL;DR: The gtest job failed on exactly one test — tcp/test_ucp_rma.put_nonblocking/1 <tcp,cuda_copy,rocm_copy/flush_ep> — and the only code this PR (#12045) touches on that path is the topology-distance logic, where the new unregistered-device fallback returns ucs_topo_default_distance (infinite BW / zero latency) and ucs_topo_is_same_numa_node() compares a NUMA node against the device-id sentinel UCS_SYS_DEVICE_ID_UNKNOWN; both inflate TCP lane bandwidth estimates and change the 2-RMA-lane put split. Fix the sentinel to UCS_NUMA_NODE_UNDEFINED and don't return "infinitely near" distance for devices that are registered but whose sysfs path lookup fails.

Full analysis

Summary: make test in the gtest stage exited 2 — 8917/8918 tests passed, tcp/test_ucp_rma.put_nonblocking/1, where GetParam() = tcp,cuda_copy,rocm_copy/flush_ep FAILED (log: make: *** [Makefile:4992: test] Error 1).

Root cause: Evidence from the log: this is a genuine assertion failure, not a timeout/hang — the suite ran to completion (8918 tests from 1268 test suites ran. (3023655 ms total), Global test environment tear-down reached, no gaps in timestamps before the exit), and only the TCP RMA put path failed. test_ucp_rma runs with MAX_RMA_LANES=2 (test/gtest/ucp/test_ucp_rma.cc:39), so its data-verification depends on multi-lane bandwidth estimation, which is fed by ucs_topo_get_distance(). Two things in this PR's topo rework alter exactly that:

  1. ucs_topo_get_distance_sysfs() now jumps to out_default and returns ucs_topo_default_distance (bandwidth = INFINITY, latency = 0, topo.c:86-89) whenever a device lookup returns NULL — i.e. an unregistered device is reported as infinitely fast/near instead of max distance.
  2. ucs_topo_is_same_numa_node() tests numa1 != UCS_SYS_DEVICE_ID_UNKNOWN on a value of type ucs_numa_node_t, whose "no value" sentinel is UCS_NUMA_NODE_UNDEFINED == -1 (src/ucs/memory/numa.h:16). Non-PCI/virtual TCP netdevs typically have an undefined NUMA node, so both sides compare equal and are classified as same-NUMA, giving dist.node (17000 MB/s) instead of dist.sys (220 MB/s).

Caveat: the exact gtest assertion line is outside the retained log tail (only the last ~500 lines of a 50-minute run are available), so the data-mismatch message itself could not be read; the above is the only PR-owned code on the failing path.

Implicated commit: PR #12045 head [REDACTED:Hex High Entropy String] (Guy Ealey Morag); specifically 81d44fb5 "UCS/TOPO: Simplify system device access under topo lock" and 54d772f8 "UCS/TOPO: Fall back to default distance for unregistered devices"

File: src/ucs/sys/topo/base/topo.c:710 (ucs_topo_is_same_numa_node), and src/ucs/sys/topo/base/topo.c:830-857 (unregistered-device fallback)

Suggested fix:

  1. Change the sentinel comparison to the NUMA type: return (numa1 == numa2) && (numa1 != UCS_NUMA_NODE_UNDEFINED);
  2. For devices that are not UCS_SYS_DEVICE_ID_UNKNOWN but fail lookup (device == NULL), do not return ucs_topo_default_distance; keep the pre-PR behaviour (ucs_topo_max_distance, or propagate the error) so an invalid device cannot be scored as the fastest lane. Reserve the default-distance fallback for the genuine UCS_SYS_DEVICE_ID_UNKNOWN case only.
  3. Re-run with --gtest_filter='tcp/test_ucp_rma.put_nonblocking/1' and UCX_LOG_LEVEL=info/UCX_PROTO_INFO=y to capture the assertion text and the selected RMA lanes/bandwidths; please also paste the assertion lines here, since the build log tail truncates them.

Related: #12045 (this PR); topology/lane-selection neighbours: #12126, #11985, #12206

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 8a281c0d-0cbd-48c8-971a-89a489fb59a2 in the triage console for the audit trail.

@tvegas1
tvegas1 merged commit 325dae2 into openucx:master Nov 10, 2026
167 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants