Skip to content

UCP/CONTEXT: Add PCIe switch grouping for GPU-NIC assignment - #12202

Merged
tvegas1 merged 14 commits into
openucx:masterfrom
tvegas1:gpunic_pci_switch
Nov 16, 2026
Merged

tvegas1 merged 14 commits into
openucx:masterfrom
tvegas1:gpunic_pci_switch

Conversation

@tvegas1

@tvegas1 tvegas1 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What

Add a UCX_GPU_NIC_ASSIGNMENT_GROUP_BY=auto|numa|pci_switch option that chooses how GPUs and NICs are grouped for GPU-NIC assignment. The default, auto, picks pci_switch on x86_64 and numa otherwise. With pci_switch, GPUs and NICs that share no PCIe switch are grouped by NUMA node.

Non-x86 stays on numa: on Vera, flip spreads the NICs of a NUMA node across its GPUs regardless of PCIe locality, which PCIe switch groups would defeat.

Why

On many x86 systems, a NUMA node holds several GPUs and many NICs, but each GPU has only a few local NICs, on its own PCIe switch, so NUMA is too coarse a grouping criterion.

With UCX_GPU_NIC_ASSIGNMENT_MODE=shared and 4 rails, PCIe-switch grouping reaches full 4-rail bandwidth on every GPU, up to about 5× more than without an assignment. NUMA grouping with shared gives no gain.

It also makes it safe to raise UCX_MAX_RMA_RAILS / UCX_MAX_RNDV_RAILS to use more paths: extra rails come only from the NICs on the GPU's own PCIe switch, never from other GPUs' NICs, up to the group size.

The deepest shared switch, not the immediate one, is needed on DGX-like systems, where each GPU sits behind a second PCIe switch that its NIC doesn't share; there, only the deepest shared switch pairs each GPU with its PXB NIC.

How

  • Each GPU uses the deepest PCIe switch it shares with any NIC. GPUs that choose the same switch share a group.
  • Each NIC joins exactly one group: the one whose GPU chose the deepest switch above that NIC.
  • GPUs that share no switch with any NIC, GPUs whose switch group got no NIC, and NICs below none of the chosen switches, are grouped by NUMA node, after the switch groups. On a host without such switches, for example with GPUs on their own root ports, this gives the same groups as numa. A GPU left in a NUMA group without NICs keeps an empty assignment, so its bulk traffic does not use NICs assigned to other GPUs.
  • auto is resolved by the UCS topo layer (UCS_TOPO_GROUP_BY_AUTO). The UCP config stores ucs_topo_group_by_t directly, and its names (ucs_topo_group_by_names) live next to the enum in UCS.
  • Switch depth comes from the common sysfs path (ucs_topo_get_common_pci_bridge_depth_nolock(), called with the topo lock held) and is passed to ucs_topo_build_groups_inner() as a callback so tests can mock it.
  • Nine new test_topo_groups gtests cover flat switches, nested switches, a NIC on an upstream switch only, the deepest switch winning, a switch group left without NICs, unmatched devices grouped by NUMA node, topologies with no shared switch, NICs without GPUs, and the per-architecture auto choice.
  • The UCP gpu-nic tag and RMA tests run with both numa and pci_switch grouping.

Add UCX_GPU_NIC_ASSIGNMENT_GROUP_BY=auto|numa|pci_switch. With pci_switch,
each GPU uses the deepest PCIe switch it shares with a NIC, GPUs on the same
switch share a group, and each NIC joins exactly one group. auto selects
pci_switch on x86_64 and numa otherwise.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucs/sys/topo/base/topo.c Outdated
const char *p;
unsigned depth;

if (ucs_topo_get_common_path(sys_dev1, sys_dev2, &path1, &path2,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this runs with ucs_topo_global_ctx.lock held (ucs_topo_build_groups -> ..._inner -> callback), and on failure ucs_topo_get_common_path() logs with ucs_topo_sys_device_get_name(), which takes the same non-recursive lock - can we deadlock here when the sysfs path lookup fails and debug logging is on?

Impact: self-deadlock at ucp_init() when realpath() of a device fails (e.g. restricted /sys/bus/pci/devices) with UCX_LOG_LEVEL=debug. Same latent issue already exists via ucs_topo_is_pci_bridge(), but this PR adds O(gpus*nics) new calls on this path, so it is worth confirming.

@guy-ealey-morag guy-ealey-morag Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a real bug I suggested a fix for as part of #12045
I think it's okay to skip it here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, this is an existing issue also reachable through ucs_topo_is_pci_bridge(). It is fixed by #12045, so not addressed here.

Comment thread src/ucs/sys/topo/base/topo.c Outdated
if (!ucs_topo_is_sys_root(common_path) &&
!ucs_topo_is_pci_root(common_path)) {
/* Count the path components below "/sys/devices/pciXXXX:XX" */
p = strchr(common_path + strlen(UCS_TOPO_SYSFS_DEVICES_ROOT) + 1, '/');

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: can we use ucs_string_count_char(common_path + strlen(UCS_TOPO_SYSFS_DEVICES_ROOT), '/') - 1 instead of the loop? it is shorter and also avoids reading past the terminator if common_path is ever shorter than /sys/devices/<component>.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment thread src/ucp/core/ucp_context.c Resolved
Comment thread src/ucp/core/ucp_context.c Resolved
Comment thread test/gtest/ucs/test_topo_groups.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gaps: ucs_topo_get_pci_switch_depth() itself (the sysfs/common-path parsing) has no test; the new gtests all inject the fake depth function. Real-topology coverage depends on a CI machine with GPUs behind PCIe switches plus an explicit UCX_GPU_NIC_ASSIGNMENT_GROUP_BY=pci_switch run, which I could not confirm exists. Also, no UCP-level test exercises UCX_GPU_NIC_ASSIGNMENT_GROUP_BY end to end (config parse -> ucs_topo_build_groups -> assignment).

@tvegas1

tvegas1 commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Performance

x86 node with 8 GPUs and 4 NICs per GPU PCIe switch; ucp_put_bw on GPU memory, 4 MB messages, two nodes:

UCX_MAX_RMA_RAILS=4 UCX_GPU_NIC_ASSIGNMENT_MODE=shared UCX_GPU_NIC_ASSIGNMENT_GROUP_BY=pci_switch
GPU No assignment pci_switch + shared
0 46.6 GB/s 46.6 GB/s
7 8.8 GB/s 46.6 GB/s (5.3×)

Without an assignment, the NICs of the whole NUMA node tie on bandwidth, and the device-ordinal tie-break starts at a different NIC for each GPU. Only GPU0 lands on its own NICs; on a single node, the other GPUs gain 1.0–5.6×, depending on how many of their local NICs the tie-break happens to pick. numa + shared gives no gain, because the NUMA group contains the same tied NICs.

Comment thread src/ucp/core/ucp_context.c Resolved
Comment thread src/ucp/core/ucp_context.c Resolved
Comment thread src/ucp/core/ucp_context.c Outdated
group_by = UCS_TOPO_GROUP_BY_PCI_SWITCH;
break;
default:
#if defined(__x86_64__)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a reliable condition to depend on here?
We could also move this logic to UCS and try to choose the correct grouping method depending on the actual pci tree. For example, choose numa only if the GPUs are depth 0 (or anything along these lines)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It think it should be reasonably simple and not break the main ARM case (need to confirm if x86 has any c2c cases)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated: pci_switch grouping now falls back to NUMA grouping for GPUs that share no PCIe switch with a NIC, together with the NICs below no chosen PCIe switch. So on x86 hosts without such switches (e.g. a GPU on its own root port) it groups like numa, and the __x86_64__ condition no longer matters for correctness.

Non-x86 stays on numa on purpose: on Vera, flip spreads the NICs of a NUMA node across its GPUs regardless of PCIe locality (NICs with and without SSD access), and PCIe switch groups would defeat that.

Comment thread src/ucp/core/ucp_context.c Resolved
Comment thread src/ucp/core/ucp_types.h Resolved
Comment thread src/ucs/sys/topo/base/topo.c Outdated
const char *p;
unsigned depth;

if (ucs_topo_get_common_path(sys_dev1, sys_dev2, &path1, &path2,

@guy-ealey-morag guy-ealey-morag Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a real bug I suggested a fix for as part of #12045
I think it's okay to skip it here

Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread src/ucs/sys/topo/base/topo_groups.h Resolved
Comment thread src/ucs/sys/topo/base/topo_groups.c Resolved
Comment thread src/ucs/sys/topo/base/topo_groups.c Resolved
Rename the PCIe bridge depth helpers, validate that the common path is
below a PCI root, split PCIe switch grouping into smaller functions with a
single group_by dispatcher, and add a NIC-only PCIe switch grouping test.
@tvegas1
tvegas1 requested a review from a team as a code owner November 4, 2026 21:09
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_context.c Resolved
Comment thread src/ucp/core/ucp_context.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gaps (unchanged from the previous review): the real ucs_topo_get_common_pci_bridge_depth() sysfs parsing has no test (all new gtests inject a fake depth function), and there is no UCP-level test driving UCX_GPU_NIC_ASSIGNMENT_GROUP_BY from config parse through ucs_topo_build_groups() to the assignment. The second gap is partly mitigated on x86_64 by the new default, which is why the comment above asks to confirm/retain the numa-grouping coverage.

@tvegas1

tvegas1 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

The sysfs depth parsing was checked on real topologies with UCX_GPU_NIC_ASSIGNMENT_GROUP_BY=pci_switch (see the performance results above). The grouping logic is covered by the test_topo_groups.pci_switch_* gtests.

Run the UCP gpu-nic tag and RMA tests with both numa and pci_switch
grouping, and declare initialized variables first in
ucs_topo_get_common_pci_bridge_depth().
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_context.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread src/ucs/sys/topo/base/topo_groups.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gaps: the real ucs_topo_get_common_pci_bridge_depth() sysfs/common-path parsing has no test (all new gtests inject the fake depth function), and no UCP-level test drives UCX_GPU_NIC_ASSIGNMENT_GROUP_BY from config parse through ucs_topo_build_groups() into the assignment. Also, no test covers the "no GPU shares a switch with any NIC" case at UCP level, which is exactly the default-path risk above (the UCS-level pci_switch_no_shared_switch test only asserts the group shape, not the resulting assignment).

With pci_switch grouping, a GPU sharing no PCIe switch with any NIC was
assigned no NICs, so hosts with GPUs on their own root port lost every
network lane of GPU memory. Group such GPUs, and the NICs below no
decided PCIe switch, by NUMA node instead.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread src/ucs/sys/topo/base/topo_groups.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

src/ucs/sys/topo/base/topo_groups.c

NULL arrays passed to the depth collector: when num_gpus is 0 we pass NULL depths/switch_depths here, and it only works because the inner loop is empty. Can we move this call into the num_gpus > 0 block above?

Impact: no crash today (pci_switch_nic_only_group covers it), but the contract is implicit and this is the kind of pattern Coverity flags as "passing null pointer to a function that dereferences it".

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucs/sys/topo/base/topo_groups.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gaps (unchanged from earlier rounds): the real ucs_topo_get_common_pci_bridge_depth() sysfs/common-path parsing has no test (all new gtests inject the fake depth function), and no UCP-level test drives UCX_GPU_NIC_ASSIGNMENT_GROUP_BY from config parse through ucs_topo_build_groups() into the assignment. Additionally, no test exercises the "GPU whose only NIC is claimed by a deeper switch group" shape from finding 1.

@svc-ucx

svc-ucx commented Nov 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Static_check Static checks) · commit 1a23e7de

TL;DR: The Static_check stage failed because clang's static analyzer flagged a possible NULL pointer dereference at src/ucs/sys/topo/base/topo_groups.c:406 — switch_depths is left NULL when num_gpus == 0 but is still passed down and indexed; fix it by bailing out to the NUMA path (or allocating unconditionally) when there are no GPUs.

Full analysis

Summary: Static checks → cstools reports step failed: cslinker/csgrep found two CLANG_WARNING: core.NullDereference findings in topo_groups.c, causing the non-empty cs.err check to exit the stage with error.

Root cause: The compile itself succeeded; the failure is the static-analysis gate. In ucs_topo_groups_build_groups_by_pci_switch() the state arrays are only allocated under if (num_gpus > 0) (topo_groups.c:440-455), so for num_gpus == 0 switch_depths, gpu_groups and depths all remain NULL. They are then unconditionally passed to ucs_topo_groups_get_pci_bridge_depths() and ucs_topo_groups_add_nics_by_pci_switch(). Inside the latter, num_gpus is re-derived from ucs_array_length(&inventory->gpus) (line 396), which the analyzer cannot prove is still 0 (it's a load through a pointer, so it may alias/change). It therefore assumes the for (gpu_idx = 0; gpu_idx < num_gpus; ...) loop at line 405 executes and reports the switch_depths[gpu_idx] read at line 406 as a null dereference. It is a latent-but-real contract violation (NULL passed where an array is expected) rather than a true runtime bug on the current call paths, but the gate treats it as an error. The duplicate warning is the same finding reported from two analyzed translation units/configs.

Implicated commit: 937c9fb2 "UCS/TOPO: Group devices without a shared PCIe switch by NUMA node" (Thomas Vegas, 2026-11-09) — the commit that introduced the conditional allocation / unmatched-NUMA fallback; the surrounding PCIe-switch grouping code came from [REDACTED:Hex High Entropy String] (same author).

File: src/ucs/sys/topo/base/topo_groups.c:406 (root cause at src/ucs/sys/topo/base/topo_groups.c:440-458)

Suggested fix: Make the no-GPU case explicit so NULL is never passed into the helpers. Preferred, in ucs_topo_groups_build_groups_by_pci_switch() right after ucs_topo_init_group(&unmatched):

if (num_gpus == 0) {
    /* No GPUs: nothing to group by PCIe switch, fall back to NUMA */
    return ucs_topo_groups_build_groups_by_numa(devices, inventory, groups);
}

then drop the if (num_gpus > 0) wrapper around the three allocations. Alternatives: allocate with ucs_max(num_gpus, 1) so the pointers are always valid (matching what depths already does for num_nics), and/or pass num_gpus explicitly as a parameter to ucs_topo_groups_add_nics_by_pci_switch()/ucs_topo_groups_get_pci_bridge_depths() instead of re-reading ucs_array_length(&inventory->gpus), which also lets the analyzer correlate the loop bound with the allocation. Avoid silencing this with an assert only — ucs_assert is compiled out in release builds and won't reliably clear the analyzer.

Related: PR #12202 (branch gpunic_pci_switch); prior topo-groups work PRs #11794, #11869, #11939, #11990.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id fd9a571c-13cb-45e5-b65f-bb07749f21da in the triage console for the audit trail.

Also share the PCI root scan format, keep the NUMA node group index
explicitly, skip the PCIe bridge depths without GPUs, and test the
pci_switch grouping with the shared assignment mode only.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucs/sys/topo/base/topo_groups.c Resolved
Comment thread src/ucs/sys/topo/base/topo_groups.c Resolved
Comment thread src/ucs/sys/topo/base/topo.c Resolved
@svc-ucx

svc-ucx commented Nov 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Coverity coverity release on coverity_rh7) · commit 8909d96a

TL;DR: The Coverity gate failed with 2 new FORWARD_NULL defects in src/ucs/sys/topo/base/topo_groups.c (lines 314 and 321): Coverity thinks the freshly-initialized dynamic array numa_nodes (buffer == NULL) is dereferenced inside ucs_topo_groups_add_elements_by_numa_node → ucs_topo_groups_get_or_add_group_by_numa_node. Guard the ucs_array_for_each over numa_nodes with an emptiness check + ucs_assert(ucs_array_begin(...) != NULL) (the same pattern already used at lines 51–55), or pre-ucs_array_reserve() the array before passing it down.

Full analysis

Summary: coverity release on coverity_rh7 job failed: cov-analyze reported nerrors=2 and the script exited 2 — ##[error]Coverity found 2 issues: .../topo_groups.c:314 and :321, both FORWARD_NULL (Explicit null dereferenced).

Root cause: In ucs_topo_groups_build_groups_by_numa() the local array is created with ucs_topo_groups_numa_node_array_t numa_nodes = UCS_ARRAY_DYNAMIC_INITIALIZER; (i.e. { NULL, 0, 0, 0 }, see src/ucs/datastruct/array.h:52). It is then passed to ucs_topo_groups_add_elements_by_numa_node(), which calls ucs_topo_groups_get_or_add_group_by_numa_node(), where ucs_array_for_each(entry, numa_nodes) assigns entry = numa_nodes->buffer and dereferences it (topo_groups.c:248-249). At runtime this is safe because length == 0 makes the loop condition false, but Coverity's interprocedural model takes the "loop body executed" path with buffer == NULL, flagging a null dereference on both call sites (the second one additionally via no_write_call: the first call only populates the array "on some paths"). So it's a modelling false positive, but the CI gate fails on any new defect. The project already works around exactly this pattern in the same file (ucs_topo_groups_sys_dev_sort(), lines 51–55: ucs_array_is_empty() early return + ucs_assert(ucs_array_begin(...) != NULL)), and the new code did not follow it.

Implicated commit: 8909d96a "UCS/TOPO: Group GPUs of PCIe switches left without NICs by NUMA node" (Thomas Vegas); the shared numa_nodes array / helper split was introduced by 937c9fb2 "UCS/TOPO: Group devices without a shared PCIe switch by NUMA node" (Thomas Vegas) in the same PR #12202.

File: src/ucs/sys/topo/base/topo_groups.c:311 (initializer), :248-249 (the modelled deref), reported at :314 and :321

Suggested fix: Make the NULL-buffer case explicitly unreachable for Coverity. Preferred, minimal change in ucs_topo_groups_get_or_add_group_by_numa_node():

/* Check if the group already exists for the given NUMA node. */
if (!ucs_array_is_empty(numa_nodes)) {
    ucs_assert(ucs_array_begin(numa_nodes) != NULL);
    ucs_array_for_each(entry, numa_nodes) {
        if (entry->numa_node == numa_node) {
            *group_p = &ucs_array_elem(groups, entry->group_idx);
            return UCS_OK;
        }
    }
}

This mirrors the existing convention at topo_groups.c:51-55 and fixes both reported defects at once (the same guard could alternatively be added once in ucs_topo_groups_add_elements_by_numa_node()). An equivalent alternative is to ucs_array_reserve(&numa_nodes, <initial_capacity>) in ucs_topo_groups_build_groups_by_numa() and check the status before the two calls, so the buffer is provably non-NULL (and it also avoids repeated reallocation). Avoid simply suppressing via a Coverity annotation, since the guarded-loop form is already used in this file.

Related: PR #12202 (this PR); prior work in the same area: #11939 (UCS/TOPO: Build topology groups by NUMA node), #11990, #11869, #11794. No existing issue found for this Coverity signature.

Also restore the index loop over the NUMA nodes in ucs_topo_groups_get_or_add_group_by_numa_node().
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.


UCS_ARRAY_DECLARE_TYPE(ucs_topo_groups_sys_dev_array_t, size_t,
ucs_sys_device_t);
typedef struct {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: pls add a blank line before the typedef.

Suggested change
typedef struct {
typedef struct {

* @return Number of PCIe hierarchy levels shared below the PCI root, or 0 if
* the devices do not share a PCIe bridge.
*/
typedef unsigned (*ucs_topo_pci_bridge_depth_func_t)(ucs_sys_device_t sys_dev1,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls document that the callback is invoked with the topology lock held; we pass the _nolock implementation, so a future one must not take the lock itself.

"gpu1", UCS_TOPO_DEVICE_CLASS_ACC, 1);
const ucs_sys_device_t nic1 = add_switch_device(
"nic1", UCS_TOPO_DEVICE_CLASS_NET, 1);
#if defined(__x86_64__)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can avoid repeating the arch rule here? The test re-implements the same #if as ucs_topo_groups_build_groups(), so a change in the auto policy would keep passing — exposing the resolved default from ucs and using it here would at least test the plumbing.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@@ -1529,14 +1561,16 @@ static void ucs_topo_release_devices()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: every call re-resolves both device paths with realpath() (2 syscalls + 3 PATH_MAX buffers), so pci_switch grouping does O(gpus*nics + gpus^2) resolutions while ucs_topo_build_groups() holds the topo spinlock — ~180 realpath() calls on an 8-GPU/8-NIC node, and this is now the x86_64 default. Can we resolve each device's sysfs path once and compare prefixes in memory?

}

/* The PCIe switches decided above a NIC are all its ancestors, so the deepest
* one is unique */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: pls align the consecutive assignments on one column, per docs/CodeStyle.md ("Indent multiple consecutive assignments on the column").

Suggested change
* one is unique */
const size_t num_gpus = ucs_array_length(&inventory->gpus);
const size_t num_nics = ucs_array_length(&inventory->nics);
unsigned *depths = NULL, *switch_depths = NULL;
size_t *gpu_groups = NULL;

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gaps (unchanged from earlier rounds): the real ucs_topo_get_common_pci_bridge_depth_nolock() sysfs/common-path parsing has no test (all new gtests inject the fake depth function), and no UCP-level test drives UCX_GPU_NIC_ASSIGNMENT_GROUP_BY from config parse through ucs_topo_build_groups() into the assignment. Additionally, the new pci_switch UCP variants will silently skip on CI hardware where grouping leaves the test GPU unmatched (is_buffer_gpu_assigned()), so a green CI run does not prove the new default path was exercised.

@guy-ealey-morag

guy-ealey-morag commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Is there a way to use only one of the UCS and UCP group_by enums? they are identical

@tvegas1

tvegas1 commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Is there a way to use only one of the UCS and UCP group_by enums? they are identical

I think your comment might have crossed with my push with addressed this comment, pls let me know otherwise.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Coverage gap: the real ucs_topo_get_common_pci_bridge_depth_nolock() sysfs/common-path parsing is untested — all of the new gtests inject a fake depth function, so the actual parsing logic is never exercised.

Coverage gap: there is no UCP-level test that drives UCX_GPU_NIC_ASSIGNMENT_GROUP_BY end to end, from config parsing through ucs_topo_build_groups() into the resulting GPU/NIC assignment.

Coverage gap / risk: the new pci_switch UCP variants silently skip when is_buffer_gpu_assigned() is false, so a green CI run does not prove the new x86_64 default path was exercised. Additionally, the finer pci_switch groups make "group has fewer NICs than GPUs" more likely with flip/round_robin, which drops all net lanes for the unassigned GPU via ucp_proto_multi_filter_gpu_nic_lanes() returning UCS_ERR_NO_ELEM.

Process note: this diff adds roughly 700 lines (about half tests), exceeding the REVIEW.md >500-added-line scope rule. No prior review round flagged it; consider whether the rule should be enforced and the change split.

@svc-ucx

svc-ucx commented Nov 12, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit a21a1973

TL;DR: The roce on worker 0 gtest run was killed by the 900-second gtest watchdog while rc/test_ucp_tag_match_rndv_align.recv_align/1 <rc_v/put_zcopy,proto_v1> sat in ucp_test::progress() with zero output — a genuine hang of a multi-hundred-MB proto_v1 PUT_ZCOPY rendezvous over rc_verbs/RoCE, not a code path touched by PR #12202; retrigger and, if it recurs, debug the put_zcopy rndv/ATP completion on swx-rain04.

Full analysis

Summary: gtest aborted (make: *** [Makefile:5014: test] Aborted (core dumped)) because the watchdog fired on rc/test_ucp_tag_match_rndv_align.recv_align/1 <rc_v/put_zcopy,proto_v1>.

Root cause: Log timeline shows the previous variant finished at 20:00:10 (recv_align/0 <get_zcopy> OK, 26 s) and recv_align/1 <put_zcopy> started immediately; the next log line is at 20:15:10 — exactly watchdog_timeout = 900. (test/gtest/common/test_helpers.cc:31) — with the failure raised at test_helpers.cc:57 and pthread_kill(SIGABRT). A 15-minute gap covering 100 % of the test's runtime means the transfer hung, not that it was slow. The backtrace pins the stall in the send-side wait loop: test_ucp_tag::send_b → send → wait → ucp_test::progress from test_ucp_tag_match.cc:1046, i.e. the rendezvous PUT_ZCOPY data transfer / ATP for a large message (sizes[] up to limit_buffer_size(2500 MB) with RNDV_THRESH=0, RNDV_ALIGN_THRESH=0, RNDV_SCHEME=put_zcopy, proto_v1) never completed, so no UCT PUT completion ever arrived. The GET_ZCOPY twin of the same test passed, so the stall is specific to the PUT_ZCOPY rendezvous path on this RoCE host. PR #12202's code is inert in this run: ucp_context_gpu_nic_assignment_init() returns immediately in auto mode on non-UCS_CPU_MODEL_NVIDIA_VERA CPUs (src/ucp/core/ucp_context.c:3061-3068), no GPU/cuda devices were available in this job (srd,cuda_copy,rocm_copy is not available), and the variant forces proto_v1, which the assignment explicitly does not support. So the PR did not change the lane/rndv selection exercised here.

Implicated commit: unknown — no commit in PR #12202 (a21a1973, Thomas Vegas) reaches the failing code path; most likely an environment/flaky hang of the large-message put_zcopy rendezvous. Note this test has prior CI-timeout history (commit acbe7ae7, Thomas Vegas, "GTEST/UCP/TAG: Avoid transfer timeouts on CI for BF").

File: test/gtest/ucp/test_ucp_tag_match.cc:1046 (hang site); src/ucp/rndv/rndv.c:1935 ucp_rndv_progress_rma_put_zcopy / ucp_rndv_put_completion (path that never completed)

Suggested fix: Retrigger the roce on worker 0 job — this is almost certainly not a PR regression. If it reproduces: (1) rerun just --gtest_filter=rc/test_ucp_tag_match_rndv_align.* with UCX_LOG_LEVEL=debug/UCX_PROTO_INFO=y and capture the core to see which lane's uct_ep_put_zcopy never completed; (2) check swx-rain04's RoCE rails (ibv_devinfo port state, PFC/ECN config, rx_discards/out_of_sequence counters on mlx5_0/mlx5_1) — with MAX_RNDV_RAILS=2 a silently dead second rail makes the PUT stall forever with no error; (3) as a test-robustness measure, bound the largest size for the put_zcopy variant (the existing ucs::limit_buffer_size()/test_time_multiplier() scaling) rather than raising watchdog_timeout, which would only delay the abort.

Related: PR #12202 (openucx/ucx) — the PR under test; prior fix for timeouts in this same test: commit acbe7ae7 "GTEST/UCP/TAG: Avoid transfer timeouts on CI for BF".

Comment thread src/ucp/core/ucp_context.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucs/sys/topo/base/topo_groups.c Resolved
Comment thread src/ucs/sys/topo/base/topo_groups.c Resolved
@svc-ucx

svc-ucx commented Nov 13, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 1) · commit 0a0daeca

TL;DR: The RoCE gtest job failed on a single test, ib/test_md.alloc/0 (mlx5_0), out of 9106 (make: *** [Makefile:5014: test] Error 1); the IB memory-domain alloc/mkey-pack path it exercises is untouched by PR #12202 (which only changes UCS topo grouping and UCP context), so this is almost certainly an unrelated/flaky failure in the MD allocation loop — re-run the job and, if it reproduces, run that single test standalone to capture the assertion text.

Full analysis

Summary: roce on worker 1 gtest run finished with [ FAILED ] ib/test_md.alloc/0, where GetParam() = mlx5_0 (9105 passed, 1 failed), making make test exit non-zero (Bash exited with code '2').

Root cause: From the log, only the gtest end-of-run summary is still retrievable — the per-test [ RUN ] ib/test_md.alloc/0 block with the actual assertion was emitted ~50 minutes earlier and is outside the retained log window, so the precise assertion cannot be quoted. What the evidence does establish: (1) the failing test is test_md.alloc (test/gtest/uct/test_md.cc:467-550), which loops 300 times doing uct_mem_alloc(UCT_ALLOC_METHOD_MD) plus uct_md_mkey_pack_v2(INVALIDATE_RMA|INVALIDATE_AMO) against the IB MD, tolerating UCS_ERR_NO_MEMORY only up to 50% of iterations (EXPECT_LT(...,0.5) << "Too many OUT_OF_RESOURCE failures", line 547); (2) there is no hang or timeout — the log shows continuous per-test progress with sub-second gaps right up to the tear-down at 16:31:14; (3) the PR under test (#12202 "UCP/CONTEXT: Add PCIe switch grouping for GPU-NIC assignment", head commit 0a0daec) only touches src/ucs/sys/topo grouping and src/ucp/core/ucp_context.c — neither is on the uct_mem_alloc/ibv_reg_mr/mkey-pack code path used by test_md.alloc, and all topology-sensitive MD tests (test_md.sys_device, test_md.mem_query) passed. That combination points to a transient/environmental failure (allocation or registration resource pressure on the shared worker after a 3069-second, 9106-test run) rather than a regression from this PR.

Implicated commit: [REDACTED:Hex High Entropy String] (Thomas Vegas, "UCS/TOPO: Move the grouping criterion names next to ucs_topo_group_by_t") — PR head, but no evidence it is causal; its changes are confined to ucs/topo + ucp_context.c, not the UCT IB MD alloc path.

File: test/gtest/uct/test_md.cc:467-550 (failing test body; resource-failure threshold at test/gtest/uct/test_md.cc:547)

Suggested fix: Re-trigger the roce on worker 1 job to confirm it is flaky — the PR does not touch the IB MD allocation path. To get an actionable signature if it recurs, run the test standalone on the roce worker and keep the full output:
./build-test/test/gtest/gtest --gtest_filter='ib/test_md.alloc/0' --gtest_repeat=20 (with UCX_LOG_LEVEL=debug). If the failure is the "Too many OUT_OF_RESOURCE failures" expectation, that threshold is sensitive to transient host memory/MR-resource pressure at the end of a ~51-minute suite; the durable fix is to turn the >50% UCS_ERR_NO_MEMORY case into a UCS_TEST_SKIP_R("no memory resources") instead of a hard failure. Separately, CI would be much easier to triage if the gtest XML/--gtest_brief summary were published as a build artifact, since the Azure log tail discards the failing test's assertion.

Related: PR #12202 (#12202, the PR under test); no existing issue found tracking ib/test_md.alloc flakiness.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id ac395626-5879-4035-99f9-87728e6d3c06 in the triage console for the audit trail.

Comment thread src/ucs/sys/topo/base/topo_groups.c Outdated
ucs_topo_group_t *group;

num_groups = 0;
ucs_array_for_each_index(group, group_idx, groups) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: group_idx is not used here, pls iterate with ucs_array_for_each(group, groups)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 9605221.

goto out;
}

ucs_topo_groups_remove_groups_without_nics(groups);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's assume the following case:

NUMA-node 0
├─ root port A ─ PCIe switch sw0 ─┬─ gpu0
│                                 ├─ nic0
│                                 └─ nic1
└─ root port B ─ gpu1

With GPU_NIC_ASSIGNMENT_MODE=shared and GPU_NIC_ASSIGNMENT_GROUP_BY=numa, both GPUs get both NICs {gpu0, gpu1 | nic0, nic1}. It is suboptimal for gpu1, but it works.

With the same shared mode and GPU_NIC_ASSIGNMENT_GROUP_BY=pci_switch (the default on x86):

  • there is one switch group sw0 = {gpu0 | nic0, nic1};
  • gpu1 doesn't share a switch with any NIC, so it goes to the NUMA fallback;
  • but the fallback only gets NICs that no switch group took, and both NICs are already in sw0;
  • so gpu1 is left without NICs, and network BW multi protocols are rejected for its memory.

Can such a GPU stay unassigned instead, like a GPU with an unknown NUMA node, or get the NICs of its NUMA node? The first is a small change: call ucs_topo_groups_remove_groups_without_nics() after the NUMA fallback, not before it and I guess it should be enough.

Suggested change
ucs_topo_groups_remove_groups_without_nics(groups);
status = ucs_topo_groups_build_groups_by_numa(devices, &unmatched, groups);
+ if (status != UCS_OK) {
+ goto out;
+ }
+
+ /* GPUs left without NICs stay unassigned, like GPUs of unknown NUMA node */
+ ucs_topo_groups_remove_groups_without_nics(groups);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intended.

gpu1 is grouped by its NUMA node together with every NIC that is not below a GPU's PCIe switch (see pci_switch_unmatched_devices: lonely_gpu gets lonely_nic and root_nic), so it ends up without NICs only when all NICs of its NUMA node belong to switch groups.

Keeping it assigned with no NICs then prevents its bulk traffic from leaking onto NICs assigned to other GPUs (here gpu0's nic0/nic1), while left unassigned it would use all NICs.

This matches numa grouping on a NUMA node without NICs, and flip/round_robin with fewer NICs than GPUs. It is documented in UCX_GPU_NIC_ASSIGNMENT_GROUP_BY and covered by pci_switch_group_without_nics.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks fro explanation

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

});
}

static std::map<ucs_sys_device_t, std::vector<int>> s_pci_path;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: pls move s_pci_path to the member block at the end of the fixture, next to m_devices/m_groups.

@svc-nvidia-pr-review

Copy link
Copy Markdown

src/ucs/sys/topo/base/topo_groups.c

ucs_topo_groups_get_pci_bridge_depths(): each pci_bridge_depth() call does 2 realpath() plus 3 PATH_MAX allocations, and we call it num_nics * num_gpus + up to num_gpus^2 / 2 times while ucs_topo_build_groups() holds the topo spinlock. Can we resolve each device's sysfs path once and compare the cached paths instead?

Impact: on an 8-GPU / 10-NIC node that is ~108 calls → ~200 realpath() (thousands of lstats) with the global topo spinlock held; other threads busy-wait for the whole duration. It is init-time only, hence not a blocker, but it is new work added under that lock.

src/ucs/sys/topo/base/topo_groups.c

ucs_topo_groups_build_groups(), the #if defined(__x86_64__) block: why is the default keyed on the build architecture? pci_switch already degrades to numa when no GPU shares a switch with a NIC, so an aarch64 host whose GPUs and NICs do sit behind PCIe switches keeps numa grouping for no clear reason.

Impact: two different default behaviors for identical hardware topology depending on the build target, and the gtest has to #ifdef its expected result to follow it.

Residual coverage gaps: ucs_topo_get_common_pci_bridge_depth_nolock()'s sysfs/%n parsing is still untested — every new gtest injects the fake depth function — and no UCP-level test drives UCX_GPU_NIC_ASSIGNMENT_GROUP_BY from config parse through ucs_topo_build_groups() into the resulting assignment. Combined with the is_buffer_gpu_assigned() skip, a green CI run does not prove the new x86_64 default path was exercised.

@tvegas1
tvegas1 merged commit e5d73ad into openucx:master Nov 16, 2026
167 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants