Repository navigation
UCP/CONTEXT: Add PCIe switch grouping for GPU-NIC assignment - #12202
Conversation
Add UCX_GPU_NIC_ASSIGNMENT_GROUP_BY=auto|numa|pci_switch. With pci_switch, each GPU uses the deepest PCIe switch it shares with a NIC, GPUs on the same switch share a group, and each NIC joins exactly one group. auto selects pci_switch on x86_64 and numa otherwise.
|
🤖 Starting review — findings will be posted here when done. |
| const char *p; | ||
| unsigned depth; | ||
|
|
||
| if (ucs_topo_get_common_path(sys_dev1, sys_dev2, &path1, &path2, |
There was a problem hiding this comment.
this runs with ucs_topo_global_ctx.lock held (ucs_topo_build_groups -> ..._inner -> callback), and on failure ucs_topo_get_common_path() logs with ucs_topo_sys_device_get_name(), which takes the same non-recursive lock - can we deadlock here when the sysfs path lookup fails and debug logging is on?
Impact: self-deadlock at ucp_init() when realpath() of a device fails (e.g. restricted /sys/bus/pci/devices) with UCX_LOG_LEVEL=debug. Same latent issue already exists via ucs_topo_is_pci_bridge(), but this PR adds O(gpus*nics) new calls on this path, so it is worth confirming.
There was a problem hiding this comment.
This is a real bug I suggested a fix for as part of #12045
I think it's okay to skip it here
There was a problem hiding this comment.
Agreed, this is an existing issue also reachable through ucs_topo_is_pci_bridge(). It is fixed by #12045, so not addressed here.
| if (!ucs_topo_is_sys_root(common_path) && | ||
| !ucs_topo_is_pci_root(common_path)) { | ||
| /* Count the path components below "/sys/devices/pciXXXX:XX" */ | ||
| p = strchr(common_path + strlen(UCS_TOPO_SYSFS_DEVICES_ROOT) + 1, '/'); |
There was a problem hiding this comment.
minor: can we use ucs_string_count_char(common_path + strlen(UCS_TOPO_SYSFS_DEVICES_ROOT), '/') - 1 instead of the loop? it is shorter and also avoids reading past the terminator if common_path is ever shorter than /sys/devices/<component>.
|
Residual coverage gaps: |
Performancex86 node with 8 GPUs and 4 NICs per GPU PCIe switch;
Without an assignment, the NICs of the whole NUMA node tie on bandwidth, and the device-ordinal tie-break starts at a different NIC for each GPU. Only GPU0 lands on its own NICs; on a single node, the other GPUs gain 1.0–5.6×, depending on how many of their local NICs the tie-break happens to pick. |
| group_by = UCS_TOPO_GROUP_BY_PCI_SWITCH; | ||
| break; | ||
| default: | ||
| #if defined(__x86_64__) |
There was a problem hiding this comment.
Is this a reliable condition to depend on here?
We could also move this logic to UCS and try to choose the correct grouping method depending on the actual pci tree. For example, choose numa only if the GPUs are depth 0 (or anything along these lines)
There was a problem hiding this comment.
It think it should be reasonably simple and not break the main ARM case (need to confirm if x86 has any c2c cases)
There was a problem hiding this comment.
Updated: pci_switch grouping now falls back to NUMA grouping for GPUs that share no PCIe switch with a NIC, together with the NICs below no chosen PCIe switch. So on x86 hosts without such switches (e.g. a GPU on its own root port) it groups like numa, and the __x86_64__ condition no longer matters for correctness.
Non-x86 stays on numa on purpose: on Vera, flip spreads the NICs of a NUMA node across its GPUs regardless of PCIe locality (NICs with and without SSD access), and PCIe switch groups would defeat that.
| const char *p; | ||
| unsigned depth; | ||
|
|
||
| if (ucs_topo_get_common_path(sys_dev1, sys_dev2, &path1, &path2, |
There was a problem hiding this comment.
This is a real bug I suggested a fix for as part of #12045
I think it's okay to skip it here
Rename the PCIe bridge depth helpers, validate that the common path is below a PCI root, split PCIe switch grouping into smaller functions with a single group_by dispatcher, and add a NIC-only PCIe switch grouping test.
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gaps (unchanged from the previous review): the real |
|
The sysfs depth parsing was checked on real topologies with |
Run the UCP gpu-nic tag and RMA tests with both numa and pci_switch grouping, and declare initialized variables first in ucs_topo_get_common_pci_bridge_depth().
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gaps: the real |
With pci_switch grouping, a GPU sharing no PCIe switch with any NIC was assigned no NICs, so hosts with GPUs on their own root port lost every network lane of GPU memory. Group such GPUs, and the NICs below no decided PCIe switch, by NUMA node instead.
|
🤖 Starting review — findings will be posted here when done. |
|
NULL arrays passed to the depth collector: when Impact: no crash today ( |
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gaps (unchanged from earlier rounds): the real |
|
🤖 CI Triage Agent — TL;DR: The Static_check stage failed because clang's static analyzer flagged a possible NULL pointer dereference at Full analysisSummary: Root cause: The compile itself succeeded; the failure is the static-analysis gate. In Implicated commit: File: Suggested fix: Make the no-GPU case explicit so NULL is never passed into the helpers. Preferred, in if (num_gpus == 0) {
/* No GPUs: nothing to group by PCIe switch, fall back to NUMA */
return ucs_topo_groups_build_groups_by_numa(devices, inventory, groups);
}then drop the Related: PR #12202 (branch
|
Also share the PCI root scan format, keep the NUMA node group index explicitly, skip the PCIe bridge depths without GPUs, and test the pci_switch grouping with the shared assignment mode only.
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The Coverity gate failed with 2 new FORWARD_NULL defects in Full analysisSummary: Root cause: In Implicated commit: File: Suggested fix: Make the NULL-buffer case explicitly unreachable for Coverity. Preferred, minimal change in /* Check if the group already exists for the given NUMA node. */
if (!ucs_array_is_empty(numa_nodes)) {
ucs_assert(ucs_array_begin(numa_nodes) != NULL);
ucs_array_for_each(entry, numa_nodes) {
if (entry->numa_node == numa_node) {
*group_p = &ucs_array_elem(groups, entry->group_idx);
return UCS_OK;
}
}
}This mirrors the existing convention at Related: PR #12202 (this PR); prior work in the same area: #11939 (UCS/TOPO: Build topology groups by NUMA node), #11990, #11869, #11794. No existing issue found for this Coverity signature. |
Also restore the index loop over the NUMA nodes in ucs_topo_groups_get_or_add_group_by_numa_node().
|
🤖 Starting review — findings will be posted here when done. |
|
|
||
| UCS_ARRAY_DECLARE_TYPE(ucs_topo_groups_sys_dev_array_t, size_t, | ||
| ucs_sys_device_t); | ||
| typedef struct { |
There was a problem hiding this comment.
minor: pls add a blank line before the typedef.
| typedef struct { | |
| typedef struct { |
| * @return Number of PCIe hierarchy levels shared below the PCI root, or 0 if | ||
| * the devices do not share a PCIe bridge. | ||
| */ | ||
| typedef unsigned (*ucs_topo_pci_bridge_depth_func_t)(ucs_sys_device_t sys_dev1, |
There was a problem hiding this comment.
pls document that the callback is invoked with the topology lock held; we pass the _nolock implementation, so a future one must not take the lock itself.
| "gpu1", UCS_TOPO_DEVICE_CLASS_ACC, 1); | ||
| const ucs_sys_device_t nic1 = add_switch_device( | ||
| "nic1", UCS_TOPO_DEVICE_CLASS_NET, 1); | ||
| #if defined(__x86_64__) |
There was a problem hiding this comment.
maybe we can avoid repeating the arch rule here? The test re-implements the same #if as ucs_topo_groups_build_groups(), so a change in the auto policy would keep passing — exposing the resolved default from ucs and using it here would at least test the plumbing.
|
🤖 Starting review — findings will be posted here when done. |
| @@ -1529,14 +1561,16 @@ static void ucs_topo_release_devices() | |||
|
|
|||
There was a problem hiding this comment.
minor: every call re-resolves both device paths with realpath() (2 syscalls + 3 PATH_MAX buffers), so pci_switch grouping does O(gpus*nics + gpus^2) resolutions while ucs_topo_build_groups() holds the topo spinlock — ~180 realpath() calls on an 8-GPU/8-NIC node, and this is now the x86_64 default. Can we resolve each device's sysfs path once and compare prefixes in memory?
| } | ||
|
|
||
| /* The PCIe switches decided above a NIC are all its ancestors, so the deepest | ||
| * one is unique */ |
There was a problem hiding this comment.
minor: pls align the consecutive assignments on one column, per docs/CodeStyle.md ("Indent multiple consecutive assignments on the column").
| * one is unique */ | |
| const size_t num_gpus = ucs_array_length(&inventory->gpus); | |
| const size_t num_nics = ucs_array_length(&inventory->nics); | |
| unsigned *depths = NULL, *switch_depths = NULL; | |
| size_t *gpu_groups = NULL; |
|
Residual coverage gaps (unchanged from earlier rounds): the real |
|
Is there a way to use only one of the UCS and UCP |
I think your comment might have crossed with my push with addressed this comment, pls let me know otherwise. |
|
🤖 Starting review — findings will be posted here when done. |
|
Coverage gap: the real Coverage gap: there is no UCP-level test that drives Coverage gap / risk: the new Process note: this diff adds roughly 700 lines (about half tests), exceeding the REVIEW.md >500-added-line scope rule. No prior review round flagged it; consider whether the rule should be enforced and the change split. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: gtest aborted ( Root cause: Log timeline shows the previous variant finished at Implicated commit: unknown — no commit in PR #12202 ( File: test/gtest/ucp/test_ucp_tag_match.cc:1046 (hang site); src/ucp/rndv/rndv.c:1935 Suggested fix: Retrigger the Related: PR #12202 (openucx/ucx) — the PR under test; prior fix for timeouts in this same test: commit |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The RoCE gtest job failed on a single test, Full analysisSummary: Root cause: From the log, only the gtest end-of-run summary is still retrievable — the per-test Implicated commit: [REDACTED:Hex High Entropy String] (Thomas Vegas, "UCS/TOPO: Move the grouping criterion names next to ucs_topo_group_by_t") — PR head, but no evidence it is causal; its changes are confined to File: test/gtest/uct/test_md.cc:467-550 (failing test body; resource-failure threshold at test/gtest/uct/test_md.cc:547) Suggested fix: Re-trigger the Related: PR #12202 (#12202, the PR under test); no existing issue found tracking
|
| ucs_topo_group_t *group; | ||
|
|
||
| num_groups = 0; | ||
| ucs_array_for_each_index(group, group_idx, groups) { |
There was a problem hiding this comment.
minor: group_idx is not used here, pls iterate with ucs_array_for_each(group, groups)
| goto out; | ||
| } | ||
|
|
||
| ucs_topo_groups_remove_groups_without_nics(groups); |
There was a problem hiding this comment.
Let's assume the following case:
NUMA-node 0
├─ root port A ─ PCIe switch sw0 ─┬─ gpu0
│ ├─ nic0
│ └─ nic1
└─ root port B ─ gpu1
With GPU_NIC_ASSIGNMENT_MODE=shared and GPU_NIC_ASSIGNMENT_GROUP_BY=numa, both GPUs get both NICs {gpu0, gpu1 | nic0, nic1}. It is suboptimal for gpu1, but it works.
With the same shared mode and GPU_NIC_ASSIGNMENT_GROUP_BY=pci_switch (the default on x86):
- there is one switch group
sw0 = {gpu0 | nic0, nic1}; gpu1doesn't share a switch with any NIC, so it goes to the NUMA fallback;- but the fallback only gets NICs that no switch group took, and both NICs are already in
sw0; - so
gpu1is left without NICs, and network BW multi protocols are rejected for its memory.
Can such a GPU stay unassigned instead, like a GPU with an unknown NUMA node, or get the NICs of its NUMA node? The first is a small change: call ucs_topo_groups_remove_groups_without_nics() after the NUMA fallback, not before it and I guess it should be enough.
| ucs_topo_groups_remove_groups_without_nics(groups); | |
| status = ucs_topo_groups_build_groups_by_numa(devices, &unmatched, groups); | |
| + if (status != UCS_OK) { | |
| + goto out; | |
| + } | |
| + | |
| + /* GPUs left without NICs stay unassigned, like GPUs of unknown NUMA node */ | |
| + ucs_topo_groups_remove_groups_without_nics(groups); |
There was a problem hiding this comment.
This is intended.
gpu1 is grouped by its NUMA node together with every NIC that is not below a GPU's PCIe switch (see pci_switch_unmatched_devices: lonely_gpu gets lonely_nic and root_nic), so it ends up without NICs only when all NICs of its NUMA node belong to switch groups.
Keeping it assigned with no NICs then prevents its bulk traffic from leaking onto NICs assigned to other GPUs (here gpu0's nic0/nic1), while left unassigned it would use all NICs.
This matches numa grouping on a NUMA node without NICs, and flip/round_robin with fewer NICs than GPUs. It is documented in UCX_GPU_NIC_ASSIGNMENT_GROUP_BY and covered by pci_switch_group_without_nics.
|
🤖 Starting review — findings will be posted here when done. |
| }); | ||
| } | ||
|
|
||
| static std::map<ucs_sys_device_t, std::vector<int>> s_pci_path; |
There was a problem hiding this comment.
Minor: pls move s_pci_path to the member block at the end of the fixture, next to m_devices/m_groups.
|
Impact: on an 8-GPU / 10-NIC node that is ~108 calls → ~200
Impact: two different default behaviors for identical hardware topology depending on the build target, and the gtest has to Residual coverage gaps: |
What
Add a
UCX_GPU_NIC_ASSIGNMENT_GROUP_BY=auto|numa|pci_switchoption that chooses how GPUs and NICs are grouped for GPU-NIC assignment. The default,auto, pickspci_switchon x86_64 andnumaotherwise. Withpci_switch, GPUs and NICs that share no PCIe switch are grouped by NUMA node.Non-x86 stays on
numa: on Vera,flipspreads the NICs of a NUMA node across its GPUs regardless of PCIe locality, which PCIe switch groups would defeat.Why
On many x86 systems, a NUMA node holds several GPUs and many NICs, but each GPU has only a few local NICs, on its own PCIe switch, so NUMA is too coarse a grouping criterion.
With
UCX_GPU_NIC_ASSIGNMENT_MODE=sharedand 4 rails, PCIe-switch grouping reaches full 4-rail bandwidth on every GPU, up to about 5× more than without an assignment. NUMA grouping withsharedgives no gain.It also makes it safe to raise
UCX_MAX_RMA_RAILS/UCX_MAX_RNDV_RAILSto use more paths: extra rails come only from the NICs on the GPU's own PCIe switch, never from other GPUs' NICs, up to the group size.The deepest shared switch, not the immediate one, is needed on DGX-like systems, where each GPU sits behind a second PCIe switch that its NIC doesn't share; there, only the deepest shared switch pairs each GPU with its PXB NIC.
How
numa. A GPU left in a NUMA group without NICs keeps an empty assignment, so its bulk traffic does not use NICs assigned to other GPUs.autois resolved by the UCS topo layer (UCS_TOPO_GROUP_BY_AUTO). The UCP config storesucs_topo_group_by_tdirectly, and its names (ucs_topo_group_by_names) live next to the enum in UCS.ucs_topo_get_common_pci_bridge_depth_nolock(), called with the topo lock held) and is passed toucs_topo_build_groups_inner()as a callback so tests can mock it.test_topo_groupsgtests cover flat switches, nested switches, a NIC on an upstream switch only, the deepest switch winning, a switch group left without NICs, unmatched devices grouped by NUMA node, topologies with no shared switch, NICs without GPUs, and the per-architectureautochoice.numaandpci_switchgrouping.