Skip to content

UCT/GDAKI: Include only usable IB HCAs in GDA matrix. - #11727

Merged
ofirfarjun7 merged 9 commits into
openucx:masterfrom
ColinNV:gdaki_fix
Oct 8, 2026
Merged

ofirfarjun7 merged 9 commits into
openucx:masterfrom
ColinNV:gdaki_fix

Conversation

@ColinNV

@ColinNV ColinNV commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor

What?

Exclude inactive or unreachable IB HCAs from GDA matrix, also ensure a limit of auto attempts at least one HCA.

Why?

Fixes one problem that showed up attempting to enable the NIXL device gtests on DL.

Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Resolved
Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Resolved
Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Resolved
michal-shalev
michal-shalev previously approved these changes Aug 6, 2026
Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Resolved
Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Resolved
Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Resolved
Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Resolved
@guy-ealey-morag

Copy link
Copy Markdown
Contributor

Notice that there are formatting issues in CI:

--- a/src/uct/ib/mlx5/gdaki/gdaki.c
+++ b/src/uct/ib/mlx5/gdaki/gdaki.c
@@ -1508,12 +1508,10 @@ uct_gdaki_dev_matrix_init(const uct_ib_md_t *ib_md, size_t *dmat_length_p)
     for (gpu_index = 0; gpu_index < gpu_count; gpu_index++) {
         /* Update PCI distance in IB device scores */
         num_reachable_ibdevs = 0;
-        for (ibdev_index = 0; ibdev_index < num_active_ibdevs;
-             ibdev_index++) {
+        for (ibdev_index = 0; ibdev_index < num_active_ibdevs; ibdev_index++) {
             ibdesc = &dmat[scores[ibdev_index].index];
-            scores[ibdev_index].reachable =
-                    ucs_topo_is_reachable(ibdesc->sys_dev,
-                                          gpus[gpu_index].sys_dev);
+            scores[ibdev_index].reachable = ucs_topo_is_reachable(
+                    ibdesc->sys_dev, gpus[gpu_index].sys_dev);
             if (!scores[ibdev_index].reachable) {
                 continue;
             }
@@ -1541,8 +1539,8 @@ uct_gdaki_dev_matrix_init(const uct_ib_md_t *ib_md, size_t *dmat_length_p)
         ucs_qsort_r(scores, num_active_ibdevs, sizeof(*scores),
                     uct_gdaki_dev_matrix_score, NULL);
 
-        num_selected_ibdevs = (int)ucs_min(
-                ib_per_cuda, (unsigned long)num_reachable_ibdevs);
+        num_selected_ibdevs = (int)ucs_min(ib_per_cuda,
+                                           (unsigned long)num_reachable_ibdevs);
         for (ibdev_index = 0; ibdev_index < num_selected_ibdevs;
              ibdev_index++) {
             ibdesc = &dmat[scores[ibdev_index].index];' = 'no modified files to format' ']' 

guy-ealey-morag
guy-ealey-morag previously approved these changes Aug 6, 2026
michal-shalev
michal-shalev previously approved these changes Aug 6, 2026
tomerg-nvidia
tomerg-nvidia previously approved these changes Aug 7, 2026
Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Resolved
Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Resolved
Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Outdated

for (ibdev_index = 0; ibdev_index < ib_per_cuda; ibdev_index++) {
ibdesc = &dmat[scores[ibdev_index].index];
num_gpu_selected_ibdevs = (int)ucs_min(ib_per_cuda,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe better cast (int)ib_per_cuda?

Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Resolved
@ColinNV
ColinNV marked this pull request as draft September 10, 2026 07:34
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Outdated

for (ibdev_index = 0; ibdev_index < ib_per_cuda; ibdev_index++) {
ibdesc = &dmat[scores[ibdev_index].index];
num_gpu_selected_ibdevs = ucs_min((int)ib_per_cuda, num_gpu_reachable_ibdevs);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker: casting to int truncates inf/large config values. ib_per_cuda comes from gda_max_hca_per_gpu, which is UCS_CONFIG_TYPE_ULUNITS. With UCX_IB_GDA_MAX_HCA_PER_GPU=inf, ib_per_cuda == UCS_ULUNITS_INF == (unsigned long)-1, so (int)ib_per_cuda == -1, ucs_min() returns -1, the selection loop body never runs, cuda_map stays 0 for every device, and uct_gdaki_query_devices() reports UCS_ERR_NO_DEVICE. Any explicit value above INT_MAX has the same effect. Please clamp in unsigned long instead:

Suggested change
num_gpu_selected_ibdevs = ucs_min((int)ib_per_cuda, num_gpu_reachable_ibdevs);
num_gpu_selected_ibdevs = ucs_min(
ib_per_cuda, (unsigned long)num_gpu_reachable_ibdevs);

num_gpu_reachable_ibdevs = 0;
for (score_index = 0; score_index < num_sys_active_ibdevs; score_index++) {
ibdesc = &dmat[scores[score_index].ibdev_index];
scores[score_index].reachable = ucs_topo_is_reachable(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reachability filter appears to be a no-op for GPUs that are only visible through NVML. uct_gdaki_enum_gpus() resolves non-CUDA-visible GPUs with ucs_topo_find_device_by_bus_id(), which does not call ucs_topo_sys_device_enable_aux_path() (only uct_cuda_get_sys_dev() does). Without the MEM sibling role, ucs_topo_is_reachable() returns 1 unconditionally, so those GPUs still consume usecount on NICs they cannot use, and the ucs_assert_always(score_index == 0) direct-NIC path is still reachable for them. Should we enable the aux path for those GPUs as well?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Today we assume all gpus has dnic, so increasing usecount shouldn't affect visible gpus selection

Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Outdated
ibdesc = &dmat[scores[ibdev_index].index];
num_gpu_selected_ibdevs = ucs_min((int)ib_per_cuda, num_gpu_reachable_ibdevs);

/* At this point there are num_gpu_reachable_ibdevs reachable

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: the 5-line comment restates the code. Per docs/CodeStyle.md ("Comments: usually one line ... focus on why"), and the ucs_assert() on the next line already documents the invariant — please shorten to one line.

Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Outdated
for (ibdev_index = 0; ibdev_index < ibdev_count; ibdev_index++) {
ibdesc = &dmat[scores[ibdev_index].index];
num_gpu_reachable_ibdevs = 0;
for (score_index = 0; score_index < num_sys_active_ibdevs; score_index++) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: this line and the ucs_min() line below exceed 80 columns (83/86 chars); the codestyle/clang-format check will flag them. Please run clang-format on the diff.

Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Outdated
}

if (num_gpu_reachable_ibdevs == 0) {
ucs_debug("GPU %04x:%02x:%02x.%u has no reachable active IB "

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: manual BDF formatting in the log message. ucs_topo_sys_device_bdf_name() / ucs_topo_sys_device_get_name() already exist and would remove the four casts, e.g. ucs_topo_sys_device_bdf_name(gpus[gpu_index].sys_dev, ...).

@svc-nvidia-pr-review

Copy link
Copy Markdown

Coverage note: no test is added. The changed path needs a multi-NIC/multi-GPU host (with a down port or a direct-NIC/sibling topology) to exercise, so a gtest is not practical; test/gtest/ucs/test_topo.cc covers ucs_topo_is_reachable() itself but not this selection logic. Worth clarifying which CI machine exercised the inactive-port and unreachable-GPU paths.

@ColinNV
ColinNV marked this pull request as ready for review October 5, 2026 13:59
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Outdated

for (ibdev_index = 0; ibdev_index < ib_per_cuda; ibdev_index++) {
ibdesc = &dmat[scores[ibdev_index].index];
num_gpu_selected_ibdevs = ucs_min((int)ib_per_cuda, num_gpu_reachable_ibdevs);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ib_per_cuda can be UCS_ULUNITS_INF (UCX_IB_GDA_MAX_HCA_PER_GPU=inf), and (int)-1 makes the loop below select no device at all. pls clamp in unsigned long:

Suggested change
num_gpu_selected_ibdevs = ucs_min((int)ib_per_cuda, num_gpu_reachable_ibdevs);
num_gpu_selected_ibdevs = (int)ucs_min(
ib_per_cuda, (unsigned long)num_gpu_reachable_ibdevs);

Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Outdated
}

if (num_gpu_reachable_ibdevs == 0) {
ucs_debug("GPU %04x:%02x:%02x.%u has no reachable active IB "

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls use the existing bus-id macros here, like in uct_gdaki_enum_gpus():

Suggested change
ucs_debug("GPU %04x:%02x:%02x.%u has no reachable active IB "
ucs_debug("GPU " UCS_SYS_BUS_ID_FMT
" has no reachable active IB devices for GDA",
UCS_SYS_BUS_ID_ARG(&gpus[gpu_index].bus_id));

Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Outdated
for (ibdev_index = 0; ibdev_index < ibdev_count; ibdev_index++) {
ibdesc = &dmat[scores[ibdev_index].index];
num_gpu_reachable_ibdevs = 0;
for (score_index = 0; score_index < num_sys_active_ibdevs; score_index++) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: over 80 columns.

Suggested change
for (score_index = 0; score_index < num_sys_active_ibdevs; score_index++) {
for (score_index = 0; score_index < num_sys_active_ibdevs;
score_index++) {

Comment thread src/uct/ib/mlx5/gdaki/gdaki.c Outdated
ibdesc = &dmat[scores[ibdev_index].index];
num_gpu_selected_ibdevs = ucs_min((int)ib_per_cuda, num_gpu_reachable_ibdevs);

/* At this point there are num_gpu_reachable_ibdevs reachable

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: pls shorten, the assert below already states the invariant.

Suggested change
/* At this point there are num_gpu_reachable_ibdevs reachable
/* Reachable devices are sorted first, so the loop selects only
* devices reachable from this GPU */

to that value, and they were sorted to the beginning of the
scores array; therefore the loop only iterates over the IB
devices that are reachable for this GPU. */
for (score_index = 0; score_index < num_gpu_selected_ibdevs;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not check ucs_topo_is_reachable in this loop and skip if unreachable? then no need changes in score

Artemy-Mellanox
Artemy-Mellanox previously approved these changes Oct 6, 2026
ofirfarjun7
ofirfarjun7 previously approved these changes Oct 6, 2026
@Artemy-Mellanox
Artemy-Mellanox dismissed stale reviews from ofirfarjun7, guy-ealey-morag, and themself via 883f50f October 6, 2026 14:36
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

for (score_index = 0; score_index < num_sys_active_ibdevs;
score_index++) {
ibdesc = &dmat[scores[score_index].ibdev_index];
scores[score_index].reachable = ucs_topo_is_reachable(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GPUs enumerated only through NVML (cuda_idx < 0) get their sys_dev from ucs_topo_find_device_by_bus_id() without ucs_topo_sys_device_enable_aux_path(), so sibling_sys_dev stays unknown and ucs_topo_is_reachable() returns 1 for every HCA. Those GPUs are therefore not filtered and still add usecount to direct-NIC devices they cannot reach — is that intended?

const uct_gdaki_dev_score_t *a = pa;
const uct_gdaki_dev_score_t *b = pb;

/* Sort reachable devices before unreachable ones */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe skip unreachable devices directly in the selection loop (continue when not reachable, and count the selected ones) instead of sorting by reachability? that would drop the reachable field, both new counters, and this branch.

@ofirfarjun7
ofirfarjun7 merged commit 68efede into openucx:master Oct 8, 2026
162 checks passed
@ColinNV
ColinNV deleted the gdaki_fix branch October 8, 2026 07:25
ofirfarjun7 added a commit that referenced this pull request Oct 14, 2026
UCT/GDAKI: Include only usable IB HCAs in GDA matrix. (#11727)

* UCT/GDAKI: Include only usable IB HCAs in GDA matrix.

* UCT/GDAKI: Improve debug logging and return code.

* UCT/GDAKI: Apply review comments.

* UCT/GDAKI: Apply review comment.

* UCT/GDAKI: Include only usable IB HCAs in GDA matrix. - 2

---------

Co-authored-by: Colin Hirsch <chirsch@nvidia.com>
Co-authored-by: Artemy Kovalyov <artemyko@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants