Repository navigation
UCT/GDAKI: Include only usable IB HCAs in GDA matrix. - #11727
Conversation
|
Notice that there are formatting issues in CI: |
|
|
||
| for (ibdev_index = 0; ibdev_index < ib_per_cuda; ibdev_index++) { | ||
| ibdesc = &dmat[scores[ibdev_index].index]; | ||
| num_gpu_selected_ibdevs = (int)ucs_min(ib_per_cuda, |
There was a problem hiding this comment.
maybe better cast (int)ib_per_cuda?
c892fd6
|
🤖 Starting review — findings will be posted here when done. |
|
|
||
| for (ibdev_index = 0; ibdev_index < ib_per_cuda; ibdev_index++) { | ||
| ibdesc = &dmat[scores[ibdev_index].index]; | ||
| num_gpu_selected_ibdevs = ucs_min((int)ib_per_cuda, num_gpu_reachable_ibdevs); |
There was a problem hiding this comment.
blocker: casting to int truncates inf/large config values. ib_per_cuda comes from gda_max_hca_per_gpu, which is UCS_CONFIG_TYPE_ULUNITS. With UCX_IB_GDA_MAX_HCA_PER_GPU=inf, ib_per_cuda == UCS_ULUNITS_INF == (unsigned long)-1, so (int)ib_per_cuda == -1, ucs_min() returns -1, the selection loop body never runs, cuda_map stays 0 for every device, and uct_gdaki_query_devices() reports UCS_ERR_NO_DEVICE. Any explicit value above INT_MAX has the same effect. Please clamp in unsigned long instead:
| num_gpu_selected_ibdevs = ucs_min((int)ib_per_cuda, num_gpu_reachable_ibdevs); | |
| num_gpu_selected_ibdevs = ucs_min( | |
| ib_per_cuda, (unsigned long)num_gpu_reachable_ibdevs); |
| num_gpu_reachable_ibdevs = 0; | ||
| for (score_index = 0; score_index < num_sys_active_ibdevs; score_index++) { | ||
| ibdesc = &dmat[scores[score_index].ibdev_index]; | ||
| scores[score_index].reachable = ucs_topo_is_reachable( |
There was a problem hiding this comment.
The reachability filter appears to be a no-op for GPUs that are only visible through NVML. uct_gdaki_enum_gpus() resolves non-CUDA-visible GPUs with ucs_topo_find_device_by_bus_id(), which does not call ucs_topo_sys_device_enable_aux_path() (only uct_cuda_get_sys_dev() does). Without the MEM sibling role, ucs_topo_is_reachable() returns 1 unconditionally, so those GPUs still consume usecount on NICs they cannot use, and the ucs_assert_always(score_index == 0) direct-NIC path is still reachable for them. Should we enable the aux path for those GPUs as well?
There was a problem hiding this comment.
Today we assume all gpus has dnic, so increasing usecount shouldn't affect visible gpus selection
| ibdesc = &dmat[scores[ibdev_index].index]; | ||
| num_gpu_selected_ibdevs = ucs_min((int)ib_per_cuda, num_gpu_reachable_ibdevs); | ||
|
|
||
| /* At this point there are num_gpu_reachable_ibdevs reachable |
There was a problem hiding this comment.
minor: the 5-line comment restates the code. Per docs/CodeStyle.md ("Comments: usually one line ... focus on why"), and the ucs_assert() on the next line already documents the invariant — please shorten to one line.
| for (ibdev_index = 0; ibdev_index < ibdev_count; ibdev_index++) { | ||
| ibdesc = &dmat[scores[ibdev_index].index]; | ||
| num_gpu_reachable_ibdevs = 0; | ||
| for (score_index = 0; score_index < num_sys_active_ibdevs; score_index++) { |
There was a problem hiding this comment.
minor: this line and the ucs_min() line below exceed 80 columns (83/86 chars); the codestyle/clang-format check will flag them. Please run clang-format on the diff.
| } | ||
|
|
||
| if (num_gpu_reachable_ibdevs == 0) { | ||
| ucs_debug("GPU %04x:%02x:%02x.%u has no reachable active IB " |
There was a problem hiding this comment.
minor: manual BDF formatting in the log message. ucs_topo_sys_device_bdf_name() / ucs_topo_sys_device_get_name() already exist and would remove the four casts, e.g. ucs_topo_sys_device_bdf_name(gpus[gpu_index].sys_dev, ...).
|
Coverage note: no test is added. The changed path needs a multi-NIC/multi-GPU host (with a down port or a direct-NIC/sibling topology) to exercise, so a gtest is not practical; |
|
🤖 Starting review — findings will be posted here when done. |
|
|
||
| for (ibdev_index = 0; ibdev_index < ib_per_cuda; ibdev_index++) { | ||
| ibdesc = &dmat[scores[ibdev_index].index]; | ||
| num_gpu_selected_ibdevs = ucs_min((int)ib_per_cuda, num_gpu_reachable_ibdevs); |
There was a problem hiding this comment.
ib_per_cuda can be UCS_ULUNITS_INF (UCX_IB_GDA_MAX_HCA_PER_GPU=inf), and (int)-1 makes the loop below select no device at all. pls clamp in unsigned long:
| num_gpu_selected_ibdevs = ucs_min((int)ib_per_cuda, num_gpu_reachable_ibdevs); | |
| num_gpu_selected_ibdevs = (int)ucs_min( | |
| ib_per_cuda, (unsigned long)num_gpu_reachable_ibdevs); |
| } | ||
|
|
||
| if (num_gpu_reachable_ibdevs == 0) { | ||
| ucs_debug("GPU %04x:%02x:%02x.%u has no reachable active IB " |
There was a problem hiding this comment.
pls use the existing bus-id macros here, like in uct_gdaki_enum_gpus():
| ucs_debug("GPU %04x:%02x:%02x.%u has no reachable active IB " | |
| ucs_debug("GPU " UCS_SYS_BUS_ID_FMT | |
| " has no reachable active IB devices for GDA", | |
| UCS_SYS_BUS_ID_ARG(&gpus[gpu_index].bus_id)); |
| for (ibdev_index = 0; ibdev_index < ibdev_count; ibdev_index++) { | ||
| ibdesc = &dmat[scores[ibdev_index].index]; | ||
| num_gpu_reachable_ibdevs = 0; | ||
| for (score_index = 0; score_index < num_sys_active_ibdevs; score_index++) { |
There was a problem hiding this comment.
minor: over 80 columns.
| for (score_index = 0; score_index < num_sys_active_ibdevs; score_index++) { | |
| for (score_index = 0; score_index < num_sys_active_ibdevs; | |
| score_index++) { |
| ibdesc = &dmat[scores[ibdev_index].index]; | ||
| num_gpu_selected_ibdevs = ucs_min((int)ib_per_cuda, num_gpu_reachable_ibdevs); | ||
|
|
||
| /* At this point there are num_gpu_reachable_ibdevs reachable |
There was a problem hiding this comment.
minor: pls shorten, the assert below already states the invariant.
| /* At this point there are num_gpu_reachable_ibdevs reachable | |
| /* Reachable devices are sorted first, so the loop selects only | |
| * devices reachable from this GPU */ |
| to that value, and they were sorted to the beginning of the | ||
| scores array; therefore the loop only iterates over the IB | ||
| devices that are reachable for this GPU. */ | ||
| for (score_index = 0; score_index < num_gpu_selected_ibdevs; |
There was a problem hiding this comment.
why not check ucs_topo_is_reachable in this loop and skip if unreachable? then no need changes in score
883f50f
|
🤖 Starting review — findings will be posted here when done. |
| for (score_index = 0; score_index < num_sys_active_ibdevs; | ||
| score_index++) { | ||
| ibdesc = &dmat[scores[score_index].ibdev_index]; | ||
| scores[score_index].reachable = ucs_topo_is_reachable( |
There was a problem hiding this comment.
GPUs enumerated only through NVML (cuda_idx < 0) get their sys_dev from ucs_topo_find_device_by_bus_id() without ucs_topo_sys_device_enable_aux_path(), so sibling_sys_dev stays unknown and ucs_topo_is_reachable() returns 1 for every HCA. Those GPUs are therefore not filtered and still add usecount to direct-NIC devices they cannot reach — is that intended?
| const uct_gdaki_dev_score_t *a = pa; | ||
| const uct_gdaki_dev_score_t *b = pb; | ||
|
|
||
| /* Sort reachable devices before unreachable ones */ |
There was a problem hiding this comment.
maybe skip unreachable devices directly in the selection loop (continue when not reachable, and count the selected ones) instead of sorting by reachability? that would drop the reachable field, both new counters, and this branch.
UCT/GDAKI: Include only usable IB HCAs in GDA matrix. (#11727) * UCT/GDAKI: Include only usable IB HCAs in GDA matrix. * UCT/GDAKI: Improve debug logging and return code. * UCT/GDAKI: Apply review comments. * UCT/GDAKI: Apply review comment. * UCT/GDAKI: Include only usable IB HCAs in GDA matrix. - 2 --------- Co-authored-by: Colin Hirsch <chirsch@nvidia.com> Co-authored-by: Artemy Kovalyov <artemyko@nvidia.com>
What?
Exclude inactive or unreachable IB HCAs from GDA matrix, also ensure a limit of
autoattempts at least one HCA.Why?
Fixes one problem that showed up attempting to enable the NIXL device gtests on DL.