Repository navigation
UCT/GDAKI: Assign ConnectX-9 HCAs to GPUs by board - #12368
Artemy-Mellanox wants to merge 1 commit into
Conversation
## What? On Vera Rubin without fabric planes, give each GPU one HCA of each board. UCX_IB_GDA_BOARD_ASSIGN=n disables it. Move the Vera Rubin check from UCP to UCS. ## Why? Without planes, equally close HCAs are dealt by IB device index, so the two GPUs of a NUMA node get one board each. ## How? Every two HCAs of a NUMA node with consecutive PCI addresses are a board. A plane is one PF of the HCAs of the first boards of the NUMA nodes, or of their second boards, and a GPU takes a device from each plane. Four HCAs per NUMA node, in boards of two, are assumed, not checked. Non-ConnectX-9 devices are not assigned. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
🤖 Starting review — findings will be posted here when done. |
| return 0; | ||
| } | ||
|
|
||
| for (i = 0; i < num_scores; ++i) { |
There was a problem hiding this comment.
Planes are clobbered even when detection fails. uct_gdaki_fabric_detect documents and implements the contract "returns 0 ... in which case the scores are left as they are" (it only writes scores[i].plane after num_planes != 0). uct_gdaki_board_detect instead resets every scores[i].plane to UCT_GDAKI_PLANE_NONE before it knows whether any plane will be found. If the group loop assigns nothing (e.g. the CX9 devices have no NUMA node, so ucs_topo_groups_add_elements_by_numa_node skips them and the groups hold no NICs), the function returns 0 with all planes left at PLANE_NONE. The caller then takes the "no planes" path (num_used_planes = 1), but every device becomes reachable = 0 and the selection loop requires reachable, so no GPU gets any NIC — GDA device assignment silently ends up empty. Please leave the scores unchanged when returning 0, as uct_gdaki_fabric_detect does.
|
|
||
| ucs_array_for_each(group, &groups) { | ||
| ucs_array_for_each_index(nic, nic_idx, &group->nics) { | ||
| for (pf = 0; pf < nic->num_sys_devs; ++pf) { |
There was a problem hiding this comment.
The documented 4-HCA layout is assumed but never checked: plane = (pf * UCT_GDAKI_VR_NUM_BOARDS) + (nic_idx / UCT_GDAKI_VR_NICS_PER_BOARD) is only unique while a group has at most NUM_BOARDS * NICS_PER_BOARD NICs. With more NICs per NUMA node and multi-PF NICs the ranks collide (pf 1 / nic_idx 0 and pf 0 / nic_idx 4 both give plane 2), mixing different boards and PFs into one plane. The GDA_BOARD_ASSIGN help text states the layout as an assumption, so the code should verify it — can we check ucs_array_length(&group->nics) == UCT_GDAKI_VR_NUM_BOARDS * UCT_GDAKI_VR_NICS_PER_BOARD and return 0 otherwise?
| return 0; | ||
| } | ||
|
|
||
| for (i = 0; i < num_scores; ++i) { |
There was a problem hiding this comment.
Non-CX9 devices become unusable: only CX9 devices enter filter, so any other active device keeps plane == UCT_GDAKI_PLANE_NONE and is dropped from assignment entirely once board planes are detected. With the option defaulting to y, a mixed host (e.g. CX9 + BlueField) loses the non-CX9 devices for GDA. Is this intended? GDA_FABRIC_DETECT documents this effect for planes; please document it in GDA_BOARD_ASSIGN too.
| } | ||
|
|
||
| if (dmat[scores[i].ibdev_index].direct_nic) { | ||
| ucs_debug("ib device %u is a direct nic", scores[i].ibdev_index); |
There was a problem hiding this comment.
Log message identifies the device by index and omits the decision. Per docs/LoggingStyle.md, please identify the object by relevant values such as the device name, and print the decision that follows the unexpected value, e.g.:
| ucs_debug("ib device %u is a direct nic", scores[i].ibdev_index); | |
| ucs_debug("not using board assignment, ib device %s is a direct nic", | |
| ucs_topo_sys_device_get_name( | |
| dmat[scores[i].ibdev_index].sys_dev)); |
| return 1; | ||
| } | ||
|
|
||
| static size_t |
There was a problem hiding this comment.
Include left behind: ucs_arch_get_cpu_model()/ucs_cpu_model_name() were the only users of <ucs/arch/cpu.h> in this file, and both moved out. Please remove the now-dead include.
| } uct_ib_device_spec_t; | ||
|
|
||
|
|
||
| #define UCT_IB_DEVICE_PCI_ID_CX9 {0x15b3, 0x1025} |
There was a problem hiding this comment.
Vendor id literal duplicated: UCT_IB_MELLANOX_PCI_VENDOR_ID already exists (ib_device.c:34); the new macro re-spells 0x15b3. Minor: please move UCT_IB_MELLANOX_PCI_VENDOR_ID from ib_device.c to this header and use it here instead of repeating the literal.
| unsigned long gda_max_hca_per_gpu; /**< Threshold of IB per GPU */ | ||
| int gda_fabric_detect; /**< Give all GPUs the same | ||
| fabric planes in GDA */ | ||
| int gda_board_assign; |
There was a problem hiding this comment.
New config field has no doc comment, unlike all its neighbors:
| int gda_board_assign; | |
| int gda_board_assign; /**< Assign HCAs to GPUs by | |
| HCA board in GDA */ |
|
Residual coverage gap: no test exercises |
|
|
||
| if (mode == UCP_GPU_NIC_ASSIGNMENT_MODE_AUTO) { | ||
| if (!ucp_gpu_nic_assignment_is_vera_rubin()) { | ||
| if (!ucs_topo_is_vera_rubin()) { |
There was a problem hiding this comment.
wdyt about eventually using the UCP gpu-nic assignment here, instead of encoding boards as planes?
flip on the same NUMA groups already gives each GPU one HCA per board (GPU0: nic0+nic3, GPU1: nic1+nic2), while the matrix gives nic0+nic2 / nic1+nic3, so the GDA devices of a GPU differ from its UCP lanes.
It could look like:
- move
ucp_gpu_nic_assignment_build()anducp_gpu_nic_assignment_lookup()to UCS, likeucs_topo_is_vera_rubin()here, in a separate PR (trivial as no actual UCP dep) - in
uct_gdaki_dev_matrix_init(), build the groups from the active IB devices withUCS_TOPO_GROUP_BY_PCI_SWITCH, which likely falls back to NUMA on Vera Rubin - run
flipon them, and set thecuda_mapbit of each IB device in the GPU's NIC bitmap, capped byGDA_MAX_HCA_PER_GPU - with
pci_switchthis mostly matches the distance sort, so the GDA-specific matching could go away; the remaining cases (fabric planes, direct NIC, and sharing a NIC with GPUs thatflipleaves without one) could likely be added to the gpu-nic assignment as options - this drops the 2x2 constants,
order,uct_gdaki_board_set_plane()and the CX9 check, and new platforms would need only topology rules in UCS instead of more GDA special cases
There was a problem hiding this comment.
Notice that ucp_gpu_nic_assignment_build() will have to keep some logic in UCP because it reads the context resources, so for example net_device_filter can be calculated in UCP and given as an arg to UCS.
|
🤖 CI Triage Agent — TL;DR: The ASAN/RoCE gtest run failed on exactly one test — Full analysisSummary: Root cause: Evidence in the log: a single AMO failure, no ASAN report, no sanitizer abort, no watchdog/timeout, no gaps in the timestamps (the run ended normally with the gtest summary and Implicated commit: Not the PR commit. The most plausible pre-existing suspects, if it reproduces, are e4d1e59 "UCT/IB/MLX5: Fix atomic MR offset without a flush memory key (#12265)" (dmitrygx) and the test's own MEMIC-hardening history (69718f6, Michal Shalev). PR commit 8e73d93 (Artemy Kovalyov) is unrelated to this code path. File: test/gtest/uct/test_atomic_key_reg_rdma_mem_type.cc:29-41 (test body); test/gtest/uct/uct_test.cc:1037 (retry only on Suggested fix:
Related: PR under test: #12368 · prior hardening of this exact test: #10475 · atomic-MR-offset change in the same area: #12265
|
What?
On Vera Rubin without fabric planes, give each GPU one HCA of each board. UCX_IB_GDA_BOARD_ASSIGN=n disables it. Move the Vera Rubin check from UCP to UCS.
Why?
Without planes, equally close HCAs are dealt by IB device index, so the two GPUs of a NUMA node get one board each.
How?
Every two HCAs of a NUMA node with consecutive PCI addresses are a board. A plane is one PF of the HCAs of the first boards of the NUMA nodes, or of their second boards, and a GPU takes a device from each plane. Four HCAs per NUMA node, in boards of two, are assumed. Non-ConnectX-9 devices are not assigned.