Skip to content

UCT/GDAKI: Assign ConnectX-9 HCAs to GPUs by board - #12368

Open
Artemy-Mellanox wants to merge 1 commit into
openucx:masterfrom
Artemy-Mellanox:topic/matrix_board_assign
Open

Artemy-Mellanox wants to merge 1 commit into
openucx:masterfrom
Artemy-Mellanox:topic/matrix_board_assign

Conversation

@Artemy-Mellanox

Copy link
Copy Markdown
Contributor

What?

On Vera Rubin without fabric planes, give each GPU one HCA of each board. UCX_IB_GDA_BOARD_ASSIGN=n disables it. Move the Vera Rubin check from UCP to UCS.

Why?

Without planes, equally close HCAs are dealt by IB device index, so the two GPUs of a NUMA node get one board each.

How?

Every two HCAs of a NUMA node with consecutive PCI addresses are a board. A plane is one PF of the HCAs of the first boards of the NUMA nodes, or of their second boards, and a GPU takes a device from each plane. Four HCAs per NUMA node, in boards of two, are assumed. Non-ConnectX-9 devices are not assigned.

## What?

On Vera Rubin without fabric planes, give each GPU one HCA of each
board. UCX_IB_GDA_BOARD_ASSIGN=n disables it. Move the Vera Rubin check
from UCP to UCS.

## Why?

Without planes, equally close HCAs are dealt by IB device index, so the
two GPUs of a NUMA node get one board each.

## How?

Every two HCAs of a NUMA node with consecutive PCI addresses are a
board. A plane is one PF of the HCAs of the first boards of the NUMA
nodes, or of their second boards, and a GPU takes a device from each
plane. Four HCAs per NUMA node, in boards of two, are assumed, not
checked. Non-ConnectX-9 devices are not assigned.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@Artemy-Mellanox
Artemy-Mellanox requested a review from a team as a code owner November 17, 2026 14:38
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

return 0;
}

for (i = 0; i < num_scores; ++i) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Planes are clobbered even when detection fails. uct_gdaki_fabric_detect documents and implements the contract "returns 0 ... in which case the scores are left as they are" (it only writes scores[i].plane after num_planes != 0). uct_gdaki_board_detect instead resets every scores[i].plane to UCT_GDAKI_PLANE_NONE before it knows whether any plane will be found. If the group loop assigns nothing (e.g. the CX9 devices have no NUMA node, so ucs_topo_groups_add_elements_by_numa_node skips them and the groups hold no NICs), the function returns 0 with all planes left at PLANE_NONE. The caller then takes the "no planes" path (num_used_planes = 1), but every device becomes reachable = 0 and the selection loop requires reachable, so no GPU gets any NIC — GDA device assignment silently ends up empty. Please leave the scores unchanged when returning 0, as uct_gdaki_fabric_detect does.


ucs_array_for_each(group, &groups) {
ucs_array_for_each_index(nic, nic_idx, &group->nics) {
for (pf = 0; pf < nic->num_sys_devs; ++pf) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documented 4-HCA layout is assumed but never checked: plane = (pf * UCT_GDAKI_VR_NUM_BOARDS) + (nic_idx / UCT_GDAKI_VR_NICS_PER_BOARD) is only unique while a group has at most NUM_BOARDS * NICS_PER_BOARD NICs. With more NICs per NUMA node and multi-PF NICs the ranks collide (pf 1 / nic_idx 0 and pf 0 / nic_idx 4 both give plane 2), mixing different boards and PFs into one plane. The GDA_BOARD_ASSIGN help text states the layout as an assumption, so the code should verify it — can we check ucs_array_length(&group->nics) == UCT_GDAKI_VR_NUM_BOARDS * UCT_GDAKI_VR_NICS_PER_BOARD and return 0 otherwise?

return 0;
}

for (i = 0; i < num_scores; ++i) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-CX9 devices become unusable: only CX9 devices enter filter, so any other active device keeps plane == UCT_GDAKI_PLANE_NONE and is dropped from assignment entirely once board planes are detected. With the option defaulting to y, a mixed host (e.g. CX9 + BlueField) loses the non-CX9 devices for GDA. Is this intended? GDA_FABRIC_DETECT documents this effect for planes; please document it in GDA_BOARD_ASSIGN too.

}

if (dmat[scores[i].ibdev_index].direct_nic) {
ucs_debug("ib device %u is a direct nic", scores[i].ibdev_index);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log message identifies the device by index and omits the decision. Per docs/LoggingStyle.md, please identify the object by relevant values such as the device name, and print the decision that follows the unexpected value, e.g.:

Suggested change
ucs_debug("ib device %u is a direct nic", scores[i].ibdev_index);
ucs_debug("not using board assignment, ib device %s is a direct nic",
ucs_topo_sys_device_get_name(
dmat[scores[i].ibdev_index].sys_dev));

return 1;
}

static size_t

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include left behind: ucs_arch_get_cpu_model()/ucs_cpu_model_name() were the only users of <ucs/arch/cpu.h> in this file, and both moved out. Please remove the now-dead include.

} uct_ib_device_spec_t;


#define UCT_IB_DEVICE_PCI_ID_CX9 {0x15b3, 0x1025}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vendor id literal duplicated: UCT_IB_MELLANOX_PCI_VENDOR_ID already exists (ib_device.c:34); the new macro re-spells 0x15b3. Minor: please move UCT_IB_MELLANOX_PCI_VENDOR_ID from ib_device.c to this header and use it here instead of repeating the literal.

Comment thread src/uct/ib/base/ib_md.h
unsigned long gda_max_hca_per_gpu; /**< Threshold of IB per GPU */
int gda_fabric_detect; /**< Give all GPUs the same
fabric planes in GDA */
int gda_board_assign;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New config field has no doc comment, unlike all its neighbors:

Suggested change
int gda_board_assign;
int gda_board_assign; /**< Assign HCAs to GPUs by
HCA board in GDA */

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: no test exercises uct_gdaki_board_detect or GDA_BOARD_ASSIGN. The topology-group side is unit-testable (test/gtest/ucs/test_topo_groups.cc already builds mock Vera Rubin NUMA/NIC layouts), but plane assignment lives inside gdaki and needs real CX9 + Vera Rubin hardware. Which CI job or hardware run validated the 2-board layout and the num_planes == 0 fallback?


if (mode == UCP_GPU_NIC_ASSIGNMENT_MODE_AUTO) {
if (!ucp_gpu_nic_assignment_is_vera_rubin()) {
if (!ucs_topo_is_vera_rubin()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wdyt about eventually using the UCP gpu-nic assignment here, instead of encoding boards as planes?

flip on the same NUMA groups already gives each GPU one HCA per board (GPU0: nic0+nic3, GPU1: nic1+nic2), while the matrix gives nic0+nic2 / nic1+nic3, so the GDA devices of a GPU differ from its UCP lanes.

It could look like:

  • move ucp_gpu_nic_assignment_build() and ucp_gpu_nic_assignment_lookup() to UCS, like ucs_topo_is_vera_rubin() here, in a separate PR (trivial as no actual UCP dep)
  • in uct_gdaki_dev_matrix_init(), build the groups from the active IB devices with UCS_TOPO_GROUP_BY_PCI_SWITCH, which likely falls back to NUMA on Vera Rubin
  • run flip on them, and set the cuda_map bit of each IB device in the GPU's NIC bitmap, capped by GDA_MAX_HCA_PER_GPU
  • with pci_switch this mostly matches the distance sort, so the GDA-specific matching could go away; the remaining cases (fabric planes, direct NIC, and sharing a NIC with GPUs that flip leaves without one) could likely be added to the gpu-nic assignment as options
  • this drops the 2x2 constants, order, uct_gdaki_board_set_plane() and the CX9 check, and new platforms would need only topology rules in UCS instead of more GDA special cases

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice that ucp_gpu_nic_assignment_build() will have to keep some logic in UCP because it reads the context resources, so for example net_device_filter can be calculated in UCP and given as an arg to UCS.

@svc-ucx

svc-ucx commented Nov 17, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 1) · commit 8e73d93e

TL;DR: The ASAN/RoCE gtest run failed on exactly one test — rc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/0 (atomic fetch-add on IB device memory, UCS_MEMORY_TYPE_RDMA/MEMIC) — while all other 9113 tests passed; this is an IB device-memory AMO test with a long history of being hardened against MEMIC allocation failures and is unrelated to PR #12368's GDAKI HCA→GPU board assignment, so re-run the job and (to stop recurrences) make the device-memory allocation path skip/retry instead of failing the suite.

Full analysis

Summary: make test in build-test/test/gtest exited with error 1 because 1 of 9114 gtests failed: rc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/0, where GetParam() = rc_mlx5/mlx5_0:1 (Azure build 145723, job "AddressSanitizer roce on worker 1").

Root cause: Evidence in the log: a single AMO failure, no ASAN report, no sanitizer abort, no watchdog/timeout, no gaps in the timestamps (the run ended normally with the gtest summary and TOP-20 longest tests listing). The failing fixture (test/gtest/uct/test_atomic_key_reg_rdma_mem_type.cc) allocates a 8-byte buffer in IB device memory (UCS_MEMORY_TYPE_RDMA, i.e. MEMIC) with num_retries = 10 and then drives 4 sender threads × 1000 uct_ep_atomic64_fetch(ADD) ops to it. Its only failure paths are (a) ASSERT_UCS_OK after device-memory allocation fails in uct_test::entity::mem_alloc() — which only retries on UCS_ERR_NO_MEMORY, so any other status (e.g. UCS_ERR_IO_ERROR from MEMIC) fails immediately, and (b) an error status/error log from the atomic operations on the device-memory atomic key. This test has been patched three times specifically for flaky MEMIC allocation (#10475, 69718f6, [REDACTED:Hex High Entropy String], d7d57e1). Nothing in the PR under test (8e73d93, "UCT/GDAKI: Assign ConnectX-9 HCAs to GPUs by board", touching src/uct/ib/mlx5/gdaki and the HCA/GPU assignment part of src/uct/ib/base/ib_md.c) affects rc_mlx5 atomic-key registration on device memory, so this is almost certainly an unrelated environment-sensitive/flaky failure rather than a regression from this PR. Caveat: the per-step log is truncated to its last ~500 lines, so the actual assertion text for the failing test was not retained — the exact sub-cause (MEMIC alloc failure vs. AMO error status) cannot be pinned down from this build alone.

Implicated commit: Not the PR commit. The most plausible pre-existing suspects, if it reproduces, are e4d1e59 "UCT/IB/MLX5: Fix atomic MR offset without a flush memory key (#12265)" (dmitrygx) and the test's own MEMIC-hardening history (69718f6, Michal Shalev). PR commit 8e73d93 (Artemy Kovalyov) is unrelated to this code path.

File: test/gtest/uct/test_atomic_key_reg_rdma_mem_type.cc:29-41 (test body); test/gtest/uct/uct_test.cc:1037 (retry only on UCS_ERR_NO_MEMORY)

Suggested fix:

  1. Re-trigger the "AddressSanitizer roce on worker 1" job — the failure is a single device-memory AMO test unrelated to the GDAKI board-assignment change; do not block the PR on it.
  2. To make it diagnosable, run the suite with the failing test only and keep the full log: GTEST_FILTER='rc_mlx5/uct_atomic_key_reg_rdma_mem_type.*' make -C build-test/test/gtest test.
  3. Harden the test path: in uct_test::entity::mem_alloc() retry on any failure status (not just UCS_ERR_NO_MEMORY) when mem_type == UCS_MEMORY_TYPE_RDMA, and/or convert the final failure for device memory into UCS_TEST_SKIP_R("cannot allocate device memory") instead of ASSERT_UCS_OK, so a transient MEMIC shortage on a shared CI node cannot fail the whole 9k-test run.

Related: PR under test: #12368 · prior hardening of this exact test: #10475 · atomic-MR-offset change in the same area: #12265

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 1121bc27-d79c-4970-a215-accc59861320 in the triage console for the audit trail.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants