Skip to content

GTEST/UCP: Skipped GPU-NIC assignment tests when there is no NICs to assign. - #12320

Merged
rakhmets merged 4 commits into
openucx:masterfrom
rakhmets:topic/skip-gtest-if-no-assignable-nics
Nov 16, 2026
Merged

rakhmets merged 4 commits into
openucx:masterfrom
rakhmets:topic/skip-gtest-if-no-assignable-nics

Conversation

@rakhmets

@rakhmets rakhmets commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

What?

Skipped GPU-NIC assignment tests when there is no NICs to assign.

Why?

[ RUN      ] rcx_cuda/test_ucp_rma_gpu_nic.put_blocking/0 <rc_x,cuda_copy/flip>
/rakhmetshin/ucx/contrib/../test/gtest/ucp/ucp_test.cc:689: Failure
...
     ucp_context.c:3056 UCX  DEBUG gpu-nic assignment mode flip
     ucp_context.c:2969 UCX  DEBUG rc_mlx5/mlx5_6:1 is ignored for gpu-nic assignment: dpu device
     ucp_context.c:2954 UCX  DEBUG ud_mlx5/mlx5_6:1 is ignored for gpu-nic assignment: auxiliary transport
     ucp_context.c:3025 UCX  DEBUG no assignable network devices for gpu-nic assignment mode flip, set UCX_GPU_NIC_ASSIGNMENT_MODE=off to disable it
...
Error: Invalid parameter
[  FAILED  ] rcx_cuda/test_ucp_rma_gpu_nic.put_blocking/0, where GetParam() = rc_x,cuda_copy/flip (137 ms)

@rakhmets
rakhmets requested a review from a team as a code owner November 11, 2026 18:56
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_context.c Resolved
Comment thread src/ucp/core/ucp_context.c Resolved
@rakhmets
rakhmets force-pushed the topic/skip-gtest-if-no-assignable-nics branch from c5f9d77 to a1805e8 Compare November 11, 2026 19:20
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@rakhmets rakhmets changed the title UCP/CORE: Changed error code when gpu2nic assignement is n/a. UCP/CORE: Changed error code when gpu2nic assignment is n/a. Nov 11, 2026
@rakhmets
rakhmets force-pushed the topic/skip-gtest-if-no-assignable-nics branch from a1805e8 to 688486a Compare November 11, 2026 19:22
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Review complete — no issues found.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Review complete — no issues found.

Comment thread src/ucp/core/ucp_context.c Resolved
@svc-ucx

svc-ucx commented Nov 12, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit 688486a9

TL;DR: The build failed on a single gtest, ib/test_md.alloc/1 (mlx5_1), which is a pure UCT memory-domain allocation test and cannot be affected by this PR's UCP gpu2nic error-code change — it is an unrelated failure; retrigger CI and fix the test's error reporting (it swallows the real error message and reads mem before checking status).

Full analysis

Summary: make test in the "roce on worker 0" job exited with code 2 because 1 of 9101 gtests failed: [ FAILED ] ib/test_md.alloc/1, where GetParam() = mlx5_1.

Root cause: The only failing test is test_md.alloc instantiated for the ib component on device mlx5_1 — a UCT-level test that opens an IB MD directly and loops 300 uct_mem_alloc(UCT_ALLOC_METHOD_MD) / uct_mem_free() cycles. No UCP context, no GPU/NIC assignment and no CUDA is involved; in fact every UCP GPU-NIC test in this run was skipped (rcx_cuda/test_ucp_rma_rndv_gpu_nic.* → [SKIP] (CUDA is not supported)), so the code changed by this PR (src/ucp/core/ucp_context.c / ucp_gpu_nic_assignment, commit 688486a) was never exercised on this worker. The failure is therefore environmental/flaky with respect to PR #12320 — the IB MD allocation returned a status other than UCS_OK/UCS_ERR_NO_MEMORY, tripping ASSERT_UCS_OK(status) (or the EXPECT_LT(num_alloc_failures/iterations, 0.5) "Too many OUT_OF_RESOURCE failures" guard) at test/gtest/uct/test_md.cc:529/547.

Notably, the console log contains no diagnostic at all for this failure: the test installs ignore_alloc_failure_log_handler (test_md.cc:448-465) which suppresses exactly the messages that would explain it ("failed to allocate", "exceeds maximal supported size"), and the gtest assertion text was in the part of the ~51-minute log that is truncated. There is also a latent bug: EXPECT_GT(mem.length, 0ul) and address = mem.address (test_md.cc:525-527) read mem before ASSERT_UCS_OK(status) at line 529, so on any non-NO_MEMORY error the test reads uninitialized memory and reports a misleading assertion.

Implicated commit: Not the PR head — 688486a (Raul Akhmetshin, "UCP/CORE: Changed error code when gpu2nic assignment is n/a.") touches only UCP context code. The most recent change on this test's actual code path is 84fb2f0 (Yaser Afshar, 2026-11-10, "UCT/ZE/COPY: Allocate DMA-BUF-exportable memory for RDMA (#11903)"), which modified both src/uct/ib/base/ib_md.c and test/gtest/uct/test_md.cc two days before this build — the candidate to check if the failure reproduces.

File: test/gtest/uct/test_md.cc:519-548 (test body); test/gtest/uct/test_md.cc:448-465 (log suppression)

Suggested fix:

  1. Retrigger the "roce on worker 0" job — this failure is not caused by PR GTEST/UCP: Skipped GPU-NIC assignment tests when there is no NICs to assign. #12320. If it does not reproduce, it is a known-style flake in the IB MD alloc loop.
  2. Make the test diagnosable so the next occurrence is actionable:
    • Move ASSERT_UCS_OK(status); to immediately after the UCS_ERR_NO_MEMORY check (before EXPECT_GT(mem.length, 0ul) and address = mem.address) so an error status doesn't read uninitialized mem.
    • Add the failing iteration/size/mem_type/ucs_status_string(status) to the assertion message, e.g. ASSERT_UCS_OK(status) << " size=" << orig_size << " mem_type=" << ucs_memory_type_names[mem_type];.
    • In ignore_alloc_failure_log_handler, re-emit the suppressed message via UCS_TEST_MESSAGE (or only suppress when status == UCS_ERR_NO_MEMORY) so the real rdma-core/IB error appears in the CI log instead of being silently dropped.
  3. If it reproduces on mlx5_1 only, bisect against 84fb2f0 (UCT/ZE/COPY: Allocate DMA-BUF-exportable memory for RDMA #11903) on the IB MD alloc path.

Related: PR #12320 (this build) — #12320; candidate regression source #11903 (commit 84fb2f0); no existing issue found for ib/test_md.alloc flakiness.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/ucp/test_ucp_context.cc Resolved
Comment thread src/ucp/core/ucp_context.c Resolved
Comment thread src/ucp/core/ucp_context.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/ucp/ucp_test.cc Resolved
@rakhmets rakhmets changed the title UCP/CORE: Changed error code when gpu2nic assignment is n/a. GTEST/UCP: Skipped GPU-NIC assignment tests when there is no NICs to assign. Nov 13, 2026
@svc-ucx

svc-ucx commented Nov 13, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit a6906249

TL;DR: The roce on worker 0 job wasn't slow — ucx_info -epw -u t in the test_unused_env_var step hung for 35 minutes with zero output until the Azure agent cancelled the job; the hang is in UCP worker/EP creation inside ucx_info and is unrelated to this PR's gtest-only change, so re-run and wrap these devel checks in timeout so a hang fails fast with diagnostics.

Full analysis

Summary: Step "Run ./contrib/test_jenkins.sh" was cancelled after ./src/tools/info/ucx_info -epw -u t (task test_unused_env_var) produced no output for ~35 minutes.

Root cause: Hang, not a time limit shortage. The log shows continuous build/install activity up to 2026-11-13T11:08:38.2134380Z + grep unused, i.e. the start of
UCX_IB_PORTS=mlx5_0:1 ./src/tools/info/ucx_info -epw -u t | grep "unused" | grep -q -E "UCX_IB_PORTS".
The next log line is 2026-11-13T11:43:29 "The agent has received a shutdown signal" — a single 34m51s gap that is essentially the whole remaining runtime. The job cleanup confirms the process was still alive and blocked: Terminate orphan process: pid (1810115) (test_jenkins.sh), pid (1956321) (ucx_info), plus the two grep children. So ucx_info blocked while creating the UCP worker/endpoint (-e -p -w with -u t) on this RoCE host; the agent cancellation/The operation was canceled messages are symptoms of that hang.

This is not attributable to the PR under test: #12320 ("GTEST/UCP: Skipped GPU-NIC assignment tests when there is no NICs to assign") only touches test/gtest/ucp/test_ucp_gpu_nic_assignment.cc, which is not linked into or executed by ucx_info. src/tools/info/ucx_info.c has not changed since commit a8e12e5f, and src/ucp/core/ucp_gpu_nic_assignment.c (last touched by abf206ae, Guy Ealey Morag) contains only bounded loops over topology groups, with no blocking calls. The failure is therefore a latent/flaky hang in UCP endpoint bring-up on the RoCE agent (agent-04), surfaced as a wall-clock kill.

Implicated commit: unknown — the PR head [REDACTED:Hex High Entropy String] (gtest-only change) cannot cause this; no commit in ucx_info/UCP init path correlates.

File: contrib/test_jenkins.sh:1161 (test_unused_env_var, the hung ucx_info -epw -u t invocation)

Suggested fix:

  1. Re-trigger the roce on worker 0 job — on current evidence this is an infrastructure/flaky hang unrelated to PR GTEST/UCP: Skipped GPU-NIC assignment tests when there is no NICs to assign. #12320, and the PR should not be blocked on it.
  2. Make the hang diagnosable instead of fatal to the whole job: the $TIMEOUT wrapper is only applied to gtest targets (run_gtest_make, run_malloc_hook_gtest, etc.), not to the quick devel checks. Wrap the ucx_info invocations, e.g. in test_unused_env_var:
    timeout -s ABRT 120 env UCX_IB_PORTS=mlx5_0:1 ./src/tools/info/ucx_info -epw -u t | grep "unused" | grep -q -E "UCX_IB_PORTS"
    With UCX_HANDLE_ERRORS=bt already exported by set_ucx_common_test_env, an ABRT on timeout yields a backtrace pinpointing where UCP EP creation blocks (device query / rdmacm / wireup) rather than a 35-minute silent stall.
  3. If it reproduces, capture a live stack of the stuck ucx_info on the RoCE agent (gdb -p <pid> -batch -ex "thread apply all bt") and check the agent's RoCE devices (mlx5_0:1 state) — a wedged port/ibv_* query on that host is the most likely blocker given UCX_IB_ROCE_SUBNET_PREFIX_LEN=inf and the RoCE-only configuration.

Related: #12320 (PR under test, gtest-only); GPU-NIC assignment series #11868, #11976, #11980, #11990, #11985 (reviewed and ruled out)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 11ecfd80-26cc-45e6-8ed1-fe9dcb844d46 in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/ucp/ucp_test.h Resolved
Comment thread test/gtest/ucp/ucp_test.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage note: the new skip path is only taken on hosts where ucp_init() fails with "no assignable network devices", so CI machines that do have assignable NICs will not exercise it; the generic init_skip_filter() mechanism has no other user. Worth confirming with a manual run (e.g. UCX_NET_DEVICES restricted to a non-assignable device) that the gpu-nic tests skip rather than fail.

@rakhmets
rakhmets enabled auto-merge (squash) November 13, 2026 16:04
@svc-ucx

svc-ucx commented Nov 14, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests new on worker 1) · commit 71c32d0f

TL;DR: The gtest job failed on a single unrelated test, rc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/1 on device mlx5_1 — a UCT IB device-memory (MEMIC / UCS_MEMORY_TYPE_RDMA) atomic test that exercises the indirect atomic-key path recently changed by #12265; nothing in PR #12320 (which only adds skip conditions to UCP GPU-NIC-assignment gtests) touches this code, so this is a pre-existing flake/regression on the rc_mlx5 atomic-key-over-DM path, not a PR breakage.

Full analysis

Summary: make test in the "Tests new on worker 1" job exited 2 because 1 of 8986 gtests failed: [ FAILED ] rc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/1, where GetParam() = rc_mlx5/mlx5_1:1 (Azure build 145054, 2026-11-14T00:56:24Z, make: *** [Makefile:5014: test] Error 1).

Root cause: Evidence from the log: exactly one failing test, it is a UCT (not UCP) test, it runs only on the second IB device (/1 = mlx5_1), and the remaining ~8985 tests — including every UCP GPU-NIC-assignment test touched by this PR (rcx_cuda/test_ucp_tag_mem_type_gpu_nic.* all SKIP/OK) — passed. So the failure is independent of the PR diff.

The failing test (test/gtest/uct/test_atomic_key_reg_rdma_mem_type.cc:29) allocates an 8-byte buffer in IB device memory (UCS_MEMORY_TYPE_RDMA, i.e. MEMIC via ibv_alloc_dm, with num_retries=10) and runs 64-bit fetch-add through the indirect atomic mkey. Two failure modes exist on that path, both plausible here:

  1. MEMIC allocation exhaustion on mlx5_1 → ASSERT_UCS_OK in uct_test::entity::mem_alloc() (test/gtest/uct/uct_test.cc:1050). This test has a documented history of exactly this flakiness (commits d7d57e1, [REDACTED:Hex High Entropy String], 69718f6 "Retry when it cannot allocate MEMIC memory" / "add retries and rand sleep").
  2. A wrong/short atomic mkey mapping for DM. In uct_ib_mlx5_devx_reg_ksm_data() (src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:524-545) DM uses a zero-based key (address == NULL) while iova = memh->address + atomic_offset, and uct_ib_mlx5_devx_reg_ksm_data_contig() derives ksm_length from mr->length only — the 8 * atomic_mr_id offset is added to the IOVA but not accounted for in the mapped length. atomic_mr_id comes from md->flush_rkey >> 8 (src/uct/ib/base/ib_md.h:428), so it is per-device and can legitimately differ between mlx5_0 and mlx5_1 — which matches the failure appearing only on /1. This is precisely the area changed three days before this run by UCT/IB/MLX5: Fix atomic MR offset without a flush memory key #12265 (and the log shows its new companion test dc_mlx5/test_dc.relaxed_order_required_hides_invalid_flush_rkey running).

Note: the whole-build log is truncated to its tail, so the per-assertion failure text printed mid-run was not retrievable; the distinction between (1) and (2) needs that line.

Implicated commit: Not the PR commit [REDACTED:Hex High Entropy String]. Prime suspect for a regression: e4d1e597 — "UCT/IB/MLX5: Fix atomic MR offset without a flush memory key (#12265)", dmitrygx, 2026-11-08 (touches src/uct/ib/base/ib_md.h and src/uct/ib/mlx5/dv/ib_mlx5dv_md.c). Otherwise a known flake with prior fixes 69718f6d / [REDACTED:Hex High Entropy String] (Michal Shalev).

File: test/gtest/uct/test_atomic_key_reg_rdma_mem_type.cc:29-41; implicated product code src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:329-355 (uct_ib_mlx5_devx_reg_ksm_data_contig) and :598-621 (uct_ib_mlx5_devx_reg_atomic_key), with src/uct/ib/base/ib_md.h:428-432 (uct_ib_md_get_atomic_mr_id).

Suggested fix:

  1. Do not block PR GTEST/UCP: Skipped GPU-NIC assignment tests when there is no NICs to assign. #12320 on this — re-run the job; the failure is in an unrelated UCT suite.
  2. Reproduce in isolation on the same agent to capture the real assertion text:
    UCX_LOG_LEVEL=debug GTEST_FILTER='rc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/1' make -C build-test/test/gtest test
  3. If the message is Allocation failed - Out of memory / MEMIC-related: raise num_retries above 10 at test_atomic_key_reg_rdma_mem_type.cc:34 or UCS_TEST_SKIP when ibv_alloc_dm cannot satisfy the request (device memory is shared with other concurrently-running gtest workers on this host).
  4. If the message is a reply/validation mismatch or remote-access error: fix the DM atomic-key range in uct_ib_mlx5_devx_reg_ksm_data_contig() to include the atomic IOVA offset in ksm_length (currently ksm_length = mr_length + address - ksm_address, with address == NULL for DM), i.e. extend the mapped length by iova_offset before the atomic_align round-up, and verify against a device whose flush_rkey >> 8 is non-zero.

Related: PR #12320 (this PR, unrelated to the failure); PR #12265 / commit e4d1e59 (atomic MR offset, prime regression suspect); PR #12324 "AZP/TESTS: Limit RNR retries in the no-SRQ gtest task" (indicates ongoing instability in this gtest task); prior MEMIC-flakiness fixes 69718f6, [REDACTED:Hex High Entropy String], d7d57e1.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 3a025861-4212-43d4-b1fe-b624a3c5e9f5 in the triage console for the audit trail.

@rakhmets
rakhmets merged commit 704cbc4 into openucx:master Nov 16, 2026
167 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants