Skip to content

UCP: Reject max eager, rndv and RMA rails above UCP_MAX_LANES - #12068

Merged
brminich merged 3 commits into
openucx:masterfrom
guy-ealey-morag:topic/ucp-cap-max-rails
Oct 19, 2026
Merged

brminich merged 3 commits into
openucx:masterfrom
guy-ealey-morag:topic/ucp-cap-max-rails

Conversation

@guy-ealey-morag

@guy-ealey-morag guy-ealey-morag commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

What?

  • Fail ucp_init() with UCS_ERR_INVALID_PARAM when UCX_MAX_EAGER_RAILS, UCX_MAX_RNDV_RAILS or UCX_MAX_RMA_RAILS exceeds UCP_MAX_LANES (64).
  • Shrink the per-device lane counters in wireup bandwidth-lane selection to uint8_t.

Why?

The code already assumes at most UCP_MAX_LANES rails: the protocol layer stores the value in an 8-bit field and asserts it is at most UCP_PROTO_MAX_LANES, so larger values were never usable.
Rejecting them at init makes that limit explicit, and lets the per-device lane counters in ucp_wireup_add_bw_lanes_pairwise() shrink to uint8_t, this saves lots of stack space especially after #12054 is merged.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_context.c Resolved
Comment thread src/ucp/core/ucp_context.c Resolved
Comment thread src/ucp/core/ucp_context.c Resolved
@svc-ucx

svc-ucx commented Oct 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 5c28235c

TL;DR: Not a code regression from PR #12068 — the mlx5_0 HCA on runner swx-rain03 wedged: a UMR QP completion never arrived (30 s ibv_poll_cq timeout), poisoning the IB memory domain so the next test (rc_mlx5/uct_p2p_rma_test.put_bcopy/0) hung for exactly 900 s until the gtest watchdog aborted the run. Retry the job on a different node; separately, harden uct_ib_mlx5_devx_umr_post_sync() to fail the MD fatally instead of leaving a broken UMR QP behind.

Full analysis

Summary: gga_mlx5/test_zcopy_comp.issue1440/0 failed on a UMR mkey registration timeout, and the following test rc_mlx5/uct_p2p_rma_test.put_bcopy/0 hung for 15 minutes until the watchdog fired SIGABRT, aborting make -C test/gtest test (make: *** [Makefile:4925: test] Aborted (core dumped), Bash exit 2).

Root cause: Hardware/driver-level failure on device mlx5_0, with a cascading-failure bug in the UMR error path — not the PR's change.

Evidence chain from the log:

  1. 20:51:23 — ib_mlx5dv_md.c:1449 mlx5_0: ibv_poll_cq(UMR CQ, registration, UMR mkey 0x...650 index 0x14e) timed out — the 30 s poll loop at ib_mlx5dv_md.c:1447-1453 expired. A UMR WQE should complete in microseconds; 30 s of no completion means the UMR QP/CQ stopped progressing, i.e. the QP went to error state.
  2. The cleanup path then hit ibv_post_send(UMR QP, invalidation, ...) returned 12: Cannot allocate memory (ib_mlx5dv_md.c:1440) — ENOMEM on post_send is the classic symptom of posting to a QP already in error/flush state.
  3. ibv_dealloc_pd() failed: Device or resource busy (ib_md.c:1542) — the MD could not be torn down, so MRs/QPs were leaked and the device was left in a broken state for the rest of the shard.
  4. The hang: the largest timestamp gap in the whole log is 20:51:23.58 → 21:06:23.55 — exactly 900 s of total silence with [ RUN ] rc_mlx5/uct_p2p_rma_test.put_bcopy/0 as the last line. That is the gtest watchdog timeout (test_helpers.cc:57-59, ETIMEDOUT → pthread_kill), confirmed by Connection timed out - abort testing and Caught signal 6. The backtrace frame libuct_ib_mlx5.so.0(+0xc116c) shows the abort landed inside the mlx5 transport. So put_bcopy was hung on the already-wedged mlx5_0, not legitimately slow — raising the 200 m timeout would not help.

Why this is unrelated to the PR: commit 5c28235c touches only src/ucp/core/ucp_context.c (clamping max_eager_rails/max_rndv_rails/RMA rails to UCP_MAX_LANES). That is UCP-level config validation; it cannot affect UCT IB DevX UMR mkey registration or uct_p2p_rma_test, which exercise the UCT API directly with no UCP context involved.

Secondary code-level weakness worth fixing: uct_ib_mlx5_devx_umr_post_sync() returns UCS_ERR_TIMED_OUT/UCS_ERR_IO_ERROR to its caller, but nothing resets or tears down the UMR QP, so every later registration on that MD inherits the broken QP — turning one transient HW hiccup into a 15-minute hang plus a leaked PD that kills the entire 8811-test shard.

Implicated commit: none — [REDACTED:Hex High Entropy String] (Guy Ealey Morag, src/ucp/core/ucp_context.c) does not touch the failing code path. The infrastructure trigger is the mlx5_0 device on node swx-rain03.

File: src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:1447-1453 (UMR poll timeout) and :1438-1444 (ENOMEM on post_send); hang manifested via test/gtest/common/test_helpers.cc:57

Suggested fix:

  1. Immediate: re-run the AddressSanitizer roce on worker 0 job; this is a node-local HW failure, not a PR regression. Do not block PR UCP: Reject max eager, rndv and RMA rails above UCP_MAX_LANES #12068 on it. Check swx-rain03 — dmesg for mlx5 firmware/CQ errors and whether mlx5_0 has leaked QPs/PDs from the aborted gtest process; the container was force-removed, so a leaked PD may persist and poison subsequent builds on that runner.
  2. Robustness (separate PR): on UMR post/poll failure, mark the MD as failed and destroy the UMR QP/CQ rather than returning a plain error and continuing. Wrap the error path in uct_ib_mlx5_devx_umr_post_sync() so the QP is reset (or the MD flagged unusable) before returning, so one wedged UMR operation fails fast instead of hanging all later tests on that device.
  3. CI hygiene: the 900 s watchdog correctly caught the hang but only after burning 15 minutes; consider having the gtest harness abort the shard immediately once an MD reports ibv_dealloc_pd() ... Device or resource busy, since every subsequent test on that device is doomed.

Related: PR #12068 (UCP: Reject max eager, rndv and RMA rails above UCP_MAX_LANES) — the PR under test, unaffected. No existing issue found for the ibv_poll_cq(UMR CQ ...) timed out signature; worth opening one against src/uct/ib/mlx5/dv/ib_mlx5dv_md.c for the non-recoverable UMR error path.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 7934292d-12c1-457b-975d-44554c78a526 in the triage console for the audit trail.

iyastreb
iyastreb previously approved these changes Oct 15, 2026
@brminich

Copy link
Copy Markdown
Contributor

can you implement some uint test pls?

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@guy-ealey-morag

guy-ealey-morag commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor Author

can you implement some unit test pls?

@brminich Added unit tests.

Comment thread src/ucp/core/ucp_context.c Resolved
Comment thread src/ucp/core/ucp_context.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@brminich
brminich merged commit 010d096 into openucx:master Oct 19, 2026
165 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants