Skip to content

UCP/RNDV: Skip inter-node memtype-copy without exportable memory - v1.23.x - #12044

Merged
tvegas1 merged 1 commit into
openucx:v1.23.xfrom
tvegas1:rndv_skip_memtype_copy_v1.23.x
Oct 8, 2026
Merged

tvegas1 merged 1 commit into
openucx:v1.23.xfrom
tvegas1:rndv_skip_memtype_copy_v1.23.x

Conversation

@tvegas1

@tvegas1 tvegas1 commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

What?

Backport #11754.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@tvegas1
tvegas1 enabled auto-merge (squash) October 7, 2026 17:25
@svc-ucx

svc-ucx commented Oct 7, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 1) · commit d022971a

TL;DR: The roce worker 1 gtest run aborted on assertion (address != 0) == (rkey_length > 0) in ucp_proto_rndv_check_rkey_length() after the mlx5_0 HCA's UMR queue stalled (30 s ibv_poll_cq(UMR CQ …) timed out), making rkey packing fail; ucp_proto_rndv_rts_pack() returns rkey_size = 0 on pack failure but leaves rts->address non-zero, so a malformed RTS is sent and the receiver asserts. Fix: handle the pack failure in ucp_proto_rndv_rts_pack() (zero rts->address/abort the request) instead of silently sending address != 0, rkey_length == 0.

Full analysis

Summary: shm_gga/test_ucp_wireup_1sided.* tests failed repeatedly with "Input/output error", then the gtest binary crashed (make: *** [Makefile:4758: test] Aborted (core dumped)) on a UCX assertion in ucp_proto_rndv_check_rkey_length() during one_sided_wireup_rndv/6.

Root cause: Two layers, clearly visible in the log:

  1. Environment/device trigger — starting at 18:17:42 the mlx5_0 UMR path on host swx-rain03 stops completing work requests:

    • ib_mlx5dv_md.c:1449 mlx5_0: ibv_poll_cq(UMR CQ, invalidation, UMR mkey … index 0x3511) timed out (the 30 s timeout in uct_ib_mlx5_devx_umr_post_sync)
    • followed by ibv_post_send(UMR QP, …) returned 12: Bad address on every subsequent attempt (the UMR QP is wedged after the timeout), plus failed to dereg from md[2]=mlx5_0: Operation timed out and ibv_dealloc_pd() failed: Device or resource busy.
      This is not caused by the PR's code path; it's an HCA/firmware/driver-level stall of the GGA UMR QP. Note src/uct/ib/mlx5/dv/ib_mlx5dv_md.c was touched twice recently around ODP/DDP ([REDACTED:Hex High Entropy String], and 75cf374 which temporarily reverted the same change), which makes this area suspect on this fleet.
  2. The fatal abort is a UCX robustness bug exposed by (1). With the UMR QP broken, ucp_rkey_pack_memh() fails → proto_common.inl:436 failed to pack remote key: Input/output error and ucp_proto_request_pack_rkey() returns 0. In ucp_proto_rndv_rts_pack() the rts->address field has already been assigned the local buffer pointer before the pack call, and the return value is not checked, so an RTS with address != 0 and rkey_length == 0 goes out. The receiver then hits:
    proto_rndv.c:912 Assertion '(address != 0) == (rkey_length > 0)' failed: rts rts->address=0x55b0c01d7c20 rkey_length=0
    via ucp_rndv_rts_handler → ucp_tag_rndv_process_rts → ucp_proto_rndv_receive_start. (Since this is a loopback/shm,gga test, sender and receiver are the same process.) In a release build this would instead be a silently malformed RTS.

The large timestamp gaps (18:21:17 → 18:23:04 → 18:24:54) are just the assertion/backtrace symbolization after the abort, not a hang.

Implicated commit: The abort path is pre-existing (missing error check in ucp_proto_rndv_rts_pack, introduced with the protov2 RTS packing code). The device-level UMR stall is most plausibly related to [REDACTED:Hex High Entropy String] (ofarjon, "UCT/IB: Allow ODP with DDP for FW >= 49.1014 - v1.23") which re-landed a change previously reverted by 75cf3740. The PR under test (d022971a, Thomas Vegas, "UCP/RNDV: Skip inter-node memtype-copy without exportable memory - v1.23.x") only affects md_map selection and is not implicated by any log evidence.

File: src/ucp/rndv/proto_rndv.inl:284-293 (RTS pack, missing pack-failure handling); assertion at src/ucp/rndv/proto_rndv.c:912; failure origin src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:1447-1453.

Suggested fix:

  1. Make ucp_proto_rndv_rts_pack() consistent on failure — e.g.:
    rkey_size = ucp_proto_request_pack_rkey(...);
    if (rkey_size == 0) {
        rts->address = 0;   /* keep RTS self-consistent: no rkey => no address */
    } else {
        rts->address = (uintptr_t)req->send.state.dt_iter.type.contig.buffer;
    }
    (or better, propagate the error and complete the request with UCS_ERR_IO_ERROR rather than sending an RTS that cannot be served). Only set rts->address after a successful pack.
  2. Harden the UMR path: after uct_ib_mlx5_devx_umr_post_sync() times out, mark the UMR QP as broken and return UCS_ERR_IO_ERROR immediately for subsequent requests instead of repeatedly posting into a wedged QP (the log shows ~8 repeated 30 s stalls, which is also why this job burned ~7 minutes).
  3. Re-run the PR and, if the UMR timeouts reproduce on swx-rain03, check that node's HCA firmware against the ODP/DDP gating added in [REDACTED:Hex High Entropy String] — this looks like an infrastructure/firmware issue, not a PR regression.

Related: PR #12044 (this build), PR #11754 (master version of the backported change), PR #12028 / #11973 / #11956 (the ODP-with-DDP land/revert/re-land cycle in ib_mlx5dv_md.c).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 3c389587-4ce3-4d6d-bbd3-07c6c8ab798d in the triage console for the audit trail.

@tvegas1
tvegas1 merged commit 9ec898d into openucx:v1.23.x Oct 8, 2026
162 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants