Skip to content

UCT/IB/MLX5: Fix atomic MR offset without a flush memory key - #12265

Merged
dmitrygx merged 12 commits into
openucx:masterfrom
dmitrygx:topic/ucx/nondevx-atomic-offset
Nov 8, 2026
Merged

dmitrygx merged 12 commits into
openucx:masterfrom
dmitrygx:topic/ucx/nondevx-atomic-offset

Conversation

@dmitrygx

@dmitrygx dmitrygx commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

What?

uct_ib_md_get_atomic_mr_id() returns 0 when the flush rkey is invalid, so
the mlx5 memory domain without DevX, and a DevX memory domain whose flush
memory key failed to register, pass atomic MR id 0 to peers instead of the
process id. They also set the flush rkey to UCT_IB_MD_INVALID_FLUSH_RKEY
now, and uct_ib_mlx5_flush_rkey_make() is removed.

Why?

Atomics, and the first put after a fence, use the atomic half of the rkey:
uct_ib_resolve_atomic_rkey() adds 8 * the atomic MR id from the peer address
to the remote address. The id is bits 8-15 of the flush rkey (since
005d879). Only the indirect atomic keys of the DevX MD are created at that
offset.

  • The non-DevX mlx5 MD packs its memory key as the atomic key (since
    e4b8123), but passed getpid() & 0xff as the atomic MR id. So rc_mlx5 and
    dc_mlx5 peers, with or without DevX, sent every atomic, and every put after
    a fence when their transport fence is enabled (e.g. on devices with PCIe
    atomics, or with relaxed ordering, where an interface flush also fences the
    next put), to 8 * (pid & 0xff) bytes past the requested address. They
    updated other memory, or failed with a remote access error
    past the end of the memory region. For example, with UCX_IB_MLX5_DEVX=n a
    fetch-add to a buffer at 0x...820000 went to 0x...820310
    (pid & 0xff = 98). This MD is used when DevX is disabled or unavailable.
  • A DevX MD with KSM whose flush memory key failed to register used the same
    value and created its atomic keys at that offset. rc_mlx5 and dc_mlx5 pass
    the id, but rc_verbs passes it only with a valid flush rkey, so rc_verbs
    atomics to such a peer went 8 * (pid & 0xff) bytes before the target.
New UCT tests on master without the fix: atomics miss their target, the put after a fence lands at word 78 (pid & 0xff of the test process) instead of word 0, and the rc_verbs atomic to a peer with an invalid flush rkey lands at word 166 instead of word 256
Note: Google Test filter = rc_*/uct_p2p_rma_test_atomic_key.*:dc_*/uct_p2p_rma_test_atomic_key.*:rc_*/uct_amo_add_xor_test.add64*:dc_*/uct_amo_add_xor_test.add64*:rc_verbs/test_rc_verbs_invalid_flush_rkey.*
[==========] Running 65 tests from 7 test suites.
...
[ RUN      ] rc_mlx5/uct_amo_add_xor_test.add64_no_devx/0 <rc_mlx5/mlx5_8:1>
test/gtest/uct/test_amo_add_xor.cc:44: Failure
Expected equality of these values:
  exp_result
    Which is: 377777819489973502
  *(T*)recvbuf.ptr()
    Which is: 6694522428030182270
[  FAILED  ] rc_mlx5/uct_amo_add_xor_test.add64_no_devx/0, where GetParam() = rc_mlx5/mlx5_8:1 (87 ms)
...
[ RUN      ] dc_mlx5/uct_amo_add_xor_test.add64_no_devx/0 <dc_mlx5/mlx5_8:1>
test/gtest/uct/test_amo_add_xor.cc:44: Failure
Expected equality of these values:
  exp_result
    Which is: 7822019642905982991
  *(T*)recvbuf.ptr()
    Which is: 3058830362627264175
[  FAILED  ] dc_mlx5/uct_amo_add_xor_test.add64_no_devx/0, where GetParam() = dc_mlx5/mlx5_8:1 (88 ms)
...
[ RUN      ] rc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/0 <rc_mlx5/mlx5_8:1>
test/gtest/uct/ib/test_ib_xfer.cc:70: Failure
Expected equality of these values:
  (i == 0) ? VALUE : 0
    Which is: 6510615555426900570
  words[i]
    Which is: 0
word 0
test/gtest/uct/ib/test_ib_xfer.cc:70: Failure
Expected equality of these values:
  (i == 0) ? VALUE : 0
    Which is: 0
  words[i]
    Which is: 6510615555426900570
word 78
test/gtest/uct/ib/test_ib_xfer.cc:70: Failure
Expected equality of these values:
  (i == 0) ? VALUE : 0
    Which is: 6510615555426900570
  words[i]
    Which is: 0
word 0
test/gtest/uct/ib/test_ib_xfer.cc:70: Failure
Expected equality of these values:
  (i == 0) ? VALUE : 0
    Which is: 0
  words[i]
    Which is: 6510615555426900570
word 78
test/gtest/uct/ib/test_ib_xfer.cc:70: Failure
Expected equality of these values:
  (i == 0) ? VALUE : 0
    Which is: 6510615555426900570
  words[i]
    Which is: 0
word 0
test/gtest/uct/ib/test_ib_xfer.cc:70: Failure
Expected equality of these values:
  (i == 0) ? VALUE : 0
    Which is: 0
  words[i]
    Which is: 6510615555426900570
word 78
[  FAILED  ] rc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/0, where GetParam() = rc_mlx5/mlx5_8:1 (38 ms)
...
[ RUN      ] dc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/0 <dc_mlx5/mlx5_8:1>
test/gtest/uct/ib/test_ib_xfer.cc:70: Failure
Expected equality of these values:
  (i == 0) ? VALUE : 0
    Which is: 6510615555426900570
  words[i]
    Which is: 0
word 0
test/gtest/uct/ib/test_ib_xfer.cc:70: Failure
Expected equality of these values:
  (i == 0) ? VALUE : 0
    Which is: 0
  words[i]
    Which is: 6510615555426900570
word 78
test/gtest/uct/ib/test_ib_xfer.cc:70: Failure
Expected equality of these values:
  (i == 0) ? VALUE : 0
    Which is: 6510615555426900570
  words[i]
    Which is: 0
word 0
test/gtest/uct/ib/test_ib_xfer.cc:70: Failure
Expected equality of these values:
  (i == 0) ? VALUE : 0
    Which is: 0
  words[i]
    Which is: 6510615555426900570
word 78
test/gtest/uct/ib/test_ib_xfer.cc:70: Failure
Expected equality of these values:
  (i == 0) ? VALUE : 0
    Which is: 6510615555426900570
  words[i]
    Which is: 0
word 0
test/gtest/uct/ib/test_ib_xfer.cc:70: Failure
Expected equality of these values:
  (i == 0) ? VALUE : 0
    Which is: 0
  words[i]
    Which is: 6510615555426900570
word 78
[  FAILED  ] dc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/0, where GetParam() = dc_mlx5/mlx5_8:1 (39 ms)
...
[ RUN      ] rc_verbs/test_rc_verbs_invalid_flush_rkey.atomic_add/0 <rc_verbs/mlx5_8:1>
test/gtest/uct/ib/test_rc.cc:855: Failure
Expected equality of these values:
  (i == target) ? value : 0
    Which is: 0
  words[i]
    Which is: 6510615555426900570
word 166
test/gtest/uct/ib/test_rc.cc:855: Failure
Expected equality of these values:
  (i == target) ? value : 0
    Which is: 6510615555426900570
  words[i]
    Which is: 0
word 256
[  FAILED  ] rc_verbs/test_rc_verbs_invalid_flush_rkey.atomic_add/0, where GetParam() = rc_verbs/mlx5_8:1 (39 ms)
...
[==========] 65 tests from 7 test suites ran. (9828 ms total)
[  PASSED  ] 30 tests.
[  FAILED  ] 35 tests, listed below:
[  FAILED  ] rc_mlx5/uct_amo_add_xor_test.add64_no_devx/0, where GetParam() = rc_mlx5/mlx5_8:1
[  FAILED  ] rc_mlx5/uct_amo_add_xor_test.add64_no_devx/1, where GetParam() = rc_mlx5/rdma_vf_rail0:1
[  FAILED  ] rc_mlx5/uct_amo_add_xor_test.add64_no_devx/2, where GetParam() = rc_mlx5/rdma_vf_rail1:1
[  FAILED  ] rc_mlx5/uct_amo_add_xor_test.add64_no_devx/3, where GetParam() = rc_mlx5/rdma_vf_rail2:1
[  FAILED  ] rc_mlx5/uct_amo_add_xor_test.add64_no_devx/4, where GetParam() = rc_mlx5/rdma_vf_rail3:1
[  FAILED  ] dc_mlx5/uct_amo_add_xor_test.add64_no_devx/0, where GetParam() = dc_mlx5/mlx5_8:1
[  FAILED  ] dc_mlx5/uct_amo_add_xor_test.add64_no_devx/1, where GetParam() = dc_mlx5/rdma_vf_rail0:1
[  FAILED  ] dc_mlx5/uct_amo_add_xor_test.add64_no_devx/2, where GetParam() = dc_mlx5/rdma_vf_rail1:1
[  FAILED  ] dc_mlx5/uct_amo_add_xor_test.add64_no_devx/3, where GetParam() = dc_mlx5/rdma_vf_rail2:1
[  FAILED  ] dc_mlx5/uct_amo_add_xor_test.add64_no_devx/4, where GetParam() = dc_mlx5/rdma_vf_rail3:1
[  FAILED  ] rc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/0, where GetParam() = rc_mlx5/mlx5_8:1
[  FAILED  ] rc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/1, where GetParam() = rc_mlx5/mlx5_8:1/loopback
[  FAILED  ] rc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/2, where GetParam() = rc_mlx5/rdma_vf_rail0:1
[  FAILED  ] rc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/3, where GetParam() = rc_mlx5/rdma_vf_rail0:1/loopback
[  FAILED  ] rc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/4, where GetParam() = rc_mlx5/rdma_vf_rail1:1
[  FAILED  ] rc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/5, where GetParam() = rc_mlx5/rdma_vf_rail1:1/loopback
[  FAILED  ] rc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/6, where GetParam() = rc_mlx5/rdma_vf_rail2:1
[  FAILED  ] rc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/7, where GetParam() = rc_mlx5/rdma_vf_rail2:1/loopback
[  FAILED  ] rc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/8, where GetParam() = rc_mlx5/rdma_vf_rail3:1
[  FAILED  ] rc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/9, where GetParam() = rc_mlx5/rdma_vf_rail3:1/loopback
[  FAILED  ] dc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/0, where GetParam() = dc_mlx5/mlx5_8:1
[  FAILED  ] dc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/1, where GetParam() = dc_mlx5/mlx5_8:1/loopback
[  FAILED  ] dc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/2, where GetParam() = dc_mlx5/rdma_vf_rail0:1
[  FAILED  ] dc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/3, where GetParam() = dc_mlx5/rdma_vf_rail0:1/loopback
[  FAILED  ] dc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/4, where GetParam() = dc_mlx5/rdma_vf_rail1:1
[  FAILED  ] dc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/5, where GetParam() = dc_mlx5/rdma_vf_rail1:1/loopback
[  FAILED  ] dc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/6, where GetParam() = dc_mlx5/rdma_vf_rail2:1
[  FAILED  ] dc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/7, where GetParam() = dc_mlx5/rdma_vf_rail2:1/loopback
[  FAILED  ] dc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/8, where GetParam() = dc_mlx5/rdma_vf_rail3:1
[  FAILED  ] dc_mlx5/uct_p2p_rma_test_atomic_key.fence_put_no_devx/9, where GetParam() = dc_mlx5/rdma_vf_rail3:1/loopback
[  FAILED  ] rc_verbs/test_rc_verbs_invalid_flush_rkey.atomic_add/0, where GetParam() = rc_verbs/mlx5_8:1
[  FAILED  ] rc_verbs/test_rc_verbs_invalid_flush_rkey.atomic_add/1, where GetParam() = rc_verbs/rdma_vf_rail0:1
[  FAILED  ] rc_verbs/test_rc_verbs_invalid_flush_rkey.atomic_add/2, where GetParam() = rc_verbs/rdma_vf_rail1:1
[  FAILED  ] rc_verbs/test_rc_verbs_invalid_flush_rkey.atomic_add/3, where GetParam() = rc_verbs/rdma_vf_rail2:1
[  FAILED  ] rc_verbs/test_rc_verbs_invalid_flush_rkey.atomic_add/4, where GetParam() = rc_verbs/rdma_vf_rail3:1

35 FAILED TESTS
test_ucp_rma_weak_fence on master without the fix
Note: Google Test filter = *test_ucp_rma_weak_fence.*
[==========] Running 4 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from rcx/test_ucp_rma_weak_fence
[ RUN      ] rcx/test_ucp_rma_weak_fence.put/0 <rc_x>
[       OK ] rcx/test_ucp_rma_weak_fence.put/0 (889 ms)
[ RUN      ] rcx/test_ucp_rma_weak_fence.put/1 <rc_x/no_devx>
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 1000000
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 100000
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 10000
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 1000
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 100
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 10
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 1
[  FAILED  ] rcx/test_ucp_rma_weak_fence.put/1, where GetParam() = rc_x/no_devx (992 ms)
[----------] 2 tests from rcx/test_ucp_rma_weak_fence (1881 ms total)

[----------] 2 tests from dcx/test_ucp_rma_weak_fence
[ RUN      ] dcx/test_ucp_rma_weak_fence.put/0 <dc_x>
[       OK ] dcx/test_ucp_rma_weak_fence.put/0 (959 ms)
[ RUN      ] dcx/test_ucp_rma_weak_fence.put/1 <dc_x/no_devx>
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 1000000
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 100000
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 10000
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 1000
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 100
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 10
test/gtest/ucp/test_ucp_rma.cc:1209: Failure
Value of: mem_buffer::compare(expected.ptr(), rbuf.ptr(), rbuf.size(), UCS_MEMORY_TYPE_HOST)
  Actual: false
Expected: true
size is 1
[  FAILED  ] dcx/test_ucp_rma_weak_fence.put/1, where GetParam() = dc_x/no_devx (1105 ms)
[----------] 2 tests from dcx/test_ucp_rma_weak_fence (2064 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 2 test suites ran. (3945 ms total)
[  PASSED  ] 2 tests.
[  FAILED  ] 2 tests, listed below:
[  FAILED  ] rcx/test_ucp_rma_weak_fence.put/1, where GetParam() = rc_x/no_devx
[  FAILED  ] dcx/test_ucp_rma_weak_fence.put/1, where GetParam() = dc_x/no_devx

 2 FAILED TESTS

How?

  • uct_ib_md_get_atomic_mr_id() checks the flush rkey, so the atomic MR id is
    0 without a flush memory key in any memory domain: the atomic keys of the
    non-DevX MD are the memory keys themselves, and the DevX MD then creates its
    atomic keys without an offset. Peers use the id as bits 8-15 of the flush
    rkey only when the address carries a valid one, so remote flush does not
    change.
  • The address format does not change. A peer of an older version without DevX
    still passes a non-zero id, so atomics to it stay misplaced until it is
    updated.
  • New test uct_p2p_rma_test_atomic_key.fence_put_no_devx for rc_verbs,
    rc_mlx5 and dc_mlx5 with IB_MLX5_DEVX=n: put short/bcopy/zcopy after an
    interface fence, to the first word of a buffer which covers the largest
    atomic MR offset; only that word may change. RC_FENCE=weak makes the fence
    take effect on any device.
  • New variant uct_amo_add_xor_test.add64_no_devx of the AMO test, with
    IB_MLX5_DEVX=n, on IB transports.
  • New test test_ucp_rma_weak_fence for rc_x and dc_x, with and without
    IB_MLX5_DEVX=n: a put, a weak fence and a put of the same size, so on the
    same lane; the target must hold the second put and the guard area after it
    must not change. A strong fence does not cover this case: the MD without
    DevX does not use relaxed ordering, so the puts after a strong fence use the
    memory key.
  • New test test_rc_verbs_invalid_flush_rkey for rc_verbs: before
    connecting, the receiver gets an invalid flush rkey with non-zero bits 8-15,
    as a DevX memory domain whose flush memory key failed to register had, so
    rc_verbs passes no atomic MR id; an atomic add must still update the
    requested word.

Peers add 8 * atomic MR id to the remote address of atomics, and of the
first put after a fence, which use the atomic half of the rkey. The id is
bits 8-15 of the flush rkey. Only the indirect atomic keys of the DevX MD,
which are created at that offset, need it.

The mlx5 MD without DevX has no flush memory key, but it set the flush rkey
to an invalid value with the process id in bits 8-15. The atomic MR id
comes from these bits since 005d879, and this MD packs its memory key as
the atomic key since e4b8123. So rc_mlx5 and dc_mlx5 peers sent atomics,
and puts after a fence when the transport fence is enabled (e.g. on devices
with PCIe atomics), to 8 * (pid & 0xff) bytes past the requested address:
they updated other memory, or failed with a remote access error past the
end of the memory region.

The DevX MD set the same value when it failed to register its flush memory
key, and created its atomic keys at that offset, but rc_verbs passes the
atomic MR id only with a valid flush rkey, so rc_verbs peers used no offset.

Set the flush rkey to UCT_IB_MD_INVALID_FLUSH_RKEY in both cases, so that
the atomic MR id is 0 without a flush memory key.
Add test_rc_atomic_key for rc_verbs, rc_mlx5 and dc_mlx5: a fetch-add, and
put_short, put_bcopy and put_zcopy after an interface fence, to the first
word of a buffer which covers the largest atomic MR offset. Only that word
must change. Run them with the default memory domain and with
IB_MLX5_DEVX=n; RC_FENCE=weak makes the fence take effect on any device.

Add test_ucp_rma_weak_fence for rc_x and dc_x, with and without
IB_MLX5_DEVX=n: a put, a weak fence and a put of the same size, so on the
same lane. The target buffer must hold the data of the second put, and the
guard area after it must not change.
@dmitrygx
dmitrygx requested a review from a team as a code owner November 5, 2026 18:33
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/mlx5/dv/ib_mlx5dv_md.c Resolved
Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

test/gtest/uct/ib/test_rc.cc

minor: in size_flush_remote, now that an invalid flush_rkey always has atomic_mr_id 0, the (mr_id != 0) term is dead (ep_addr_len depends only on flush_rkey validity) - pls drop it together with rc_iface_mr_id(), which exists only for this check.

Residual coverage gap: the DevX path where uct_ib_mlx5_devx_init_flush_mr() fails (atomic keys exist, flush key does not - the rc_verbs reason quoted in the new ib_md.h comment) is not exercised by any of the new tests and is hard to simulate without injecting a CREATE_MKEY failure; worth stating in the PR description if a test is not practical.

An invalid flush rkey now always gives atomic MR id 0, so the rc_verbs
endpoint address length depends only on whether the flush rkey is valid.
@svc-ucx

svc-ucx commented Nov 5, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer BlueField on worker 0) · commit 7a7f56ae

TL;DR: The BlueField ASAN gtest shard didn't run out of time — it hung in its very first test (shm_ib/test_ucp_am_nbx_seg_size.single/1 <shm,ib/proto_v1>): a 12095‑byte eager AM was posted on the new sender's IB lane and never arrived, and the teardown flush never completed either, so the shard self-aborted. The only change on this branch is commit 15d4db40, which flips the mlx5 non‑DEVX MD's flush_rkey to UCT_IB_MD_INVALID_FLUSH_RKEY — overloading one field for both "atomic MR id" and "remote-flush key validity", which changes what RC/DC advertise to the peer in addition to the intended atomic_mr_id = 0.

Full analysis

Summary: make -C test/gtest test (shard 1/4, ASAN, aarch64/BlueField) aborted after the first UCP test hung: test_ucp_am.cc:594 EXPECT_EQ(m_recv_counter=0, m_send_counter=1), then Connection timed out - abort testing and signal 6 from ucp_test_base::entity::close_all_eps().

Root cause: Evidence of a hang, not slowness: [ RUN ] shm_ib/test_ucp_am_nbx_seg_size.single/1 and [ INFO ] seg size 24191 data size 12095 at 20:05:28, then 14.5 minutes of total silence until ucp_test.cc:310 UCX ERROR request 0xffff4a60a420 did not complete on time at 20:20:04, a second 11‑minute silent gap to the next timed-out request at 20:31:20, then abort. So the AM message was posted on the (shm,ib) sender created by this test with a different IB_SEG_SIZE, was never delivered to the receiver, and the subsequent close_all_eps() flush never completed — i.e. the IB lane was non-functional for both data and flush. That points at the UCT/IB MD/endpoint-address state, which is exactly what this branch changes: src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:3438-3440 now sets md->super.flush_rkey = UCT_IB_MD_INVALID_FLUSH_RKEY for the non‑DEVX mlx5 MD. Because uct_ib_md_get_atomic_mr_id() (flush_rkey >> 8) and uct_ib_md_is_flush_rkey_valid() (flush_rkey & 0xff) read the same field (src/uct/ib/base/ib_md.h:176-183, 424-433), forcing atomic_mr_id = 0 simultaneously marks the remote-flush key invalid, which changes the flush_rkey/atomic_mr_id the RC/DC endpoint address propagates to the peer. Note this exact coupling has bitten before: 48163083 "UCT/IB/RC: Adapt EP address size without flush_rkey" had to be reverted by 362bbc49. I could not prove the chain to packet loss from the logs alone (no UCX debug log/core in the artifacts), so treat the attribution as high-suspicion rather than proven; the smoking gun is that the first IB test in the shard hung on a branch whose only change is in the IB MD rkey path.

Implicated commit: 15d4db40 — "UCT/IB/MLX5: Fix atomic MR offset without a flush memory key" (Dmitrii, 2026-11-05), the sole change in PR #12265 (head 7a7f56ae); it touches src/uct/ib/base/ib_md.h and src/uct/ib/mlx5/dv/ib_mlx5dv_md.c.

File: src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:3438-3440 (and src/uct/ib/base/ib_md.h:176-183, 430-433)

Suggested fix: Stop overloading flush_rkey to carry the atomic MR id. Add an explicit field to uct_ib_md_t (e.g. uint8_t atomic_mr_id), have uct_ib_md_get_atomic_mr_id() return it, set it to 0 in uct_ib_mlx5dv_md_open() (non‑DEVX, where the atomic key is the direct key, so offset must be 0) and to flush_rkey >> 8 in the DEVX path — leaving flush_rkey validity semantics (and therefore the advertised remote-flush capability and EP address contents) untouched. Meanwhile, to confirm: re-run the BlueField ASAN job on this commit and on the merge base; if it reproduces only with the patch, check whether that agent's IB MD is DEVX or ib_mlx5dv_md (ucs_debug lines "opened DEVX md" vs. the dv open path) and capture UCX_LOG_LEVEL=debug for the wireup of the extra test_ucp_am_nbx_seg_size entity. Do not raise the 200m timeout — the log shows a hang, not a job that needed more wall time.

Related: PR #12265 (this change); prior revert of the same coupling: 48163083 / 362bbc49 ("Adapt EP address size without flush_rkey" and its revert); adjacent non‑DEVX MD work in PR #12112 ("IB/MLX5: Enable relaxed ordering without DEVX").

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@dmitrygx

dmitrygx commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

minor: in size_flush_remote, now that an invalid flush_rkey always has atomic_mr_id 0, the (mr_id != 0) term is dead

Done in 898921b: removed the term and rc_iface_mr_id().

Residual coverage gap: the DevX path where uct_ib_mlx5_devx_init_flush_mr() fails

Right, the new tests do not cover it; I added this to the description. I checked that path by forcing the flush memory key registration to fail: rc_verbs atomics to such a peer got a remote access error on master and passed with this fix.

Comment thread src/uct/ib/base/ib_md.h Resolved
Check the flush rkey in uct_ib_md_get_atomic_mr_id() instead of relying on
every memory domain to use UCT_IB_MD_INVALID_FLUSH_RKEY as its invalid
value.
Comment thread test/gtest/uct/ib/test_rc.cc Resolved
Comment thread test/gtest/uct/ib/test_rc.cc Resolved
This reverts commit 898921b: a pending change to the same test updates
this expectation.
Keep only the put after a fence without DevX in test_rc_atomic_key, and
check atomics with a variant of uct_amo_add_xor_test without DevX. The
variants with DevX pass without the fix.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
Comment thread test/gtest/uct/test_amo_add_xor.cc Resolved
Comment thread test/gtest/uct/ib/test_rc.cc Resolved
A DevX memory domain whose flush memory key failed to register has an
invalid flush rkey, and rc_verbs then does not pass the atomic MR id to the
peer. Give the receiver such a flush rkey, with non-zero bits 8-15, and
check that an atomic add still updates the requested address.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/base/ib_md.h Resolved
Comment thread test/gtest/uct/test_amo_add_xor.cc Resolved
@svc-ucx

svc-ucx commented Nov 6, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer BlueField on worker 0) · commit 3877670e

TL;DR: The ASAN BlueField gtest shard died when ud_mlx5/test_uct_peer_failure_keepalive.killed/0 made zero forward progress — the loopback UD AM was never delivered and uct_ud_iface_flush() never completed, so the gtest watchdog SIGABRT'd the process; this is a UD/mlx5 data-path hang on the aarch64 BlueField worker and is unrelated to the atomic-MR-offset change in PR #12265, so retrigger and (if reproducible) investigate the mlx5 BlueFlame/UD posting path rather than raising timeouts.

Full analysis

Summary: make -C test/gtest test (shard 1/4, AddressSanitizer BlueField) aborted in the first test, ud_mlx5/test_uct_peer_failure_keepalive.killed/0, after ~17 min of no progress.

Root cause: From the stage log timeline:

  • 09:45:32 — [ RUN ] ud_mlx5/test_uct_peer_failure_keepalive.killed/0 <ud_mlx5/mlx5_0:1> — last meaningful line.
  • 09:48:53 and 09:52:13 — two identical failures at test/gtest/uct/test_peer_failure.cc:194 (m_am_count[index] is 0, expected 1). These are the two send_recv_am() calls made from new_receiver() during test_uct_peer_failure::init(). Each stall is ~200 s, i.e. wait_for_flag() timed out — the AM short posted on the ud_mlx5 EP was never received, even though sender and receiver are the same process/device (loopback). Notably no UD retransmit/ENDPOINT_TIMEOUT/CQE-error messages were printed at all, so no TX or RX completions were being reaped either.
  • 09:52:13 → 10:00:32 — an 8 m 19 s gap with zero output, then test/gtest/common/test_helpers.cc:57: Failed / Connection timed out - abort testing — that is the gtest watchdog firing (ETIMEDOUT → pthread_kill), followed by Caught signal 6.
  • The backtrace confirms where it was stuck: test_peer_failure.cc:1484 (flush() in test_ep_check) → uct_test::flush() (uct_test.cc:810) → uct_ud_iface_flush() (ud_iface.c:816 / ud_iface.h:495), i.e. the flush loop spun on UCS_INPROGRESS because UD TX outstanding never drained.

So the largest gap is most of the runtime → this is a hang, not slowness. The ud_mlx5 transport on swx-rain04-bf1 posted work requests that never produced completions and never delivered data.

This is not in the code path changed by PR #12265 ("Fix atomic MR offset without a flush memory key"): the atomic MR offset / flush_rkey / atomic_mr_id machinery is only consumed by the RC/DC indirect-atomic path (uct_ib_resolve_atomic_rkey, uct_ib_md_atomic_offset, uct_ib_md_get_atomic_mr_id — src/uct/ib/base/ib_md.h:315-432) and uct_ib_verbs_mkey_pack (ib_md.c:767). UD has no atomics and its RX/TX buffers use the plain memh->lkey, so the PR cannot explain a UD AM never arriving.

Implicated commit: unknown — the PR under test (3877670, topic/ucx/nondevx-atomic-offset) is not implicated by the evidence. The only recent change touching the aarch64 mlx5 WQE/BlueFlame posting path, which is the natural suspect for "WQE posted, nothing ever transmitted, no completions" on an aarch64 BlueField, is 6e3cfd3a "UCT/IB/MLX5: Add AArch64 ST64B BF copy mode" (Thomas Vegas) — treat as a hypothesis to rule out, not a conclusion.

File: test/gtest/uct/test_peer_failure.cc:194 (observed failure) and src/uct/ib/ud/base/ud_iface.c:816 (uct_ud_iface_flush, where it hung); candidate area src/uct/ib/mlx5/ib_mlx5.c:801-836 (uct_ib_mlx5_txwq_init_bf_copy, ST64B selection on aarch64).

Suggested fix:

  1. Re-trigger the AddressSanitizer BlueField job. Do not raise the 200 m timeout or the gtest watchdog — the process was hung, not slow.
  2. If it reproduces, confirm whether it is node/HW state or software: check the mlx5_0:1 port state and error counters on swx-rain04-bf1, and re-run the single test with UCX_LOG_LEVEL=debug plus UCX_IB_BF_COPY_MODE=generic (defaults to auto, which selects st64b on aarch64 CPUs reporting LS64 — src/uct/ib/mlx5/ib_mlx5.c:807-816). If generic passes and auto/st64b hangs, the ST64B BlueFlame copy path on this BlueField is the bug and should be disabled by default for these devices.
  3. Independently, consider making uct_test::flush() bail out on its deadline with a clear "UD flush timed out" failure instead of letting the watchdog abort the whole shard, so one stuck transport doesn't take down 8982 tests.

Related: #12265 (PR under test, not implicated); commit 6e3cfd3a "UCT/IB/MLX5: Add AArch64 ST64B BF copy mode" (candidate to rule out)

IB_MLX5_DEVX has no effect on other transports, where add64_no_devx
repeats add64.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Bit 0 is FLUSH_EP in test_ucp_rma, so helpers like is_ep_flush() would
read the no_devx variant as flush_ep.
The uct_p2p_rma_test based IB tests live there. Rename it to
uct_p2p_rma_test_atomic_key, like its neighbors.
@svc-nvidia-pr-review

Copy link
Copy Markdown

test/gtest/uct/ib/test_rc.cc

size_flush_remote's flush_rkey_enabled || (mr_id != 0) term is dead code: for m_entity_flush_rkey, flush_rkey_enabled == uct_ib_md_is_flush_rkey_valid(), and an invalid rkey now yields mr_id == 0. The rc_iface_mr_id() helper, which exists only for this term, is therefore also unnecessary. (Previously reported, still unaddressed.)

Real-hardware coverage gap: fence_put_no_devx and the no_devx UCP variant only exercise the fix when IB_MLX5_DEVX=n is honored. A CI job that forces DevX on (UCX_IB_MLX5_DEVX=y is respected via ?=) will silently run the already-working path.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/uct/ib/test_rc.cc Resolved
Comment thread test/gtest/uct/ib/test_rc.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Previously reported and still unaddressed (do not re-open): size_flush_remote's flush_rkey_enabled || (mr_id != 0) term is now dead, together with the rc_iface_mr_id() helper that exists only for it. In the GitHub flow this warrants a +1 reaction on the existing thread, not a new comment.

test/gtest/uct/ib/test_ib_xfer.cc

Residual note (not worth posting): the comment "the mlx5 memory domain without DevX has no indirect atomic keys" is a slightly misleading justification — without DevX the packed atomic key is the direct MR key, which is exactly why the offset is harmful. Also, fence_put_no_devx passes vacuously if an iface advertises none of PUT_SHORT/BCOPY/ZCOPY.

test/gtest/uct/test_amo_add_xor.cc

Residual note (not worth posting): the comment "the mlx5 memory domain without DevX has no indirect atomic keys" is a slightly misleading justification — without DevX the packed atomic key is the direct MR key, which is exactly why the offset is harmful.

check_skip_test() runs before init(), when there are no entities, so the
atomics check always passed. Check it again once the entities exist, before
the flush rkey changes. Also align the constants.
Without DevX, the atomic key is the memory key itself, so the atomic MR
offset is 0 on both sides and the test cannot fail.
tvegas1
tvegas1 previously approved these changes Nov 6, 2026
@dmitrygx

dmitrygx commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

on a non-DevX md the packed atomic key is the direct rkey, so this never skips, yet atomic_mr_offset is 0 on both sides and the test cannot fail

Done in d3c698f: the test now skips when the packed atomic key is invalid or equal to the direct key, i.e. when there is no indirect atomic key. Checking the packed key rather than UCT_IB_MLX5_MD_FLAG_INDIRECT_ATOMICS also covers IB_INDIRECT_ATOMIC=n, where the flag is set but no indirect key is created, and needs no mlx5 headers.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

test/gtest/uct/ib/test_rc.cc

Previously reported and still unaddressed: the flush_rkey_enabled || (mr_id != 0) term in size_flush_remote together with the rc_iface_mr_id() helper (the term is effectively dead on the test side). Do not re-open as a new thread; a +1 reaction on the existing thread is the right action.

test/gtest/uct/ib/test_rc.cc

Previously reported and still unaddressed: the skip condition in test_rc_verbs_invalid_flush_rkey.atomic_add — atomic_rkey == uct_ib_md_direct_rkey(...) does not skip on a non-DevX md that has a companion strict-order MR, where the test cannot fail anyway. Do not re-open as a new thread; a +1 reaction on the existing thread is the right action.

Residual coverage gaps (all previously noted): the DevX path where uct_ib_mlx5_devx_init_flush_mr() fails is not exercised by any new test, and fence_put_no_devx / the no_devx UCP variant only cover the fix on jobs where IB_MLX5_DEVX=n is actually honored.

@svc-ucx

svc-ucx commented Nov 6, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests new on worker 1) · commit d3c698f9

TL;DR: The new on worker 1 gtest job aborted on a UCS registration-cache invariant assertion — rcache.c:405 Assertion '!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed — while ucp_mem_unmap() released the last reference of a parent memh region that was still in the page table; this is an rcache refcount bug in the recently merged region-merge rework (e536ef0), not a result of PR #12265.

Full analysis

Summary: tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh> crashed (core dumped, make: *** [Makefile:4992: test] Aborted) on an assertion inside ucs_mem_region_destroy_internal().

Root cause: From the backtrace in the Azure log:

rcache.c:405  Assertion `!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed
 0 ucs_mem_region_destroy_internal()  src/ucs/memory/rcache.c:405
 3 ucs_rcache_region_put_unsafe()     src/ucs/memory/rcache.inl:128
 4 ucp_memh_put_rcache()              src/ucp/core/ucp_mm.c:444
 5 ucp_memh_cleanup()                 src/ucp/core/ucp_mm.c:473
 6 ucp_mem_unmap()                    src/ucp/core/ucp_mm.c:1298
 7 test_ucp_memheap::test_xfer()      test/gtest/ucp/test_ucp_memheap.cc:120

ucp_mem_unmap() → ucp_memh_cleanup() releases the user memh's parent rcache region (ucp_mm.c:473). By design a region in the page table holds its own +1 reference (rcache.c:1108: region->refcount = 2; /* Page-table + user */), so a user put must never drive the count to 0 while UCS_RCACHE_REGION_FLAG_PGTABLE is still set. It did here, i.e. the region's reference count was one too low (a reference was lost/double-released, or the memh's parent pointer was already stale) — a broken rcache refcount/page-table invariant, which then destroys a region that is still linked in the page table (latent use-after-free).

Contributing/suspect code paths:

  • ucs_rcache_check_adj_regions() / ucs_rcache_check_overlap_one() (rcache.c:843, rcache.c:757) collect regions straight out of the page table (tmp_list) and call ops->merge() + ucs_rcache_region_invalidate_internal() on them without taking a reference first.
  • Reference counting is inconsistent: ucs_rcache_lookup_unsafe() does a plain region->refcount++ and ucs_rcache_region_put_unsafe() a plain --region->refcount (rcache.inl:103, rcache.inl:127), while rcache-internal paths use ucs_atomic_add32() / ucs_atomic_fsub32() (rcache.c:459, rcache.c:1135). UCP calls the non-atomic variants under only context->mt_lock, not pgt_lock, so an interleaved atomic decrement from an invalidation path can swallow the increment and leave the count one too low — exactly the observed symptom.
  • The failing test is TCP/cuda_copy/rocm_copy with host memory and no IB atomics, so the PR under test (non-DevX atomic MR offset) is not in the code path. The PR's own commits (7a7f56a, 8af21f6) only add a variant bit to the weak-fence test; variant /2 here is plain USER_MEMH.

Note on timing: the ~2 min gap between the assert line (21:07:06) and the backtrace (21:08:57) is just the UCS debug handler resolving symbols — not a hang.

Implicated commit: e536ef0 — "UCS/RCACHE: Adds capability for merging adjacent rcache regions (#11958)", william-gallagher-nv, 2026-11-04 (two days before this build; the only recent change to both src/ucs/memory/rcache.c region create/merge/destroy and src/ucp/core/ucp_mm.c merge/put paths). PR #12265 is not implicated.

File: src/ucs/memory/rcache.c:405 (assert), broken invariant produced in src/ucs/memory/rcache.c:757-891 (ucs_rcache_check_overlap_one / ucs_rcache_check_adj_regions) and released at src/ucp/core/ucp_mm.c:444

Suggested fix:

  1. Take an explicit reference (ucs_rcache_region_hold()) on every region collected from the page table in ucs_rcache_check_adj_regions()/ucs_rcache_find_regions() before calling ops->merge() + ucs_rcache_region_invalidate_internal(), and put it afterwards, so a merged-with region can never lose its page-table reference accounting.
  2. Make ucs_rcache_lookup_unsafe()/ucs_rcache_region_put_unsafe() use the same atomic primitives as ucs_rcache_region_hold()/ucs_rcache_region_put_internal() (ucs_atomic_add32/ucs_atomic_fsub32) to remove the lost-update window between UCP's mt_lock-only fast path and rcache's pgt_lock paths.
  3. Harden ucs_mem_region_destroy_internal(): if UCS_RCACHE_REGION_FLAG_PGTABLE is still set, remove the region from the page table (ucs_pgtable_remove) and log an error instead of freeing a linked region; also fix rcache.c:1110-1116, where a ucs_rcache_fill_pfn() failure does ucs_free(region) while the region is still inserted in the page table.
  4. Until fixed, hold off landing PR UCS/RCACHE: Default merging rcache regions to on #12267 ("Default merging rcache regions to on") and re-run this job on an unmodified master to confirm the regression is independent of UCT/IB/MLX5: Fix atomic MR offset without a flush memory key #12265.

Related: #11958 (merge-adjacent capability, merged as e536ef0), #12267 (would enable merging by default), #12265 (PR under test — unaffected code path)

@dmitrygx
dmitrygx merged commit e4d1e59 into openucx:master Nov 8, 2026
167 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants