Repository navigation
UCT/IB/MLX5: Fix atomic MR offset without a flush memory key - #12265
Conversation
Peers add 8 * atomic MR id to the remote address of atomics, and of the first put after a fence, which use the atomic half of the rkey. The id is bits 8-15 of the flush rkey. Only the indirect atomic keys of the DevX MD, which are created at that offset, need it. The mlx5 MD without DevX has no flush memory key, but it set the flush rkey to an invalid value with the process id in bits 8-15. The atomic MR id comes from these bits since 005d879, and this MD packs its memory key as the atomic key since e4b8123. So rc_mlx5 and dc_mlx5 peers sent atomics, and puts after a fence when the transport fence is enabled (e.g. on devices with PCIe atomics), to 8 * (pid & 0xff) bytes past the requested address: they updated other memory, or failed with a remote access error past the end of the memory region. The DevX MD set the same value when it failed to register its flush memory key, and created its atomic keys at that offset, but rc_verbs passes the atomic MR id only with a valid flush rkey, so rc_verbs peers used no offset. Set the flush rkey to UCT_IB_MD_INVALID_FLUSH_RKEY in both cases, so that the atomic MR id is 0 without a flush memory key.
Add test_rc_atomic_key for rc_verbs, rc_mlx5 and dc_mlx5: a fetch-add, and put_short, put_bcopy and put_zcopy after an interface fence, to the first word of a buffer which covers the largest atomic MR offset. Only that word must change. Run them with the default memory domain and with IB_MLX5_DEVX=n; RC_FENCE=weak makes the fence take effect on any device. Add test_ucp_rma_weak_fence for rc_x and dc_x, with and without IB_MLX5_DEVX=n: a put, a weak fence and a put of the same size, so on the same lane. The target buffer must hold the data of the second put, and the guard area after it must not change.
|
🤖 Starting review — findings will be posted here when done. |
|
minor: in Residual coverage gap: the DevX path where uct_ib_mlx5_devx_init_flush_mr() fails (atomic keys exist, flush key does not - the rc_verbs reason quoted in the new ib_md.h comment) is not exercised by any of the new tests and is hard to simulate without injecting a CREATE_MKEY failure; worth stating in the PR description if a test is not practical. |
An invalid flush rkey now always gives atomic MR id 0, so the rc_verbs endpoint address length depends only on whether the flush rkey is valid.
|
🤖 CI Triage Agent — TL;DR: The BlueField ASAN gtest shard didn't run out of time — it hung in its very first test ( Full analysisSummary: Root cause: Evidence of a hang, not slowness: Implicated commit: File: Suggested fix: Stop overloading Related: PR #12265 (this change); prior revert of the same coupling: |
|
🤖 Starting review — findings will be posted here when done. |
Done in 898921b: removed the term and
Right, the new tests do not cover it; I added this to the description. I checked that path by forcing the flush memory key registration to fail: rc_verbs atomics to such a peer got a remote access error on master and passed with this fix. |
Check the flush rkey in uct_ib_md_get_atomic_mr_id() instead of relying on every memory domain to use UCT_IB_MD_INVALID_FLUSH_RKEY as its invalid value.
This reverts commit 898921b: a pending change to the same test updates this expectation.
Keep only the put after a fence without DevX in test_rc_atomic_key, and check atomics with a variant of uct_amo_add_xor_test without DevX. The variants with DevX pass without the fix.
|
🤖 Starting review — findings will be posted here when done. |
A DevX memory domain whose flush memory key failed to register has an invalid flush rkey, and rc_verbs then does not pass the atomic MR id to the peer. Give the receiver such a flush rkey, with non-zero bits 8-15, and check that an atomic add still updates the requested address.
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The ASAN BlueField gtest shard died when Full analysisSummary: Root cause: From the stage log timeline:
So the largest gap is most of the runtime → this is a hang, not slowness. The ud_mlx5 transport on This is not in the code path changed by PR #12265 ("Fix atomic MR offset without a flush memory key"): the atomic MR offset / Implicated commit: unknown — the PR under test (3877670, File: Suggested fix:
Related: #12265 (PR under test, not implicated); commit |
IB_MLX5_DEVX has no effect on other transports, where add64_no_devx repeats add64.
|
🤖 Starting review — findings will be posted here when done. |
Bit 0 is FLUSH_EP in test_ucp_rma, so helpers like is_ep_flush() would read the no_devx variant as flush_ep.
The uct_p2p_rma_test based IB tests live there. Rename it to uct_p2p_rma_test_atomic_key, like its neighbors.
|
Real-hardware coverage gap: |
|
🤖 Starting review — findings will be posted here when done. |
|
Previously reported and still unaddressed (do not re-open):
Residual note (not worth posting): the comment "the mlx5 memory domain without DevX has no indirect atomic keys" is a slightly misleading justification — without DevX the packed atomic key is the direct MR key, which is exactly why the offset is harmful. Also,
Residual note (not worth posting): the comment "the mlx5 memory domain without DevX has no indirect atomic keys" is a slightly misleading justification — without DevX the packed atomic key is the direct MR key, which is exactly why the offset is harmful. |
check_skip_test() runs before init(), when there are no entities, so the atomics check always passed. Check it again once the entities exist, before the flush rkey changes. Also align the constants.
Without DevX, the atomic key is the memory key itself, so the atomic MR offset is 0 on both sides and the test cannot fail.
Done in d3c698f: the test now skips when the packed atomic key is invalid or equal to the direct key, i.e. when there is no indirect atomic key. Checking the packed key rather than |
|
🤖 Starting review — findings will be posted here when done. |
|
Previously reported and still unaddressed: the
Previously reported and still unaddressed: the skip condition in Residual coverage gaps (all previously noted): the DevX path where |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Root cause: From the backtrace in the Azure log:
Contributing/suspect code paths:
Note on timing: the ~2 min gap between the assert line (21:07:06) and the backtrace (21:08:57) is just the UCS debug handler resolving symbols — not a hang. Implicated commit: e536ef0 — "UCS/RCACHE: Adds capability for merging adjacent rcache regions (#11958)", william-gallagher-nv, 2026-11-04 (two days before this build; the only recent change to both File: Suggested fix:
Related: #11958 (merge-adjacent capability, merged as e536ef0), #12267 (would enable merging by default), #12265 (PR under test — unaffected code path) |
What?
uct_ib_md_get_atomic_mr_id()returns 0 when the flush rkey is invalid, sothe mlx5 memory domain without DevX, and a DevX memory domain whose flush
memory key failed to register, pass atomic MR id 0 to peers instead of the
process id. They also set the flush rkey to
UCT_IB_MD_INVALID_FLUSH_RKEYnow, and
uct_ib_mlx5_flush_rkey_make()is removed.Why?
Atomics, and the first put after a fence, use the atomic half of the rkey:
uct_ib_resolve_atomic_rkey()adds 8 * the atomic MR id from the peer addressto the remote address. The id is bits 8-15 of the flush rkey (since
005d879). Only the indirect atomic keys of the DevX MD are created at that
offset.
e4b8123), but passed
getpid() & 0xffas the atomic MR id. So rc_mlx5 anddc_mlx5 peers, with or without DevX, sent every atomic, and every put after
a fence when their transport fence is enabled (e.g. on devices with PCIe
atomics, or with relaxed ordering, where an interface flush also fences the
next put), to
8 * (pid & 0xff)bytes past the requested address. Theyupdated other memory, or failed with a remote access error
past the end of the memory region. For example, with
UCX_IB_MLX5_DEVX=nafetch-add to a buffer at
0x...820000went to0x...820310(pid & 0xff = 98). This MD is used when DevX is disabled or unavailable.
value and created its atomic keys at that offset. rc_mlx5 and dc_mlx5 pass
the id, but rc_verbs passes it only with a valid flush rkey, so rc_verbs
atomics to such a peer went
8 * (pid & 0xff)bytes before the target.New UCT tests on master without the fix: atomics miss their target, the put after a fence lands at word 78 (pid & 0xff of the test process) instead of word 0, and the rc_verbs atomic to a peer with an invalid flush rkey lands at word 166 instead of word 256
test_ucp_rma_weak_fence on master without the fix
How?
uct_ib_md_get_atomic_mr_id()checks the flush rkey, so the atomic MR id is0 without a flush memory key in any memory domain: the atomic keys of the
non-DevX MD are the memory keys themselves, and the DevX MD then creates its
atomic keys without an offset. Peers use the id as bits 8-15 of the flush
rkey only when the address carries a valid one, so remote flush does not
change.
still passes a non-zero id, so atomics to it stay misplaced until it is
updated.
uct_p2p_rma_test_atomic_key.fence_put_no_devxfor rc_verbs,rc_mlx5 and dc_mlx5 with
IB_MLX5_DEVX=n: put short/bcopy/zcopy after aninterface fence, to the first word of a buffer which covers the largest
atomic MR offset; only that word may change.
RC_FENCE=weakmakes the fencetake effect on any device.
uct_amo_add_xor_test.add64_no_devxof the AMO test, withIB_MLX5_DEVX=n, on IB transports.test_ucp_rma_weak_fencefor rc_x and dc_x, with and withoutIB_MLX5_DEVX=n: a put, a weak fence and a put of the same size, so on thesame lane; the target must hold the second put and the guard area after it
must not change. A strong fence does not cover this case: the MD without
DevX does not use relaxed ordering, so the puts after a strong fence use the
memory key.
test_rc_verbs_invalid_flush_rkeyfor rc_verbs: beforeconnecting, the receiver gets an invalid flush rkey with non-zero bits 8-15,
as a DevX memory domain whose flush memory key failed to register had, so
rc_verbs passes no atomic MR id; an atomic add must still update the
requested word.