Repository navigation
UCT/IB: Disable atomic offset for direct nic - #12280
Artemy-Mellanox wants to merge 1 commit into
Conversation
|
🤖 Starting review — findings will be posted here when done. |
| return NULL; | ||
| } | ||
|
|
||
| md->flags &= ~UCT_IB_MLX5_MD_FLAG_ATOMIC_OFFSET; |
There was a problem hiding this comment.
blocker: this flips md state after iface/ep addresses were already packed with the old value (UCP packs the DC iface address before any memory is registered), so the peer's atomic_mr_offset and the atomic key we hand out can disagree and atomics land at the wrong address. direct_nic_sys_dev is already known in uct_ib_mlx5_devx_md_open_common() — can we decide the flag there instead of clearing it at registration time?
| int atomic_offset; | ||
| void *ptr; | ||
|
|
||
| atomic_offset = !!(md->flags & UCT_IB_MLX5_MD_FLAG_ATOMIC_OFFSET); |
There was a problem hiding this comment.
uct_rc_verbs_ep_get_address() still derives this from md->config.enable_indirect_atomic; over the same devx md it will advertise an atomic offset while mkey_pack no longer creates offset keys. Should it use the same source of truth?
| uct_ib_mlx5_md_t *md = uct_ib_mlx5_iface_md( | ||
| &iface->super.super); | ||
| uct_rc_mlx5_ep_ext_address_t *ext_addr; | ||
| int atomic_offset; |
There was a problem hiding this comment.
minor: pls init atomic_offset in the definition instead of a separate assignment (CodeStyle: "Variables defined with a value should be first").
|
Test/verification gap: there is no test coverage for the direct-NIC path anywhere in |
| #endif | ||
| UCT_IB_MEM_FLAG_GVA = UCS_BIT(5), /**< The memory handle is a | ||
| GVA region */ | ||
| UCT_IB_MEM_DIRECT_NIC = UCS_BIT(6), /**< The memory handle was |
There was a problem hiding this comment.
what case fails on master? direct-NIC memhs already skip the atomic key (#11111), so the peer uses the direct key without offset, and host memory never takes the direct-NIC path.
if indirect atomics break for all memory with data-direct, can we decide it at md open from direct_nic_sys_dev? clearing it at the first registration leaves old memhs with offset keys while new peers get offset 0.
it's strange to change md flags after iface addresses were already packed: peers that connected before keep the offset, and new ones get none. can we decide it once at md open?
|
🤖 CI Triage Agent — TL;DR: The gtest shard aborted because device Full analysisSummary: Root cause: Chain of events in the log:
The PR's only functional change, Implicated commit: unknown — the failure is a device-level fault on File: Suggested fix:
Related: PR #12280 (this change), parent #12265 "UCT/IB/MLX5: Fix atomic MR offset without a flush memory key"; no existing issue found for the |
What?
Disable atomic offset when direct-NIC used.
Why?
It fails with PCI relaxed order since atomic rkey in this case hold strict order key w/o offset and remote side has no information it shouldn't apply offset to atomic operations.
Currently, there is no support for indirect atomics over direct-NIC.