Skip to content

UCT/IB: Disable atomic offset for direct nic - #12280

Open
Artemy-Mellanox wants to merge 1 commit into
openucx:masterfrom
Artemy-Mellanox:topic/fix_dnic_reg
Open

Artemy-Mellanox wants to merge 1 commit into
openucx:masterfrom
Artemy-Mellanox:topic/fix_dnic_reg

Conversation

@Artemy-Mellanox

@Artemy-Mellanox Artemy-Mellanox commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What?

Disable atomic offset when direct-NIC used.

Why?

It fails with PCI relaxed order since atomic rkey in this case hold strict order key w/o offset and remote side has no information it shouldn't apply offset to atomic operations.

Currently, there is no support for indirect atomics over direct-NIC.

@Artemy-Mellanox
Artemy-Mellanox requested a review from a team as a code owner November 9, 2026 08:25
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

return NULL;
}

md->flags &= ~UCT_IB_MLX5_MD_FLAG_ATOMIC_OFFSET;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker: this flips md state after iface/ep addresses were already packed with the old value (UCP packs the DC iface address before any memory is registered), so the peer's atomic_mr_offset and the atomic key we hand out can disagree and atomics land at the wrong address. direct_nic_sys_dev is already known in uct_ib_mlx5_devx_md_open_common() — can we decide the flag there instead of clearing it at registration time?

int atomic_offset;
void *ptr;

atomic_offset = !!(md->flags & UCT_IB_MLX5_MD_FLAG_ATOMIC_OFFSET);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uct_rc_verbs_ep_get_address() still derives this from md->config.enable_indirect_atomic; over the same devx md it will advertise an atomic offset while mkey_pack no longer creates offset keys. Should it use the same source of truth?

uct_ib_mlx5_md_t *md = uct_ib_mlx5_iface_md(
&iface->super.super);
uct_rc_mlx5_ep_ext_address_t *ext_addr;
int atomic_offset;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: pls init atomic_offset in the definition instead of a separate assignment (CodeStyle: "Variables defined with a value should be first").

@svc-nvidia-pr-review

Copy link
Copy Markdown

Test/verification gap: there is no test coverage for the direct-NIC path anywhere in test/ (no match for direct_nic/data_direct), and the only test change makes has_atomic_ksm() read the saved md flag, keeping the existing test insensitive to the new runtime clearing. How was this verified on a data-direct setup?

Comment thread src/uct/ib/base/ib_md.h
#endif
UCT_IB_MEM_FLAG_GVA = UCS_BIT(5), /**< The memory handle is a
GVA region */
UCT_IB_MEM_DIRECT_NIC = UCS_BIT(6), /**< The memory handle was

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what case fails on master? direct-NIC memhs already skip the atomic key (#11111), so the peer uses the direct key without offset, and host memory never takes the direct-NIC path.

if indirect atomics break for all memory with data-direct, can we decide it at md open from direct_nic_sys_dev? clearing it at the first registration leaves old memhs with offset keys while new peers get offset 0.

it's strange to change md flags after iface addresses were already packed: peers that connected before keep the offset, and new ones get none. can we decide it once at md open?

@svc-ucx

svc-ucx commented Nov 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit a12a6f4e

TL;DR: The gtest shard aborted because device mlx5_1 on swx-rain04 stopped completing UMR BIND_MW work requests — every exported-mkey registration burned a 30 s timeout, permanently wedged the MD's internal UMR QP, and a later ud_verbs test then hung 15 minutes and aborted the whole run. The UMR error path has no recovery (it posts to an already-full SQ and destroys an mkey the HW still owns), so a single device hiccup escalates into a full job failure.

Full analysis

Summary: make -C test/gtest test aborted (signal 6, core dumped) after ud_verbs/uct_p2p_am_test.am_short_iov_keep_data/2 <ud_verbs/mlx5_1:1> hung for 900 s, preceded by 5 udx/test_ucp_loopback.envelope failures caused by UMR mkey registration timeouts on mlx5_1.

Root cause: Chain of events in the log:

  1. uct_ib_mlx5_devx_umr_mkey_bind() posts an IBV_WR_BIND_MW to the MD's internal self-loopback UMR QP. ibv_post_send succeeds, but no CQE ever arrives — ib_mlx5dv_md.c:1439 mlx5_1: ibv_poll_cq(UMR CQ, registration, UMR mkey ... index 0x49c) timed out after the hard-coded 30 s (ib_mlx5dv_md.c:1438). Normal data-path traffic on the same device works fine (am_bw : 22493 MB/sec, tag_mr_b : 7.6 Mpps), so only the UMR/BIND_MW engine is dead — a device/firmware-level fault, not a UCX logic error.
  2. The error path makes it unrecoverable: on timeout uct_ib_mlx5_devx_umr_mkey_bind() (:1540) calls uct_ib_mlx5_devx_umr_mkey_invalidate(), which posts another WR to the same QP whose previous WQE is still outstanding → ibv_post_send(UMR QP, invalidation, ...) returned 12: Resource temporarily unavailable (SQ full, :1430). The mkey is then freed via uct_ib_mlx5_devx_umr_mkey_destroy() while the HW may still reference it, and the UMR QP is never reset. Each subsequent test repeats this: 30 s lost, another pool mkey leaked (index 0x49c, then 0x4c5), 5 tests failed with Error: Input/output error.
  3. Terminal hang: the largest timestamp gap in the log is exactly 900 s with zero output — 08:50:29 ([ RUN ] ud_verbs/uct_p2p_am_test.am_short_iov_keep_data/2) → 09:05:29 (Connection timed out - abort testing). The backtrace shows the process spinning in uct_ud_verbs_iface_poll_tx() → ibv_poll_cq() (ud_verbs.c:402) from uct_test::short_progress_loop(), i.e. mlx5_1 never completed the UD send. This is a hang, not slowness — the device was left in a bad state by step 2. test_helpers.cc:57 then aborts the entire 9076-test shard.

The PR's only functional change, md->flags &= ~UCT_IB_MLX5_MD_FLAG_ATOMIC_OFFSET; at ib_mlx5dv_md.c:816, is almost certainly not the trigger: uct_ib_mlx5_direct_nic_reg_mr() returns NULL early unless a valid dmabuf_fd is passed (:797), and this RoCE run uses host memory only (cuda_copy,rocm_copy is not available), so that line never executed. It is, however, a latent bug worth fixing separately — it is an unsynchronized read-modify-write of process-wide shared md->flags from a per-registration fast path (this job runs OMP_NUM_THREADS=4), it is never restored, and one direct-NIC dmabuf registration permanently changes atomic-key layout for all later plain host registrations on that MD.

Implicated commit: unknown — the failure is a device-level fault on swx-rain04/mlx5_1, not reproduced by the PR diff. Separately, a12a6f4e (Artemy Kovalyov, "UCT/IB: Disable atomic offset for direct nic") introduces the global-flag-mutation concern noted above.

File: src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:1437-1444 (poll timeout), :1538-1542 (unrecoverable bind error path), :1428-1434 (post to full SQ); secondary: src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:816

Suggested fix:

  1. Unblock CI: retry the job on a different RoCE worker; mlx5_1 on swx-rain04 needs inspection (check dmesg, firmware, and data-direct/UMR state) — the UMR BIND_MW engine is not completing while the normal data path is healthy.
  2. Make the UMR path fail safely in uct_ib_mlx5_devx_umr_post_sync() / uct_ib_mlx5_devx_umr_mkey_bind(): when UCS_ERR_TIMED_OUT is returned, do not post the invalidation WR to the same QP and do not call mlx5dv_destroy_mkey() on an mkey with an outstanding WQE. Instead mark md->umr as broken, tear down and recreate the UMR CQ/QP (or disable the UMR export path on this MD and return UCS_ERR_IO_ERROR), so registration fails cleanly instead of leaking mkeys and wedging the device for the rest of the run.
  3. Log wc.status / move the 30 s constant at :1438 into a configurable MD knob so a dead device fails fast rather than costing 30 s per registration.
  4. In this PR: move the md->flags &= ~UCT_IB_MLX5_MD_FLAG_ATOMIC_OFFSET decision out of uct_ib_mlx5_direct_nic_reg_mr() into MD open/capability detection (set it once from md->direct_nic_sys_dev), or record it per-memh instead of mutating shared md->flags at registration time.

Related: PR #12280 (this change), parent #12265 "UCT/IB/MLX5: Fix atomic MR offset without a flush memory key"; no existing issue found for the ibv_poll_cq(UMR CQ, registration) timed out signature — worth filing one for the missing UMR error recovery.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants