Skip to content

UCT/IB/MLX5: Handle relaxed-only fence ordering - #11721

Merged
roiedanino merged 16 commits into
openucx:masterfrom
roiedanino:cx10-ordering-pr3
Sep 2, 2026
Merged

roiedanino merged 16 commits into
openucx:masterfrom
roiedanino:cx10-ordering-pr3

Conversation

@roiedanino

@roiedanino roiedanino commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

What?

Handle UCT fence ordering for IB devices whose data memory keys are limited to relaxed ordering.

  • RC mlx5 uses strong-order WQE fence flags when the local memory domain requires relaxed-only data keys.
  • DC mlx5 advertises its existing remote-flush key whenever relaxed-only data keys require it, and posts a remote-read barrier on the first operation after a UCT fence.
  • DC rejects an address without the required remote-flush key during reachability, allowing wireup to select another transport. Endpoint creation keeps a defensive validation.
  • DC tracks remote-read barriers per endpoint so endpoints sharing one DCI can pipeline their barriers.
  • RC verbs is unavailable when the memory domain requires relaxed-only data keys that this path cannot support.

Why?

Strict data memory keys previously supplied ordering that some RC and DC fence sequences relied on. Devices limited to relaxed-order data keys remove that implicit ordering, so the IB transports must provide the required ordering explicitly.

How?

RC maps the local relaxed-only requirement to the existing strong-order WQE fence flags. DC keeps the existing flush-key address capability, resolves RC_FENCE=auto to weak fence mode for relaxed-only memory domains, and performs a per-endpoint remote-read barrier before later operations proceed. No UCT or UCP API and no new wire capability are added. Incompatible DC addresses are rejected during reachability instead of failing after transport selection.

@roiedanino
roiedanino marked this pull request as ready for review August 3, 2026 13:47
@roiedanino

Copy link
Copy Markdown
Contributor Author

@svc-nvidia-pr-review

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/rc/base/rc_iface.h Resolved
Comment thread src/uct/ib/mlx5/gga/gga_mlx5.c Resolved
Comment thread test/gtest/uct/ib/test_ib_xfer.cc Resolved
@gleon99

gleon99 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@roiedanino after refining, please ping @nbellalou for review.

@roiedanino roiedanino changed the title UCT/IB: Preserve fence semantics for relaxed-only MKEYs feat(ib/mlx5): add relaxed-order fence-flush support for RC and DC transports Aug 6, 2026
Signed-off-by: Roie Danino <rdanino@nvidia.com>
Signed-off-by: Roie Danino <rdanino@nvidia.com>
Signed-off-by: Roie Danino <rdanino@nvidia.com>
@roiedanino roiedanino changed the title feat(ib/mlx5): add relaxed-order fence-flush support for RC and DC transports UCT/IB/MLX5: Handle relaxed-only fence ordering Aug 6, 2026
@roiedanino
roiedanino marked this pull request as draft August 6, 2026 13:47
@roiedanino roiedanino self-assigned this Aug 6, 2026
Comment thread src/uct/ib/mlx5/rc/rc_mlx5.h Outdated
Comment on lines +43 to +47
UCT_RC_MLX5_EP_ADDR_FLAG_NO_ATOMIC_OFFSET = UCS_BIT(1)
UCT_RC_MLX5_EP_ADDR_FLAG_NO_ATOMIC_OFFSET = UCS_BIT(1),
UCT_RC_MLX5_EP_ADDR_FLAG_RELAXED_ORDER = UCS_BIT(2),
UCT_RC_MLX5_EP_ADDR_FLAG_ORDERING_CAP = UCS_BIT(3)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Align

Comment thread src/uct/ib/mlx5/dc/dc_mlx5_ep.h Outdated
Comment on lines +86 to +87
uint8_t dci_channel_index;
uct_ib_fence_info_t fi;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

align

Signed-off-by: Roie Danino <rdanino@nvidia.com>
Signed-off-by: Roie Danino <rdanino@nvidia.com>
@roiedanino
roiedanino marked this pull request as ready for review August 10, 2026 14:23
* compatibility */
};


Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert

Comment thread src/uct/ib/mlx5/rc/rc_mlx5_iface.c Outdated
Comment on lines +634 to +635
return UCT_RC_MLX5_TM_ENABLED(iface) ? UCT_RC_MLX5_IFACE_ADDR_TYPE_TM :
UCT_RC_MLX5_IFACE_ADDR_TYPE_BASIC;
return UCT_RC_MLX5_TM_ENABLED(iface) ? UCT_RC_MLX5_IFACE_ADDR_TYPE_TM :
UCT_RC_MLX5_IFACE_ADDR_TYPE_BASIC;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert

Comment on lines -753 to 758
iface->config.put_fence_flag =
iface->config.send_fence_flag =
strong_order ? UCT_IB_MLX5_WQE_CTRL_FLAG_STRONG_ORDER : 0;
iface->config.put_fence_flag =
strong_order ? UCT_IB_MLX5_WQE_CTRL_FLAG_STRONG_ORDER : 0;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need send_fence_flag if it follows the same logic as put_fence_flag

Signed-off-by: Roie Danino <rdanino@nvidia.com>
Signed-off-by: Roie Danino <rdanino@nvidia.com>
Signed-off-by: Roie Danino <rdanino@nvidia.com>
Signed-off-by: Roie Danino <rdanino@nvidia.com>
Signed-off-by: Roie Danino <rdanino@nvidia.com>
Signed-off-by: Roie Danino <rdanino@nvidia.com>
Comment thread src/uct/ib/mlx5/dc/dc_mlx5.h Outdated
typedef enum {
/* Indicates that this specific dci is shared, regardless of policy */
UCT_DC_DCI_FLAG_SHARED = UCS_BIT(0),
UCT_DC_DCI_FLAG_SHARED = UCS_BIT(0)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert

Comment thread src/uct/ib/mlx5/dc/dc_mlx5_ep.c Outdated


#define REVERSE_SL_MASK UCS_MASK(4)
#define REVERSE_SL_MASK UCS_MASK(4)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert

Signed-off-by: Roie Danino <rdanino@nvidia.com>
Comment thread src/uct/ib/mlx5/dc/dc_mlx5_ep.h Outdated
Comment on lines +61 to +63
UCT_DC_MLX5_EP_FLAG_INVALIDATED = UCS_BIT(12)
UCT_DC_MLX5_EP_FLAG_INVALIDATED = UCS_BIT(12),
#else
UCT_DC_MLX5_EP_FLAG_INVALIDATED = 0
UCT_DC_MLX5_EP_FLAG_INVALIDATED = 0,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert

Signed-off-by: Roie Danino <rdanino@nvidia.com>
Signed-off-by: Roie Danino <rdanino@nvidia.com>
Signed-off-by: Roie Danino <rdanino@nvidia.com>
@roiedanino

Copy link
Copy Markdown
Contributor Author

@Artemy-Mellanox @evgeny-leksikov
Can you please review? thanks!

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.


status = uct_dc_mlx5_ep_flush_remote_post(
ep, uct_dc_mlx5_ep_fence_flush_handler, NULL, fence_state);
if (status != UCS_INPROGRESS) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The status == UCS_ERR_NO_RESOURCE branch is only reachable via the UCT_RC_IFACE_GET_TX_DESC failure inside uct_dc_mlx5_ep_flush_remote_post; that path is easy to miss because flush_remote_post otherwise always returns UCS_INPROGRESS. Consider a one-line comment noting the NO_RESOURCE comes from TX-desc exhaustion, to make the control flow obvious.

#endif
}

UCS_TEST_SKIP_COND_P(test_rc, relaxed_order_required_rejects_verbs,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test only exercises the iface_open UNSUPPORTED path; it does not cover the new uct_rc_verbs_query_tl_devices filtering (device excluded from resource list). The dc side has relaxed_order_required_hides_invalid_flush_rkey for the query path; a symmetric check for rc_verbs would close the gap.

Comment thread src/uct/ib/mlx5/dc/dc_mlx5_ep.h Outdated
UCT_DC_MLX5_CHECK_DCI_RES(_iface, _ep) \
UCT_RC_CHECK_NUM_RDMA_READ_RET(&(_iface)->super.super, \
UCS_ERR_NO_RESOURCE) \
if (ucs_unlikely((_ep)->flags & UCT_DC_MLX5_EP_FLAG_FENCE_FLUSH)) { \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe common macro for check_fence to use here and in UCT_DC_CHECK_RES_PTR

Comment thread src/uct/ib/mlx5/dc/dc_mlx5.c Outdated
ucs_debug("%s: dc_mlx5 does not support relaxed-only memory keys "
"without a valid flush rkey",
uct_ib_device_name(&ib_md->dev));
*tl_devices_p = NULL;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't return UCS_ERR_NO_DEVICE not enough here?

Comment thread src/uct/ib/mlx5/dc/dc_mlx5_ep.c Resolved
Comment thread src/uct/ib/mlx5/dc/dc_mlx5_ep.c Outdated
uct_dc_mlx5_ep_fence_state(ep);
uct_rc_iface_send_op_t *op = fence_state->fence_op;

ucs_assertv(op != NULL, "fence flush operation for endpoint %p was not found",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why asserr+silent skip? maybe assert_always?

Comment thread src/uct/ib/mlx5/dc/dc_mlx5_ep.c Resolved
Signed-off-by: Roie Danino <rdanino@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Review complete — no issues found.

@roiedanino
roiedanino enabled auto-merge (squash) September 2, 2026 13:00
@svc-ucx

svc-ucx commented Sep 2, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests gpu on worker 1) · commit a0afdf03

TL;DR: The gpu on worker 1 gtest run failed on exactly one test — rc_gda/test_ucp_device_xfer.counter/9 (rc,rc_gda,cuda_copy,rocm_copy/cuda/thread/lazy_without_req) — the remote device counter never reached the expected value, which points at the fence/ordering change this PR introduces for relaxed-only ordering (uct_rc_mlx5_ep_fence_put/_fence_get in rc_mlx5.inl); the fence flag / atomic-rkey selection for the relaxed-only case needs to be re-checked for the RC/GDA atomic (counter-inc) path.

Full analysis

Summary: make test in build-test/test/gtest exited with error 1 (Makefile:4714) because 1 of 10254 tests failed: [ FAILED ] rc_gda/test_ucp_device_xfer.counter/9, where GetParam() = rc,rc_gda,cuda_copy,rocm_copy/cuda/thread/lazy_without_req. All other tests passed (10233 passed, 20 skipped) and there is no hang — the log shows continuous per-test progress right up to the tear-down at 14:21:16.

Root cause: The only failing case is the GDA device-side remote counter test in the "lazy" send mode (with_no_delay = false, with_request = false, thread level, CUDA rx memory). That test (test/gtest/ucp/test_ucp_device.cc:775) launches TEST_UCP_DEVICE_KERNEL_COUNTER_INC from the GPU and then wait_for_counter() polls until the remote counter equals num_iters * num_threads (10 × 8 = 80), asserting equality in EXPECT_EQ at test_ucp_device.cc:724. A failure there means increments were lost / not made visible in the polled window — i.e. an ordering/visibility problem on the remote atomic path, not a test-harness or infrastructure problem (no SIGTERM, no timeout, no other transport affected). This is precisely the area the PR changes: the branch carries [REDACTED:Hex High Entropy String] "UCT/IB/MLX5: Handle relaxed-only fence ordering" plus a follow-up 98081a75 "UCT/IB/MLX5: Fix relaxed-only fence handling" (both Roie Danino) touching src/uct/ib/mlx5/rc/rc_mlx5.inl, where the fence decision also selects between the atomic-capable rkey (uct_ib_resolve_atomic_rkey) and the plain/relaxed rkey (uct_ib_md_direct_rkey). In the relaxed-only configuration, choosing the relaxed/direct key or dropping the fence flag makes remote atomic increments unordered or ineffective, which matches the observed "counter never reaches target" symptom for the lazy (no explicit request/flush) variant only.

Caveat on evidence: the retrievable Azure log is only the tail of the task output, so the gtest assertion text/line for the failing case itself is not in the fetched text — the attribution above is based on the failing test identity plus the PR's changed code, not on a captured assertion message.

Implicated commit: [REDACTED:Hex High Entropy String] "UCT/IB/MLX5: Handle relaxed-only fence ordering" (Roie Danino), with follow-up 98081a75 "UCT/IB/MLX5: Fix relaxed-only fence handling" on branch cx10-ordering-pr3 (head a0afdf03)

File: src/uct/ib/mlx5/rc/rc_mlx5.inl:41-62 (uct_rc_mlx5_ep_fence_put / uct_rc_mlx5_ep_fence_get, iface->config.put_fence_flag / atomic_fence_flag); failing test at test/gtest/ucp/test_ucp_device.cc:775 (assert helper at test/gtest/ucp/test_ucp_device.cc:712-727)

Suggested fix:

  1. In the relaxed-only ordering path, make sure an atomic/counter operation still gets an atomic-capable rkey and a real fence: when only relaxed-ordering mkeys exist, do not fall back to uct_ib_md_direct_rkey() for the fenced case, and keep atomic_fence_flag non-zero (strong/small fence) instead of clearing it — i.e. relaxed-only must force the fence rather than skip it.
  2. Add a targeted check that iface->config.put_fence_flag/atomic_fence_flag are derived consistently with the new relaxed-only capability bits (see dp_ordering_devx / dp_ordering_force handling at src/uct/ib/mlx5/rc/rc_mlx5_common.c:1305-1314) so the RC/GDA QP used by rc_gda gets the same ordering guarantees as the host RMA path.
  3. Reproduce locally with just that variant to confirm it is deterministic and PR-induced rather than flaky:
    ./gtest --gtest_filter='rc_gda/test_ucp_device_xfer.counter/9' --gtest_repeat=20 on the same GDA-capable node, once on a0afdf03 and once on the merge base.
  4. If it reproduces only with lazy_without_req, instrument the counter-inc WQE path to verify the fence bit and rkey actually posted from the device for that mode.

Related: PR #11721 (this PR); prerequisite work #11649 ("UCT/IB: Support relaxed-only memory keys", a9568621) and #11399 ("UCT/IB/MLX5: Add initiator small fence WQE control flag")

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 22f9733c-11b3-4216-af0a-5f7192c229ee in the triage console for the audit trail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants