Repository navigation
UCP/FT: TX/RX tokens based protocols - #11810
evgeny-leksikov wants to merge 78 commits into
Conversation
|
🤖 CI Triage Agent — TL;DR: The BlueField gtest job failed with 22 real test failures — all in this PR's own failover/PSN area ( Full analysisSummary: Root cause: Genuine functional regressions in the PR's fault-tolerance/PSN work, not flakiness or a timeout (the log shows continuous test progress up to the summary, no stalls). The failure set maps exactly onto the code this branch changes:
Note: the other 53 "failed" jobs (all Implicated commit: File: Suggested fix:
Related: PR #11810 (branch
|
|
🤖 CI Triage Agent — TL;DR: The build failed to compile, not a test failure: Full analysisSummary: "ucx_perftest over MAD RTE / Setup Server" job failed during Root cause: This job configures UCX without debug/assertions (log shows Implicated commit: 3478631 — "UCP/RKEY: Retain keys for failover replay", Zihao Zhao (the commit that introduced File: src/ucp/core/ucp_rkey.c:1163 (function Suggested fix: Annotate the assertion-only variable so release builds accept it: void ucp_rkey_retain(ucp_rkey_h rkey)
{
uint32_t UCS_V_UNUSED old_refcount = ucs_atomic_fadd32(&rkey->refcount, 1);
ucs_assertv(old_refcount > 0, "rkey %p refcount %u", rkey, old_refcount);
}
Related: PR #11810 (openucx/ucx) — the failing PR itself; no pre-existing issue found for this warning. |
|
🤖 CI Triage Agent — TL;DR: The ICC/ICX build of Full analysisSummary: Root cause: The build uses Line 1066 in Implicated commit: The failing branch is File: Suggested fix: Drop the redundant parentheses: - if ((ctx == NULL)) {
+ if (ctx == NULL) {Then re-push. Optionally grep the rest of the PR's new code for the same pattern, since GCC-only local builds won't catch it: Related: PR #11810 (
|
|
🤖 CI Triage Agent — TL;DR: The new gtest Full analysisSummary: gtest run on the "althca on worker 2" job failed with 1 of 3975 tests failing: Root cause: The test calls
The ucx gtest base class counts every unhidden Implicated commit: 1c1c1b6 "UCP/FT: resolve post-rebase extract→purge conflicts; drop leftover rc_mlx5 TLS" — Evgeny Leksikov (the only commit touching File: test/gtest/uct/v2/test_uct_outstanding_purge.cc:54-82 (error sources: src/uct/ib/mlx5/ib_mlx5_ext.c:257 and :264) Suggested fix: Suppress the expected error logs in the negative test, mirroring the existing pattern: UCS_TEST_F(test_uct_ib_mlx5_ext_outstanding_purge, invalid_params) {
uct_ep_outstanding_purge_params_t params = {};
uint64_t rx_token = 0;
scoped_log_handler wrap_err(wrap_errors_logger); /* <-- add this */
EXPECT_EQ(UCS_ERR_INVALID_PARAM,
uct_ib_mlx5_ext_ep_outstanding_purge(NULL, NULL));
...
}(Alternatively demote the parameter-validation messages in Related: PR #11810 (UCP/FT: TX/RX tokens based protocols); prior work #11668, #11580
|
|
🤖 Starting review — findings will be posted here when done. |
|
PR scope: this PR is far over the 500-added-line limit set in REVIEW.md, and feature PRs must not bundle unrelated refactoring. It adds two ~1150/~415-line new source files plus large changes across wireup.c, ucp_ep.c, the UCT v2 API, and mlx5 transport, and mixes in unrelated refactors (e.g. ucp_wireup_msg_prepare/ucp_wireup_msg_send signature rework, put_offload RESUME/remote_flush gating, test_ib_mlx5_ext init restructure, AGENTS.md/UCX_HANDLE_ERRORS doc change). Please split into reviewable pieces: at minimum the UCT API + mlx5 token/defer-completion support, the wireup token wire-format, and the UCP failover engine as separate PRs, with pure refactors extracted on their own.
Hot path: uct_rc_mlx5_common_post_send now calls uct_rc_mlx5_txwq_record_token (an uct_ib_mtu_value lookup + ucs_div_round_up) for every non-NOP RC WQE, regardless of whether failover was ever enabled on the endpoint. This adds a division to the innermost TX path for all RC users. Can we gate the token bookkeeping on ep->super.failover_flags & UCT_RC_EP_FAILOVER_FLAG_ENABLED (or precompute packets-per-MTU) so non-failover traffic is unaffected?
uct_ep_failover_enable is a permanent, non-revertible arm with no disable. uct_rc_mlx5_ep_failover_enable only sets UCT_RC_EP_FAILOVER_FLAG_ENABLED and there is no matching disable entry point. Combined with the always-on token recording, once any lane enables it the ep pays the cost for its lifetime. Is a disable path needed, and should the new public UCT function document that it cannot be undone?
uct_rc_mlx5_ep_failover_enable is a permanent, non-revertible arm with no disable. Combined with the always-on token recording, once any lane enables it the ep pays the cost for its lifetime. Is a disable path needed?
Minor: confusing nested version gate in ucp_ep_recovery_arm. The outer
Minor: UCT_EP_INVALIDATE_PARAM_FIELD_FLAGS documented but
Test coverage: ucp_ep_recovery_send_request reuses worker->am_message_id as the recovery generation id with a
Test coverage: no test drives a stale/mismatched request_id for the ucp_ep_failover_apply_rx_tokens correlation logic (mismatched request_id ignored). If practical, add a focused check, otherwise note the coverage gap. |
|
🤖 CI Triage Agent — TL;DR: Full analysisSummary: gtest run on "althca on worker 2" ended with Root cause: Implicated commit: 1c1c1b6 — Evgeny Leksikov, "UCP/FT: resolve post-rebase extract→purge conflicts; drop leftover rc_mlx5 TLS" (the only commit touching this test file, on the PR branch) File: test/gtest/uct/v2/test_uct_outstanding_purge.cc:54-82 (error source: src/uct/ib/mlx5/ib_mlx5_ext.c:253-269) Suggested fix: Suppress the expected error logs in the negative test, e.g.: UCS_TEST_F(test_uct_ib_mlx5_ext_outstanding_purge, invalid_params) {
uct_ep_outstanding_purge_params_t params = {};
uint64_t rx_token = 0;
scoped_log_handler wrap_err(wrap_errors_logger); /* <-- add this */
...
}(scope it to cover all five Related: PR #11810 (UCP/FT: TX/RX tokens based protocols) — the branch introducing this test; prior purge work: #11668, #11580. |
…s exchange Append an optional trailer to the LANES_ADDR messages, holding the generation of the recovery round and a token section per direction: the TX tokens of the lanes the message provides, and the RX tokens of the lanes it answers. Both sections carry one length per lane and are empty until the lanes provide tokens, which lands in a follow-up patch together with their consumer, as does matching the generation against the tokens of an answer. Add LANES_ADDR_ACK, the third leg the exchange needs so that the peer which replies also receives the tokens derived from its own. The trailer is sent only towards peers of UCP_WIREUP_ADDR_TOKEN_MIN_DST_VERSION and above, and the ACK, which exists only to carry it, is withheld from older peers.
The token sections are parsed ahead of the packed addresses, so a malformed section must be rejected rather than shift the addresses. Feed crafted sections to the parser, including truncated ones, which the empty sections of the wire itself do not exercise.
Master bumped UCP_API_MINOR to 24. A gate of 23 would send the trailer to peers that still parse addresses immediately after lanes_info. Keep the original meaning of the LANES_ADDR lane maps in comments, and drop the claim that the trailer carries a length of its own.
Adopt the dst_version 24 tokens_info layout and skip_token_section parser, and keep FT TX collect, RX derive, and apply_rx_tokens on that handshake so extract stays compatible with the upstreamable wire protocol.
Take master's UCT on overlapping mlx5/token files so this PR can stay UCP-focused. Keep failover extract, INPROGRESS freeze, and FT tests.
564d70d to
ab9c9a3
Compare
|
🤖 Starting review — findings will be posted here when done. |
| return UCS_OK; | ||
| } | ||
|
|
||
| if (ucp_ep_failover_is_uct_ep(ucp_ep, lane, uct_ep)) { |
There was a problem hiding this comment.
Dead code: the first ucp_ep_failover_is_uct_ep check already returns UCS_INPROGRESS for this uct_ep, so the second identical check inside the wireup_ep branch (around L484) looks unreachable. Since ucp_ep_set_lanes_failed() between them does not turn a lane into a failover uct_ep, the second check never returns UCS_INPROGRESS. Can we drop it, or is a state change between them intended?
| - most fixtures support memtype skipping via | ||
| `check_skip_test()` patterns; prefer `UCS_TEST_SKIP_R(...)` over silent | ||
| short-circuits so the runner reports skipped cases. | ||
| - New tests must be listed in the relevant `Makefile.am` source list. |
There was a problem hiding this comment.
Test-default behavior change bundled into a feature PR: changing the default UCX_HANDLE_ERRORS from freeze to bt affects the whole gtest suite, not just failover tests, and is unrelated to the failover feature. Per REVIEW.md ("Feature PRs must not refactor code unrelated to the feature"), this should be split out.
| uct/test_pending.cc \ | ||
| uct/test_progress.cc \ | ||
| uct/test_uct_ep.cc \ | ||
| uct/test_uct_iface.cc \ |
There was a problem hiding this comment.
Test-default behavior change bundled into a feature PR: changing the default UCX_HANDLE_ERRORS from freeze to bt affects the whole gtest suite, not just failover tests, and is unrelated to the failover feature. Per REVIEW.md ("Feature PRs must not refactor code unrelated to the feature"), this should be split out.
| @@ -164,7 +164,8 @@ ucp_proto_put_offload_bcopy_send_func(ucp_request_t *req, | |||
| return status; | |||
There was a problem hiding this comment.
Unrelated behavioral change: adding UCP_PROTO_COMMON_INIT_FLAG_RESUME and a new ucp_rkey_need_remote_flush gate to ucp_proto_put_offload_bcopy_probe is a real functional change to PUT offload protocol selection that will affect non-failover PUT users. It should be justified in the description or split out; as-is it is hard to review in the middle of a failover PR.
|
Residual coverage gap: the token wire-format parsing has a focused gtest ( Residual coverage gap: no test drives the asymmetric-failure Dominant recommendation remains splitting the PR; the individual failover-engine correctness paths are difficult to review with confidence at this size. |
|
🤖 CI Triage Agent — TL;DR: The "Basic compile" job failed with Full analysisSummary: Root cause: The branch's C code references symbols that do not exist in the merged tree:
This is a bad merge resolution: the head commit is a merge of upstream master, and upstream commit Implicated commit: File: Suggested fix:
Related: PR #11810 (UCP/FT: TX/RX tokens based protocols); upstream conflict source #11821 (UCT/IB/MLX5: Track RC send PSN with path MTU); #11804 (UCT: add token query support) also touched
|
Take master's remaining mlx5/iface files so this PR stays UCP-first and builds against the token purge API without DEFER UCT bits.
|
🤖 Starting review — findings will be posted here when done. |
uct_invoke_completion is declared there; dropping the include broke the devel -Werror build after taking master's UCT headers.
| @@ -2389,10 +2949,9 @@ static void ucp_wireup_msg_dump(ucp_worker_h worker, uct_am_trace_type_t type, | |||
| char *p, *end; | |||
There was a problem hiding this comment.
minor — ucp_wireup_msg_dump skips only ucp_wireup_msg_lanes_info_t before calling ucp_address_unpack. When the peer is >= UCP_WIREUP_ADDR_TOKEN_MIN_DST_VERSION, a ucp_wireup_msg_tokens_info_t + variable-length token sections sit between the lanes_info and the packed address (as ucp_wireup_parse_lanes_addr correctly accounts for). The dump therefore points addr_ptr into the token trailer and will misparse / log "failed to unpack address" whenever UCX_LOG_LEVEL=data AM tracing is enabled on a token-capable connection. Impact is limited to the debug/trace path (no data-path effect), but the dump should mirror the trailer-skipping logic (e.g. reuse ucp_wireup_skip_token_section) or bail out for token-carrying messages.
| /* Re-evaluate am_lane from scratch so a recovered AM lane can become the | ||
| * short-protocol lane again; keeping a post-failover fallback here races | ||
| * with proto short selection (lane mismatch assert). */ | ||
| cfg_key.am_lane = UCP_NULL_LANE; |
There was a problem hiding this comment.
nit — In ucp_ep_reconfig_clear_failed_lanes, cfg_key.am_lane = UCP_NULL_LANE; is dead code: ucp_ep_config_key_find_am_lane() takes a const ucp_ep_config_key_t * and does not read am_lane, so the first assignment is immediately overwritten with no observable effect. Drop the redundant first assignment (or, if the intent is to guard a future reader of am_lane, add a comment clarifying why the reset must precede the call).
|
🤖 Starting review — findings will be posted here when done. |
| @@ -349,7 +349,7 @@ typedef struct ucp_worker { | |||
| uct_worker_cb_id_t rkey_ptr_cb_id; /* RKEY PTR worker callback queue ID */ | |||
There was a problem hiding this comment.
minor: am_message_id isn't used by failover — recovery uses recovery_arg->request_id. The comment change from /* For matching long AMs */ to /* AM/failover message ID generator */ is misleading since no failover code path uses worker->am_message_id (only long-AM matching in proto_common.inl and ucp_request.c consumes it). Please revert the comment to /* For matching long AMs */ to avoid implying a coupling that doesn't exist.
|
Latent path worth confirming: in |
The leftover mlx5_ext checks do not compile against master's uct_rc_mlx5_ep flags.
|
🤖 Starting review — findings will be posted here when done. |
| /* Re-evaluate am_lane from scratch so a recovered AM lane can become the | ||
| * short-protocol lane again; keeping a post-failover fallback here races | ||
| * with proto short selection (lane mismatch assert). */ | ||
| cfg_key.am_lane = UCP_NULL_LANE; |
There was a problem hiding this comment.
minor: the first assignment cfg_key.am_lane = UCP_NULL_LANE; is dead — ucp_ep_config_key_find_am_lane() scans lane types and does not read key->am_lane, so it is immediately overwritten. The comment about "re-evaluate from scratch" is already satisfied by the call alone; pls drop the redundant = UCP_NULL_LANE line.
| [UCP_OP_ID_RNDV_SEND] = "rndv_send", | ||
| [UCP_OP_ID_RNDV_RECV] = "rndv_recv", | ||
| [UCP_OP_ID_RNDV_RECV_DROP] = "rndv_recv_drop", | ||
| [UCP_OP_ID_FAILOVER_AM_BCOPY] = "failover_am_bcopy", |
There was a problem hiding this comment.
minor: the new [UCP_OP_ID_FAILOVER_AM_BCOPY] = "..." entry breaks the = column alignment used by the rest of the ucp_operation_names table (it sits one column left of [UCP_OP_ID_RNDV_RECV_DROP] =). pls re-align for consistency with the surrounding designated initializers.
| [UCP_OP_ID_RNDV_RECV] = "rendezvous data fetch", | ||
| [UCP_OP_ID_RNDV_RECV_DROP] = "rendezvous data drop", | ||
| [UCP_OP_ID_LAST] = NULL | ||
| [UCP_OP_ID_FAILOVER_AM_BCOPY] = "failover replay active message", |
There was a problem hiding this comment.
minor: the new ucp_operation_descs failover entry breaks the = column alignment used by the rest of the table. pls re-align for consistency with the surrounding designated initializers.
|
🤖 CI Triage Agent — TL;DR: The Coverity release job failed on a single new REVERSE_INULL defect in Full analysisSummary: Root cause: In Implicated commit: PR #11810 branch File: src/ucp/wireup/wireup.c:1456 (and the duplicate check at :1465); dereference site src/ucp/core/ucp_ep.inl:225,234 Suggested fix: Remove the redundant ucp_ep_update_remote_id(ep, msg->src_ep_id);
/* Dual-REQ: lower ep_id is the initiator. */
if ((ep->ext->recovery_arg != NULL) &&
(ep->ext->recovery_arg->state == UCP_EP_RECOVERY_STATE_WAIT_REPLY)) {
if (ucp_ep_local_id(ep) < msg->src_ep_id) {
ucs_debug("ep %p: ignore LANES_ADDR_REQ from higher ep id 0x%" PRIx64
" while we are initiator", ep, msg->src_ep_id);
return;
}
if (ucp_ep_local_id(ep) > msg->src_ep_id) {
ep->ext->recovery_arg->state = UCP_EP_RECOVERY_STATE_IDLE;
}
}If a NULL Related: PR #11810 (#11810); prior fix-up commits on the same file/branch: 1127ba9, 6b7c7bf ("Fix maybe-uninitialized lanes_info"), c2dd315.
|
Keep the PR UCP-only: remove unused enable_lanes, restore master's gtest HANDLE_ERRORS default, and drop the outstanding-purge UCT test.
Coverity REVERSE_INULL: ucp_ep_update_remote_id already uses ep->ext.
|
🤖 Starting review — findings will be posted here when done. |
| @@ -2389,10 +2948,9 @@ static void ucp_wireup_msg_dump(ucp_worker_h worker, uct_am_trace_type_t type, | |||
| char *p, *end; | |||
There was a problem hiding this comment.
ucp_wireup_msg_dump skips lanes_info but not the token trailer added here, so with a token-capable peer the address dump starts mid-trailer. When the token trailer is present (peer dst_version >= UCP_WIREUP_ADDR_TOKEN_MIN_DST_VERSION), the dump interprets the token bytes as the start of the packed address and prints garbage / possibly log-spams an unpack error. Can we reuse the same trailer-skip logic (ucp_wireup_parse_lanes_addr / ucp_wireup_skip_token_section) here? Debug-only, so minor.
|
The The dominant recommendation stands: this PR is far over the 500-added-line limit and bundles the UCT API/mlx5 token support, the wireup token wire-format, the UCP failover engine, and unrelated refactors (the put_offload RESUME flag + ucp_rkey_need_remote_flush gate, the ucp_wireup_msg_prepare/msg_send signature rework, and test restructuring). Splitting into separate reviewable PRs would let each correctness path be reviewed with confidence — the individual failover state-machine transitions (abort vs. replay-completion ownership of super_req, flush_ops_count accounting) are hard to verify at this size. Coverage gap: No test drives a stale/mismatched request_id through ucp_ep_failover_apply_rx_tokens. Coverage gap: No test exercises the dual-REQ tie-break (ucp_ep_local_id(ep) < msg->src_ep_id) in ucp_wireup_process_lanes_addr_request. Coverage gap: The old-peer/no-trailer cross-version path (ucp_wireup_ep_supports_tokens false) is only implicitly covered. |
|
🤖 CI Triage Agent — TL;DR: The ASan/roce gtest shard didn't time out on the CI step — the very first test ( Full analysisSummary: Root cause: A hang, not slowness. Log evidence:
The stall is in code introduced by this PR branch ( Implicated commit: File: Suggested fix:
Related: PR #11810 (this PR, "UCP/FT: TX/RX tokens based protocols"), PR #11843 ("UCP/WIREUP: Add a token trailer and an ACK message to the lane address exchange", merged into this branch as
|
|
out of date |
What?