Skip to content

UCP/FT/TF: Keep the lane and one peer request id until the error completion - #12183

Merged
gleon99 merged 17 commits into
openucx:masterfrom
evgeny-leksikov:ucp_ft_ep_hold
Nov 2, 2026
Merged

gleon99 merged 17 commits into
openucx:masterfrom
evgeny-leksikov:ucp_ft_ep_hold

Conversation

@evgeny-leksikov

Copy link
Copy Markdown
Contributor

What?

Keep the lane and one peer request id until the error completion

Why?

part of #12129

Keep the lane and one peer request id until the error completion, and do not treat a CM endpoint slot as recovery state.
The ucs_diag calls on the hold and purge path were only used while debugging.
The calls bumped to ucs_diag while debugging go back to ucs_debug.
Match the clang-format 12 diff check used by CI.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 27, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR · commit ed88dfa3

TL;DR: The "Basic compile" stage failed with -Werror compile errors in src/ucp/wireup/wireup.c because the PR's commit changed the signatures of ucp_ep_tf_derive_rx() and ucp_ep_tf_lanes_purge_outstanding() in ucp_ep.h/ucp_ep.c but never updated the three call sites in wireup.c. Fix by passing the new require_open and release_lanes arguments at those call sites.

Full analysis

Summary: Azure Pipelines build 141003 "Basic compile" job aborted at make -j4: make[2]: *** [Makefile:1741: wireup/libucp_la-wireup.lo] Error 1 — three -Werror argument-mismatch errors while compiling src/ucp/wireup/wireup.c.

Root cause: Pure API/caller skew introduced by this PR, not a flaky infra failure. The branch commit added two parameters to the token-failover API:

  • ucp_ep_tf_derive_rx() gained a 5th parameter int require_open (now 7 params total) — src/ucp/core/ucp_ep.h:1220-1223, implemented at ucp_ep.c:2899.
  • ucp_ep_tf_lanes_purge_outstanding() gained a 5th parameter ucp_lane_map_t release_lanes — src/ucp/core/ucp_ep.h:1245-1247, implemented at ucp_ep.c:3149.

The callers in wireup.c were left at the old 4-argument form, so GCC reported:

  • wireup.c:1279-1280 — "too few arguments to function 'ucp_ep_tf_derive_rx'", plus void** being passed where int require_open is expected and uint8_t* where void** is expected (the arguments shifted by one).
  • wireup.c:1505 and wireup.c:1682 — "too few arguments to function 'ucp_ep_tf_lanes_purge_outstanding'".

Supporting evidence from git history: ucp_ep.h was last touched by [REDACTED:Hex High Entropy String] (2026-10-27T20:08:36Z, ~1h15m before the build), while wireup.c was last touched by 40d8c4a6 (2026-10-15) — i.e. the header/implementation changed in this PR without the caller being updated.

Implicated commit: [REDACTED:Hex High Entropy String] — "UCP/EP: Hold a failed lane until its error completion", Evgeny Leksikov (head of branch ucp_ft_ep_hold, PR #12183)

File: src/ucp/wireup/wireup.c:1279 (also :1505 and :1682); declarations at src/ucp/core/ucp_ep.h:1220 and src/ucp/core/ucp_ep.h:1245

Suggested fix: Update the three call sites to the new signatures. Per the doc comments in ucp_ep.h and the semantics in ucp_ep.c:3208-3217:

  1. wireup.c:1279 — this derive happens on the reply path while building the ACK, and the header states "A reply passes 0 so a released lane is still looked up by QPN", so pass 0:
status = ucp_ep_tf_derive_rx(ep, lane, tx_tokens[lane].token,
                             tx_tokens[lane].len, 0 /* require_open */,
                             &rx_token, &rx_len);
  1. wireup.c:1505 (reply path, from_ack == 0) — the lane must stay held so the later ACK can still read that queue pair while building the ACK token, so pass an empty release_lanes:
ucp_ep_tf_lanes_purge_outstanding(ep, lanes_info->requested_lane_map,
                                  request_id, 0, 0 /* release_lanes */);
  1. wireup.c:1682 (ACK path, from_ack == 1) — this is the terminal step, so release the lanes:
ucp_ep_tf_lanes_purge_outstanding(ep, lanes_info->requested_lane_map,
                                  request_id, 1,
                                  lanes_info->requested_lane_map);

Please confirm items 2 and 3 against the intended hold/release design, since the release semantics are behavioural rather than purely mechanical — only item 1 is unambiguously documented by the header comment. A local ./autogen.sh && ./configure && make -j (or just compiling src/ucp) would have caught this before push; consider whether the CI "Basic compile" job can be made a required fast-fail gate.

Related: PR #12183 (this change); possibly related open PR #12129 "FT/TF: wireup recovery bugfixes" touching the same token/recovery wireup path.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 961f9550-b01b-4da4-b0c8-af46e299d1ec in the triage console for the audit trail.

Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/proto/proto_repost.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

The wireup side of this change is missing from the diff: ucp_ep_tf_derive_rx() and ucp_ep_tf_lanes_purge_outstanding() changed signature but their callers in wireup/wireup.c (lines 1279, 1505, 1682) still pass the old argument count, and the new helpers (ucp_ep_tf_accept_peer_req, ucp_ep_tf_peer_req_is_newer, ucp_ep_tf_drop_invalidated, ucp_ep_tf_lanes_to_invalidate, ucp_ep_tf_invalidate_transport, ucp_ep_tf_is_held) have no caller at all. Pls add them together with the code that uses them, otherwise docs/CodeStyle.md asks for static.

Also, the close_req accessor conversion and the formatting-only hunks are not related to token failover - pls split them out per REVIEW.md.

Residual coverage gap: no test changes. The recovery scheduling change (sent_lanes/resend) and the new held-lane purge lifetime (purged, release_lanes) are only exercised by DISABLED_test_ucp_fault_tolerance_tf, which does not run by default, so none of the new logic has active CI coverage; the seen_peer_id path is unreachable until the wireup side lands.

@evgeny-leksikov evgeny-leksikov added the WIP-DNM Work in progress / Do not review label Oct 27, 2026
Derive still requires an open exchange, and a purge still releases every lane it walks.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

UCP_LANE_TYPE_FAILED is 10, the AM_BW and RMA_BW bits. The lookup matched those lanes instead of a failed one.
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/wireup/wireup.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/wireup/wireup.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
@svc-ucx

svc-ucx commented Oct 27, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests althca on worker 0) · commit 03c4f517

TL;DR: The gtest test_obj_size.size failed because sizeof(ucp_ep_ext_t) grew from 216 to 224 bytes after this PR added the tf_request_id field to ucp_ep_ext_t; either shrink/pack the new field into existing padding or bump the expected size in test_obj_size.cc.

Full analysis

Summary: Tests althca on worker 0 failed in the release gtest run: test_obj_size.size asserted (size_t)216 == sizeof(ucp_ep_ext_t) but got 224 (make: *** [test] Error 1, bash exit code 2).

Root cause: This is a hard-coded structure-size regression guard, not a flaky/timeout failure. test/gtest/common/test_obj_size.cc:53 pins sizeof(ucp_ep_ext_t) to 216 bytes. The PR branch commit [REDACTED:Hex High Entropy String] ("UCP/EP: Hold a failed lane until its error completion") added a new 4-byte field uint32_t tf_request_id to ucp_ep_ext_t (src/ucp/core/ucp_ep.h:580), right after ucp_rsc_index_t cm_idx / before uint32_t lane_generation. The pre-existing 3-byte padding after the 1-byte cm_idx was already consumed by lane_generation, so the new uint32_t adds 4 bytes plus 4 bytes of tail padding to re-align the following 8-byte ucs_ptr_map_key_t local_ep_id — total +8 bytes, 216 → 224. The test only runs in the release configuration (it skips when debug-data/stats/asserts are enabled), which is why only this release job caught it.

Implicated commit: [REDACTED:Hex High Entropy String] — Evgeny Leksikov, "UCP/EP: Hold a failed lane until its error completion" (HEAD of branch ucp_ft_ep_hold, merge commit [REDACTED:Hex High Entropy String])

File: test/gtest/common/test_obj_size.cc:53 (expected value) / src/ucp/core/ucp_ep.h:573-641, new field at src/ucp/core/ucp_ep.h:580

Suggested fix: Preferred: avoid growing the struct. tf_request_id is a monotonically increasing failover request counter and is already compared via the wrap-aware ucp_ep_tf_peer_req_is_newer() helper, so it does not need 32 bits of headroom in the hot EP extension — narrow it to uint16_t/uint8_t and place it adjacent to cm_idx so it fits in the existing padding hole, or fold it into the recovery_arg/close_req union-backed recovery object (ucp_ep_recovery_arg_t, which is heap-allocated per-EP under recovery) instead of ucp_ep_ext_t. Note the comment says it must survive across recovery object lifetime, so if it genuinely must live in the ext struct, pack it alongside lane_generation (e.g. make both bitfields of one uint32_t, or reuse the low bits already documented as free in the close_req/recovery_arg union). If the 8-byte growth is intentional and accepted, update the guard to EXPECTED_SIZE(ucp_ep_ext_t, 224); and call it out explicitly in the PR description, since this test exists to make per-EP memory growth a deliberate review decision.

Related: PR #12183 (this PR). Prior commits in the same failover series that touched ucp_ep_ext_t/recovery state: #12048, #12008, #11995, #12005. No existing issue tracks this specific size assertion.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 2f39e491-4a0a-4521-b646-d593850efb0a in the triage console for the audit trail.

Comment thread src/ucp/core/ucp_ep.h Resolved
A 32-bit tf_request_id grew ucp_ep_ext by 8 bytes and failed the object
size check. Keep the id between cm_idx and lane_generation so it fills
that padding.
The id kept on the recovery object, and in the ext slot after that
object is freed, is the same width as the id this endpoint sends.
seen_peer_id is defined with the other locals in ucp_ep_recovery_arm.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/proto/proto_repost.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

The earlier finding still stands and I would +1 the existing thread rather than reopen it: ucp_ep_tf_accept_peer_req, ucp_ep_tf_peer_req_is_newer, ucp_ep_tf_drop_invalidated, ucp_ep_tf_lanes_to_invalidate, ucp_ep_tf_invalidate_transport and ucp_ep_tf_is_held still have no callers. The same applies to the two new parameters: require_open is only ever passed 1 (wireup.c:1279) and release_lanes is always equal to lanes at both call sites (wireup.c:1507, wireup.c:1685), so the keep-held and QPN-lookup paths the docs describe are unreachable.

I verified the close_req accessor is functionally safe: ucp_ep_set_close_request() is only reached under ucp_ep_is_cm_local_connected() (ucp_ep.c:3428, :3490), which implies a CM lane, so returning NULL for non-CM eps does not hide a live close request. It does weaken the assert in ucp_ep_set_close_request() for non-CM eps, but that path is unreachable.

Coverage gap: still no test changes. The sent_lanes/resend scheduling and the purged/release_lanes hold lifetime are only exercised by DISABLED_test_ucp_fault_tolerance_tf, which does not run by default; seen_peer_id stays 0 in the enabled fixture, which is the only reason the direct ep->ext->recovery_arg reads in the gtest still work today.

@svc-ucx

svc-ucx commented Oct 28, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 1) · commit b75eb7e7

TL;DR: The roce on worker 1 gtest run finished with exactly one failure — dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/0, a UCT device‑memory (MEMIC / UCS_MEMORY_TYPE_RDMA) atomic test that is historically flaky on shared RoCE runners — and it is unrelated to PR #12183, which only touches UCP fault‑tolerance EP/lane hold logic; re-run the job and, if it reproduces, harden the MEMIC allocation/skip logic in that test.

Full analysis

Summary: make test in build-test/test/gtest exited with Error 1 because 1 of 8951 gtests failed: [ FAILED ] dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/0, where GetParam() = dc_mlx5/mlx5_0:1.

Root cause: From the log, the run was healthy end‑to‑end (8950 passed, 3522 skipped, no timeout, no crash, no hang — the last test suites executed continuously up to teardown at 09:39:14, so this is not a wall‑clock/hang symptom). The single failing case is uct_atomic_key_reg_rdma_mem_type.fadd64, which allocates an 8‑byte buffer in UCS_MEMORY_TYPE_RDMA (mlx5 device memory / MEMIC) with UCT_MD_MEM_ACCESS_ALL and 10 allocation retries, then runs 4 concurrent uct_ep_atomic64_fetch(ADD) senders over dc_mlx5/mlx5_0:1 (test/gtest/uct/test_atomic_key_reg_rdma_mem_type.cc:29-41). This test exercises the IB atomic-key registration path for device memory and has a documented history of environment‑sensitive failures on busy CI machines — three separate hardening commits exist for exactly this file (69718f6d "Add retries and rand sleep when it cannot allocate MEMIC memory", [REDACTED:Hex High Entropy String] "Retry when it cannot allocate MEMIC memory", d7d57e17 "Limit MEMIC allocation to prevent out-of-memory test failures", all by Michal Shalev / Leonid Genkin). Nothing in PR #12183 (ucp_ft_ep_hold: keep lane and one peer request id until error completion, UCP core) can influence the UCT dc_mlx5 atomic/MEMIC path, and no UCP test failed in this run.
Caveat: the Azure log is only retrievable as the last ~500 lines, which stops after the summary table; the assertion text / error log line emitted at the moment of failure is above that window, so the precise failing expectation (MEMIC allocation error vs. atomic reply mismatch) could not be read.

Implicated commit: unknown — not attributable to [REDACTED:Hex High Entropy String] (PR #12183). If it is a real regression rather than flakiness, the likeliest merge-base candidates are the RC/DC mlx5 TX-path changes merged shortly before this build: [REDACTED:Hex High Entropy String] (Evgeny Leksikov, "UCT/IB/MLX5: Revert outstanding-purge transmit range + NOP in invalidate", #12133, 2026‑10‑28 07:29Z) and b0d5d1a4 (Guangguan Wang, "UCT/IB/RC: Remove an endpoint from ep_list in uct_rc_ep_t cleanup", #12059).

File: test/gtest/uct/test_atomic_key_reg_rdma_mem_type.cc:33 (mapped_buffer in UCS_MEMORY_TYPE_RDMA with 10 retries), test instantiated for dc_mlx5 at line 43.

Suggested fix:

  1. Re-trigger the roce on worker 1 job to confirm flakiness (expected: pass on re-run, since the rest of the 8951 tests are green).
  2. To get the actual assertion, fetch the full (non-truncated) raw Azure log for job ce213e0a-73ad-5e2a-1c22-[REDACTED:Hex High Entropy String] and grep for uct_atomic_key_reg_rdma_mem_type / Failure, or reproduce locally with GTEST_FILTER='dc_mlx5/uct_atomic_key_reg_rdma_mem_type.*' make -C test/gtest test.
  3. If it reproduces as a MEMIC/device-memory allocation error, continue the existing hardening pattern in that file: raise the retry count above 10 and/or UCS_TEST_SKIP when the device-memory allocation cannot be satisfied (MEMIC is a tiny shared resource and worker 1 runs many concurrent jobs), instead of failing the test.
  4. Do not gate PR UCP/FT/TF: Keep the lane and one peer request id until the error completion #12183 on this result — it is an unrelated UCT dc_mlx5 atomic test, not a UCP fault-tolerance test.

Related: PR #12183 (#12183) is the PR under test; prior hardening of the same test: commits 69718f6d, [REDACTED:Hex High Entropy String], d7d57e17, original test added in 6c7715d0 (Roie Danino, "TEST/GTEST/UCT: Added gtest for RDMA memory atomic key reg").

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id e93c056e-5f35-45c5-b7c6-9b999e5ad60d in the triage console for the audit trail.

A 16-bit peer id that wraps past 0xffff is still newer than the last one
accepted. A recovery round that publishes nothing returns to the worker
keepalive loop and does not burn a retry. A CM endpoint does not store
that id over the close request. Per-lane purge skips are traced, and a
repost log names the operation.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual gap: still no test changes; the sent_lanes/resend scheduling and the purged/release_lanes hold lifetime are only reachable from DISABLED_test_ucp_fault_tolerance_tf.

Note (not raised as a blocking comment): new debug/trace messages use name=value formatting (failed=0x..., ready=0x..., lanes=0x...), which docs/LoggingStyle.md asks to avoid; the surrounding code already does this.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual gap: no test changes. The new held-lane purge scheduling (ucp_ep_tf_schedule_token_purge, purged) and the sent_lanes/resend scheduling are reachable only from the failover fixtures that do not run by default, and the seen_peer_id path is still unreachable (no caller of ucp_ep_tf_accept_peer_req), so the tagged-pointer slot always stays 0 in CI.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

The reply path invalidates transport endpoints directly, so nothing
calls ucp_ep_tf_hold_invalidated_lanes.
Comment thread src/ucp/wireup/wireup.c Resolved
Comment thread src/ucp/wireup/wireup.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Deliberately not repeated (already-open findings that still apply; would +1 the existing threads rather than open new ones): the unused ucp_ep_tf_accept_peer_req/ucp_ep_tf_peer_req_is_newer/ucp_ep_tf_drop_invalidated helpers, require_open always 1 and release_lanes always equal to lanes, the unrelated formatting/close_req hunks, the name=value log formatting, and the missing test coverage (the new purged/sent_lanes/resend logic is still only reachable from the failover fixtures that do not run by default).

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/wireup/wireup.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
A client-manager lane is already asserted, releasing a purged lane clears
the purged flag, and the peer-id compare is a 16-bit counter. The purge
filter prototype joins the other static declarations.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_ep.c
status = uct_ep_invalidate(uct_ep, &params);
ucs_debug("ep %p: lane %d invalidate uct_ep %p: %s", ep, lane, uct_ep,
ucs_status_string(status));
if (status == UCS_OK) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uct_ep_invalidate() returns UCS_ERR_NO_RESOURCE on rc/dc mlx5 when TX CQ/WQE credits are exhausted (UCT_RC_CHECK_TX_CQ_RES), which is transient. Such a lane is left out of armed, and both callers then hand it to ucp_ep_failover_reconfig() (wireup.c:1346, wireup.c:1452), canceling its outstanding operations instead of replaying them. Can we retry NO_RESOURCE on a later round instead of falling back to the software discard path?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will be implemented in separate PR

Comment thread src/ucp/core/ucp_ep.c Resolved
Comment thread src/ucp/core/ucp_ep.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: no test changes. ucp_ep_tf_schedule_token_purge()/purged, the sent_lanes/resend scheduling, and the seen_peer_id tagged-pointer slot are reachable only from the failover fixtures that do not run by default; seen_peer_id stays 0 in CI, which is why the direct ep->ext->recovery_arg reads in test_ucp_fault_tolerance.cc:884,1268-1274 still pass.

A purge without a token reports every outstanding send. The held-lane
lookup reads the recovery state once per endpoint.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@evgeny-leksikov

Copy link
Copy Markdown
Contributor Author

@dmitrygx pls take a look

Comment thread src/ucp/core/ucp_ep.c
}

/* Low bit set: the ext slot holds a peer request id, not a pointer. */
#define UCP_EP_SLOT_PEER_ID_BIT ((uintptr_t)1)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not keep seen_peer_id in ucp_ep_ext_t next to the new tf_request_id instead of tagging the union slot? it costs a few bytes per ep but drops the tag bit, the 3 helpers and the asserts, and keeps ext->recovery_arg a plain pointer (the gtest and ucs_assert(ucp_ep_recovery_arg(ep) == NULL) still read it directly).

Problem: seen_peer_id is stashed back into ep->ext->recovery_arg with a low-bit tag, which costs UCP_EP_SLOT_PEER_ID_BIT, three helpers (ucp_ep_ext_recovery_is_peer_id, ucp_ep_tf_seen_peer_id, ucp_ep_tf_seen_peer_id_set), two ucs_assert(!ucp_ep_has_cm_lane()), and turns a plain pointer field into something that must never be read directly. The PR already added ep->ext->tf_request_id to ucp_ep_ext_t for exactly the same "must outlive recovery_arg" reason, so the two lifetimes are solved two different ways.

Impact: every remaining direct read of ep->ext->recovery_arg becomes wrong once a non-zero peer id is recorded — including test/gtest/ucp/test_ucp_fault_tolerance.cc:884,1268-1274 (ASSERT_EQ(nullptr, ep->ext->recovery_arg) would fail). It works today only because nothing calls ucp_ep_tf_accept_peer_req().

Comment thread src/ucp/core/ucp_ep.c
@@ -2558,11 +2667,11 @@ static ucp_ep_lane_tf_t *ucp_ep_tf_get(ucp_ep_h ep, ucp_lane_index_t lane)
{

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
{
ucp_ep_recovery_arg_t *arg = ucp_ep_recovery_arg(ep);
if (!ucp_ep_err_mode_eq(ep, UCP_ERR_HANDLING_MODE_FAILOVER) ||
(arg == NULL) || (lane >= UCP_MAX_LANES)) {
return NULL;
}
return &arg->tf[lane];

minor: ucp_ep_recovery_arg() already returns NULL when the ep has a CM lane, so the ucp_ep_get_cm_lane() check can go and the accessor is called once. same double call in ucp_ep_recovery_probe_complete() and ucp_ep_tf_claim(). The accessor is currently called once in the condition and once in the return, and each call re-does ucp_ep_has_cm_lane() → ucp_ep_config(); the accessor also handles cfg_index == UCP_WORKER_CFG_INDEX_NULL, and ucp_ep_tf_get() is called in per-lane loops.

Comment thread src/ucp/core/ucp_ep.c
ucs_debug("ep %p: recovery round (retries_left=%u, failed=0x%" PRIx64 ")",
ep, ep->ext->recovery_arg->retries_left, (uint64_t)failed);
ucs_assert(arg->state == UCP_EP_RECOVERY_STATE_IDLE);
resend = arg->resend;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
resend = arg->resend;
lanes = resend ? failed : (failed & ~arg->sent_lanes);

minor: a resend republishes all failed lanes, this says it directly. lanes = failed & ~arg->sent_lanes; followed by if (resend) lanes |= failed & arg->sent_lanes; is just failed when resend is set; the two-step form hides that.

Comment thread src/ucp/core/ucp_ep.c
* would drop it with the recovery state. */
recovered = ucp_ep_recovery_get_ready_lanes(ep, failed) &
~ucp_ep_tf_held_lanes(ep);
ready = ucp_ep_recovery_get_ready_lanes(ep, failed);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: status is no longer in the same alignment group after the inserted ucs_debug() line, pls drop the extra spaces. git-clang-format will not fix it because that line is untouched.

@svc-nvidia-pr-review

Copy link
Copy Markdown

PR-level: added lines are around 520 (≈430 in ucp_ep.c alone, plus ucp_ep.h, ucp_worker.c, wireup.c, proto_repost.c), which is over the 500 added-line hard limit in REVIEW.md. Dropping the still-unused helpers and the unrelated close_req/formatting hunks (already requested in an open thread) would bring it back under; otherwise pls split the scheduled-purge change from the recovery-scheduling (sent_lanes/resend) change.

Comment thread src/ucp/core/ucp_ep.c
if (failed == 0) {
/* Recovery completed between rounds, the ep operates normally. */
ucs_assert(ep->ext->recovery_arg == NULL);
ucs_assert(ucp_ep_recovery_arg(ep) == NULL);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Handle pending invalidated lanes here? The request handler now allocates recovery_arg before the err completion marks the lane failed, so a KA round between those events reaches this assertion with failed == 0 and a non-NULL recovery argument.

Comment thread src/ucp/core/ucp_worker.c
if (ucp_ep_tf_hold(ucp_ep, lane, uct_ep) == UCS_OK) {
/* Purge after return, once the outstanding range is
* recorded. */
ucp_ep_tf_schedule_token_purge(ucp_ep, lane);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls add a test that delays the err completion while processing lanes addr msgs and KA. To cover for the new deferred purge or the interval where a lane is invalidated, but not yet failed.

Comment thread src/ucp/core/ucp_ep.c
Comment on lines +2070 to +2083
ucp_ep_recovery_arg_t *ucp_ep_recovery_arg(ucp_ep_h ep)
{
if (ucp_ep_has_cm_lane(ep) ||
ucp_ep_ext_recovery_is_peer_id(ep->ext->recovery_arg)) {
return NULL;
}

return ep->ext->recovery_arg;
}

ucp_request_t *ucp_ep_close_req(ucp_ep_h ep)
{
return ucp_ep_has_cm_lane(ep) ? ep->ext->close_req : NULL;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider moving to .inl file?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not fast path, imo better to keep in .h/.c

@roiedanino roiedanino left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three recovery-path concerns inline.

Comment thread src/ucp/core/ucp_ep.c
ep->ext->recovery_arg->state = UCP_EP_RECOVERY_STATE_IDLE;
/* Nothing was published. Let this keepalive round continue. */
arg->state = UCP_EP_RECOVERY_STATE_IDLE;
arg->resend = resend;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If request A was sent and newly failed lane B cannot be prepared, this path leaves resend clear. Every later round attempts B only, so a lost request or reply for A is never retried. Could we preserve a retry path for sent_lanes and test this sequence?

Comment thread src/ucp/core/ucp_ep.c
if (status != UCS_OK) {
ucs_error("ep %p: outstanding purge on lane %d failed: %s", ep, lane,
ucs_status_string(status));
goto out;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If uct_ep_outstanding_purge() returns UCS_ERR_NO_MEMORY, this one-shot callback exits after freeing its argument, leaving the held transport endpoint without a scheduled purge. ucp_ep_tf_schedule_token_purge() also drops an allocation failure while the UCT error handler returns UCS_INPROGRESS; could both paths retry or propagate the failure, with allocation-failure tests?

Comment thread src/ucp/core/ucp_ep.c
arg->resend = 0;
/* A lane that never went out stays eligible. A resend also republishes
* lanes whose previous request is still outstanding. */
lanes = failed & ~arg->sent_lanes;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A new failure can send request B while reply A is still outstanding. A delayed reply to A then calls ucp_ep_recovery_on_reply_received() and moves B's WAIT_REPLY state to PROBING without checking the echoed request ID. Could we correlate replies with the outstanding request or track both exchanges, and test this ordering?

@gleon99
gleon99 enabled auto-merge (squash) November 2, 2026 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants