Repository navigation
UCP/FT/TF: Keep the lane and one peer request id until the error completion - #12183
Conversation
Keep the lane and one peer request id until the error completion, and do not treat a CM endpoint slot as recovery state.
The ucs_diag calls on the hold and purge path were only used while debugging.
The calls bumped to ucs_diag while debugging go back to ucs_debug.
Match the clang-format 12 diff check used by CI.
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The "Basic compile" stage failed with Full analysisSummary: Azure Pipelines build 141003 "Basic compile" job aborted at Root cause: Pure API/caller skew introduced by this PR, not a flaky infra failure. The branch commit added two parameters to the token-failover API:
The callers in
Supporting evidence from git history: Implicated commit: File: Suggested fix: Update the three call sites to the new signatures. Per the doc comments in
status = ucp_ep_tf_derive_rx(ep, lane, tx_tokens[lane].token,
tx_tokens[lane].len, 0 /* require_open */,
&rx_token, &rx_len);
ucp_ep_tf_lanes_purge_outstanding(ep, lanes_info->requested_lane_map,
request_id, 0, 0 /* release_lanes */);
ucp_ep_tf_lanes_purge_outstanding(ep, lanes_info->requested_lane_map,
request_id, 1,
lanes_info->requested_lane_map);Please confirm items 2 and 3 against the intended hold/release design, since the release semantics are behavioural rather than purely mechanical — only item 1 is unambiguously documented by the header comment. A local Related: PR #12183 (this change); possibly related open PR #12129 "FT/TF: wireup recovery bugfixes" touching the same token/recovery wireup path.
|
|
The wireup side of this change is missing from the diff: Also, the Residual coverage gap: no test changes. The recovery scheduling change ( |
Derive still requires an open exchange, and a purge still releases every lane it walks.
|
🤖 Starting review — findings will be posted here when done. |
UCP_LANE_TYPE_FAILED is 10, the AM_BW and RMA_BW bits. The lookup matched those lanes instead of a failed one.
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The gtest Full analysisSummary: Root cause: This is a hard-coded structure-size regression guard, not a flaky/timeout failure. Implicated commit: File: Suggested fix: Preferred: avoid growing the struct. Related: PR #12183 (this PR). Prior commits in the same failover series that touched
|
A 32-bit tf_request_id grew ucp_ep_ext by 8 bytes and failed the object size check. Keep the id between cm_idx and lane_generation so it fills that padding.
The id kept on the recovery object, and in the ext slot after that object is freed, is the same width as the id this endpoint sends.
seen_peer_id is defined with the other locals in ucp_ep_recovery_arm.
|
🤖 Starting review — findings will be posted here when done. |
|
The earlier finding still stands and I would I verified the Coverage gap: still no test changes. The |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Root cause: From the log, the run was healthy end‑to‑end (8950 passed, 3522 skipped, no timeout, no crash, no hang — the last test suites executed continuously up to teardown at 09:39:14, so this is not a wall‑clock/hang symptom). The single failing case is Implicated commit: unknown — not attributable to File: test/gtest/uct/test_atomic_key_reg_rdma_mem_type.cc:33 (mapped_buffer in Suggested fix:
Related: PR #12183 (#12183) is the PR under test; prior hardening of the same test: commits
|
A 16-bit peer id that wraps past 0xffff is still newer than the last one accepted. A recovery round that publishes nothing returns to the worker keepalive loop and does not burn a retry. A CM endpoint does not store that id over the close request. Per-lane purge skips are traced, and a repost log names the operation.
|
🤖 Starting review — findings will be posted here when done. |
|
Residual gap: still no test changes; the Note (not raised as a blocking comment): new debug/trace messages use |
|
Residual gap: no test changes. The new held-lane purge scheduling ( |
|
🤖 Starting review — findings will be posted here when done. |
The reply path invalidates transport endpoints directly, so nothing calls ucp_ep_tf_hold_invalidated_lanes.
|
Deliberately not repeated (already-open findings that still apply; would |
|
🤖 Starting review — findings will be posted here when done. |
A client-manager lane is already asserted, releasing a purged lane clears the purged flag, and the peer-id compare is a 16-bit counter. The purge filter prototype joins the other static declarations.
|
🤖 Starting review — findings will be posted here when done. |
| status = uct_ep_invalidate(uct_ep, ¶ms); | ||
| ucs_debug("ep %p: lane %d invalidate uct_ep %p: %s", ep, lane, uct_ep, | ||
| ucs_status_string(status)); | ||
| if (status == UCS_OK) { |
There was a problem hiding this comment.
uct_ep_invalidate() returns UCS_ERR_NO_RESOURCE on rc/dc mlx5 when TX CQ/WQE credits are exhausted (UCT_RC_CHECK_TX_CQ_RES), which is transient. Such a lane is left out of armed, and both callers then hand it to ucp_ep_failover_reconfig() (wireup.c:1346, wireup.c:1452), canceling its outstanding operations instead of replaying them. Can we retry NO_RESOURCE on a later round instead of falling back to the software discard path?
There was a problem hiding this comment.
will be implemented in separate PR
|
Residual coverage gap: no test changes. |
A purge without a token reports every outstanding send. The held-lane lookup reads the recovery state once per endpoint.
|
🤖 Starting review — findings will be posted here when done. |
|
@dmitrygx pls take a look |
| } | ||
|
|
||
| /* Low bit set: the ext slot holds a peer request id, not a pointer. */ | ||
| #define UCP_EP_SLOT_PEER_ID_BIT ((uintptr_t)1) |
There was a problem hiding this comment.
why not keep seen_peer_id in ucp_ep_ext_t next to the new tf_request_id instead of tagging the union slot? it costs a few bytes per ep but drops the tag bit, the 3 helpers and the asserts, and keeps ext->recovery_arg a plain pointer (the gtest and ucs_assert(ucp_ep_recovery_arg(ep) == NULL) still read it directly).
Problem: seen_peer_id is stashed back into ep->ext->recovery_arg with a low-bit tag, which costs UCP_EP_SLOT_PEER_ID_BIT, three helpers (ucp_ep_ext_recovery_is_peer_id, ucp_ep_tf_seen_peer_id, ucp_ep_tf_seen_peer_id_set), two ucs_assert(!ucp_ep_has_cm_lane()), and turns a plain pointer field into something that must never be read directly. The PR already added ep->ext->tf_request_id to ucp_ep_ext_t for exactly the same "must outlive recovery_arg" reason, so the two lifetimes are solved two different ways.
Impact: every remaining direct read of ep->ext->recovery_arg becomes wrong once a non-zero peer id is recorded — including test/gtest/ucp/test_ucp_fault_tolerance.cc:884,1268-1274 (ASSERT_EQ(nullptr, ep->ext->recovery_arg) would fail). It works today only because nothing calls ucp_ep_tf_accept_peer_req().
| @@ -2558,11 +2667,11 @@ static ucp_ep_lane_tf_t *ucp_ep_tf_get(ucp_ep_h ep, ucp_lane_index_t lane) | |||
| { | |||
There was a problem hiding this comment.
| { | |
| ucp_ep_recovery_arg_t *arg = ucp_ep_recovery_arg(ep); | |
| if (!ucp_ep_err_mode_eq(ep, UCP_ERR_HANDLING_MODE_FAILOVER) || | |
| (arg == NULL) || (lane >= UCP_MAX_LANES)) { | |
| return NULL; | |
| } | |
| return &arg->tf[lane]; |
minor: ucp_ep_recovery_arg() already returns NULL when the ep has a CM lane, so the ucp_ep_get_cm_lane() check can go and the accessor is called once. same double call in ucp_ep_recovery_probe_complete() and ucp_ep_tf_claim(). The accessor is currently called once in the condition and once in the return, and each call re-does ucp_ep_has_cm_lane() → ucp_ep_config(); the accessor also handles cfg_index == UCP_WORKER_CFG_INDEX_NULL, and ucp_ep_tf_get() is called in per-lane loops.
| ucs_debug("ep %p: recovery round (retries_left=%u, failed=0x%" PRIx64 ")", | ||
| ep, ep->ext->recovery_arg->retries_left, (uint64_t)failed); | ||
| ucs_assert(arg->state == UCP_EP_RECOVERY_STATE_IDLE); | ||
| resend = arg->resend; |
There was a problem hiding this comment.
| resend = arg->resend; | |
| lanes = resend ? failed : (failed & ~arg->sent_lanes); |
minor: a resend republishes all failed lanes, this says it directly. lanes = failed & ~arg->sent_lanes; followed by if (resend) lanes |= failed & arg->sent_lanes; is just failed when resend is set; the two-step form hides that.
| * would drop it with the recovery state. */ | ||
| recovered = ucp_ep_recovery_get_ready_lanes(ep, failed) & | ||
| ~ucp_ep_tf_held_lanes(ep); | ||
| ready = ucp_ep_recovery_get_ready_lanes(ep, failed); |
There was a problem hiding this comment.
minor: status is no longer in the same alignment group after the inserted ucs_debug() line, pls drop the extra spaces. git-clang-format will not fix it because that line is untouched.
|
PR-level: added lines are around 520 (≈430 in |
| if (failed == 0) { | ||
| /* Recovery completed between rounds, the ep operates normally. */ | ||
| ucs_assert(ep->ext->recovery_arg == NULL); | ||
| ucs_assert(ucp_ep_recovery_arg(ep) == NULL); |
There was a problem hiding this comment.
Handle pending invalidated lanes here? The request handler now allocates recovery_arg before the err completion marks the lane failed, so a KA round between those events reaches this assertion with failed == 0 and a non-NULL recovery argument.
| if (ucp_ep_tf_hold(ucp_ep, lane, uct_ep) == UCS_OK) { | ||
| /* Purge after return, once the outstanding range is | ||
| * recorded. */ | ||
| ucp_ep_tf_schedule_token_purge(ucp_ep, lane); |
There was a problem hiding this comment.
pls add a test that delays the err completion while processing lanes addr msgs and KA. To cover for the new deferred purge or the interval where a lane is invalidated, but not yet failed.
| ucp_ep_recovery_arg_t *ucp_ep_recovery_arg(ucp_ep_h ep) | ||
| { | ||
| if (ucp_ep_has_cm_lane(ep) || | ||
| ucp_ep_ext_recovery_is_peer_id(ep->ext->recovery_arg)) { | ||
| return NULL; | ||
| } | ||
|
|
||
| return ep->ext->recovery_arg; | ||
| } | ||
|
|
||
| ucp_request_t *ucp_ep_close_req(ucp_ep_h ep) | ||
| { | ||
| return ucp_ep_has_cm_lane(ep) ? ep->ext->close_req : NULL; | ||
| } |
There was a problem hiding this comment.
consider moving to .inl file?
There was a problem hiding this comment.
not fast path, imo better to keep in .h/.c
roiedanino
left a comment
There was a problem hiding this comment.
Three recovery-path concerns inline.
| ep->ext->recovery_arg->state = UCP_EP_RECOVERY_STATE_IDLE; | ||
| /* Nothing was published. Let this keepalive round continue. */ | ||
| arg->state = UCP_EP_RECOVERY_STATE_IDLE; | ||
| arg->resend = resend; |
There was a problem hiding this comment.
If request A was sent and newly failed lane B cannot be prepared, this path leaves resend clear. Every later round attempts B only, so a lost request or reply for A is never retried. Could we preserve a retry path for sent_lanes and test this sequence?
| if (status != UCS_OK) { | ||
| ucs_error("ep %p: outstanding purge on lane %d failed: %s", ep, lane, | ||
| ucs_status_string(status)); | ||
| goto out; |
There was a problem hiding this comment.
If uct_ep_outstanding_purge() returns UCS_ERR_NO_MEMORY, this one-shot callback exits after freeing its argument, leaving the held transport endpoint without a scheduled purge. ucp_ep_tf_schedule_token_purge() also drops an allocation failure while the UCT error handler returns UCS_INPROGRESS; could both paths retry or propagate the failure, with allocation-failure tests?
| arg->resend = 0; | ||
| /* A lane that never went out stays eligible. A resend also republishes | ||
| * lanes whose previous request is still outstanding. */ | ||
| lanes = failed & ~arg->sent_lanes; |
There was a problem hiding this comment.
A new failure can send request B while reply A is still outstanding. A delayed reply to A then calls ucp_ep_recovery_on_reply_received() and moves B's WAIT_REPLY state to PROBING without checking the echoed request ID. Could we correlate replies with the outstanding request or track both exchanges, and test this ordering?
What?
Keep the lane and one peer request id until the error completion
Why?
part of #12129