Skip to content

UCT/UD: Revert stale private connection cleanup in v1.23.x-fur - #12249

Merged
shasson5 merged 1 commit into
openucx:v1.23.x-furfrom
shasson5:v1.23-fur-revert-ud-stale-5346155
Nov 4, 2026
Merged

shasson5 merged 1 commit into
openucx:v1.23.x-furfrom
shasson5:v1.23-fur-revert-ud-stale-5346155

Conversation

@shasson5

@shasson5 shasson5 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

What?

Revert 9aa2c33e58e40306244461117a56af23458f7bac (#11899) and its dependent follow-up changes from 027a92f7cd94d6ea35eb2214f4fb21b7a4674835, restoring the previous private UD endpoint lifecycle.

The change is based directly on v1.23.x-fur at the HPC-X 2.52 UCX pin, 8094f699daa0d53ea84d11b6cefd7dffa9c9eb46.

Why?

Address MPI/UCC collective hangs caused by stale private-endpoint cleanup while retaining the HPC-X pinned baseline.

)"

This reverts commit 9aa2c33.

Remove the dependent test timeout adjustments from
027a92f together with the tests
introduced by the reverted change.
@shasson5
shasson5 merged commit 2b81997 into openucx:v1.23.x-fur Nov 4, 2026
5 checks passed
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.


if (ep->flags & UCT_UD_EP_FLAG_PRIVATE) {
uct_ud_ep_purge(ep, UCS_ERR_ENDPOINT_TIMEOUT);
ucs_assert(ucs_queue_is_empty(&ep->tx.window));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uct_ud_ep_timer() reaches the peer-timeout flow only when tx.window is non-empty (e.g. a private ep still retransmitting an unacked CREP to a peer that is gone), so this assert can fail in debug builds. Can we keep the purge here?

Suggested change
ucs_assert(ucs_queue_is_empty(&ep->tx.window));
uct_ud_ep_purge(ep, UCS_ERR_ENDPOINT_TIMEOUT);

void *ud_ep GTEST_ATTRIBUTE_UNUSED_;
return !ucs_ptr_array_lookup(&iface->eps, ep_id, ud_ep);
}, [this]() { short_progress_loop(); }, 60.0);
}, []() { usleep(1000); }, 60.0);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why revert this hunk to a plain sleep? short_progress_loop() works for both remaining callers and additionally covers endpoints that are released from the worker progress callback.

#include <ucs/vfs/base/vfs_obj.h>


/* Must be less then peer_timeout to avoid false positive errors taking into

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: then -> than

Suggested change
/* Must be less then peer_timeout to avoid false positive errors taking into
/* Must be less than peer_timeout to avoid false positive errors taking into

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage note: the revert removes stale_private_ep_reuse and private_ep_keepalive, so the stale-private-ep-matched-by-conn_sn case is no longer covered by any test. If the PR description does not already state why the flow is reverted and whether the original issue is tracked by a follow-up, that is worth asking in the PR conversation.

@shasson5

shasson5 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/azp run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants