Skip to content

UCT/UD: Revert stale private endpoint handling in v1.23.x - #12210

Closed
shasson5 wants to merge 1 commit into
openucx:v1.23.xfrom
shasson5:v1.23-revert-ud-stale-eps
Closed

shasson5 wants to merge 1 commit into
openucx:v1.23.xfrom
shasson5:v1.23-revert-ud-stale-eps

Conversation

@shasson5

@shasson5 shasson5 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

What?

Revert #11899 and its test-hardening follow-up #11948 from v1.23.x.

Why?

Restore the private UD endpoint behavior from before the stale-endpoint cleanup backport in the 1.23 release branch.

How?

Remove private-endpoint liveness probes and their state, and restore the earlier timeout cleanup and test-helper behavior.

Revert 027a92f and 9aa2c33, introduced by openucx#11948 and openucx#11899.

Restore private UD endpoint timeout handling before the stale-endpoint cleanup backport, including the earlier test-helper behavior.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

void *ud_ep GTEST_ATTRIBUTE_UNUSED_;
return !ucs_ptr_array_lookup(&iface->eps, ep_id, ud_ep);
}, [this]() { short_progress_loop(); }, 60.0);
}, []() { usleep(1000); }, 60.0);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls keep the progress loop here; stale_crep_on_reused_ep_id destroys an ep with an unacked CREQ, so it is released only after uct_ud_ep_deferred_timeout_handler() purges the TX window, and that handler runs on the worker progress queue which usleep() never drives.

Suggested change
}, []() { usleep(1000); }, 60.0);
}, [this]() { short_progress_loop(); }, 60.0);

Note: wait_for_ep_destroyed() still has two callers after this PR (test_ud_timer.ep_destroy and test_ud.stale_crep_on_reused_ep_id), so reverting to usleep() will cause a 60s stall and then ASSERT_EQ(ep_index, uct_ud_ep_id_to_index(ep(m_e1)->ep_id)) will fail because the index is not reused.


if (ep->flags & UCT_UD_EP_FLAG_PRIVATE) {
uct_ud_ep_purge(ep, UCS_ERR_ENDPOINT_TIMEOUT);
ucs_assert(ucs_queue_is_empty(&ep->tx.window));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the timeout flow runs only when the TX window is not empty (uct_ud_ep_assert_tx_window_nonempty() in uct_ud_ep_timer), so this ucs_assert(ucs_queue_is_empty(&ep->tx.window)) can fire for a private ep — why not keep uct_ud_ep_purge(ep, UCS_ERR_ENDPOINT_TIMEOUT) before the destroy?

#include <ucs/vfs/base/vfs_obj.h>


/* Must be less then peer_timeout to avoid false positive errors taking into

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo reintroduced by the moved comment (minor):

Suggested change
/* Must be less then peer_timeout to avoid false positive errors taking into
/* Must be less than peer_timeout to avoid false positive errors taking into

minor: keep "less than".

@svc-nvidia-pr-review

Copy link
Copy Markdown

PR-level — behavior being given up: the revert removes both parts of the private-ep liveness handling (uct_ud_iface_cep_remove_ep() on timeout and the idle probe), so a private endpoint whose peer is gone stays in the conn-match UNEXP queue for the iface lifetime and can again be matched by a new connection with the same conn_sn; the two regression tests (stale_private_ep_reuse, private_ep_keepalive) are deleted with it. pls state in the description why the revert is needed and link the issue tracking the stale private-ep reuse, so the fix is not lost.

Residual coverage gap: after the revert there is no test exercising the private-ep timeout path (UCT_UD_EP_FLAG_PRIVATE branch of uct_ud_ep_deferred_timeout_handler), which is what makes the assert finding hard to catch in CI.

@iyastreb

iyastreb commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Why need to revert it?

@iyastreb
iyastreb self-requested a review November 3, 2026 05:27
@shasson5

shasson5 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Why need to revert it?

no need (opened to check a non related issue).

@shasson5 shasson5 closed this Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants