Skip to content

UCP/WIREUP: EP close handshake - #12030

Draft
iyastreb wants to merge 5 commits into
openucx:masterfrom
iyastreb:iyastreb/ep-close-handshake
Draft

iyastreb wants to merge 5 commits into
openucx:masterfrom
iyastreb:iyastreb/ep-close-handshake

Conversation

@iyastreb

@iyastreb iyastreb commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

What?

Negotiate the close of endpoints connected by worker address: ucp_ep_close_nbx() sends EP_REMOVED to the peer and completes once the peer acknowledges it, so both sides destroy their endpoints. Enabled by default for PEER/FAILOVER error handling modes (UCX_EP_CLOSE_NEGOTIATE=on|off|auto); NONE mode is unchanged.

Why?

Without a disconnect handshake, a plain close keeps the local endpoint, its lanes and QPs, and the peer's passive endpoint alive until worker destroy. Repeated connect/disconnect cycles leak one endpoint pair per cycle until QP creation fails
(ai-dynamo/nixl#2025).

How?

  • Closer: after the close flush, send EP_REMOVED on the AM lane and keep the close request pending. Keepalive or a transport error completes it if the peer never answers.
  • Peer: acknowledge with EP_REMOVED, flush, then fail the endpoint with UCS_ERR_CONNECTION_RESET (passive endpoints are destroyed). A closed endpoint waiting for the ack treats an incoming EP_REMOVED as the ack, which makes simultaneous close work.
  • No new message or wire change; replies are sent only to peers with a known release version.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 5, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests althca on worker 2) · commit 3c164dd7

TL;DR: The build failed on a single gtest assertion in tcp/test_ucp_am_psn.some_duplicates/0 — the PSN duplicate filter drops any AM whose msg_id <= ep->ext->am.psn, and with MAX_EAGER_LANES=2 + ZCOPY_THRESH=0 the 64 KB multi-fragment messages complete out of order over the two TCP lanes, so legitimate messages get discarded as "duplicates". This is a pre-existing flaky/order-sensitive test, not caused by the EP-close-handshake PR (all of its new test_ucp_wireup_close_negotiate cases passed in the same run).

Full analysis

Summary: make test exited 2 because 1 of 4044 gtests failed: tcp/test_ucp_am_psn.some_duplicates/0 (GetParam() = tcp); 4043 tests passed and the rest of the suite, including the PR's new wireup close-handshake tests, was green.

Root cause: The AM PSN receive path tracks a single monotonically-increasing per-EP sequence number. ucp_am_is_duplicate_psn() (src/ucp/core/ucp_am.c:1758) drops a fragment whenever ftr->msg_id <= ep->ext->am.psn, but ep_ext->am.psn is advanced to the msg_id of whichever message finishes assembly first (ucp_am.c:1478 in ucp_am_handle_unfinished(), and ucp_am.c:1594 for the single-fragment path). The test (test/gtest/ucp/test_ucp_am.cc:2442 → test_psn_send_recv(200, 100)) posts 300 sends of 65536-byte buffers back-to-back without waiting, while test_ucp_am_base::init() forces MAX_EAGER_LANES=2 (test_ucp_am.cc:58) and the test forces ZCOPY_THRESH=0/RNDV_THRESH=inf. Each message is therefore striped as multiple AM_FIRST_PSN/AM_MIDDLE_PSN fragments across two independent TCP lanes with hundreds of messages in flight. If message N+1 completes assembly before N, am.psn jumps to N+1 and the still-arriving fragments of the lower-numbered, genuinely-new message are silently discarded (ucs_debug("dropped duplicate message ...")). m_recv_counter then never reaches 200, failing EXPECT_EQ(total_msgs, m_recv_counter) / the wait_for_value() at test_ucp_am.cc:2428-2433. The duplicate-suppression design assumes in-order, one-message-at-a-time delivery, which multi-lane eager striping does not guarantee.

Implicated commit: Not the PR under test. The failing logic originates in ef265cdc "UCP/PROTO: Introduce Multi Eager PSN Protocol (#11160)" by shasson5 (which added both ucp_am_is_duplicate_psn and test_ucp_am_psn), with later changes in 4cb9dbb7 (Evgeny Leksikov, AM/PSN failover) and 2bafbb02 (Evgeny Leksikov, UCP_AM_HDR_FLAG_RESEND). The PR's own commit 3c164dd7 (Ilia Yastrebov, EP close handshake) touches ucp_ep.c/wireup.c/test_ucp_wireup.cc/test_ucp_am.cc but nothing in the PSN receive path.

File: src/ucp/core/ucp_am.c:1745-1766 (ucp_am_is_duplicate_psn), with the PSN update sites at src/ucp/core/ucp_am.c:1478 and src/ucp/core/ucp_am.c:1594; test at test/gtest/ucp/test_ucp_am.cc:2393-2446.

Suggested fix:

  1. Short term — unblock the PR: re-run the althca worker 2 job to confirm it's intermittent, and don't treat it as a regression from UCP/WIREUP: EP close handshake #12030. If it reproduces, mark test_ucp_am_psn.some_duplicates flaky or constrain it to a single eager lane by adding "MAX_EAGER_LANES=1" to the UCS_TEST_P config list at test_ucp_am.cc:2442 so message completion order is deterministic.
  2. Proper fix — make duplicate detection order-independent instead of a single high-water mark: track a small sliding window / bitmap of recently-completed msg_ids per EP (e.g. psn_base + 64-bit received mask) and only drop a fragment if its msg_id falls at or below the base and is already marked received. Advance the base only when the window is contiguous. Change ucp_am_is_duplicate_psn() to consult that window rather than the bare ftr->msg_id <= ep->ext->am.psn comparison, and update the two ep_ext->am.psn = first_ftr->super.msg_id; assignments to insert into the window rather than overwrite it.
  3. Add an assertion/trace counter for dropped-as-duplicate fragments so this failure mode is visible in the gtest log instead of only surfacing as a receive-count mismatch.

Related: PR #12030 (the PR under test, "UCP/WIREUP: EP close handshake"); PR #11160 (introduced the PSN protocol and this test); PR #11452 (UCP/AM/FT: introduce resend flag for first fragment retransmissions); PR #11810 (open, "UCP/FT: TX/RX tokens based protocols" — touches the same PSN/duplicate area).

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 6, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests althca on worker 0) · commit e7516823

TL;DR: The test_obj_size.size gtest failed because sizeof(ucp_ep_ext_t) is now 224 bytes while test/gtest/common/test_obj_size.cc:53 still asserts 216 — the PR's "EP close handshake" commits added a field to ucp_ep_ext_t without updating the size guard test.

Full analysis

Summary: Release-config gtest test_obj_size.size failed in the althca on worker 0 job: expected sizeof(ucp_ep_ext_t) == 216, got 224.

Root cause: test_obj_size is a deliberate struct-size regression guard. The log shows:

test/gtest/common/test_obj_size.cc:53: Failure
Expected equality of these values:
  (size_t)216
  sizeof(ucp_ep_ext_t)
    Which is: 224

ucp_ep_ext_t (src/ucp/core/ucp_ep.h:563-619) grew by 8 bytes on this branch. Git history on src/ucp/core/ucp_ep.h shows the only changes since the last upstream commit (aaf97b49, Thomas Vegas) are this PR's own commits — 3c164dd7 "UCP/WIREUP: EP close handshake" and 3d7e0074 "UCP/WIREUP: FIxes" (both Ilia Yastrebov), merged into HEAD [REDACTED:Hex High Entropy String] "UCP: Merged with main". The close-handshake work added per-endpoint state (an extra pointer/64-bit member alongside close_req / recovery_arg) to the endpoint extension, pushing it from 216 → 224, but test_obj_size.cc was not updated to match. The test is only enabled in the release configuration (it skips under debug-data/stats/assert), which is why this job catches it and debug jobs don't.

Note: the same log also contains gcc: error: test_ucp_config-test_ucp_config.o: No such file or directory during the ucx_config_noib step. That is a separate, non-fatal pre-existing wart (the step's exit status is swallowed by the tee/grep pipeline) and is not the cause of the job failure — the build failed only on make: *** [test] Error 1 from gtest.

Implicated commit: 3c164dd7 "UCP/WIREUP: EP close handshake" (Ilia Yastrebov), carried into [REDACTED:Hex High Entropy String]; possibly refined by 3d7e0074 "UCP/WIREUP: FIxes" (same author)

File: test/gtest/common/test_obj_size.cc:53 (expectation) / src/ucp/core/ucp_ep.h:563-619 (ucp_ep_ext_t definition)

Suggested fix: Pick one of:

  1. Preferred if the growth is intentional and justified: update the guard to the new size and document why in the commit message:

    EXPECTED_SIZE(ucp_ep_ext_t, 224);

    Reviewers for this file normally expect a rationale, since ucp_ep_ext_t is allocated per endpoint and affects scalability at high EP counts.

  2. Preferred if growth can be avoided: keep ucp_ep_ext_t at 216 bytes by not adding a new top-level member. Options:

    • Store the close-handshake state inside the already-allocated close_req (ucp_request_t) or in the recovery_arg struct, both of which are only live during close/recovery.
    • Add the new state to the existing union { ep_match; flush_state; } if its lifetime is mutually exclusive with those members.
    • Pack small scalars into the padding next to cm_idx (ucp_rsc_index_t, which currently leaves alignment padding before local_ep_id) rather than adding a new 8-byte slot.

Either way, re-run the release gtest (GTEST_FILTER=*test_obj_size*) locally before pushing, as debug/assert builds silently skip this test.

Related: PR #12030 (#12030); adjacent recent ucp_ep_ext_t/recovery_arg changes in #12005 and #11995 (Evgeny Leksikov) and #11971 (Thomas Vegas)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id c656204b-100b-4b5a-907c-3fc7471610f7 in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

iyastreb added a commit to ai-dynamo/nixl that referenced this pull request Oct 13, 2026
## What?
Force-close UCP EPs when error handling mode is set.
Fixes #2025

## Why?
To avoid resource leaks, because currently UCP keeps closed non-failed
EPs forever.
The better alternative is to implement a close handshake in UCP
(openucx/ucx#12030)
But it requires a significant change in UCX.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved UCX connection cleanup after failures and disconnections,
including handling remote-disconnect errors without treating them as
failures.
* Remote metadata invalidation now reports backend disconnect errors
instead of silently ignoring them.
* Fixed metadata invalidation across agent ranges so the final agent is
included.
* Updated connection error handling to reflect endpoint state and stop
processing callbacks after a failure.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Ilia Yastrebov <iyastrebov@nvidia.com>
Co-authored-by: Colin Hirsch <chirsch@nvidia.com>
alok-mrvl pushed a commit to alok-mrvl/nixl that referenced this pull request Nov 11, 2026
## What?
Force-close UCP EPs when error handling mode is set.
Fixes ai-dynamo#2025

## Why?
To avoid resource leaks, because currently UCP keeps closed non-failed
EPs forever.
The better alternative is to implement a close handshake in UCP
(openucx/ucx#12030)
But it requires a significant change in UCX.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved UCX connection cleanup after failures and disconnections,
including handling remote-disconnect errors without treating them as
failures.
* Remote metadata invalidation now reports backend disconnect errors
instead of silently ignoring them.
* Fixed metadata invalidation across agent ranges so the final agent is
included.
* Updated connection error handling to reflect endpoint state and stop
processing callbacks after a failure.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Ilia Yastrebov <iyastrebov@nvidia.com>
Co-authored-by: Colin Hirsch <chirsch@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants