Repository navigation
Conversation
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The build failed on a single gtest assertion in Full analysisSummary: Root cause: The AM PSN receive path tracks a single monotonically-increasing per-EP sequence number. Implicated commit: Not the PR under test. The failing logic originates in File: Suggested fix:
Related: PR #12030 (the PR under test, "UCP/WIREUP: EP close handshake"); PR #11160 (introduced the PSN protocol and this test); PR #11452 ( |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Release-config gtest Root cause:
Note: the same log also contains Implicated commit: File: Suggested fix: Pick one of:
Either way, re-run the release gtest ( Related: PR #12030 (#12030); adjacent recent
|
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Starting review — findings will be posted here when done. |
## What? Force-close UCP EPs when error handling mode is set. Fixes #2025 ## Why? To avoid resource leaks, because currently UCP keeps closed non-failed EPs forever. The better alternative is to implement a close handshake in UCP (openucx/ucx#12030) But it requires a significant change in UCX. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved UCX connection cleanup after failures and disconnections, including handling remote-disconnect errors without treating them as failures. * Remote metadata invalidation now reports backend disconnect errors instead of silently ignoring them. * Fixed metadata invalidation across agent ranges so the final agent is included. * Updated connection error handling to reflect endpoint state and stop processing callbacks after a failure. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Ilia Yastrebov <iyastrebov@nvidia.com> Co-authored-by: Colin Hirsch <chirsch@nvidia.com>
## What? Force-close UCP EPs when error handling mode is set. Fixes ai-dynamo#2025 ## Why? To avoid resource leaks, because currently UCP keeps closed non-failed EPs forever. The better alternative is to implement a close handshake in UCP (openucx/ucx#12030) But it requires a significant change in UCX. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved UCX connection cleanup after failures and disconnections, including handling remote-disconnect errors without treating them as failures. * Remote metadata invalidation now reports backend disconnect errors instead of silently ignoring them. * Fixed metadata invalidation across agent ranges so the final agent is included. * Updated connection error handling to reflect endpoint state and stop processing callbacks after a failure. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Ilia Yastrebov <iyastrebov@nvidia.com> Co-authored-by: Colin Hirsch <chirsch@nvidia.com>
What?
Negotiate the close of endpoints connected by worker address:
ucp_ep_close_nbx()sendsEP_REMOVEDto the peer and completes once the peer acknowledges it, so both sides destroy their endpoints. Enabled by default forPEER/FAILOVERerror handling modes (UCX_EP_CLOSE_NEGOTIATE=on|off|auto);NONEmode is unchanged.Why?
Without a disconnect handshake, a plain close keeps the local endpoint, its lanes and QPs, and the peer's passive endpoint alive until worker destroy. Repeated connect/disconnect cycles leak one endpoint pair per cycle until QP creation fails
(ai-dynamo/nixl#2025).
How?
EP_REMOVEDon the AM lane and keep the close request pending. Keepalive or a transport error completes it if the peer never answers.EP_REMOVED, flush, then fail the endpoint withUCS_ERR_CONNECTION_RESET(passive endpoints are destroyed). A closed endpoint waiting for the ack treats an incomingEP_REMOVEDas the ack, which makes simultaneous close work.