Skip to content

UCP/PROTO: Introduce Multi Eager PSN Protocol - #11160

Merged
shasson5 merged 22 commits into
openucx:masterfrom
shasson5:ft2
Feb 7, 2026
Merged

shasson5 merged 22 commits into
openucx:masterfrom
shasson5:ft2

Conversation

@shasson5

@shasson5 shasson5 commented Jan 28, 2026 •

Copy link
Copy Markdown
Contributor

What?

Introduce Multi Eager PSN Protocol

Why?

Fault Tolerance support.
This PR depends on #11120.
Basic tests exist and more will be added later.

Comment thread src/ucp/core/ucp_am.c Resolved
Comment thread src/ucp/core/ucp_am.c Resolved
@evgeny-leksikov

Copy link
Copy Markdown
Contributor

LGTM 👍

@gleon99 gleon99 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Comment thread test/gtest/ucp/test_ucp_am.cc Outdated

UCS_TEST_P(test_ucp_am_psn, no_duplicates, "ZCOPY_THRESH=0", "RNDV_THRESH=inf")
{
/* TODO: enable after interval tree is merged */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't forget to address, after interval tree is merged.

Comment thread src/ucp/core/ucp_am.c Outdated

/* Get pointer to frag_tree allocated before the rdesc */
#define ucp_am_rdesc_frag_tree(_rdesc) \
((ucs_interval_tree_t*)((char*)(_rdesc) - sizeof(ucs_interval_tree_t)))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UCS_PTR_BYTE_OFFSET?

Comment thread src/ucp/core/ucp_am.c Resolved
Comment thread src/ucp/core/ucp_am.c Resolved
Comment thread src/ucp/core/ucp_am.c
Comment on lines +1416 to +1417
ucs_interval_tree_insert(ucp_am_rdesc_frag_tree(first_rdesc), offset,
offset + length - 1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Room for optimization: maybe rework interval tree to accept format [min; max)? Or add a parameter to init. then no need -1 here and avoid few +/-1 ops in intersect logic

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's consider for phase 2

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Comment thread src/ucp/core/ucp_am.c Outdated
Comment on lines 1487 to 1489
desc_offset = sizeof(ucs_interval_tree_t) + first_rdesc->payload_offset;
first_rdesc = (ucp_recv_desc_t*)payload - 1;
first_rdesc->flags = UCP_RECV_DESC_FLAG_MALLOC |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alignment

Comment thread src/ucp/core/ucp_am.c Resolved
Comment thread src/ucp/core/ucp_am.c Resolved
Comment thread src/ucp/core/ucp_types.h Resolved
Comment thread test/gtest/ucp/test_ucp_am.cc Outdated

void test_psn_send_recv(size_t total_msgs, size_t num_duplicates)
{
const size_t msg_size = 1024;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

multi fragment case is not covered at all?

@evgeny-leksikov evgeny-leksikov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM besides code style and open Q regarding to AM ordering for non-PSN

Comment thread src/ucp/core/ucp_am.c Outdated
first_rdesc->payload_offset;
first_rdesc = (ucp_recv_desc_t*)payload - 1;
first_rdesc->flags = UCP_RECV_DESC_FLAG_MALLOC |
UCP_RECV_DESC_FLAG_AM_CB_INPROGRESS;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alignment

Comment thread src/ucp/core/ucp_am.c
Comment on lines +1452 to +1453
ucs_list_for_each_safe(first_rdesc, tmp_rdesc, &ep_ext->am.started_ams,
am_first.list) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need force ordering for non-psn protocol?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so, because AM should keep order AFAIU

Comment thread src/ucp/core/ucp_am.c Resolved
Comment thread src/ucp/proto/proto_common.h Outdated
UCP_PROTO_COMMON_KEEP_MD_MAP = UCS_BIT(11)
UCP_PROTO_COMMON_KEEP_MD_MAP = UCS_BIT(11),

/* Supports failover error handling mode with PSN tracking */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

del "with PSN tracking"

Comment thread test/gtest/ucp/test_ucp_am.cc Resolved
Comment thread src/ucp/am/eager_multi.c Resolved
Comment thread src/ucp/core/ucp_am.c Resolved
Comment thread src/ucp/core/ucp_am.c Resolved
}
};

UCS_TEST_P(test_ucp_am_psn, no_duplicates, "ZCOPY_THRESH=0", "RNDV_THRESH=inf", "PROTO_REQUEST_RESET=n")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EFA CI failure is because we fallback on proto reconfig, since we could not select am/egr/multi/zcopy/psn, which is the only possible with UCP_PROTO_COMMON_INIT_FLAG_FAILOVER for AM in this test.

We failed to select it because on EFA, am max_iov is only 1.

 probing am/egr/multi/zcopy/psn
   selecting up to 1/2 lanes for am/egr/multi/zcopy/psn am_send from host memory
   lane[0] ud_verbs/rdmap0:1: max iov 1 is less than min iov 2
   lane[1] ud_verbs/rdmap1:1: no am in lane types
    selected 0 lanes
 no lanes for am/egr/multi/zcopy/psn

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants