Repository navigation
UCP/PROTO: Introduce Multi Eager PSN Protocol - #11160
Conversation
|
LGTM 👍 |
|
|
||
| UCS_TEST_P(test_ucp_am_psn, no_duplicates, "ZCOPY_THRESH=0", "RNDV_THRESH=inf") | ||
| { | ||
| /* TODO: enable after interval tree is merged */ |
There was a problem hiding this comment.
Don't forget to address, after interval tree is merged.
|
|
||
| /* Get pointer to frag_tree allocated before the rdesc */ | ||
| #define ucp_am_rdesc_frag_tree(_rdesc) \ | ||
| ((ucs_interval_tree_t*)((char*)(_rdesc) - sizeof(ucs_interval_tree_t))) |
| ucs_interval_tree_insert(ucp_am_rdesc_frag_tree(first_rdesc), offset, | ||
| offset + length - 1); |
There was a problem hiding this comment.
Room for optimization: maybe rework interval tree to accept format [min; max)? Or add a parameter to init. then no need -1 here and avoid few +/-1 ops in intersect logic
There was a problem hiding this comment.
let's consider for phase 2
| desc_offset = sizeof(ucs_interval_tree_t) + first_rdesc->payload_offset; | ||
| first_rdesc = (ucp_recv_desc_t*)payload - 1; | ||
| first_rdesc->flags = UCP_RECV_DESC_FLAG_MALLOC | |
|
|
||
| void test_psn_send_recv(size_t total_msgs, size_t num_duplicates) | ||
| { | ||
| const size_t msg_size = 1024; |
There was a problem hiding this comment.
multi fragment case is not covered at all?
evgeny-leksikov
left a comment
There was a problem hiding this comment.
LGTM besides code style and open Q regarding to AM ordering for non-PSN
| first_rdesc->payload_offset; | ||
| first_rdesc = (ucp_recv_desc_t*)payload - 1; | ||
| first_rdesc->flags = UCP_RECV_DESC_FLAG_MALLOC | | ||
| UCP_RECV_DESC_FLAG_AM_CB_INPROGRESS; |
| ucs_list_for_each_safe(first_rdesc, tmp_rdesc, &ep_ext->am.started_ams, | ||
| am_first.list) { |
There was a problem hiding this comment.
do we need force ordering for non-psn protocol?
There was a problem hiding this comment.
I think so, because AM should keep order AFAIU
| UCP_PROTO_COMMON_KEEP_MD_MAP = UCS_BIT(11) | ||
| UCP_PROTO_COMMON_KEEP_MD_MAP = UCS_BIT(11), | ||
|
|
||
| /* Supports failover error handling mode with PSN tracking */ |
There was a problem hiding this comment.
del "with PSN tracking"
| } | ||
| }; | ||
|
|
||
| UCS_TEST_P(test_ucp_am_psn, no_duplicates, "ZCOPY_THRESH=0", "RNDV_THRESH=inf", "PROTO_REQUEST_RESET=n") |
There was a problem hiding this comment.
EFA CI failure is because we fallback on proto reconfig, since we could not select am/egr/multi/zcopy/psn, which is the only possible with UCP_PROTO_COMMON_INIT_FLAG_FAILOVER for AM in this test.
We failed to select it because on EFA, am max_iov is only 1.
probing am/egr/multi/zcopy/psn
selecting up to 1/2 lanes for am/egr/multi/zcopy/psn am_send from host memory
lane[0] ud_verbs/rdmap0:1: max iov 1 is less than min iov 2
lane[1] ud_verbs/rdmap1:1: no am in lane types
selected 0 lanes
no lanes for am/egr/multi/zcopy/psn
What?
Introduce Multi Eager PSN Protocol
Why?
Fault Tolerance support.
This PR depends on #11120.
Basic tests exist and more will be added later.