Repository navigation
UCP/RMA: Add SGL support to ucp_get_nbx - #12191
michal-shalev wants to merge 1 commit into
Conversation
|
🤖 Starting review — findings will be posted here when done. |
|
Not raised (checked and fine):
|
|
🤖 Starting review — findings will be posted here when done. |
|
the diff adds ~600 lines, above the 500-line limit. can the mechanical part — extracting Checked and not raised:
Residual coverage gaps:
|
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap: no test forces Residual coverage gap: no test covers Residual coverage gap: the |
|
🤖 CI Triage Agent — TL;DR: The newly added gtest Full analysisSummary: 1 of 4108 gtests failed — Root cause: The test (added in the PR's top commit) runs with
The sibling tests are guarded and therefore pass everywhere: Implicated commit: [REDACTED:Hex High Entropy String] — "UCP/TEST: Add GET SGL short element layout and failure tests", Michal Shalev (feature it tests: 1310a95, same author) File: test/gtest/ucp/test_ucp_rma.cc:1905-1947 (test body), test_ucp_rma.cc:1911 + :1524 (first failing assert); related: src/ucp/rma/get_offload.c:280-285, src/ucp/proto/proto_multi.c:879-885 Suggested fix: Guard
Also make the leading Related: PR #12191 (this branch,
|
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The Coverity gate failed with 4 new FORWARD_NULL defects caused by the new static callback Full analysisSummary: Root cause: Not a build/test failure — a static-analysis regression introduced by this PR. static void ucp_proto_get_sgl_short_elem_completion(void *request, ucs_status_t status, void *user_data)
{
ucp_request_t *req = user_data;
ucp_request_free(request);
ucp_invoke_uct_completion(&req->send.state.uct_comp, status); /* unconditional deref */
}It is assigned to
Coverity therefore reports NULL Implicated commit: PR #12191 branch File: Suggested fix: Make the NULL impossibility explicit so Coverity prunes the modeled path. static void ucp_proto_get_sgl_short_elem_completion(void *request, ucs_status_t status, void *user_data)
{
ucp_request_t *req = user_data;
ucs_assert(req != NULL);
ucp_request_free(request);
ucp_invoke_uct_completion(&req->send.state.uct_comp, status);
}Alternatives if the assert is not sufficient: (a) annotate the modeled call with Related: PR #12191 (this PR). No pre-existing issue found for this Coverity signature.
|
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap: no test forces Residual coverage gap: no test covers Residual coverage gap: |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The new gtest Full analysisSummary: Root cause: From the backtrace, the completion was invoked by The only code exercised exclusively by this test is the new short-element path:
Nothing protects the parent from being completed/freed while those sub-request references are outstanding: if Implicated commit: File: Suggested fix:
Related: PR #12191 (this PR, "UCP/RMA: Add SGL support to ucp_get_nbx"); flush accounting context in PR #11923 ("UCP/FLUSH: Add topology-safe selective endpoint flush"); UCT side already merged in #11763 ("UCT/IB/MLX5/RC: Implement get_sgl_zcopy") |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The centos7 Full analysisSummary: Root cause: On the CentOS 7 image (old GCC, are written before use. They are filled by
This is a false positive from the old compiler, not a real logic bug — but Implicated commit: [REDACTED:Hex High Entropy String] (Michal Shalev, merge of File: src/ucp/proto/proto_multi.inl:417-419 (triggered from src/ucp/rma/get_offload.c:340,353 and src/ucp/rma/put_offload.c:460) Suggested fix: Give the compiler a definite initialization at the declarations in void *buffers[UCP_MAX_IOV] = {NULL};
size_t lengths[UCP_MAX_IOV] = {0};
uint64_t remote_addrs[UCP_MAX_IOV] = {0};Only the first element is strictly needed for the Related: PR #12191 (
|
|
Hi @michal-shalev, thanks for this. We run SGL gets in production for NIXL KV-cache reads (on our own UCX patch) and tested this PR at 1187eaa on B300 with rc_mlx5:
Are you planning the NIXL side (READs through sglXfer, like ai-dynamo/nixl#1835)? If not, we have it drafted against this branch and can send it. |
1187eaa to
5404b7b
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
No test forces
Short elements recurse into full protocol selection, so on GPU memory without GPUDirect each short element becomes a rendezvous operation. |
5404b7b to
88b3adc
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap: the emulation tests force Previously raised and still open: |
|
Hi @qierli, I couldn't reproduce the performance issue you reported. I've pushed some changes. You can retest with the latest commit. |
88b3adc to
02aabb1
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
Coverage gap: no test forces Coverage gap: no test covers Coverage gap: the emulation tests pin the nested per-element get to |
02aabb1 to
653642e
Compare
653642e to
e2f80e3
Compare
|
🤖 CI Triage Agent — TL;DR: Full analysisSummary: Azure build 145271, job "roce on worker 1", gtest Root cause: From the backtrace, the assertion fires in /* src/ucp/rma/rma.inl:189-228 */
static inline void ucp_proto_get_sgl_short_elem_completion(void *request, ucs_status_t status, void *user_data) {
ucp_request_t *req = user_data; /* raw, unprotected parent pointer */
ucp_request_free(request);
ucp_invoke_uct_completion(&req->send.state.uct_comp, status);
}
...
++req->send.state.uct_comp.count;
status_ptr = ucp_get_nbx(req->send.ep, buffer, length, remote_addr, ...);
Implicated commit: File: Suggested fix:
Related: PR #12191 (this change); same-area predecessors #12283, #12226, #11947, #11943; UCT counterpart #11763 (
|
tvegas1
left a comment
There was a problem hiding this comment.
wdyt we handle short elements once per request, and reuse the put SGL send code for get? IMO it fixes the CI crash and removes most of the get-specific code.
Proposal:
- probing can stay as it is (get zcopy probe-param with
UCT_IFACE_FLAG_V2_GET_SGL_ZCOPY, like put). Onlyget/sgl/contigpassesNULLinstead of&ucp_get_am_bcopy_proto, soget/am/bcopycan be used per element (this also keeps TCP-only endpoints working) andget/sgl/am/bcopycan be removed - replace the adjust callback with a
min_fragvalue (cap.get.min_zcopyfor get, 0 for put). A split fragment shorter thanmin_fraggrows to amin_fragwindow inside its element, starting atmin(frag_offset, elem_length - min_frag). It always fits, so get can batch exactly like put, without the one-fragment, no-redirect andUCS_INPROGRESSspecial cases - at the first progress call, before registering memory, check if some element is shorter than the lanes'
min_zcopy. If so, useucp_proto_rma_sgl_contig_progress(), extended to send each run of long elements as one internal SGL get (its descriptor points into the user arrays) and each short element as a contig get. The parent only counts child requests, so it never goes pending and is never restarted - then
ucp_proto_get_sgl_short_elem()anducp_proto_get_sgl_adjust_min_frag()can be removed
This also removes the restart issue from the inline comment in rma.inl and the possible out-of-bounds access behind the diff <= frag_offset assert, passes the memh for short elements, and works for GPU memory.
A list where short and long elements alternate would be read per element; UCX_IB_TX_INLINE_RESP=0 avoids short elements completely.
| }; | ||
| ucs_status_ptr_t status_ptr; | ||
| ucs_status_t status; | ||
|
|
There was a problem hiding this comment.
blocker: on restart (UCX_PROTO_REQUEST_RESET=y, which CI sets) the reset sets uct_comp.count back to 1 while this nested request is still running, so the parent completes too early and is reused. This is the stale completion behind the comp->count > 0 abort in the CI triage comment; see the PR-level comment for a way to avoid the nested request.
There was a problem hiding this comment.
I agree, it sounds like a problem.
Can short elements be posted on the lane with the parent's uct_comp?
| } | ||
|
|
||
| static void get_test_variants(std::vector<ucp_test_variant>& variants) { | ||
| get_op_variants(variants, UCS_MASK(2)); |
There was a problem hiding this comment.
pls add a variant of various_lengths with IB_TX_INLINE_RESP=0 (full HW path), and tests for a large element with a short last fragment, for short and long elements interleaved, and for get/sgl/offload_sw
| static void get_test_variants(std::vector<ucp_test_variant>& variants) { | ||
| get_op_variants(variants, UCS_MASK(2)); | ||
| } |
There was a problem hiding this comment.
it is the same as in test_ucp_rma_sgl class, so can be dropped and used from base class
| return ucp_ep_get_iface_attr(req->send.ep, lpriv->super.lane) | ||
| ->cap.get.min_zcopy; |
There was a problem hiding this comment.
ucp_proto_get_sgl_min_frag() looks up cap.get.min_zcopy through the ep config and the worker ifaces on every send call. Could it be cached per lane like opt_align (an offset in ucp_proto_multi_init_params_t, read once in ucp_proto_multi_init_priv()), next to max_sgl_zcopy_count?
| }; | ||
| ucs_status_ptr_t status_ptr; | ||
| ucs_status_t status; | ||
|
|
There was a problem hiding this comment.
I agree, it sounds like a problem.
Can short elements be posted on the lane with the parent's uct_comp?
| * copying them */ | ||
| if (ucs_unlikely((total_length < max_payload) && | ||
| if (ucs_unlikely((adjust_func == NULL) && (total_length < max_payload) && | ||
| (elem_count < max_frag_count) && | ||
| ((start_index + elem_count) < dt_iter->length))) { | ||
| return ucp_proto_rma_sgl_offload_send_frags(req, lpriv, max_frag_count, | ||
| next_iter, post_func); | ||
| return ucp_proto_rma_sgl_offload_send_frags(req, lpriv, frags_count, | ||
| min_frag, next_iter, | ||
| post_func, adjust_func); |
There was a problem hiding this comment.
For GET, one send never combines the end of one element with the start of the next: the batch is not redirected to ]ucp_proto_rma_sgl_offload_send_frags()], and that function sends only one fragment for GET. So the lanes get uneven shares: 3x1 MiB on 4 rails goes 1.5/0.5/0.75/0.25 MiB instead of 0.75 MiB each.
PUT splits it evenly here:
ucx/src/ucp/dt/datatype_iter.inl
Lines 665 to 666 in 546ab37
Is that intended? If not, GET can cut the element the same way, as long as the cut piece is at least min_zcopy. split_uneven_elements has exactly this case, but it only checks that all lanes are used.
What?
Add SGL datatype support to
ucp_get_nbx: newget/sgl/offloadandget/sgl/offload_swprotocols, param validation, and gtests (SGL tests now run for both put and get).Why?
Completes SGL RMA at the UCP level. UCT
get_sgl_zcopywas added in #11763, and put SGL already exists.How?
Mirrors the put SGL protocols.