Skip to content

UCP/RMA: Add SGL support to ucp_get_nbx - #12191

Open
michal-shalev wants to merge 1 commit into
openucx:masterfrom
michal-shalev:ucp-get-sgl
Open

michal-shalev wants to merge 1 commit into
openucx:masterfrom
michal-shalev:ucp-get-sgl

Conversation

@michal-shalev

@michal-shalev michal-shalev commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What?

Add SGL datatype support to ucp_get_nbx: new get/sgl/offload and get/sgl/offload_sw protocols, param validation, and gtests (SGL tests now run for both put and get).

Why?

Completes SGL RMA at the UCP level. UCT get_sgl_zcopy was added in #11763, and put SGL already exists.

How?

Mirrors the put SGL protocols.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/get_offload.c Resolved
Comment thread src/ucp/rma/get_offload.c Resolved
Comment thread src/ucp/rma/get_offload.c Resolved
Comment thread src/ucp/rma/get_offload.c Resolved
Comment thread src/ucp/rma/get_offload.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Not raised (checked and fine):

  • ucp_get_nbx now uses UCP_REQUEST_CHECK_PARAM_COMMON plus UCP_RMA_SGL_CHECK_PARAMS, matching ucp_put_nbx; send.rma.sgl.* is set only for the SGL path and completed_size is still initialized.
  • reset via ucp_proto_get_offload_zcopy_reset rewinds the SGL iterator (offset and frag_offset), which is required for the idempotent GET replay.
  • get/sgl/offload_sw does get exercised on transports without UCT_IFACE_FLAG_V2_GET_SGL_ZCOPY (self/shm), so the new sw path is not left untested.
  • Residual coverage gap: no test forces PROTOS=get/sgl/offload_sw, so on IB-only runs (rc_mlx5 always advertises the v2 get-SGL cap) only the hardware path is covered; and the get variant has no emulation counterpart to put_emulation, since there is no get/sgl/am protocol.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/get_offload.c Resolved
Comment thread src/ucp/rma/get_offload.c Resolved
Comment thread src/ucp/proto/proto_multi.h Resolved
Comment thread src/ucp/api/ucp.h Resolved
Comment thread src/ucp/rma/rma_sgl.inl Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

the diff adds ~600 lines, above the 500-line limit. can the mechanical part — extracting rma_sgl.inl out of put_offload.c and renaming UCP_PUT_SGL_CHECK* to UCP_RMA_SGL_CHECK* — go in a separate refactor-only PR, so this one only adds the get/sgl protocols and their tests?

Checked and not raised:

  • ucp_get_nbx switching to UCP_REQUEST_CHECK_PARAM_COMMON + UCP_RMA_SGL_CHECK_PARAMS mirrors ucp_put_nbx; send.state.completed_size = 0 is preserved and send.rma.sgl.* is written only on the SGL path (it does not overlap rma.rkey/remote_addr in the union).
  • min_sgl_frag is zero-initialized for every lane before the conditional, so put/sgl/offload keeps its previous length > 0 semantics via ucs_max(min_frag_length, 1).
  • Dropping UCP_PROTO_COMMON_INIT_FLAG_MIN_FRAG and setting min_frag_offs = UCP_PROTO_COMMON_OFFSET_INVALID is required here: perf->min_length would otherwise be a byte count compared against the SGL element count.
  • frag_offset read in the adjust helper is the current fragment's offset (ucp_datatype_iter_next_sgl_frags writes only next_iter), and a non-zero frag_offset implies the returned descriptor belongs to the same element, so the backward overlap stays inside the element's memh/rkey.
  • ucp_proto_get_offload_zcopy_reset → ucp_proto_offload_zcopy_reset rewinds offset and type.sgl.frag_offset, which the idempotent GET replay needs.
  • Returning UCS_ERR_INVALID_PARAM from the send func goes through ucp_proto_multi_handle_send_error → ucp_proto_request_abort → ucp_proto_request_zcopy_abort, so in-flight fragments still hold completion refs and the memh is dereg'd.
  • The new test's lane loop follows the existing entity::has_lane_with_caps() idiom.

Residual coverage gaps:

  • No test forces PROTOS=get/sgl/offload_sw, so on IB-only runs (where rc_mlx5 advertises UCT_IFACE_FLAG_V2_GET_SGL_ZCOPY) only the hardware path is exercised; the sw path is covered incidentally via self/shm.
  • No test covers max_frag < min_sgl_frag (the case in the first comment above).

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/proto/proto_multi.c Resolved
Comment thread src/ucp/rma/get_offload.c Resolved
Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: no test forces PROTOS=get/sgl/offload_sw; on IB-only runs (rc_mlx5 advertises UCT_IFACE_FLAG_V2_GET_SGL_ZCOPY) the sw path is only hit incidentally via self/shm.

Residual coverage gap: no test covers max_frag < min_sgl_frag (e.g. RMA_ZCOPY_MAX_SEG_SIZE below cap.get.min_zcopy), where ucs_assertv(diff <= frag_offset) in ucp_proto_get_sgl_adjust_min_frag would fire — already raised in the previous round.

Residual coverage gap: the test_ucp_rma_sgl_err tests skip non-host memory because a short GPU get goes through rendezvous and asserts on error; that path stays untested.

@svc-ucx

svc-ucx commented Oct 28, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests althca on worker 0) · commit 1b00c0ad

TL;DR: The newly added gtest test_ucp_rma_sgl.short_element_sync_failure forces PROTOS=get/sgl/offload (an mlx5-only HW SGL-get protocol) without checking that the hardware supports it, so it fails on the althca worker, which has no mlx5 device; the test needs the same availability guard that split_between_lanes uses.

Full analysis

Summary: 1 of 4108 gtests failed — all/test_ucp_rma_sgl.short_element_sync_failure/1 <all/host/get> — causing make: *** [test] Error 1 in "Run ./contrib/test_jenkins.sh" on the althca worker.

Root cause: The test (added in the PR's top commit) runs with "PROTOS=get/sgl/offload,reconfig", i.e. it allows only the hardware SGL-get protocol, and then relies on two mlx5-specific properties:

  1. get/sgl/offload is probed with first.tl_v2_cap_flags = UCT_IFACE_FLAG_V2_GET_SGL_ZCOPY (src/ucp/rma/get_offload.c:280-285), a capability implemented only in uct/ib/mlx5/rc (commit 5122423, "UCT/IB/MLX5/RC: Implement get_sgl_zcopy"). The althca worker has no mlx5 interface — the log shows all dc_mlx5 suites [ SKIP ] (No interface for testing) — so with rc_verbs no lane advertises this cap and no protocol can be selected for an SGL get. The very first statement of the test, test_sgl(1, UCS_KBYTE) (test_ucp_rma.cc:1911), then hits ASSERT_UCS_OK(UCS_PTR_STATUS(sptr)) at test_ucp_rma.cc:1524 with the error returned by ucp_get_nbx.
  2. Even if the offload proto were selected, the "short element" path only triggers when length < lpriv->min_sgl_frag, which is cap.get.min_zcopy (src/ucp/proto/proto_multi.c:879-885) — ~65 only on mlx5 (inline-response limit). On verbs it is 0/1, so the 8-byte element would be posted in the normal SGL batch, no sub-request would be created, the parent would complete with UCS_OK, and EXPECT_NE(UCS_OK, cb.status) / the "elements not posted" loop (test_ucp_rma.cc:1934-1940) would fail.

The sibling tests are guarded and therefore pass everywhere: split_between_lanes skips via offload_proto_selected(), short_elements_get_sw only excludes the offload proto (PROTOS=^get/sgl/offload), and the generic test_sgl skips when no GET_ZCOPY lane exists (which is why the tcp/... instantiation was skipped rather than failed). test_ucp_rma_sgl is instantiated with UCP_INSTANTIATE_TEST_CASE_TLS(test_ucp_rma_sgl, all, "all") (test_ucp_rma.cc:2269), so the test runs on non-mlx5 hardware.

Implicated commit: [REDACTED:Hex High Entropy String] — "UCP/TEST: Add GET SGL short element layout and failure tests", Michal Shalev (feature it tests: 1310a95, same author)

File: test/gtest/ucp/test_ucp_rma.cc:1905-1947 (test body), test_ucp_rma.cc:1911 + :1524 (first failing assert); related: src/ucp/rma/get_offload.c:280-285, src/ucp/proto/proto_multi.c:879-885

Suggested fix: Guard short_element_sync_failure on actual HW support before any assertion, e.g.:

  • Add a skip condition for a lane advertising UCT_IFACE_FLAG_V2_GET_SGL_ZCOPY and cap.get.min_zcopy > 8 (the element size used), similar to test_sgl's has_lane_with_caps() check; or
  • Post the wireup SGL op first and reuse the existing helper: if !offload_proto_selected(sptr) → request_wait + UCS_TEST_SKIP_R("SGL offload protocol was not selected"), exactly as split_between_lanes (test_ucp_rma.cc:1715-1719) does; or
  • Move this test into a sub-class instantiated only for mlx5 TLs (rc_x/dc_x) instead of "all".

Also make the leading test_sgl(1, UCS_KBYTE) wireup call tolerant of the restricted-PROTOS case so it skips instead of asserting.

Related: PR #12191 (this branch, ucp-get-sgl); prerequisite commits 1310a95, 83c09db, e66f515; UCT capability from "UCT/IB/MLX5/RC: Implement get_sgl_zcopy (#11763)"

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 2c2d0631-767b-4d50-a6ef-aaa58163d8ba in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
Comment thread src/ucp/proto/proto_multi.h Resolved
@svc-ucx

svc-ucx commented Oct 29, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Coverity coverity release on coverity_rh7) · commit 36d2668b

TL;DR: The Coverity gate failed with 4 new FORWARD_NULL defects caused by the new static callback ucp_proto_get_sgl_short_elem_completion() in this PR, which dereferences user_data unconditionally; Coverity's function-pointer model then resolves param->cb.send in the legacy ucp_tag_send_nb/ucp_tag_send_nbr/ucp_tag_send_sync_nb/ucp_stream_send_nb wrappers (where param.user_data is implicitly NULL) to it. Add a ucs_assert(user_data != NULL) (or explicit NULL guard) at the top of the callback to break the false path.

Full analysis

Summary: coverity release on coverity_rh7 exited with code 4 — cov-format-errors reported 4 new "Explicit null dereferenced (FORWARD_NULL)" issues, so the pipeline step failed (##[error]Coverity found 4 issues).

Root cause: Not a build/test failure — a static-analysis regression introduced by this PR. src/ucp/rma/get_offload.c:290-298 adds:

static void ucp_proto_get_sgl_short_elem_completion(void *request, ucs_status_t status, void *user_data)
{
    ucp_request_t *req = user_data;
    ucp_request_free(request);
    ucp_invoke_uct_completion(&req->send.state.uct_comp, status);  /* unconditional deref */
}

It is assigned to param.cb.send (get_offload.c:308) with user_data = req. Because cov-analyze runs with --enable-fnptr --fnptr-models, Coverity adds this new function as a resolution target for every indirect param->cb.send invocation — specifically ucp_request_imm_cmpl_param() at src/ucp/proto/proto_common.inl:283. The legacy wrappers build a ucp_request_param_t on the stack with a callback but no UCP_OP_ATTR_FIELD_USER_DATA, so param.user_data is zero-initialized to NULL:

  • src/ucp/tag/tag_send.c:184 (ucp_tag_send_nb) → defect at :190
  • src/ucp/tag/tag_send.c:198 (ucp_tag_send_nbr) → defect at :206
  • src/ucp/tag/tag_send.c:222 (ucp_tag_send_sync_nb) → defect at :228
  • src/ucp/stream/stream_send.c:99 (ucp_stream_send_nb) → defect at :108

Coverity therefore reports NULL user_data flowing into the new callback and being dereferenced at src/ucp/core/ucp_request.inl:1004 (ucp_invoke_uct_completion → comp). This is a false positive at runtime (those paths never dispatch to the SGL callback), but it is a hard CI gate.

Implicated commit: PR #12191 branch ucp-get-sgl, HEAD [REDACTED:Hex High Entropy String] (Michal Shalev); the SGL short-element code is touched by a33a3ff3, 7707911e, 3cc34f95 (all Michal Shalev, 2026-10-28).

File: src/ucp/rma/get_offload.c:290-298 (deref at :294/:297); reported call sites src/ucp/tag/tag_send.c:190,206,228 and src/ucp/stream/stream_send.c:108.

Suggested fix: Make the NULL impossibility explicit so Coverity prunes the modeled path. ucs_assert is compiled in under __COVERITY__ (see src/ucs/debug/assert.h:54), so it is both free in release builds and visible to the analyzer:

static void ucp_proto_get_sgl_short_elem_completion(void *request, ucs_status_t status, void *user_data)
{
    ucp_request_t *req = user_data;

    ucs_assert(req != NULL);
    ucp_request_free(request);
    ucp_invoke_uct_completion(&req->send.state.uct_comp, status);
}

Alternatives if the assert is not sufficient: (a) annotate the modeled call with /* coverity[var_deref_model] */ next to ucp_request_imm_cmpl_param() in src/ucp/proto/proto_common.inl:283 (same style as the existing /* coverity[offset_free] */ on line 282) — though this suppresses broadly; or (b) explicitly set .user_data = NULL is already the case, so instead set UCP_OP_ATTR_FIELD_USER_DATA is not applicable — prefer the assert in the new callback since it is the code this PR added.

Related: PR #12191 (this PR). No pre-existing issue found for this Coverity signature.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 95e9e4a6-4e8a-413f-b5df-cb716016555e in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/proto/proto_multi.inl Resolved
Comment thread src/ucp/rma/get_offload.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: no test forces PROTOS=get/sgl/offload_sw.

Residual coverage gap: no test covers max_frag < min_sgl_frag. Note the same ucs_assertv(diff <= frag_offset) can also be reached with asymmetric lanes (one lane's lpriv->max_frag below another lane's min_zcopy), not only via RMA_ZCOPY_MAX_SEG_SIZE; in a release build this walks the buffer pointer before the element start.

Residual coverage gap: test_ucp_rma_sgl_err skips non-host memory because a short GPU get goes through rendezvous and asserts on force close.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/get_offload.c Resolved
Comment thread src/ucp/rma/put_offload.c Resolved
Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_request.inl Resolved
Comment thread src/ucp/proto/proto_multi.inl Resolved
Comment thread src/ucp/proto/proto_multi.inl Resolved
Comment thread src/ucp/rma/get_offload.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/api/ucp_def.h Resolved
Comment thread src/ucp/api/ucp_def.h Resolved
Comment thread src/ucp/rma/get_offload.c Resolved
Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
@svc-ucx

svc-ucx commented Oct 29, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 1) · commit 54d5e142

TL;DR: The new gtest all/test_ucp_rma_sgl.short_and_long_elements aborts in flush_ep() on ucs_assertv(comp->count > 0) — a UCT RC flush op completed against a UCP ep-flush request whose completion counter was already 0, i.e. a UCP uct_completion_t is being released more times than it was referenced (stale/over-released completion) by the brand-new "short GET-SGL element via nested sub-request" path. Fix the reference accounting in ucp_proto_get_sgl_short_elem() (and make the parent request un-completable/un-abortable while sub-requests are outstanding) rather than touching the flush code.

Full analysis

Summary: roce on worker 1 gtest job aborted (core dumped) during all/test_ucp_rma_sgl.short_and_long_elements/1 <all/host/get>: uct_iface.h:1012 Assertion 'comp->count > 0' failed: comp=0x55e0e0d75498 (ucp_ep_flush_completion) count=0 status=0, make: *** [Makefile:4971: test] Aborted (core dumped).

Root cause: From the backtrace, the completion was invoked by uct_rc_ep_flush_op_completion_handler() (rc_ep.c:371) → uct_invoke_completion() while handling a TX CQE, on the completion of a UCP endpoint-flush request (ucp_ep_flush_completion) that had already reached count == 0. The crash site is ucp_test::flush_ep() → test_ucp_rma_sgl::test_sgl() (test_ucp_rma.cc:1541) called from short_and_long_elements (test_ucp_rma.cc:1769). UCP's own flush bookkeeping is one-completion-per-started-lane (ucp_ep_flush_request_update_uct_comp(), flush.c:21-48, 185-188) and a duplicate flush on the same lane would have tripped the started_lanes assertion first — so the extra decrement comes from a completion reference that outlived its owning request (the request slot was then recycled from the worker request mpool by the subsequent ucp_ep_flush_nbx).

The only code exercised exclusively by this test is the new short-element path: short_and_long_elements mixes elements below the iface cap.get.min_zcopy (1, 7, 3, 64 bytes) with normal ones, so it is the only test that drives ucp_proto_get_sgl_adjust_min_frag() → ucp_proto_get_sgl_short_elem() interleaved with real uct_ep_get_sgl_zcopy() posts (short_elements/many_short_elements use only short elements, multi_rail/with_callback only long ones — all of those passed in the same run). That function hand-rolls the parent's completion reference count outside the proto_multi framework:

  • ++req->send.state.uct_comp.count before a nested ucp_get_nbx() on the same endpoint, issued from inside the parent request's own progress/send callback (get_offload.c:316-318);
  • --req->send.state.uct_comp.count on the non-pointer return path (get_offload.c:325);
  • the matching release happens later in ucp_proto_get_sgl_short_elem_completion(), which does ucp_request_free(request) and then ucp_invoke_uct_completion(&req->send.state.uct_comp, status) (get_offload.c:290-299).

Nothing protects the parent from being completed/freed while those sub-request references are outstanding: if ucp_proto_rma_sgl_offload_send_frag() later returns an error, ucp_proto_multi_handle_send_error() → ucp_proto_request_abort() (proto_multi.inl:133-143) completes and frees the parent regardless of uct_comp.count, leaving the already-posted sub-requests and in-flight get_sgl_zcopy UCT ops pointing at a recycled uct_completion_t. That is exactly the signature observed: a live completion function (ucp_ep_flush_completion) with count == 0.

Implicated commit: 1310a95b "UCP/RMA: Support short elements in GET SGL via sub-requests" — Michal Shalev (test added in 1b00c0ad, PR #12191, head 54d5e142)

File: src/ucp/rma/get_offload.c:290-356 (ucp_proto_get_sgl_short_elem_completion, ucp_proto_get_sgl_short_elem, ucp_proto_get_sgl_adjust_min_frag); failing test at test/gtest/ucp/test_ucp_rma.cc:1768

Suggested fix:

  1. Make the sub-request reference accounting single-sourced: set UCP_OP_ATTR_FLAG_NO_IMM_CMPL on the nested ucp_get_nbx() param so the callback is always invoked exactly once, and drop the ++/-- pair around the call (today three different code paths — PTR, immediate-OK, immediate-error — must each get the count right).
  2. Guarantee the parent cannot be completed or aborted while sub-requests are outstanding: in ucp_proto_get_sgl_offload_*, route abort/error through a path that waits for uct_comp.count to drain (e.g. only uct_completion_update_status() + let the last sub-request completion finish the request), and add ucs_assertv(req->send.state.uct_comp.count == 0, ...) in the zcopy abort/complete path for SGL protocols to catch this in CI.
  3. Preferably avoid re-entering the public ucp_get_nbx() from inside the parent's progress callback at all — post the short elements from the parent request itself (e.g. uct_ep_get_bcopy/the existing get/sgl/offload_sw single-element post) so that one uct_completion_t owns all fragments.
  4. To confirm locally: ./gtest --gtest_filter=all/test_ucp_rma_sgl.short_and_long_elements* with UCX_LOG_LEVEL=trace_req (watch the "flush update ... comp_count" trace) and under valgrind/ASAN — the recycled-request use-after-free should show up directly.

Related: PR #12191 (this PR, "UCP/RMA: Add SGL support to ucp_get_nbx"); flush accounting context in PR #11923 ("UCP/FLUSH: Add topology-safe selective endpoint flush"); UCT side already merged in #11763 ("UCT/IB/MLX5/RC: Implement get_sgl_zcopy")

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Nov 2, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Build_Static build_source centos7) · commit f55427a7

TL;DR: The centos7 build_source job failed compiling src/ucp/ucp because GCC 4.8 on CentOS 7 raised -Werror=maybe-uninitialized for the buffers/lengths/remote_addrs arrays declared in proto_multi.inl and passed by address into the new SGL adjust path used by get_offload.c/put_offload.c; initialize those arrays (or at least element 0) to silence the old-compiler warning.

Full analysis

Summary: make failed with cc1: all warnings being treated as errors while compiling src/ucp/rma/get_offload.c and src/ucp/rma/put_offload.c (make[2] Error 1 → make Error 2).

Root cause: On the CentOS 7 image (old GCC, -O3 -Wall -Werror) the compiler cannot prove that the stack arrays declared in ucp_proto_rma_sgl_offload_send_frags():

proto_multi.inl:417:  void *buffers[UCP_MAX_IOV];
proto_multi.inl:418:  size_t lengths[UCP_MAX_IOV];
proto_multi.inl:419:  uint64_t remote_addrs[UCP_MAX_IOV];

are written before use. They are filled by ucp_datatype_iter_next_sgl_frags() whose fill count is only known via the returned frag_count, and then &buffers[0] / &lengths[0] / &remote_addrs[0] are handed to adjust_func (the new ucp_proto_{get,put}_sgl_offload_sw_progress adjust callbacks). After inlining into get_offload.c / put_offload.c, GCC's value tracking loses the frag_count == 0 early-return correlation and emits:

  • get_offload.c:353 'remote_addrs' may be used uninitialized
  • get_offload.c:340 'lengths' may be used uninitialized
  • compiler_def.h:112 (via UCS_PTR_BYTE_OFFSET) 'buffers' may be used uninitialized
  • the same three in put_offload.c:460 / ucp_worker.inl:103

This is a false positive from the old compiler, not a real logic bug — but -Werror makes it fatal. It is new code: proto_multi.inl was touched by the SGL series (4af0b6b5 "UCP: Split an SGL put between the lanes by bytes", 54d5e142 "Drop redundant max fragment arg from SGL send frag") and the failure appears on the merge commit f55427a7.

Implicated commit: [REDACTED:Hex High Entropy String] (Michal Shalev, merge of master into ucp-get-sgl); the warning originates from the SGL offload work in 4af0b6b / 54d5e14 (Michal Shalev)

File: src/ucp/proto/proto_multi.inl:417-419 (triggered from src/ucp/rma/get_offload.c:340,353 and src/ucp/rma/put_offload.c:460)

Suggested fix: Give the compiler a definite initialization at the declarations in ucp_proto_rma_sgl_offload_send_frags(), e.g.:

void *buffers[UCP_MAX_IOV]        = {NULL};
size_t lengths[UCP_MAX_IOV]       = {0};
uint64_t remote_addrs[UCP_MAX_IOV] = {0};

Only the first element is strictly needed for the adjust_func (max_frag_count == 1) path, so initializing just buffers[0]/lengths[0]/remote_addrs[0] before the ucp_datatype_iter_next_sgl_frags() call is the zero-cost variant. Alternatively wrap the declarations with UCX's existing "pretend-initialized" idiom (UCS_V_INITIALIZED) rather than disabling -Wmaybe-uninitialized. Verify with the centos7 container, since newer GCC used by the other legs does not reproduce this.

Related: PR #12191 (ucp-get-sgl); predecessor PR #11947 ("UCP: Split an SGL put between the lanes by bytes") introduced the array-passing pattern. No existing issue found for this warning.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 5810bc80-784e-48fd-90b3-51a201f6da3d in the triage console for the audit trail.

Comment thread src/ucp/rma/get_offload.c Resolved
Comment thread src/ucp/rma/get_offload.c Resolved
@qierli

qierli commented Nov 4, 2026

Copy link
Copy Markdown

Hi @michal-shalev, thanks for this. We run SGL gets in production for NIXL KV-cache reads (on our own UCX patch) and tested this PR at 1187eaa on B300 with rc_mlx5:

  • On an endpoint with no GET_ZCOPY lane (e.g. UCX_TLS=tcp), an SGL get fails the whole endpoint: cannot find remote protocol ... get into sgl host memory from host, then the error callback fires with UCS_ERR_ENDPOINT_TIMEOUT. Contig gets there work through get/am. Could SGL get have an AM fallback like put/sgl/am, or fail only the request (e.g. UCS_ERR_UNSUPPORTED)? Happy to send a follow-up PR.
  • Perf against get/sgl/offload_sw: get/sgl/offload is faster for small elements (102K × 4 KiB, GPU→GPU: 45 vs 55 ms) but about 10% slower for 40–120 KiB elements read from GPU memory (48K × 40 KiB: 121 vs 111 ms).

Are you planning the NIXL side (READs through sglXfer, like ai-dynamo/nixl#1835)? If not, we have it drafted against this branch and can send it.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/proto/proto_multi.c Resolved
Comment thread src/ucp/proto/proto_multi.h Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

No test forces PROTOS=get/sgl/offload_sw; on IB-only runs rc_mlx5 advertises UCT_IFACE_FLAG_V2_GET_SGL_ZCOPY, so the sw path is only hit incidentally via self/shm.

max_frag < min_sgl_frag is still unfiltered at probe time. A second symptom of the same root cause: with elem_len=1000, lane max_payload=20, min_zcopy=33, the second fragment passes ucs_assertv(diff <= frag_offset) but is posted with length = 33 > max_frag = 20, which UCT_CHECK_LENGTH rejects and the request aborts with UCS_ERR_INVALID_PARAM.

Short elements recurse into full protocol selection, so on GPU memory without GPUDirect each short element becomes a rendezvous operation.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/get_am.c Resolved
Comment thread src/ucp/rma/rma.inl Resolved
Comment thread src/ucp/rma/get_am.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: the emulation tests force PROTOS=...,get/sgl/am/*,get/am/*,reconfig, so the nested per-element get is pinned to AM bcopy. No test exercises get/sgl/am/bcopy with unrestricted nested selection (where an element could pick zcopy or rendezvous) — exactly the case where the nested selection, and the GPU rendezvous path called out in earlier rounds, matter.

Previously raised and still open: max_frag < min_sgl_frag has no probe-time filter or test; no test forces PROTOS=get/sgl/offload_sw; test_ucp_rma_sgl_err skips non-host memory.

@michal-shalev

Copy link
Copy Markdown
Contributor Author

Hi @qierli, I couldn't reproduce the performance issue you reported. I've pushed some changes. You can retest with the latest commit.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/get_offload.c Resolved
Comment thread src/ucp/rma/get_am.c Resolved
Comment thread src/ucp/rma/rma.inl Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Coverage gap: no test forces PROTOS=get/sgl/offload_sw; on IB-only runs rc_mlx5 advertises UCT_IFACE_FLAG_V2_GET_SGL_ZCOPY, so the sw path is only hit incidentally via self/shm.

Coverage gap: no test covers max_payload < min_sgl_frag (reachable with weight-scaled multi-lane payloads, not only RMA_ZCOPY_MAX_SEG_SIZE), where ucs_assertv(diff <= frag_offset) in ucp_proto_get_sgl_adjust_min_frag() fires and a release build walks the local buffer pointer before the element start — a GET writes there, so this is a local out-of-bounds write.

Coverage gap: the emulation tests pin the nested per-element get to get/am/*, so unrestricted nested selection (zcopy/rendezvous per element, including the GPU rendezvous path) stays untested.

@michal-shalev
michal-shalev marked this pull request as ready for review November 16, 2026 10:03
@michal-shalev michal-shalev added Ready for Review and removed WIP-DNM Work in progress / Do not review labels Nov 16, 2026
@svc-ucx

svc-ucx commented Nov 16, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 1) · commit e2f80e38

TL;DR: all/test_ucp_rma_sgl_data.various_lengths/1 <all/host/get/host> aborted on ucs_assertv(comp->count > 0) in uct_invoke_completion() — the flush request's completion had already been driven to 0 by a stale completion coming from the new "short SGL element" nested ucp_get_nbx path added in this PR, so the real UCT flush CQE then decremented an already-finished (recycled) completion. Fix the parent-request reference/counting in ucp_proto_get_sgl_short_elem() (or stop issuing a nested UCP request for short elements).

Full analysis

Summary: Azure build 145271, job "roce on worker 1", gtest all/test_ucp_rma_sgl_data.various_lengths/1 crashed with Assertion 'comp->count > 0' failed: comp=0x[REDACTED:Hex High Entropy String] (ucp_ep_flush_completion) count=0 status=0 → make: *** [Makefile:5015: test] Aborted (core dumped).

Root cause: From the backtrace, the assertion fires in uct_rc_ep_flush_op_completion_handler() (src/uct/ib/rc/base/rc_ep.c:370) while the test is inside test_ucp_rma_sgl::test_sgl() → flush_ep() (test/gtest/ucp/test_ucp_rma.cc:1711). The completion object belongs to a UCP flush request (comp->func == ucp_ep_flush_completion) and its count is already 0 before the lane's flush op completes. ucp_ep_flush_progress() initializes the count to the number of lanes and only ever decrements per started lane (src/ucp/rma/flush.c:119-200), so a second decrement must come from outside the flush flow — and the only new out-of-band decrementer on this branch is the SGL-GET short-element path added by this PR:

/* src/ucp/rma/rma.inl:189-228 */
static inline void ucp_proto_get_sgl_short_elem_completion(void *request, ucs_status_t status, void *user_data) {
    ucp_request_t *req = user_data;          /* raw, unprotected parent pointer */
    ucp_request_free(request);
    ucp_invoke_uct_completion(&req->send.state.uct_comp, status);
}
...
++req->send.state.uct_comp.count;
status_ptr = ucp_get_nbx(req->send.ep, buffer, length, remote_addr, ...);

various_lengths uses element sizes {64, 256, 1K, 4K, 512}; the 64-byte element is below cap.get.min_zcopy on rc_mlx5, so it is routed through ucp_proto_get_sgl_adjust_min_frag() → ucp_proto_get_sgl_short_elem() (src/ucp/rma/get_offload.c:297-323), which posts a nested ucp_get_nbx whose callback dereferences the parent request. If the parent SGL request is completed/released while that nested GET is still outstanding, the callback later decrements uct_comp of a recycled request slot. The test releases the SGL request and immediately calls flush_ep(), so the LIFO request mpool hands the same memory to the flush request — the stale callback goes through ucp_invoke_uct_completion() (which only ucs_assertvs count > 0, and 1→0 passes silently), prematurely completes the flush, and the subsequent genuine flush CQE then trips the hard assert in uct_invoke_completion(). Nothing in the UCT/flush code changed in this build; the only new code on the path is e2f80e38.

Implicated commit: [REDACTED:Hex High Entropy String] — "UCP/RMA: Add SGL support to ucp_get_nbx", Michal Shalev (PR #12191)

File: src/ucp/rma/rma.inl:189-228 (ucp_proto_get_sgl_short_elem() / ucp_proto_get_sgl_short_elem_completion()), used from src/ucp/rma/get_offload.c:297-323; failing test at test/gtest/ucp/test_ucp_rma.cc:1894

Suggested fix:

  1. Make the nested short-element GET hold a real reference to its parent instead of a bare pointer: link it with ucp_request_set_super() / a request id, and guarantee the parent's uct_comp reference taken by ++count cannot be dropped before the nested request completes (also cover the abort/reset paths — ucp_proto_request_zcopy_abort / ucp_proto_get_offload_zcopy_reset currently reinitialize the parent state without cancelling or accounting for outstanding nested GETs, which produces exactly this dangling reference).
  2. Add a debug check in ucp_proto_get_sgl_short_elem_completion() that the parent is still the expected SGL request (e.g. ucs_assert(req->send.state.uct_comp.func == ucp_proto_request_zcopy_completion) and count > 0) so corruption is caught at the source rather than in an unrelated flush.
  3. Preferred simplification: avoid a nested UCP request entirely — post short elements from the SGL protocol itself with &req->send.state.uct_comp (e.g. uct_ep_get_bcopy/get_zcopy on the selected lane), or extend the fragment backwards to min_zcopy in the frag_offset == 0 case too, as already done for frag_offset != 0.
  4. Re-run test_ucp_rma_sgl_data.various_lengths (and various_sizes with the 1-byte element) on an rc_mlx5/roce node, which deterministically exercises the sub-min_zcopy element path.

Related: PR #12191 (this change); same-area predecessors #12283, #12226, #11947, #11943; UCT counterpart #11763 (UCT/IB/MLX5/RC: Implement get_sgl_zcopy)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id a882b624-8613-4e28-86cc-da72e75c3958 in the triage console for the audit trail.

@tvegas1 tvegas1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wdyt we handle short elements once per request, and reuse the put SGL send code for get? IMO it fixes the CI crash and removes most of the get-specific code.

Proposal:

  • probing can stay as it is (get zcopy probe-param with UCT_IFACE_FLAG_V2_GET_SGL_ZCOPY, like put). Only get/sgl/contig passes NULL instead of &ucp_get_am_bcopy_proto, so get/am/bcopy can be used per element (this also keeps TCP-only endpoints working) and get/sgl/am/bcopy can be removed
  • replace the adjust callback with a min_frag value (cap.get.min_zcopy for get, 0 for put). A split fragment shorter than min_frag grows to a min_frag window inside its element, starting at min(frag_offset, elem_length - min_frag). It always fits, so get can batch exactly like put, without the one-fragment, no-redirect and UCS_INPROGRESS special cases
  • at the first progress call, before registering memory, check if some element is shorter than the lanes' min_zcopy. If so, use ucp_proto_rma_sgl_contig_progress(), extended to send each run of long elements as one internal SGL get (its descriptor points into the user arrays) and each short element as a contig get. The parent only counts child requests, so it never goes pending and is never restarted
  • then ucp_proto_get_sgl_short_elem() and ucp_proto_get_sgl_adjust_min_frag() can be removed

This also removes the restart issue from the inline comment in rma.inl and the possible out-of-bounds access behind the diff <= frag_offset assert, passes the memh for short elements, and works for GPU memory.

A list where short and long elements alternate would be read per element; UCX_IB_TX_INLINE_RESP=0 avoids short elements completely.

Comment thread src/ucp/rma/rma.inl
};
ucs_status_ptr_t status_ptr;
ucs_status_t status;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker: on restart (UCX_PROTO_REQUEST_RESET=y, which CI sets) the reset sets uct_comp.count back to 1 while this nested request is still running, so the parent completes too early and is reused. This is the stale completion behind the comp->count > 0 abort in the CI triage comment; see the PR-level comment for a way to avoid the nested request.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, it sounds like a problem.
Can short elements be posted on the lane with the parent's uct_comp?

}

static void get_test_variants(std::vector<ucp_test_variant>& variants) {
get_op_variants(variants, UCS_MASK(2));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls add a variant of various_lengths with IB_TX_INLINE_RESP=0 (full HW path), and tests for a large element with a short last fragment, for short and long elements interleaved, and for get/sgl/offload_sw

Comment on lines +2213 to +2215
static void get_test_variants(std::vector<ucp_test_variant>& variants) {
get_op_variants(variants, UCS_MASK(2));
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is the same as in test_ucp_rma_sgl class, so can be dropped and used from base class

Comment thread src/ucp/rma/get_offload.c
Comment on lines +293 to +294
return ucp_ep_get_iface_attr(req->send.ep, lpriv->super.lane)
->cap.get.min_zcopy;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ucp_proto_get_sgl_min_frag() looks up cap.get.min_zcopy through the ep config and the worker ifaces on every send call. Could it be cached per lane like opt_align (an offset in ucp_proto_multi_init_params_t, read once in ucp_proto_multi_init_priv()), next to max_sgl_zcopy_count?

Comment thread src/ucp/rma/rma.inl
};
ucs_status_ptr_t status_ptr;
ucs_status_t status;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, it sounds like a problem.
Can short elements be posted on the lane with the parent's uct_comp?

Comment on lines 519 to +525
* copying them */
if (ucs_unlikely((total_length < max_payload) &&
if (ucs_unlikely((adjust_func == NULL) && (total_length < max_payload) &&
(elem_count < max_frag_count) &&
((start_index + elem_count) < dt_iter->length))) {
return ucp_proto_rma_sgl_offload_send_frags(req, lpriv, max_frag_count,
next_iter, post_func);
return ucp_proto_rma_sgl_offload_send_frags(req, lpriv, frags_count,
min_frag, next_iter,
post_func, adjust_func);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For GET, one send never combines the end of one element with the start of the next: the batch is not redirected to ]ucp_proto_rma_sgl_offload_send_frags()], and that function sends only one fragment for GET. So the lanes get uneven shares: 3x1 MiB on 4 rails goes 1.5/0.5/0.75/0.25 MiB instead of 0.75 MiB each.

PUT splits it evenly here:

frag_length = ucs_min(elem_length - frag_offset,
max_total_length - total_length);

Is that intended? If not, GET can cut the element the same way, as long as the cut piece is at least min_zcopy. split_uneven_elements has exactly this case, but it only checks that all lanes are used.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants