Skip to content

UCP/RMA: Emulate SGL PUT with contiguous PUTs on Vera Rubin - #12283

Merged
michal-shalev merged 24 commits into
openucx:masterfrom
michal-shalev:vera-rubin-sgl-put
Nov 14, 2026
Merged

michal-shalev merged 24 commits into
openucx:masterfrom
michal-shalev:vera-rubin-sgl-put

Conversation

@michal-shalev

Copy link
Copy Markdown
Contributor

Why

On Vera Rubin, localized CUDA memory SGL PUT falls back to put/sgl/am/bcopy, which is slow.

What

Add put/sgl/contig. It splits an SGL PUT into one contiguous PUT per element.
It is only picked when no native SGL protocol is available.

How

  • Each element is sent with the normal contiguous PUT path, so it can use put/rndv or staging.
  • The SGL request completes when all element PUTs complete. The first error is kept.
  • put/sgl/offload and put/sgl/offload_sw set fallback_class, so this protocol is not picked when they work.

@michal-shalev michal-shalev self-assigned this Nov 9, 2026
@michal-shalev
michal-shalev requested a review from a team as a code owner November 9, 2026 15:46
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

pls add a test for the new protocol: test_ucp_rma_sgl* forces PROTOS=put/sgl/am/* and test_ucp_rma_rndv forces put/rndv,get/rndv,rndv/*, and with default config put/sgl/offload_sw (now declaring fallback_class = RMA_RNDV) suppresses the new proto wherever put_zcopy exists — so put/sgl/contig is never selected in CI.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Coverage note: test_ucp_rma_sgl_contig only covers CUDA with UCX_IB_GPU_DIRECT_RDMA=n and forces PROTOS=put/sgl/contig,put/rndv,rndv/*, so the two new fallback_class declarations in put_offload.c (i.e. that put/sgl/offload* actually suppress put/sgl/contig) are not exercised by any test, and the protocol actually selected is not asserted (e.g. via PROTO_INFO=used).

@tvegas1 tvegas1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have some perf numbers vs the previous put/sgl/am/bcopy fallback?

  • typical SGL element count and sizes in the target workload
  • bw/latency of am/bcopy vs put/sgl/contig for a few counts and sizes, especially small elements
  • UCX_PROTO_INFO=used output

Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
@svc-ucx

svc-ucx commented Nov 10, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 2) · commit 236f2e70

TL;DR: The ASAN "roce on worker 2" job failed with a single test failure, ib/test_md.alloc/1 (mlx5_1), which is the IB DEVX device-memory (on-chip DM / UCS_MEMORY_TYPE_RDMA) allocation test — a scarce, device-global resource shared by the parallel gtest workers; nothing in PR #12283 (UCP RMA SGL-PUT emulation) touches that path, so this is an unrelated/flaky infra failure. Re-run the job; if it recurs, make test_md.alloc tolerant of DM exhaustion (skip or treat UCS_ERR_NO_MEMORY on UCS_MEMORY_TYPE_RDMA as a skip instead of counting it toward the 50 % failure budget).

Full analysis

Summary: make test in ./contrib/test_jenkins.sh exited non-zero because 1 of 9006 gtests failed: [ FAILED ] ib/test_md.alloc/1, where GetParam() = mlx5_1 (all other tests passed, including every other mlx5_1 test).

Root cause: For the IB component, UCT_MD_FLAG_ALLOC is only advertised by the DEVX md after uct_ib_mlx5dv_check_dm_ksm_reg() succeeds, and uct_ib_mlx5_devx_md_query() then reports alloc_mem_types |= UCS_BIT(UCS_MEMORY_TYPE_RDMA) with max_alloc = dev_attr.max_dm_size (src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:3183-3188). So test_md.alloc for ib/mlx5_1 performs 300 iterations of ibv_alloc_dm()-backed allocations of on-chip device memory (uct_ib_mlx5_devx_device_mem_alloc, src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:2128-2230). That memory is a device-global resource, limited to max_dm_size and shared with the other gtest worker processes running concurrently on the same host under ASAN; when ibv_alloc_dm() fails the path returns UCS_ERR_NO_MEMORY, which the test counts and fails on once more than 50 % of iterations fail (EXPECT_LT((double)num_alloc_failures / iterations, 0.5) << "Too many OUT_OF_RESOURCE failures", test/gtest/uct/test_md.cc:559). The exact assertion line is not recoverable — the Azure task log is only retrievable as the last ~500 lines and the failure happened much earlier in the 2338-second run — but the end-of-run summary shows no other failure, no crash, no ASAN report, and no hang (log timestamps are continuous, ~200 ms per test, right up to the summary). Critically, PR #12283 ("UCP/RMA: Emulate SGL PUT with contiguous PUTs on Vera Rubin") changes UCP RMA protocol code only; it does not touch uct_ib_md.c, the DEVX md, DM allocation, or mkey packing, so it cannot plausibly affect this test.

Implicated commit: none for the PR under test; the test/feature area originates from #9308 "UCT/IB: Added support for device memory allocation (memic/rdma)". The failure is environmental, not caused by commit [REDACTED:Hex High Entropy String].

File: test/gtest/uct/test_md.cc:479-561 (test_md.alloc, failure check at line 559); allocation path src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:2156-2180

Suggested fix:

  1. Re-trigger the AddressSanitizer roce job — treat this as a known flake for this PR, not a regression.
  2. To stop the flake recurring, harden test_md.alloc for device memory: when mem_type == UCS_MEMORY_TYPE_RDMA, either UCS_TEST_SKIP_R("no device memory available") once ibv_alloc_dm starts failing, or exclude DM failures from the num_alloc_failures / iterations < 0.5 budget, since on-chip DM is a shared, device-wide resource and concurrent gtest workers on the same HCA will legitimately exhaust it. Optionally cap max_size for RDMA memory at a small fraction of max_dm_size (e.g. max_alloc / 64) so parallel workers don't starve each other.
  3. For faster future triage, have the CI publish the gtest output as an artifact (or print the [ FAILED ] detail block again at the end), since only the log tail is retrievable from Azure.

Related: PR #12283 (build under test), PR #9308 (introduced IB device-memory alloc + this test coverage)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id b447e06c-9e89-427d-9e07-8759301f86e2 in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
Comment thread src/ucp/rma/rma_rndv.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Nov 11, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Build build_aarch64 azurelinux40_aarch64) · commit 0d7a0ae4

TL;DR: The aarch64 build failed to compile test/gtest/ucp/test_ucp_rma.cc because two new tests still call the old helper name test_put_sgl, which was renamed to test_sgl on master; rename the two call sites (lines 2223, 2227) to test_sgl.

Full analysis

Summary: Build on aarch64 step failed with make[3]: *** [Makefile:2861: ucp/gtest-test_ucp_rma.o] Error 1 — a hard C++ compile error in the gtest UCP RMA test file (not a timeout, not a flaky test).

Root cause: Compiler output:

test/gtest/ucp/test_ucp_rma.cc:2223:5: error: 'test_put_sgl' was not declared in this scope; did you mean 'test_sgl'?
     test_put_sgl({64, 256, UCS_KBYTE, 4 * UCS_KBYTE, 512});
test/gtest/ucp/test_ucp_rma.cc:2227:5: error: 'test_put_sgl' was not declared in this scope; did you mean 'test_sgl'?
     test_put_sgl(10, UCS_KBYTE, true, true);

With -Werror this aborts the build. The PR branch vera-rubin-sgl-put added test_ucp_rma_sgl_contig tests (commits 236f2e7 "GTEST/UCP: Add test for put/sgl/contig" and cf372cb "Check put/sgl/contig is the selected protocol") that call test_put_sgl(...). Meanwhile master commit [REDACTED:Hex High Entropy String] ("TEST/UCP: Add operation variant to RMA SGL tests", #12277) renamed that base-class helper from test_put_sgl to test_sgl (all other call sites in the file, e.g. lines 1846–1954, now use test_sgl). The merge commit 2d46ac2 ("Merge branch 'master' into vera-rubin-sgl-put") took both changes but did not update the two new call sites — a textually-clean but semantically-broken merge.

Implicated commit: 2d46ac2 (merge of master into vera-rubin-sgl-put, Michal Shalev) combined with the rename in [REDACTED:Hex High Entropy String] (Michal Shalev, PR #12277); the stale call sites come from 236f2e7 / cf372cb.

File: test/gtest/ucp/test_ucp_rma.cc:2223 and test/gtest/ucp/test_ucp_rma.cc:2227

Suggested fix: Update the two stale calls to the renamed helper:

UCS_TEST_P(test_ucp_rma_sgl_contig, put) {
    test_sgl({64, 256, UCS_KBYTE, 4 * UCS_KBYTE, 512});
}

UCS_TEST_P(test_ucp_rma_sgl_contig, put_with_callback) {
    test_sgl(10, UCS_KBYTE, true, true);
}

Verify the argument lists still match the new test_sgl signature (it gained the op-variant handling in #12277 — the overloads used at lines 1871 and 1875 are identical in shape, so no further changes should be needed), then re-run the build. A quick grep -n test_put_sgl test/ before pushing will catch any other leftovers from the merge.

Related: PR #12283 (this branch), PR #12277 (the rename that introduced the conflict).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 7ad50f0d-8953-40f1-a4c7-204e0c43690e in the triage console for the audit trail.

Comment thread test/gtest/ucp/test_ucp_rma.cc Resolved
Comment thread src/ucp/rma/put_offload.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/put_offload.c Resolved
Comment thread src/ucp/rma/put_offload.c Resolved
Comment thread src/ucp/rma/put_offload.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: test_ucp_rma_sgl_contig::check_posted_request() only asserts the selected protocol when the operation is still in progress; if all elements complete inline (likely for the small 64/256/512-byte elements on the host variant) the parent also completes inline and the put/sgl/contig assertion is skipped. PROTO_INFO=used (as in test_ucp_rma_gpu_nic) would assert it unconditionally.

@svc-ucx

svc-ucx commented Nov 12, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests BlueField on worker 0) · commit f8284bbd

TL;DR: rc_verbs/test_rc.stress_iface_ops/1 hung for 15 minutes inside uct_iface_flush() and was killed by the gtest watchdog ("Connection timed out - abort testing"); the hang is in the brand-new rc_verbs no-SRQ path (UCX_RC_VERBS_SRQ_ENABLE=n task added one day earlier by commit 75371a0), not in PR #12283, so the fix belongs in the no-SRQ receive-refill/FC logic (or in excluding/adapting stress_iface_ops for that config).

Full analysis

Summary: Azure job "Tests BlueField on worker 0" failed in the new run_rc_verbs_nosrq_gtest task: rc_verbs/test_rc.stress_iface_ops/1 <rc_verbs/mlx5_1:1> never finished flushing and the gtest watchdog aborted the process (test_helpers.cc:57: Failure / Connection timed out - abort testing, SIGABRT, core dumped, make: *** [Makefile:5014: test] Aborted).

Root cause: A hang, not a slow test. Timeline from the log: last application output is [ RUN ] rc_verbs/test_rc.stress_iface_ops/1 at 11:19:24.017, then a single 900-second gap with zero output until the watchdog fires at 11:34:23.996 — i.e. the process was stuck, not progressing. The abort backtrace shows the main thread spinning in the flush loop:
test_rc::test_iface_ops() (test_rc.cc:83) → uct_test::flush() (uct_test.cc:824/818) → uct_worker_progress → uct_rc_verbs_iface_progress → uct_rc_verbs_iface_poll_rx (rc_verbs_impl.h:143).
uct_test::flush() loops until uct_iface_flush() returns UCS_OK for every entity, so some endpoint returned UCS_INPROGRESS/UCS_ERR_NO_RESOURCE forever — no TX completion ever arrived (uct_rc_ep_flush(), rc_ep.c:612-639, returns NO_RESOURCE while txqp.available <= 0 or the FC window is exhausted).

This configuration is brand new: the gtest task with UCX_RC_VERBS_SRQ_ENABLE=n UCX_RC_VERBS_RX_CQ_LEN=262144 UCX_RC_VERBS_RX_MAX_WR=64 and the whole no-SRQ code path were introduced by 75371a0 "UCT/IB/RC/VERBS: Support QPs without SRQ (#12066)" (Guangguan Wang, 2026-11-11) — the day before this build, and it is the only commit touching both contrib/test_jenkins.sh and src/uct/ib/rc/verbs/*. In that path the receive queue is per-QP and tiny: an EP preposts only min(2*rx_max_batch, rx_max_wr) WRs (rc_verbs_ep.c:701-708), the FC window is recomputed to (rx_max_wr - rx_batch_limit + 1)/(1+soft_thresh) with the explicit caveat "a fresh QP's smaller prepost may still cause RNR on its first burst" (rc_verbs_iface.c:395-400), and refill is driven only from an RX CQE (uct_rc_verbs_iface_rx_complete_no_srq) — the code itself notes "An empty QP cannot generate a completion to retry receive posting" (rc_verbs_impl.h:52-55). A peer QP that is drained therefore leaves the sender's WQE in endless RNR retry / FC-credit wait, which matches exactly the never-completing flush. PR #12283 (UCP/RMA SGL PUT emulation) cannot cause this: the test is pure UCT, and all SGL tests in this very run were skipped ("put_sgl_zcopy is not supported").

Implicated commit: 75371a0 — "UCT/IB/RC/VERBS: Support QPs without SRQ (#12066)", Guangguan Wang (2026-11-11). Not the PR under test (f8284bb).

File: src/uct/ib/rc/verbs/rc_verbs_impl.h:52-55 and src/uct/ib/rc/verbs/rc_verbs_iface.c:395-400 (no-SRQ refill / FC window); hang observed at test/gtest/uct/uct_test.cc:818-835 via test/gtest/uct/ib/test_rc.cc:83.

Suggested fix:

  1. Do not block PR UCP/RMA: Emulate SGL PUT with contiguous PUTs on Vera Rubin #12283 on this — re-run the BlueField row; the failure is in an unrelated, one-day-old no-SRQ CI task.
  2. Short term: exclude this test from the no-SRQ filter in contrib/test_jenkins.sh (run_rc_verbs_nosrq_gtest already excludes *test_rc_no_srq*/*test_rc_srq_select*; add :-rc_verbs/test_rc.stress_iface_ops*), or make stress_iface_ops skip / use a larger RC_VERBS_RX_MAX_WR when srq == NULL, since it deliberately starves resources (RC_TX_CQ_LEN=16, 10 flushes per put).
  3. Real fix in the no-SRQ path: make receive refill independent of RX CQEs — drive uct_rc_verbs_iface_repost_empty_qps_no_srq() from the general iface progress (TX poll / progress callback) rather than only from uct_rc_verbs_iface_poll_rx() when a CQE arrives, and prepost at least fc_max_wnd_size WRs on a fresh QP so a first burst cannot park the peer in indefinite RNR retry. Add a bounded flush timeout/ucs_warn with FC/rx_available state to make this diagnosable.

Related: PR #12283 (#12283) — job under test; PR #12066 "UCT/IB/RC/VERBS: Support QPs without SRQ" — suspected source of the hang. No existing issue found for this signature; worth opening one against #12066.

tvegas1
tvegas1 previously approved these changes Nov 12, 2026
Comment thread src/ucp/rma/put_offload.c Resolved
@michal-shalev

Copy link
Copy Markdown
Contributor Author

Thanks @ovidiusm for catching 2 protocol selection issues in this PR. I've pushed the fixes.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/put_offload.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

src/ucp/rma/put_offload.c

can we drop has_native() and rely on fallback_class? ucp_proto_select_disable_fallback() already disables put/sgl/contig in every range where put/sgl/offload* is available, and unlike this check it ignores protocols disabled by configuration — here a put/sgl/offload with cfg_thresh=inf (e.g. UCX_ZCOPY_THRESH=inf) hides put/sgl/contig entirely. It also makes the probe depend on the order of UCP_PROTO_FOR_EACH, since ctx holds only the protocols probed before this one.

src/ucp/rma/put_offload.c

why is put/am/bcopy excluded? pls add a short comment — I assume because put/sgl/am/bcopy already emulates SGL per element.

@svc-ucx

svc-ucx commented Nov 13, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 2) · commit 6dcc03bf

TL;DR: The ASAN roce gtest shard did not fail an assertion — it hung: rcx/test_ucp_wireup_1sided.send_disconnect_onesided_nozcopy/4 <rc_x/tag> spun in the UCP progress loop for 15 min (one UCP request never completed), the 900 s gtest watchdog fired and aborted the process. The hang is in the RC eager-bcopy send / ep-close path and is not reachable from this PR's SGL-PUT code, so treat it as a master-side/flaky hang, not a regression from #12283.

Full analysis

Summary: make test aborted (signal 6, make: *** [Makefile:5014: test] Aborted) because the gtest watchdog killed a hung test: rcx/test_ucp_wireup_1sided.send_disconnect_onesided_nozcopy/4 <rc_x/tag>.

Root cause: Timeline from the Azure log proves a hang, not slowness:

  • 10:34:45.66 — [ RUN ] rcx/test_ucp_wireup_1sided.send_disconnect_onesided_nozcopy/4 <rc_x/tag> (last meaningful application output).
  • 10:46:02.64 — after ~676 s the test's own deadline expired: ucp_test.cc:310 UCX ERROR request 0x62e00015a420 did not complete on time (test/gtest/ucp/ucp_test.cc:300-311, i.e. the thread was calling check_events()/progress in a loop the whole time).
  • 10:49:45.61 — the 900 s watchdog (test/gtest/common/test_helpers.cc:31,57 → Failed / Connection timed out - abort testing) sent the kill signal; backtrace shows the watched thread spinning in __sched_yield under gtest/ucp_test frames, i.e. inside the progress loop.

So both workers were being progressed, yet a single UCP request never completed. The test configuration is ZCOPY_THRESH=-1 + RNDV_THRESH=inf (test/gtest/ucp/test_ucp_wireup.cc:676-684): 100 × 8000-byte eager bcopy sends on rc_x, then ep_close, then the receives. That path depends on RC flow-control credits / ep-close flush completing; one of those never completed (only one "did not complete on time" line is printed, after which the process sat silent for another 223 s, consistent with blocking in teardown/close after the first timeout).

The PR under test (vera-rubin-sgl-put) only adds SGL PUT protocols: ucp_put_sgl_contig_proto / ucp_put_sgl_offload* have .dt_mask = UCS_BIT(UCP_DATATYPE_SGL) and their probes bail out unless UCP_OP_ID_PUT + dt_class == UCP_DATATYPE_SGL (src/ucp/rma/put_offload.c:686-691), and ucp_proto_select_init_protocols() only probes protocols whose dt_mask matches the select param (src/ucp/proto/proto_select.c:297). A tag/contig send over rc_x therefore never touches the new code — the diff cannot explain this hang.

Implicated commit: unknown — not the PR commit (6dcc03b); the hung path is pre-existing master code. Plausible bisect candidates touching this exact scenario: d513f98f (UCP/FLUSH: topology-safe selective endpoint flush, #11923), aaf97b49 (UCP/WIREUP: activate AM lane iface, #11971), b904de40 (UCP/WIREUP: fix worker flush hang from recovery wireup ep, #11988).

File: test/gtest/ucp/test_ucp_wireup.cc:676 (hanging test) — timeout reported at test/gtest/ucp/ucp_test.cc:310, watchdog abort at test/gtest/common/test_helpers.cc:57.

Suggested fix: Re-run the ASAN roce job to confirm it is not reproducible on this branch (the diff cannot reach the tag/rc_x eager path). If it reproduces, debug as a hang, not a timeout — do not raise the watchdog/deadline: rerun that single test with UCX_LOG_LEVEL=debug/UCX_PROTO_INFO=y and, on the request ... did not complete on time path in ucp_test::request_process(), dump the stuck request's protocol/lane state (ucp_ep_print_info + outstanding RC FC credits) so it's clear whether the sender is starved of RC flow-control grants or the ucp_ep_close flush never completes; then bisect across the flush/wireup commits listed above.

Related: PR #12283 (UCP/RMA: Emulate SGL PUT with contiguous PUTs on Vera Rubin, the branch under test, unrelated to the hang); possibly relevant flush/wireup changes: #11923, #11971, #11988.

@michal-shalev
michal-shalev enabled auto-merge (squash) November 13, 2026 13:11
@michal-shalev
michal-shalev merged commit a17aaac into openucx:master Nov 14, 2026
167 checks passed
@michal-shalev
michal-shalev deleted the vera-rubin-sgl-put branch November 14, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants