Skip to content

UCP/RMA: Share SGL multi-lane send code in proto_multi.inl - #12226

Merged
michal-shalev merged 4 commits into
openucx:masterfrom
michal-shalev:ucp-put-sgl-shared-send
Nov 9, 2026
Merged

michal-shalev merged 4 commits into
openucx:masterfrom
michal-shalev:ucp-put-sgl-shared-send

Conversation

@michal-shalev

Copy link
Copy Markdown
Contributor

What?

Move the PUT SGL fragment/batch send code into shared helpers in proto_multi.inl, and reuse them in the PUT sw protocol.

Why?

Prepare for GET SGL, which needs the same lane split. No functional change.

@michal-shalev michal-shalev self-assigned this Nov 3, 2026
@michal-shalev
michal-shalev requested a review from a team as a code owner November 3, 2026 15:06
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/proto/proto_multi.inl Resolved
Comment thread src/ucp/proto/proto_multi.inl Resolved
Comment thread src/ucp/rma/put_offload.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual gap: no gtest exercises a post_func with count != 1 through ..._send_frags() with max_frag_count == 1, i.e. the new ucs_assertv(count == 1) is only an invariant check; that is acceptable for a refactor.

Residual gap: SGL zcopy offload (put/sgl/offload) needs RC/mlx5 hardware; on the tcp instantiation only the offload_sw path is covered, so the batch path depends on the IB CI jobs.

@svc-ucx

svc-ucx commented Nov 3, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests new on worker 3) · commit f042d604

TL;DR: The gtest binary aborted on a UCS rcache assertion (!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE) at src/ucs/memory/rcache.c:396) while tcp/test_ucp_rma.put_nonblocking/2 <…/user_memh> was unmapping its memory handle — the rcache region's refcount hit 0 while the region was still in the page table, i.e. an unbalanced (double) release of the memh reference taken for a user memh; the RMA send-path refactor in this PR is the only delta in the build and must be audited for an extra ucp_memh_put*()/ucp_datatype_iter_cleanup() on the request path.

Full analysis

Summary: make test (gtest, Azure job "new on worker 3", build 142446) aborted with a core dump in tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh>:
rcache.c:396 Assertion '!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed, backtrace ucp_mem_unmap → ucp_memh_cleanup (ucp_mm.c:470) → ucp_memh_put_rcache (ucp_mm.c:441) → ucs_rcache_region_put_unsafe (rcache.inl:128) → ucs_mem_region_destroy_internal (rcache.c:396), from test_ucp_memheap::test_xfer (test_ucp_memheap.cc:120).

Root cause: Registration-cache reference-count underflow. A region inserted into the rcache page table is created with refcount = 1 plus UCS_RCACHE_REGION_FLAG_PGTABLE (src/ucs/memory/rcache.c:986-989), and that page-table reference is dropped only by ucs_rcache_region_invalidate_internal(), which first clears the flag (rcache.c:492-501). Therefore reaching refcount == 0 while the flag is still set — exactly what assert 396 caught (the refcount == 0 assert at line 394 passed) — can only happen if the region was released more times than it was acquired. The release in the trace is the user-memh path: ucp_memh_cleanup() → ucp_memh_put_rcache(context, memh->parent) for the parent reference acquired in ucp_memh_init_uct_reg() (src/ucp/core/ucp_mm.c:950). So some earlier code path in the same test already dropped that parent/memh reference (note that for a user memh ucp_memh_put() is intentionally a no-op, src/ucp/core/ucp_mm.inl:97, so any path that releases the rcache region directly, e.g. ucp_memh_put_rcache()/ucs_rcache_region_put_unsafe() on a memh it does not own, or a request whose dt_iter is cleaned up twice via the completion and the reset/abort path, produces this underflow). This is a genuine use-after-free class bug, not a timeout/infra flake — there are no large time gaps in the log, the process aborted while actively running.

Implicated commit: [REDACTED:Hex High Entropy String] — "UCP/RMA: Share SGL multi-lane send code in proto_multi.inl", Michal Shalev (the only change under test; it rewrites the put send path exercised by test_ucp_rma.put_nonblocking). Related prior work on the same files: 4af0b6b5 (#11947), [REDACTED:Hex High Entropy String] (#11888, "register user-supplied SGL memory handles on all required MDs").

File: src/ucs/memory/rcache.c:396 (assert site); real defect is in the memh ref accounting reached from src/ucp/core/ucp_mm.c:441/:470 and the RMA send path refactored in src/ucp/proto/proto_multi.inl:400-550 / src/ucp/rma/put_offload.c:261-514.

Suggested fix:

  1. Audit the PR's moved code for an unbalanced release: every ucp_memh_get()/ucs_rcache_lookup*() in the shared ucp_proto_rma_sgl_offload_send_func() / ..._send_frags() and in ucp_proto_multi_zcopy_progress()'s init/abort/reset paths (ucp_proto_request_zcopy_init, ucp_proto_offload_zcopy_reset, ucp_proto_request_zcopy_abort) must have exactly one matching put; in particular make sure a request that is reset/aborted and later completed does not run ucp_datatype_iter_cleanup(dereg=1) twice, and that user-supplied memhs (param->memh, dt_iter->type.sgl.memhs[]) are never passed to ucp_memh_put_rcache()/ucs_rcache_region_put_unsafe() (only their parent, once, from ucp_memh_cleanup()).
  2. Reproduce locally with the exact variant to confirm the PR introduced it:
    UCX_TLS=tcp,cuda_copy,rocm_copy ./gtest --gtest_filter='tcp/test_ucp_rma.put_nonblocking/2' with UCX_LOG_LEVEL=trace and rcache tracing (UCX_LOG_LEVEL=trace + ucs_rcache_region_trace output shows each get/put of the region) on both this branch and its merge base — that pinpoints the extra put.
  3. To make such bugs fail loudly and earlier, strengthen the check at rcache.c:396 into ucs_assertv printing region/refcount/name (as line 394 does), and consider adding an assert in ucp_memh_put_rcache() that memh->parent == NULL || memh->parent == memh (i.e. that only a non-user, rcache-owned memh is being released).

Related: PR #12226 (this PR), #12191 (UCP/RMA: Add SGL support to ucp_get_nbx, same shared code), #11947 (closed, split SGL put between lanes), #11958 (open, rcache adjacent-region merging — touches the same refcount logic).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 2603e5af-1482-4c2e-a3b8-a644dd820ead in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/proto/proto_multi.inl Resolved
Comment thread src/ucp/proto/proto_multi.inl Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/put_offload.c Resolved
@svc-ucx

svc-ucx commented Nov 8, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit b6634eed

TL;DR: The gtest run failed on a single unrelated test, ib/test_md.alloc/0 (UCT IB memory-domain allocation test), while PR #12226 only refactors UCP RMA/proto_multi.inl SGL send code — this is an environment/flaky resource failure (the test's "Too many OUT_OF_RESOURCE failures" guard), so re-run the job and fix the over-strict counter in the test rather than the PR.

Full analysis

Summary: roce on worker 0 job: make test in build-test/test/gtest exited non-zero because 1 of 9037 tests failed — [ FAILED ] ib/test_md.alloc/0, where GetParam() = mlx5_0.

Root cause: From the log, 9036/9037 tests passed; the only failure is ib/test_md.alloc/0, a pure UCT-level memory-domain allocation stress test (uct_mem_alloc + uct_md_mkey_pack_v2 in a 300-iteration loop on mlx5_0). Nothing in that test path touches the UCP RMA / proto_multi.inl SGL send code this PR changes; the PR's own new test (rc_mlx5/test_rc.sgl_reserve_released_on_completion/0) passed, as did every UCP RMA/tag/put test. The test is allocation-pressure sensitive: its only soft assertion is EXPECT_LT((double)num_alloc_failures / iterations, 0.5) << "Too many OUT_OF_RESOURCE failures", which trips when uct_mem_alloc returns UCS_ERR_NO_MEMORY too often (pinnable-memory / memlock pressure on the shared worker swx-rain04 — note the immediately preceding posix/uct_p2p_rma_test.put_bcopy/1 mapped a ~4.4 GB buffer in the same process, and the run lasted ~50 min). Caveat: the Azure log is only available as a tail, so the exact assertion line for this test had already scrolled out; the surviving evidence is the failure summary plus the test source.

A real contributing defect in the test: num_alloc_failures is initialized once at test_md.cc:484 but the ratio check at test_md.cc:559 is evaluated inside the ucs_for_each_bit(mem_type, ...) loop, so allocation failures accumulate across memory types and are compared against a per-type iterations count — making the threshold progressively stricter and the test needlessly flaky under memory pressure.

Implicated commit: unknown — not attributable to [REDACTED:Hex High Entropy String] (PR #12226); recent commits touching test/gtest/uct/test_md.cc are all CUDA/GDR-related (4cdc171, 20fe448, 27962e3) and unrelated to the IB alloc test.

File: test/gtest/uct/test_md.cc:559 (check), test/gtest/uct/test_md.cc:484 (counter init), test/gtest/uct/test_md.cc:516 (per-mem-type loop)

Suggested fix:

  1. Re-run the roce on worker 0 job — this failure is not caused by the PR; if it reproduces, capture the full gtest log (the tail truncates the assertion text) to confirm whether it is the OUT_OF_RESOURCE ratio check or an ASSERT_UCS_OK on uct_mem_alloc/uct_md_mkey_pack_v2.
  2. Harden the test: reset num_alloc_failures (and the iteration count it is divided by) at the start of each ucs_for_each_bit(mem_type, ...) iteration, e.g. move unsigned num_alloc_failures = 0; inside the loop at line 516, and include the failure count/status in the assertion message so CI tails are diagnosable.
  3. Optionally skip/relax the test when the process memlock limit or available pinnable memory is low, since the IB MD alloc path pins memory.

Related: PR #12226 (the PR under test, #12226); no existing issue found for ib/test_md.alloc flakiness — worth opening one referencing build 143631.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 8064073c-50eb-41af-814b-79c9eee66023 in the triage console for the audit trail.

@michal-shalev
michal-shalev merged commit a28e31c into openucx:master Nov 9, 2026
167 checks passed
@michal-shalev
michal-shalev deleted the ucp-put-sgl-shared-send branch November 9, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants