Skip to content

UCP/RNDV: Throttle rndv fragment requests (#11062) - 1.21.x - #12276

Merged
ofirfarjun7 merged 1 commit into
openucx:v1.21.xfrom
amastbaum:rndv_throttling_1-21
Nov 11, 2026
Merged

ofirfarjun7 merged 1 commit into
openucx:v1.21.xfrom
amastbaum:rndv_throttling_1-21

Conversation

@amastbaum

Copy link
Copy Markdown
Contributor

Backport #11062

@amastbaum
amastbaum requested a review from ofirfarjun7 November 8, 2026 09:47
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@amastbaum
amastbaum changed the base branch from v1.21.x to v1.21.1 November 8, 2026 11:37
@ofirfarjun7
ofirfarjun7 changed the base branch from v1.21.1 to v1.21.x November 8, 2026 21:08
@svc-ucx

svc-ucx commented Nov 8, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer gpu on worker 3) · commit 76937281

TL;DR: The ASAN gtest job failed on LeakSanitizer (not on a test assertion): the new test_ucp_peer_failure_rndv_put_ppln_abort.put_mtype_fc_pending test added by this PR leaks the sender's CUDA buffer registration (cuda_ipc memh + packed lkey) because a rndv/put/mtype fragment that is throttled on the new FC pending queue and then aborted on forced EP close never drives the parent rndv/ppln request to completion, so ucp_datatype_iter_cleanup() never releases the memh reference.

Full analysis

Summary: AddressSanitizer: 224 byte(s) leaked in 2 allocation(s) at the end of the gtest run (make: *** [Makefile:4455: test] Error 1); all 10468 tests reported PASSED, so the job failed purely on the leak report.

Root cause: Both leaked blocks are the UCT cuda_ipc registration of the sender buffer used by the new test: uct_cuda_ipc_mem_reg (cuda_ipc_md.c:476) reached via ucp_memh_get → ucp_datatype_iter_mem_reg → ucp_proto_rndv_rts_request_init (proto_rndv.inl:56), plus the packed lkey from uct_cuda_ipc_mkey_pack during RTS packing. The allocating stack is rooted in test_ucp_peer_failure.cc:1178 → rndv_fc_pending_abort_test → smoke_test → ucp_tag_send_nbx, i.e. the registration is cached in the rcache and its refcount is never dropped, so it is still held when the context/rcache is destroyed and the UCT memh is never deregistered.
The reference is only released when the top-level rndv/ppln send request completes, in ucp_proto_rndv_ppln_frag_complete() (ucp_datatype_iter_cleanup, rndv_ppln.c:228), which happens only after all fragments are accounted. In this test the sender's EP is force-closed while a rndv/put/mtype fragment sits on the brand-new worker throttling queue (worker->rndv_mtype_fc.pending_q[UCP_WORKER_RNDV_FC_OP_PUT]). That fragment is aborted through ucp_proto_rndv_mtype_fc_ep_purge() → ucp_proto_request_abort() → ucp_proto_rndv_put_mtype_abort() → ucp_proto_rndv_stub_abort(), which merely invokes the UCT completion ("Proper abort functionality is not implemented yet", proto_rndv.c:1203-1212). If the sibling in-flight fragment's completion is not delivered (its uct_comp.count is still >1 when the lane is discarded) the parent request's completed_size never reaches the full length, the parent is never completed, and its datatype-iterator/memh reference leaks — exactly one buffer registration, matching the 2 reported allocations (memh + lkey).

Implicated commit: [REDACTED:Hex High Entropy String] — amastbaum, "UCP/RNDV: Throttle rndv fragment requests (#11062)" (the only recent commit on rndv_mtype.inl, proto_rndv.c, rndv_put.c, rndv_ppln.c and test_ucp_peer_failure.cc).

File: src/ucp/rndv/proto_rndv.c:99 (ucp_proto_rndv_mtype_fc_ep_purge) / src/ucp/rndv/rndv_put.c:674 (ucp_proto_rndv_put_mtype_abort) / src/ucp/rndv/rndv_ppln.c:228 (ucp_datatype_iter_cleanup only reached on full completion); test: test/gtest/ucp/test_ucp_peer_failure.cc:1172-1179.

Suggested fix: Make the throttled-request abort path deterministically release the request's registrations instead of relying on all pipeline fragments completing:

  1. In ucp_proto_rndv_mtype_fc_ep_purge()/ucp_proto_rndv_put_mtype_abort(), for a fragment request that never got a staging buffer (!UCP_REQUEST_FLAG_PROTO_INITIALIZED), complete the parent with the abort status — i.e. route send fragments through ucp_proto_rndv_ppln_frag_complete(..., abort=1, ...) (as the recv path already does) so the super request's ucp_rkey_destroy() + ucp_datatype_iter_cleanup() run even when the remaining fragments can never complete.
  2. Equivalently/in addition, have the ppln super request's abort (ucp_proto_rndv_stub_abort for ucp_rndv_send_ppln_proto) clean up req->send.state.dt_iter and the rkey so the memh reference cannot outlive the EP.
  3. To catch regressions locally, run just UCX_RNDV_FRAG_MEM_TYPE=cuda ./gtest --gtest_filter='*test_ucp_peer_failure_rndv_put_ppln_abort.put_mtype_fc_pending*' under ASAN; consider adding a check that worker->rndv_mtype_fc.pending_q[*] is empty and that no rcache region is in-use at worker destroy.

Related: PR #12276 "UCP/RNDV: Throttle rndv fragment requests (#11062)" — no existing issue found for this leak; the abort gap is the known ucp_proto_rndv_stub_abort() FIXME (src/ucp/rndv/proto_rndv.c:1203).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 7916fc94-86b7-4ca5-b4b2-283ab3d97f16 in the triage console for the audit trail.

@ofirfarjun7 ofirfarjun7 changed the title UCP/RNDV: Throttle rndv fragment requests (#11062) UCP/RNDV: Throttle rndv fragment requests (#11062) - 1.21.x Nov 10, 2026
@ofirfarjun7
ofirfarjun7 merged commit be8757f into openucx:v1.21.x Nov 11, 2026
147 of 152 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants