Skip to content

GTEST: Fix Blossom sockaddr and CPU-mask failures - #12312

Open
shasson5 wants to merge 1 commit into
openucx:masterfrom
shasson5:blossom-test-portability-fixes
Open

shasson5 wants to merge 1 commit into
openucx:masterfrom
shasson5:blossom-test-portability-fixes

Conversation

@shasson5

Copy link
Copy Markdown
Contributor

What?

Combine #12213 and #12216: match unreachable destinations to the source address family, handle immediate route errors, and bound worker batches while checking all affinity CPUs.

Why?

Prevent address-family mismatches and descriptor exhaustion in Blossom runs.

HPCINFRA-5026, HPCINFRA-5031.

@shasson5
shasson5 requested a review from a team as a code owner November 11, 2026 09:50
@shasson5 shasson5 added the WIP-DNM Work in progress / Do not review label Nov 11, 2026
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/rdmacm/rdmacm_cm_ep.c Resolved
Comment thread test/gtest/ucp/test_ucp_worker.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: the new else branch only asserts the status, so for IPv6 CI runs where connect()/rdma_resolve_addr() fail synchronously the test no longer exercises the async error-propagation path (TEST_STATE_CLIENT_GOT_SERVER_UNAVAILABLE). Also, the strict EXPECT_EQ(UCS_ERR_UNREACHABLE, status) will fail on any host where the resolve fails with a different errno (e.g. EADDRNOTAVAIL/ENODEV).

@shasson5 shasson5 removed the WIP-DNM Work in progress / Do not review label Nov 11, 2026
@svc-ucx

svc-ucx commented Nov 11, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests new on worker 1) · commit c5af3fef

TL;DR: The gtest binary aborted on a UCS rcache assertion — tcp/test_ucp_rma.put_nonblocking/2 <…/user_memh> destroyed a registration-cache region that was still present in the page table (rcache.c:405), i.e. the last reference dropped by ucp_memh_put_rcache() was the page table's own reference. This is a reference-count/lifecycle imbalance in the rcache↔ucp_mem_h parent-release path (most recently touched by the "merge adjacent rcache regions" change), not a problem introduced by PR #12312.

Full analysis

Summary: Azure job "Tests new on worker 1" (build 144486) failed in the gtest step: make: *** [Makefile:5014: test] Aborted (core dumped) after rcache.c:405 Assertion '!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed during tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh>.

Root cause: From the backtrace:

ucp_mem_unmap()            ucp_mm.c:1298
 ucp_memh_cleanup()        ucp_mm.c:473      -> else-branch: parent came from rcache
  ucp_memh_put_rcache()    ucp_mm.c:444      -> ucs_rcache_region_put_unsafe()
   ucs_rcache_region_put_unsafe()  rcache.inl:128  -> --refcount == 0 -> destroy
    ucs_mem_region_destroy_internal() rcache.c:405  -> ASSERT: region still in pgtable

The rcache ownership contract (see rcache.c:1067-1069, :1107-1108 and rcache.c:674-677) is that a cached region holds one reference for the page-table entry plus one per user, and a region may only be destroyed after ucs_rcache_region_invalidate_internal() has removed it from the page table and dropped that reference. Here the region's refcount was 1 while UCS_RCACHE_REGION_FLAG_PGTABLE was still set, so the single put done for memh->parent at unmap time consumed the page-table's reference — i.e. the user ucp_mem_h released a reference it did not own (double release of memh->parent, or a parent pointer stored without a matching hold). The assertion is the symptom; the state it catches is a leaked/duplicated reference, and in a non-assert build this would be a use-after-free of a live page-table entry.

Two concrete suspect sites in the code I read:

  • ucs_rcache_check_adj_regions() / ucs_rcache_check_overlap_one() (rcache.c:843-891, :756-827): adjacent/overlapping regions are handed to ops->merge and then invalidated without checking that they are unused (refcount == 1), unlike ucs_rcache_lru_evict() which explicitly skips region->refcount > 1. The UCP merge callback (ucp_mem_rcache_merge_cb, ucp_mm.c:1760) mutates the old memh's flags while a live user memh may still hold it as parent.
  • ucs_rcache_create_region() error path at rcache.c:1110-1116: if ucs_rcache_fill_pfn() fails the region is ucs_free()d while still inserted in the page table and still counted in num_regions/total_size, leaving a dangling page-table entry — which later surfaces exactly as "destroying a region that is still in the page table".

Note the failing branch (blossom-test-portability-fixes, PR #12312 — test portability fixes) does not touch rcache or ucp_mm, so this is a master-side regression/flake, not a PR defect.

Implicated commit: e536ef03 — "UCS/RCACHE: Adds capability for merging adjacent rcache regions (#11958)", william-gallagher-nv, 2026-11-04 (the only recent commit touching both src/ucs/memory/rcache.c region lifecycle and src/ucp/core/ucp_mm.c memh/merge callbacks, one week before this failure).

File: src/ucs/memory/rcache.c:405 (assert) — imbalance originates at src/ucs/memory/rcache.inl:128 / src/ucp/core/ucp_mm.c:444 (ucp_memh_put_rcache), with suspect logic at src/ucs/memory/rcache.c:843-891 and src/ucs/memory/rcache.c:1110-1116.

Suggested fix:

  1. Do not block PR GTEST: Fix Blossom sockaddr and CPU-mask failures #12312 on this; re-run and reproduce on master (gtest --gtest_filter='*test_ucp_rma.put_nonblocking*' with UCX_LOG_LEVEL=trace, and compare UCX_RCACHE_MERGE_ADJACENT=n vs y) to confirm whether the merge path is involved before PR UCS/RCACHE: Default merging rcache regions to on #12267 ("Default merging rcache regions to on") is merged — that PR would make this failure the default behaviour.
  2. In ucs_rcache_check_adj_regions()/ucs_rcache_can_merge_adj_region(), skip regions that are in use (region->refcount > 1), mirroring the guard in ucs_rcache_lru_evict() (rcache.c:718-724), so a region still referenced as a ucp_mem_h parent is never merged away.
  3. Fix the ucs_rcache_fill_pfn() failure path (rcache.c:1110-1116) to tear the region down via ucs_rcache_region_invalidate_internal(rcache, region, UCS_RCACHE_REGION_PUT_FLAG_IN_PGTABLE | UCS_RCACHE_REGION_PUT_FLAG_MUST_DESTROY) instead of a bare ucs_free(), so the page-table entry and the region counters are not left dangling.
  4. Add a cheap diagnostic to catch the real owner next time: in ucs_rcache_region_put_unsafe() assert (region->refcount > 1) || !(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE) and include region->refcount/flags in the ucs_rcache_region_trace("put region") message.

Related: #11958 (merge adjacent rcache regions — implicated), #12267 (would enable merging by default — hold until this is fixed), #12104 (rcache explicit registration, same area).

@william-gallagher-nv william-gallagher-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shasson5 Looks good. A couple of minor points that you can address if you think appropriate. I also wanted to confirm the tests pass when running on Blossom.

Comment on lines +1239 to +1245
if (status == UCS_OK) {
wait_for_bits(&m_state, TEST_STATE_CLIENT_GOT_SERVER_UNAVAILABLE,
300);
EXPECT_TRUE(m_state & TEST_STATE_CLIENT_GOT_SERVER_UNAVAILABLE);
} else {
EXPECT_EQ(UCS_ERR_UNREACHABLE, status);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, why rework the validation here?

/* TCP_SYNCNT may still leave SYN retransmission timeouts in the
* minutes range on some systems. Bound TCP_USER_TIMEOUT as well when
* the transport supports it. */
modify_config("TCP_CM_USER_TIMEOUT", "1s", IGNORE_IF_NOT_EXIST);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The RDMACM tests take 10 seconds to fail. One possibility is to set RDMA_CM_TIMEOUT to 1 seconds like the TCP case. Feel free to ignore as out of scope.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants