Skip to content

UCS/RCACHE: Default merging rcache regions to on - #12267

Open
william-gallagher-nv wants to merge 1 commit into
openucx:masterfrom
william-gallagher-nv:default-rcache-merge-enable
Open

william-gallagher-nv wants to merge 1 commit into
openucx:masterfrom
william-gallagher-nv:default-rcache-merge-enable

Conversation

@william-gallagher-nv

@william-gallagher-nv william-gallagher-nv commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What?

Change default to enable merging adjacent rcache regions. Opt out available by setting UCX_RCACHE_MERGE_ADJACENT=n.

Why?

Defaulting to enabling the merging of adjacent regions provides a better user experience for applications that register large numbers of memory regions.

@william-gallagher-nv william-gallagher-nv self-assigned this Nov 5, 2026
@william-gallagher-nv
william-gallagher-nv requested a review from a team as a code owner November 5, 2026 21:58
@william-gallagher-nv william-gallagher-nv added the WIP-DNM Work in progress / Do not review label Nov 5, 2026
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucs/memory/rcache.c
ucs_offsetof(ucs_rcache_config_t, purge_on_fork), UCS_CONFIG_TYPE_BOOL},

{"RCACHE_MERGE_ADJACENT", "n", "Merge adjacent registration cache regions",
{"RCACHE_MERGE_ADJACENT", "y", "Merge adjacent registration cache regions",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for host memory ucp_mem_rcache_can_merge_cb() returns 1 regardless of the underlying allocation, so two unrelated adjacent buffers can become one region, and freeing one of them invalidates the registration of the other. was this measured on host workloads with mixed buffer lifetimes, or only on the merge-friendly case?

Comment thread src/ucs/memory/rcache.c
ucs_offsetof(ucs_rcache_config_t, purge_on_fork), UCS_CONFIG_TYPE_BOOL},

{"RCACHE_MERGE_ADJACENT", "n", "Merge adjacent registration cache regions",
{"RCACHE_MERGE_ADJACENT", "y", "Merge adjacent registration cache regions",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: now that this is the default, modify_config("RCACHE_MERGE_ADJACENT", "y") in test_ucp_rcache_merge is a no-op and nothing covers the disabled path. can we add a variant with n there?

@svc-ucx

svc-ucx commented Nov 5, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Build build_x86_64 centos7) · commit 31921dfc

TL;DR: The job never built anything — it died in the Azure Pipelines "Initialize containers" step because docker inspect on rdmz-harbor.rdmz.labs.mlnx/ucx/centos7:5 returned "No such object" on agent swx-rdmz-instinct01; this is an agent/Docker infrastructure problem, not a code regression from PR #12267, so simply re-run the pipeline (and clean the agent's Docker image cache).

Full analysis

Summary: Build build_x86_64 centos7 (build 143175) failed during container initialization with Exit code 1 returned from process: file name '/scrap/bin/docker', arguments 'inspect --format=... rdmz-harbor.rdmz.labs.mlnx/ucx/centos7:5'.

Root cause: Infrastructure/Docker-state inconsistency on the build agent. The log shows docker pull rdmz-harbor.rdmz.labs.mlnx/ucx/centos7:5 returning Status: Image is up to date (22:01:16.412) — i.e. the pull was a no-op based on the local manifest — but the immediately following docker inspect of the same tag failed with Error: No such object: rdmz-harbor.rdmz.labs.mlnx/ucx/centos7:5 (22:01:19.110). The image reference was therefore not actually resolvable locally: either the local image store is corrupt/partially GC'd, or a concurrent image prune/docker rmi on the shared agent (swx-rdmz-instinct01, slot -05, multiple concurrent jobs do docker network prune/cleanup) removed the image between the pull and the inspect. No source compilation, no test, and not even the git checkout of the PR merge commit happened before the failure (checkout ran only after the container step already errored, and "Stop Containers"/"Finalize Job" followed within 2 seconds). There is no evidence of any UCX code involvement — the whole job lasted ~7 seconds and the total log is pure agent bootstrap. This is also not a timeout: there are no gaps larger than ~3 seconds anywhere in the log.

Implicated commit: unknown — not a code regression; commit 31921df / PR #12267 code was never compiled or executed in this build.

File: unknown (failure is in the Azure agent container-initialization step, before any repo file is used; pipeline definition: buildlib/azure-pipelines-pr.yml)

Suggested fix: Re-run the failed job — this class of failure is transient. If it recurs on swx-rdmz-instinct01, have CI admins clean up that agent's Docker state (docker image prune -a then re-pull rdmz-harbor.rdmz.labs.mlnx/ucx/centos7:5, or docker rmi the stale tag to force a full re-pull) and check for cron/other jobs running docker system prune concurrently with pipeline jobs on that shared host, since the agent's own prune --filter "label=f35170" plus an unfiltered external prune would explain an image disappearing between pull and inspect. As a hardening step, consider adding a pull retry/docker image inspect pre-check (or setting a non-shared image cache per agent slot) in the container resource setup.

Related: #12094 (CI: Jenkins integration testing) — only loosely related CI infrastructure work; no existing issue tracks this specific Docker No such object flake.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucs/memory/rcache.c
ucs_offsetof(ucs_rcache_config_t, purge_on_fork), UCS_CONFIG_TYPE_BOOL},

{"RCACHE_MERGE_ADJACENT", "n", "Merge adjacent registration cache regions",
{"RCACHE_MERGE_ADJACENT", "y", "Merge adjacent registration cache regions",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why enable this for host memory too? can_merge() returns 1 for host without the alloc_base check, so adjacent but unrelated host mappings are now merged by default, and freeing one of them invalidates the registration of the other - do we have perf/memory numbers for this default?

EXPECT_EQ(size, ucp_memh_length(memh2->parent));
EXPECT_EQ(2, rcache->num_regions);

/* Verify registrations by performing zcopy put */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: can we move these puts after the REGISTERED flag is restored? otherwise the transfer runs while memh1's region is artificially marked as unregistered.

}

UCS_TEST_P(test_ucp_rcache_merge, adjacent_region_unregistered) {
UCS_TEST_P(test_ucp_rcache_merge, merge_adjacent_enabled,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: can merge_adjacent_enabled drop "RCACHE_MERGE_ADJACENT=y" and rely on the new default? then this test also covers the default being changed.

Switches UCX_RCACHE_MERGE_ADJACENT default from 'n' to 'y'. Regions of
memory that are adjacent in memory will be merged into a single region
when possible.
@william-gallagher-nv
william-gallagher-nv force-pushed the default-rcache-merge-enable branch from 15ae738 to 46561ab Compare November 12, 2026 13:30
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

size_t size,
ucs_memory_type_t mem_type)
{
mapped_buffer target = mapped_buffer(size, receiver(), 0,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: direct initialization is enough here, no need for the temporary copy.

Suggested change
mapped_buffer target = mapped_buffer(size, receiver(), 0,
mapped_buffer target(size, receiver(), 0, UCS_MEMORY_TYPE_HOST);

ucp_context_h context = sender().ucph();
ucs_rcache_t *rcache = context->rcache;
ucp_mem_map_params_t params;
const size_t size = ucs_get_page_size();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: this block duplicates the allocate + register-first-half setup of test_adjacent_registration; maybe extract a small helper for it?

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage note: test_ucp_rcache_merge is instantiated only via UCP_INSTANTIATE_TEST_CASE_TLS(..., rcx, "rc_x,cuda_copy"), so the UCP-level merge/no-merge assertions run only on that transport pair; the new default elsewhere is validated only indirectly by the rest of the suite running with merging enabled. Generic merge logic is covered by test_rcache_merge_adjacent in test/gtest/ucs/test_rcache.cc, so this is acceptable, but it is worth keeping in mind if CI shows transport-specific fallout from the default change.

@william-gallagher-nv william-gallagher-nv removed the WIP-DNM Work in progress / Do not review label Nov 12, 2026
@gleon99
gleon99 requested a review from nbellalou November 12, 2026 16:00
@gleon99

gleon99 commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

@nbellalou please review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants