Repository navigation
UCS/RCACHE: Default merging rcache regions to on - #12267
william-gallagher-nv wants to merge 1 commit into
Conversation
|
🤖 Starting review — findings will be posted here when done. |
| ucs_offsetof(ucs_rcache_config_t, purge_on_fork), UCS_CONFIG_TYPE_BOOL}, | ||
|
|
||
| {"RCACHE_MERGE_ADJACENT", "n", "Merge adjacent registration cache regions", | ||
| {"RCACHE_MERGE_ADJACENT", "y", "Merge adjacent registration cache regions", |
There was a problem hiding this comment.
for host memory ucp_mem_rcache_can_merge_cb() returns 1 regardless of the underlying allocation, so two unrelated adjacent buffers can become one region, and freeing one of them invalidates the registration of the other. was this measured on host workloads with mixed buffer lifetimes, or only on the merge-friendly case?
| ucs_offsetof(ucs_rcache_config_t, purge_on_fork), UCS_CONFIG_TYPE_BOOL}, | ||
|
|
||
| {"RCACHE_MERGE_ADJACENT", "n", "Merge adjacent registration cache regions", | ||
| {"RCACHE_MERGE_ADJACENT", "y", "Merge adjacent registration cache regions", |
There was a problem hiding this comment.
minor: now that this is the default, modify_config("RCACHE_MERGE_ADJACENT", "y") in test_ucp_rcache_merge is a no-op and nothing covers the disabled path. can we add a variant with n there?
|
🤖 CI Triage Agent — TL;DR: The job never built anything — it died in the Azure Pipelines "Initialize containers" step because Full analysisSummary: Build Root cause: Infrastructure/Docker-state inconsistency on the build agent. The log shows Implicated commit: unknown — not a code regression; commit 31921df / PR #12267 code was never compiled or executed in this build. File: unknown (failure is in the Azure agent container-initialization step, before any repo file is used; pipeline definition: Suggested fix: Re-run the failed job — this class of failure is transient. If it recurs on Related: #12094 (CI: Jenkins integration testing) — only loosely related CI infrastructure work; no existing issue tracks this specific Docker |
31921df to
15ae738
Compare
|
🤖 Starting review — findings will be posted here when done. |
| ucs_offsetof(ucs_rcache_config_t, purge_on_fork), UCS_CONFIG_TYPE_BOOL}, | ||
|
|
||
| {"RCACHE_MERGE_ADJACENT", "n", "Merge adjacent registration cache regions", | ||
| {"RCACHE_MERGE_ADJACENT", "y", "Merge adjacent registration cache regions", |
There was a problem hiding this comment.
why enable this for host memory too? can_merge() returns 1 for host without the alloc_base check, so adjacent but unrelated host mappings are now merged by default, and freeing one of them invalidates the registration of the other - do we have perf/memory numbers for this default?
| EXPECT_EQ(size, ucp_memh_length(memh2->parent)); | ||
| EXPECT_EQ(2, rcache->num_regions); | ||
|
|
||
| /* Verify registrations by performing zcopy put */ |
There was a problem hiding this comment.
minor: can we move these puts after the REGISTERED flag is restored? otherwise the transfer runs while memh1's region is artificially marked as unregistered.
| } | ||
|
|
||
| UCS_TEST_P(test_ucp_rcache_merge, adjacent_region_unregistered) { | ||
| UCS_TEST_P(test_ucp_rcache_merge, merge_adjacent_enabled, |
There was a problem hiding this comment.
minor: can merge_adjacent_enabled drop "RCACHE_MERGE_ADJACENT=y" and rely on the new default? then this test also covers the default being changed.
Switches UCX_RCACHE_MERGE_ADJACENT default from 'n' to 'y'. Regions of memory that are adjacent in memory will be merged into a single region when possible.
15ae738 to
46561ab
Compare
|
🤖 Starting review — findings will be posted here when done. |
| size_t size, | ||
| ucs_memory_type_t mem_type) | ||
| { | ||
| mapped_buffer target = mapped_buffer(size, receiver(), 0, |
There was a problem hiding this comment.
minor: direct initialization is enough here, no need for the temporary copy.
| mapped_buffer target = mapped_buffer(size, receiver(), 0, | |
| mapped_buffer target(size, receiver(), 0, UCS_MEMORY_TYPE_HOST); |
| ucp_context_h context = sender().ucph(); | ||
| ucs_rcache_t *rcache = context->rcache; | ||
| ucp_mem_map_params_t params; | ||
| const size_t size = ucs_get_page_size(); |
There was a problem hiding this comment.
minor: this block duplicates the allocate + register-first-half setup of test_adjacent_registration; maybe extract a small helper for it?
|
Residual coverage note: |
|
@nbellalou please review |
What?
Change default to enable merging adjacent rcache regions. Opt out available by setting
UCX_RCACHE_MERGE_ADJACENT=n.Why?
Defaulting to enabling the merging of adjacent regions provides a better user experience for applications that register large numbers of memory regions.