Skip to content

UCT/ROCM: fix for ipc_cache with async copy - #12268

Open
ddebonis-amd wants to merge 1 commit into
openucx:masterfrom
ddebonis-amd:ipc-cache-fix
Open

ddebonis-amd wants to merge 1 commit into
openucx:masterfrom
ddebonis-amd:ipc-cache-fix

Conversation

@ddebonis-amd

Copy link
Copy Markdown
Contributor

What?

uct_rocm_ipc_ep_zcopy now pins the cached mapping and holds it until the async copy completes.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/uct/rocm/test_rocm_signal.cc Resolved
Comment thread src/uct/rocm/base/rocm_signal.c Resolved
Comment thread src/uct/rocm/base/rocm_signal.h Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/rocm/ipc/rocm_ipc_cache.c Resolved
Comment thread test/gtest/uct/rocm/test_rocm_signal.cc Resolved
uint32_t handler_done; /**< Async handler entered once */
int handler_armed; /**< Async handler owns signal */
unsigned pin_count; /**< Consumers holding mapped_addr */
uint32_t pin_count; /**< Consumers holding mapped_addr,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Field comment documents only half the protocol. Suggestion:

Suggested change
uint32_t pin_count; /**< Consumers holding mapped_addr,
uint32_t pin_count; /**< Consumers holding mapped_addr,
incremented under the lock,
decremented atomically */

Minor: the decrement is now lockless, which is the part a reader needs to know here.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: everything here needs real ROCm hardware with two processes to exercise (IPC import/export plus the free-event invalidation path), so CI almost certainly will not hit the pin/unpin race or the new err_release paths. Beyond the gtest finding, please confirm manual multi-process testing with UCX_ROCM_IPC_SIGNAL_INVALIDATE=y and with UCX_ROCM_IPC_CACHE_IPC_HANDLES=n, since the two branches now take different release paths.

Comment thread src/uct/rocm/base/rocm_signal.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/rocm/ipc/rocm_ipc_iface.c Outdated
mp_params.elems_per_chunk = 128;
mp_params.max_elems = config->params.sigpool_max_elems;
mp_params.ops = &uct_rocm_base_signal_desc_mpool_ops;
mp_params.ops = &uct_rocm_ipc_signal_desc_mpool_ops;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ep_flush also takes descriptors from this pool through uct_rocm_base_ep_flush, which fills only the base fields and leaves region as whatever the previous user left there; it is NULL today only because progress resets it before ucs_mpool_put. pls set region = NULL on the flush path (e.g. a small ipc flush wrapper) so the invariant is not spread across two files — cuda keeps a separate flush_mpool for the same reason.

if (pin_count == 1) {
/* Before the lock is taken, another thread may pin the region again,
* or a drop that saw no pins may detach the mapping itself. */
ucs_rw_spinlock_write_lock(&uct_rocm_ipc_remote_cache.lock);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe check deactivate before taking the lock? with one pin per copy this takes the global write lock on every completion, and if deactivate is still 0 a concurrent drop_locked would see the already decremented pin_count and detach the mapping itself (it sets deactivate before reading pin_count).

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/rocm/ipc/rocm_ipc_iface.c Outdated
hsa_status_t status;
ucs_status_t completion_status;

ucs_queue_for_each_extract(rocm_signal, &iface->signal_queue, super.queue,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uct_rocm_ipc_iface_progress is a near-verbatim copy of uct_rocm_base_progress, and ep_flush plus the mpool ops below duplicate the base ones as well; can we add an optional release hook (callback + arg) to uct_rocm_base_signal_desc_t and keep using the base helpers? Root AGENTS.md asks not to duplicate implementation patterns, and fixes to the base copy (the one covered by test_rocm_signal) will no longer reach rocm_ipc.

if ((--region->pin_count == 0) &&
uint32_t pin_count;

pin_count = ucs_atomic_fsub32(&region->pin_count, 1);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not keep the pin_count decrement under the write lock as before? uct_rocm_ipc_cache_map_memhandle already takes the write lock for every zcopy, so unpin is not a new serialization point, and the lock-free variant forces this 6-line ordering proof to be re-verified on every future change of drop_locked.

uint32_t handler_done; /**< Async handler entered once */
int handler_armed; /**< Async handler owns signal */
unsigned pin_count; /**< Consumers holding mapped_addr */
uint32_t pin_count; /**< Consumers holding mapped_addr,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: pls also say that the decrement is done without the lock — "incremented under the lock" alone suggests all updates are serialized by the cache lock.

Suggested change
uint32_t pin_count; /**< Consumers holding mapped_addr,
uint32_t pin_count; /**< Consumers holding mapped_addr,
incremented under the lock,
decremented atomically */

@svc-nvidia-pr-review

Copy link
Copy Markdown

The pin_count == 0 recheck in the new unpin lock path is dead-but-harmless defensive code: pins are taken only via a pgtable hit under the write lock, and drop_locked sets deactivate and unlinks the region in the same critical section, so the "another thread may have pinned the region again" case in the comment cannot happen. Not worth a comment on its own, but the comment text slightly overstates the race.

@ddebonis-amd
ddebonis-amd marked this pull request as ready for review November 9, 2026 18:20
@ddebonis-amd
ddebonis-amd requested review from a team and mshanthagit as code owners November 9, 2026 18:20
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

return UCS_INPROGRESS;
}

static unsigned uct_rocm_ipc_iface_progress(uct_iface_h tl_iface)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this duplicates uct_rocm_base_progress() almost verbatim, and the same for uct_rocm_ipc_ep_flush() and the mpool ops below. can we instead add a release hook (e.g. void *release_arg + void (*release)(void*)) to uct_rocm_base_signal_desc_t and keep using the shared helpers? root AGENTS.md asks not to duplicate code or similar implementation patterns.

Note that only the zeroing of the new region field actually forces the fork: with super at offset 0, base progress/flush would otherwise still operate correctly on the derived descriptor, so a release hook in the base descriptor removes all three copies.

@svc-ucx

svc-ucx commented Nov 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests BlueField on worker 0) · commit 82eaf0b7

TL;DR: The BlueField gtest job did not fail on a test assertion — the Azure DevOps agent on the BlueField worker received a shutdown signal ~4 minutes into the gtest run and the whole job (including the still-running timeout 200m make -C test/gtest test) was killed as orphaned processes; this is a CI infrastructure event unrelated to PR #12268, so re-run the job and check the worker's agent/host health.

Full analysis

Summary: "Tests BlueField on worker 0" was cancelled mid-gtest with ##[error]The agent has received a shutdown signal / The operation was canceled, not by a test failure or a job time limit.

Root cause: Evidence from the log:

  • 23:18:12 — ==== Running make -C test/gtest test ==== then timeout 200m make -C test/gtest test starts.
  • 23:18:19 — shard 1 of 4 begins, [ RUN ] udx/test_ucp_tag_match_rndv_align.recv_align/0 <ud_x/get_zcopy,proto_v1>.
  • 23:21:53 — ##[error]The agent has received a shutdown signal. This can happen when the agent service is stopped, or a manually started agent is canceled.
  • 23:22:08 — Terminate orphan process: pid (2911917) (timeout), (make), (gtest).

The timeout 200m wrapper was still alive and was itself reaped as an orphan, so the 200-minute test budget was nowhere near exhausted — the job was not killed for exceeding a time limit. The termination came from the agent process going away (agent service stopped / BlueField host rebooted or the agent was OOM-killed), which tore down the entire process tree. The 3.5-minute window of silence is simply one gtest case executing; there is no watchdog/timeout message, no assertion failure, no [ FAILED ], and no abort/backtrace anywhere in the log.

Additionally, PR #12268 is "UCT/ROCM: fix for ipc_cache with async copy", and this same log reports [ INFO ] srd,cuda_copy,rocm_copy is not available — the ROCm transport is not even built/available on the BlueField worker, so the patch's code path cannot be exercised by this job. The failure is therefore not attributable to the commit under test.

Implicated commit: none — not a code regression. The tested commit [REDACTED:Hex High Entropy String] (PR #12268, ROCm ipc_cache) touches a transport that is unavailable on this worker.

File: n/a — infrastructure. Relevant harness entry point: contrib/test_jenkins.sh (the run_gtest_make default test step that was interrupted).

Suggested fix:

  1. Re-trigger the "Tests BlueField on worker 0" job — this is a transient, non-reproducible infra abort.
  2. Have CI ops inspect the BlueField worker (agent-02, workspace /scrap/azure/agent-02) around 2026-11-09T23:21:53Z: check journalctl -u vsts.agent* for an agent service stop/restart, dmesg//var/log/messages for an OOM kill of the agent or a host reset, and uptime to see whether the DPU rebooted. BlueField DPUs have limited RAM, and a 4-shard gtest run is a plausible OOM trigger — if the kernel OOM-killed the agent, reduce per-worker gtest memory pressure (fewer parallel shards or lower OMP_NUM_THREADS) for the BlueField workers.
  3. Do not raise any timeout: the timeout 200m guard had used under 4 minutes of its budget when the agent died.

Related: PR #12268 (UCT/ROCM: fix for ipc_cache with async copy) — the PR under test, not the cause. Other PRs showing the same agent has received a shutdown signal pattern on these workers: #12194 ("CI: Run the new and GPU jobs on new-01 only"), #12160, #11903, #12104, #12066 — the recurrence across unrelated PRs corroborates a worker-level infra problem rather than a code defect.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 37f33dde-dcb9-4330-bd9c-171bc582b3af in the triage console for the audit trail.

@ddebonis-amd
ddebonis-amd force-pushed the ipc-cache-fix branch 2 times, most recently from 4f48902 to 3387093 Compare November 10, 2026 16:08
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

status = uct_rocm_ipc_cache_map_memhandle(&unpacked->super,
&unpacked->mapped_addr,
&unpacked->region);
if (status != UCS_OK) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls map the failure to UCS_ERR_UNREACHABLE: ucp_ep_rkey_unpack_internal() drops only this md on UNREACHABLE but fails the whole rkey on any other status, so a peer we cannot attach to now breaks rkeys that other lanes could still use. uct_cuda_ipc_rkey_unpack() does the same for its accessibility check.

Suggested change
if (status != UCS_OK) {
if (status != UCS_OK) {
ucs_free(unpacked);
return UCS_ERR_UNREACHABLE;
}

* owner's next hsa_amd_ipc_memory_create, which then fails in
* AMDKFD_IOC_EXPORT_DMABUF with ENOENT. Copies add their own pins, so
* this hold is what keeps the mapping cached between operations. */
if (uct_rocm_ipc_cache_enabled()) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need a pin for the whole rkey lifetime? drop_locked() skips regions with pin_count != 0, so while an unpacked rkey exists the free-event handler can no longer detach the import, which is the ENOENT case this PR fixes. Copies and rkey_ptr already take their own pins.

Comment thread src/uct/rocm/ipc/rocm_ipc_iface.c Outdated
static unsigned uct_rocm_ipc_iface_progress(uct_iface_h tl_iface)
{
uct_rocm_ipc_iface_t *iface = ucs_derived_of(tl_iface, uct_rocm_ipc_iface_t);
static const unsigned max_signals = 16;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this loop, the ep_flush above and the mpool ops below are copies of uct_rocm_base_progress()/uct_rocm_base_ep_flush()/uct_rocm_base_signal_desc_mpool_ops with only the pin release added (~100 duplicated lines). can we add a release callback (or the pin handle) to uct_rocm_base_signal_desc_t and keep using the base helpers? root AGENTS.md asks not to duplicate code.

Comment thread src/uct/rocm/ipc/rocm_ipc_iface.c Outdated
return enabled;
}

int uct_rocm_ipc_cache_enabled(void)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: this duplicates uct_rocm_ipc_signal_invalidate_enabled() except for the cached field; pls fill the config once in a shared helper and cache both values.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/rocm/ipc/rocm_ipc_md.c Outdated
&unpacked->region);
if (status != UCS_OK) {
ucs_free(unpacked);
return UCX_ERR_UNREACHABLE;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return UCX_ERR_UNREACHABLE;
return UCS_ERR_UNREACHABLE;

blocker: UCX_ERR_UNREACHABLE does not exist, this file does not compile.

* across hipFree, and the next hsa_amd_ipc_memory_create fails with
* ENOENT from AMDKFD_IOC_EXPORT_DMABUF. A drop that already detached, or a
* pin that arrived after this unpin, leaves mapped_addr alone. */
if (pin_count == 1) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

detaching on the last unpin removes the cache benefit for the rndv path: the rkey is unpacked per operation in ucp_proto_rndv_send_reply and destroyed on completion, so pin_count hits 0 after every transfer and each rendezvous pays a new hsa_amd_ipc_memory_attach plus detach. can we keep the mapping until the region is actually dropped (free event or stale handle) instead?

* owner's next hsa_amd_ipc_memory_create, which then fails in
* AMDKFD_IOC_EXPORT_DMABUF with ENOENT. Copies add their own pins, so
* this hold is what keeps the mapping cached between operations. */
if (uct_rocm_ipc_cache_enabled()) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with the rkey pinning the region for its whole lifetime, the free-event handler can no longer detach (drop_locked skips the detach when pin_count != 0), so the import outlives the owner's hipFree. doesn't that bring back the ENOENT case described here whenever the owner frees before the importer releases the rkey?

void *mapped_addr;
uct_completion_t *comp;
ucs_queue_elem_t queue;
uct_rocm_base_signal_release_cb_t release; /**< Called when signal completes */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
uct_rocm_base_signal_release_cb_t release; /**< Called when signal completes */
uct_rocm_base_signal_release_cb_t release; /**< On completion */

minor: the line is 88 columns, over the 80-column limit in docs/CodeStyle.md.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

* owner's next hsa_amd_ipc_memory_create, which then fails in
* AMDKFD_IOC_EXPORT_DMABUF with ENOENT. Copies add their own pins, so
* this hold is what keeps the mapping cached between operations. */
if (uct_rocm_ipc_cache_enabled()) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls honor params->flags here: rndv unpacks its rkeys without UCT_RKEY_UNPACK_FLAG_FREE_UNUSED_MAPPING (ucp_proto_rndv_send_reply(), ucp_ep_rkey_unpack_wo_flags()), which per the API means the local mapping should stay cached after uct_rkey_release. With the unconditional detach in the last unpin, each rndv operation now unpacks, pins, releases and so re-runs hsa_amd_ipc_memory_attach/hsa_amd_ipc_memory_detach on the hot path - exactly what the handle cache is for. cuda_ipc stores this flag in its rkey handle and only frees the mapping when it is set.

@@ -48,7 +49,9 @@ void uct_rocm_ipc_destroy_cache(uct_rocm_ipc_cache_t *cache);
/**

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all callers now pass a non-NULL region_p; maybe make it mandatory and drop the optional case? Without a pin, the returned mapped_addr can be detached by a concurrent last unpin as soon as the cache lock is released, so the NULL variant is now unsafe to use.

@@ -286,14 +291,24 @@ int uct_rocm_ipc_signal_invalidate_enabled(void)
status = ucs_config_parser_fill_opts(&config, &uct_rocm_ipc_tl.config,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: unrelated whitespace change, pls drop.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual gap: as in previous rounds, none of this is exercisable without two-process ROCm hardware, so the attach/detach frequency change in the rndv path is worth measuring (e.g. ucx_perftest rocm_ipc rndv, repeated transfers to the same remote buffer) rather than only functionally testing.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.


static void uct_rocm_ipc_signal_release(void *arg)
{
uct_rocm_ipc_cache_unpin(arg, 1);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uct_rocm_ipc_signal_release passes free_unused_mapping=1, so whenever the transfer's pin happens to be the last one the cached import is detached and the next operation pays hsa_amd_ipc_memory_attach again, even though handle caching is enabled. Can we pass 0 here (and in uct_rocm_ipc_md_mem_elem_release, which used to keep the mapping)? A deactivated region is still detached on the last unpin.

* it only when UCT_RKEY_UNPACK_FLAG_FREE_UNUSED_MAPPING is set, which is
* the window path that must drop the dma-buf before the next export.
* Rendezvous leaves the flag clear so the import stays cached. */
if (uct_rocm_ipc_cache_enabled()) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why map and pin here instead of on first use? For rkeys unpacked without UCT_RKEY_UNPACK_FLAG_FREE_UNUSED_MAPPING this adds an attach for every unpacked rkey even when the rocm_ipc lane is never used, and the pin lives as long as the rkey, so the free-event handler can no longer detach the import and the owner's free is deferred until the peer releases the rkey.

if (status != UCS_OK) {
ucs_free(rkey_handle);
ucs_free(unpacked);
return UCS_ERR_UNREACHABLE;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: this failure is recoverable (ucp only marks the md unreachable), but uct_rocm_ipc_cache_map_region reports it with ucs_error, so a benign "owner already freed the buffer" race now prints an error. cuda_ipc passes a log level into uct_cuda_ipc_map_memhandle for exactly this case.

void *mapped_addr;
uct_completion_t *comp;
ucs_queue_elem_t queue;
uct_rocm_base_signal_release_cb_t release; /**< Called when signal completes */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
uct_rocm_base_signal_release_cb_t release; /**< Called when signal completes */
uct_rocm_base_signal_release_cb_t release; /**< Signal completion cb */

minor: the line is 88 columns, over the 80-column limit in docs/CodeStyle.md.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap (not worth a new thread, already raised in earlier rounds): the reattach-on-cache-hit path and the pin/unpin interleaving are only reachable with two processes on real ROCm hardware, so the attach/detach frequency change in the rndv and device-mem-element paths needs measurement rather than only a functional run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants