Repository navigation
UCT/ROCM: fix for ipc_cache with async copy - #12268
ddebonis-amd wants to merge 1 commit into
Conversation
|
🤖 Starting review — findings will be posted here when done. |
84b48fe to
bc82c59
Compare
|
🤖 Starting review — findings will be posted here when done. |
| uint32_t handler_done; /**< Async handler entered once */ | ||
| int handler_armed; /**< Async handler owns signal */ | ||
| unsigned pin_count; /**< Consumers holding mapped_addr */ | ||
| uint32_t pin_count; /**< Consumers holding mapped_addr, |
There was a problem hiding this comment.
Field comment documents only half the protocol. Suggestion:
| uint32_t pin_count; /**< Consumers holding mapped_addr, | |
| uint32_t pin_count; /**< Consumers holding mapped_addr, | |
| incremented under the lock, | |
| decremented atomically */ |
Minor: the decrement is now lockless, which is the part a reader needs to know here.
|
Residual coverage gap: everything here needs real ROCm hardware with two processes to exercise (IPC import/export plus the free-event invalidation path), so CI almost certainly will not hit the pin/unpin race or the new |
bc82c59 to
b3bf0e5
Compare
|
🤖 Starting review — findings will be posted here when done. |
| mp_params.elems_per_chunk = 128; | ||
| mp_params.max_elems = config->params.sigpool_max_elems; | ||
| mp_params.ops = &uct_rocm_base_signal_desc_mpool_ops; | ||
| mp_params.ops = &uct_rocm_ipc_signal_desc_mpool_ops; |
There was a problem hiding this comment.
ep_flush also takes descriptors from this pool through uct_rocm_base_ep_flush, which fills only the base fields and leaves region as whatever the previous user left there; it is NULL today only because progress resets it before ucs_mpool_put. pls set region = NULL on the flush path (e.g. a small ipc flush wrapper) so the invariant is not spread across two files — cuda keeps a separate flush_mpool for the same reason.
| if (pin_count == 1) { | ||
| /* Before the lock is taken, another thread may pin the region again, | ||
| * or a drop that saw no pins may detach the mapping itself. */ | ||
| ucs_rw_spinlock_write_lock(&uct_rocm_ipc_remote_cache.lock); |
There was a problem hiding this comment.
maybe check deactivate before taking the lock? with one pin per copy this takes the global write lock on every completion, and if deactivate is still 0 a concurrent drop_locked would see the already decremented pin_count and detach the mapping itself (it sets deactivate before reading pin_count).
2a4428f to
4f48902
Compare
|
🤖 Starting review — findings will be posted here when done. |
| hsa_status_t status; | ||
| ucs_status_t completion_status; | ||
|
|
||
| ucs_queue_for_each_extract(rocm_signal, &iface->signal_queue, super.queue, |
There was a problem hiding this comment.
uct_rocm_ipc_iface_progress is a near-verbatim copy of uct_rocm_base_progress, and ep_flush plus the mpool ops below duplicate the base ones as well; can we add an optional release hook (callback + arg) to uct_rocm_base_signal_desc_t and keep using the base helpers? Root AGENTS.md asks not to duplicate implementation patterns, and fixes to the base copy (the one covered by test_rocm_signal) will no longer reach rocm_ipc.
| if ((--region->pin_count == 0) && | ||
| uint32_t pin_count; | ||
|
|
||
| pin_count = ucs_atomic_fsub32(®ion->pin_count, 1); |
There was a problem hiding this comment.
why not keep the pin_count decrement under the write lock as before? uct_rocm_ipc_cache_map_memhandle already takes the write lock for every zcopy, so unpin is not a new serialization point, and the lock-free variant forces this 6-line ordering proof to be re-verified on every future change of drop_locked.
| uint32_t handler_done; /**< Async handler entered once */ | ||
| int handler_armed; /**< Async handler owns signal */ | ||
| unsigned pin_count; /**< Consumers holding mapped_addr */ | ||
| uint32_t pin_count; /**< Consumers holding mapped_addr, |
There was a problem hiding this comment.
minor: pls also say that the decrement is done without the lock — "incremented under the lock" alone suggests all updates are serialized by the cache lock.
| uint32_t pin_count; /**< Consumers holding mapped_addr, | |
| uint32_t pin_count; /**< Consumers holding mapped_addr, | |
| incremented under the lock, | |
| decremented atomically */ |
|
The |
|
🤖 Starting review — findings will be posted here when done. |
| return UCS_INPROGRESS; | ||
| } | ||
|
|
||
| static unsigned uct_rocm_ipc_iface_progress(uct_iface_h tl_iface) |
There was a problem hiding this comment.
this duplicates uct_rocm_base_progress() almost verbatim, and the same for uct_rocm_ipc_ep_flush() and the mpool ops below. can we instead add a release hook (e.g. void *release_arg + void (*release)(void*)) to uct_rocm_base_signal_desc_t and keep using the shared helpers? root AGENTS.md asks not to duplicate code or similar implementation patterns.
Note that only the zeroing of the new region field actually forces the fork: with super at offset 0, base progress/flush would otherwise still operate correctly on the derived descriptor, so a release hook in the base descriptor removes all three copies.
|
🤖 CI Triage Agent — TL;DR: The BlueField gtest job did not fail on a test assertion — the Azure DevOps agent on the BlueField worker received a shutdown signal ~4 minutes into the gtest run and the whole job (including the still-running Full analysisSummary: "Tests BlueField on worker 0" was cancelled mid-gtest with Root cause: Evidence from the log:
The Additionally, PR #12268 is "UCT/ROCM: fix for ipc_cache with async copy", and this same log reports Implicated commit: none — not a code regression. The tested commit File: n/a — infrastructure. Relevant harness entry point: Suggested fix:
Related: PR #12268 (
|
4f48902 to
3387093
Compare
|
🤖 Starting review — findings will be posted here when done. |
| status = uct_rocm_ipc_cache_map_memhandle(&unpacked->super, | ||
| &unpacked->mapped_addr, | ||
| &unpacked->region); | ||
| if (status != UCS_OK) { |
There was a problem hiding this comment.
pls map the failure to UCS_ERR_UNREACHABLE: ucp_ep_rkey_unpack_internal() drops only this md on UNREACHABLE but fails the whole rkey on any other status, so a peer we cannot attach to now breaks rkeys that other lanes could still use. uct_cuda_ipc_rkey_unpack() does the same for its accessibility check.
| if (status != UCS_OK) { | |
| if (status != UCS_OK) { | |
| ucs_free(unpacked); | |
| return UCS_ERR_UNREACHABLE; | |
| } |
| * owner's next hsa_amd_ipc_memory_create, which then fails in | ||
| * AMDKFD_IOC_EXPORT_DMABUF with ENOENT. Copies add their own pins, so | ||
| * this hold is what keeps the mapping cached between operations. */ | ||
| if (uct_rocm_ipc_cache_enabled()) { |
There was a problem hiding this comment.
why do we need a pin for the whole rkey lifetime? drop_locked() skips regions with pin_count != 0, so while an unpacked rkey exists the free-event handler can no longer detach the import, which is the ENOENT case this PR fixes. Copies and rkey_ptr already take their own pins.
| static unsigned uct_rocm_ipc_iface_progress(uct_iface_h tl_iface) | ||
| { | ||
| uct_rocm_ipc_iface_t *iface = ucs_derived_of(tl_iface, uct_rocm_ipc_iface_t); | ||
| static const unsigned max_signals = 16; |
There was a problem hiding this comment.
this loop, the ep_flush above and the mpool ops below are copies of uct_rocm_base_progress()/uct_rocm_base_ep_flush()/uct_rocm_base_signal_desc_mpool_ops with only the pin release added (~100 duplicated lines). can we add a release callback (or the pin handle) to uct_rocm_base_signal_desc_t and keep using the base helpers? root AGENTS.md asks not to duplicate code.
| return enabled; | ||
| } | ||
|
|
||
| int uct_rocm_ipc_cache_enabled(void) |
There was a problem hiding this comment.
minor: this duplicates uct_rocm_ipc_signal_invalidate_enabled() except for the cached field; pls fill the config once in a shared helper and cache both values.
3387093 to
6304513
Compare
|
🤖 Starting review — findings will be posted here when done. |
6304513 to
533f932
Compare
| &unpacked->region); | ||
| if (status != UCS_OK) { | ||
| ucs_free(unpacked); | ||
| return UCX_ERR_UNREACHABLE; |
There was a problem hiding this comment.
| return UCX_ERR_UNREACHABLE; | |
| return UCS_ERR_UNREACHABLE; |
blocker: UCX_ERR_UNREACHABLE does not exist, this file does not compile.
| * across hipFree, and the next hsa_amd_ipc_memory_create fails with | ||
| * ENOENT from AMDKFD_IOC_EXPORT_DMABUF. A drop that already detached, or a | ||
| * pin that arrived after this unpin, leaves mapped_addr alone. */ | ||
| if (pin_count == 1) { |
There was a problem hiding this comment.
detaching on the last unpin removes the cache benefit for the rndv path: the rkey is unpacked per operation in ucp_proto_rndv_send_reply and destroyed on completion, so pin_count hits 0 after every transfer and each rendezvous pays a new hsa_amd_ipc_memory_attach plus detach. can we keep the mapping until the region is actually dropped (free event or stale handle) instead?
| * owner's next hsa_amd_ipc_memory_create, which then fails in | ||
| * AMDKFD_IOC_EXPORT_DMABUF with ENOENT. Copies add their own pins, so | ||
| * this hold is what keeps the mapping cached between operations. */ | ||
| if (uct_rocm_ipc_cache_enabled()) { |
There was a problem hiding this comment.
with the rkey pinning the region for its whole lifetime, the free-event handler can no longer detach (drop_locked skips the detach when pin_count != 0), so the import outlives the owner's hipFree. doesn't that bring back the ENOENT case described here whenever the owner frees before the importer releases the rkey?
| void *mapped_addr; | ||
| uct_completion_t *comp; | ||
| ucs_queue_elem_t queue; | ||
| uct_rocm_base_signal_release_cb_t release; /**< Called when signal completes */ |
There was a problem hiding this comment.
| uct_rocm_base_signal_release_cb_t release; /**< Called when signal completes */ | |
| uct_rocm_base_signal_release_cb_t release; /**< On completion */ |
minor: the line is 88 columns, over the 80-column limit in docs/CodeStyle.md.
|
🤖 Starting review — findings will be posted here when done. |
| * owner's next hsa_amd_ipc_memory_create, which then fails in | ||
| * AMDKFD_IOC_EXPORT_DMABUF with ENOENT. Copies add their own pins, so | ||
| * this hold is what keeps the mapping cached between operations. */ | ||
| if (uct_rocm_ipc_cache_enabled()) { |
There was a problem hiding this comment.
pls honor params->flags here: rndv unpacks its rkeys without UCT_RKEY_UNPACK_FLAG_FREE_UNUSED_MAPPING (ucp_proto_rndv_send_reply(), ucp_ep_rkey_unpack_wo_flags()), which per the API means the local mapping should stay cached after uct_rkey_release. With the unconditional detach in the last unpin, each rndv operation now unpacks, pins, releases and so re-runs hsa_amd_ipc_memory_attach/hsa_amd_ipc_memory_detach on the hot path - exactly what the handle cache is for. cuda_ipc stores this flag in its rkey handle and only frees the mapping when it is set.
| @@ -48,7 +49,9 @@ void uct_rocm_ipc_destroy_cache(uct_rocm_ipc_cache_t *cache); | |||
| /** | |||
There was a problem hiding this comment.
all callers now pass a non-NULL region_p; maybe make it mandatory and drop the optional case? Without a pin, the returned mapped_addr can be detached by a concurrent last unpin as soon as the cache lock is released, so the NULL variant is now unsafe to use.
| @@ -286,14 +291,24 @@ int uct_rocm_ipc_signal_invalidate_enabled(void) | |||
| status = ucs_config_parser_fill_opts(&config, &uct_rocm_ipc_tl.config, | |||
There was a problem hiding this comment.
minor: unrelated whitespace change, pls drop.
|
Residual gap: as in previous rounds, none of this is exercisable without two-process ROCm hardware, so the attach/detach frequency change in the rndv path is worth measuring (e.g. |
533f932 to
59703ac
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
|
||
| static void uct_rocm_ipc_signal_release(void *arg) | ||
| { | ||
| uct_rocm_ipc_cache_unpin(arg, 1); |
There was a problem hiding this comment.
uct_rocm_ipc_signal_release passes free_unused_mapping=1, so whenever the transfer's pin happens to be the last one the cached import is detached and the next operation pays hsa_amd_ipc_memory_attach again, even though handle caching is enabled. Can we pass 0 here (and in uct_rocm_ipc_md_mem_elem_release, which used to keep the mapping)? A deactivated region is still detached on the last unpin.
| * it only when UCT_RKEY_UNPACK_FLAG_FREE_UNUSED_MAPPING is set, which is | ||
| * the window path that must drop the dma-buf before the next export. | ||
| * Rendezvous leaves the flag clear so the import stays cached. */ | ||
| if (uct_rocm_ipc_cache_enabled()) { |
There was a problem hiding this comment.
why map and pin here instead of on first use? For rkeys unpacked without UCT_RKEY_UNPACK_FLAG_FREE_UNUSED_MAPPING this adds an attach for every unpacked rkey even when the rocm_ipc lane is never used, and the pin lives as long as the rkey, so the free-event handler can no longer detach the import and the owner's free is deferred until the peer releases the rkey.
| if (status != UCS_OK) { | ||
| ucs_free(rkey_handle); | ||
| ucs_free(unpacked); | ||
| return UCS_ERR_UNREACHABLE; |
There was a problem hiding this comment.
minor: this failure is recoverable (ucp only marks the md unreachable), but uct_rocm_ipc_cache_map_region reports it with ucs_error, so a benign "owner already freed the buffer" race now prints an error. cuda_ipc passes a log level into uct_cuda_ipc_map_memhandle for exactly this case.
| void *mapped_addr; | ||
| uct_completion_t *comp; | ||
| ucs_queue_elem_t queue; | ||
| uct_rocm_base_signal_release_cb_t release; /**< Called when signal completes */ |
There was a problem hiding this comment.
| uct_rocm_base_signal_release_cb_t release; /**< Called when signal completes */ | |
| uct_rocm_base_signal_release_cb_t release; /**< Signal completion cb */ |
minor: the line is 88 columns, over the 80-column limit in docs/CodeStyle.md.
|
Residual coverage gap (not worth a new thread, already raised in earlier rounds): the reattach-on-cache-hit path and the pin/unpin interleaving are only reachable with two processes on real ROCm hardware, so the attach/detach frequency change in the rndv and device-mem-element paths needs measurement rather than only a functional run. |
What?
uct_rocm_ipc_ep_zcopy now pins the cached mapping and holds it until the async copy completes.