Skip to content

UCT/ZE_IPC: Fix correctness issues - #8

Merged
yuanwu2017 merged 15 commits into
zhangxiaoli73:cherry/add-ze-ipc-supportfrom
tsg-:fix/pr11218-ze-ipc-correctness
Aug 25, 2026
Merged

yuanwu2017 merged 15 commits into
zhangxiaoli73:cherry/add-ze-ipc-supportfrom
tsg-:fix/pr11218-ze-ipc-correctness

Conversation

@tsg-

@tsg- tsg- commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator
  • UCT/ZE_IPC: release IPC handle on mem_dereg
  • UCT/ZE_IPC: invalidate stale cache mappings, purge on MD close
  • UCT/ZE_IPC: reject multi-iov and out-of-range rkeys in zcopy
  • UCT/ZE_IPC: drop unsignaled EVENT_FD, fix teardown double-close
  • UCT/ZE_IPC: supply ep_outstanding_purge to unblock iface init
  • UCT/ZE_IPC: disable iface progress on teardown
  • TEST/UCT/ZE_IPC: add iface RMA and cache identity/purge coverage
  • UCT/ZE_IPC: reject MAX_CMD_LISTS=0, propagate completion errors
  • UCT/ZE_IPC: resolve remote allocation's own device on map
  • UCT/ZE: fix codestyle findings from Azure format-code check
  • AUTHORS: add Tushar Gohad
  • UCT/ZE_IPC: require a quiesced context when purging the cache
  • UCT/ZE_IPC: defer closing mappings that still have in-flight operations
  • UCT/ZE_IPC: close the mapping when caching it fails
  • UCT/ZE_IPC: discriminate cached mappings by allocation and process

What?

This PR adds correctness, lifecycle, and test fixes on top of
cherry/add-ze-ipc-support, the head branch for openucx#11218.

It fixes IPC handle and mapping leaks, stale mapping reuse, teardown
ordering, invalid zcopy requests, completion error propagation, and
remote device selection. It also adds ZE IPC cache and RMA coverage.

Why?

The current ZE IPC implementation can leak driver resources, reuse stale
mappings after address or PID reuse, close mappings while operations are
still in flight, and report some failed operations as successful. It can
also abort during interface initialization or leave progress callbacks
registered after teardown.

These issues affect correctness and transport lifetime, rather than only
error handling or diagnostics.

How?

Cache hits now discriminate allocations using the Level Zero allocation
ID, allocation length, and peer process creation time. Invalidated
mappings with outstanding operations move to an orphan list and are
closed after their final reference is released. Cache mappings are
purged while the ZE context is still valid during MD teardown.

The patchset also:

  • releases exported IPC handles during deregistration
  • validates zcopy IOV count and remote-key bounds
  • rejects MAX_CMD_LISTS=0
  • propagates Level Zero completion errors
  • disables progress before interface destruction
  • uses the packed remote device ordinal when opening mappings
  • removes the unsignaled EVENT_FD capability

The packed rkey layout changes, so both peers must use the same build.
ZE IPC is not yet released; the layout should be versioned before it
becomes a compatibility contract.

Verification:

  • 16 ZE IPC unit tests pass with the Level Zero stub driver
  • full patchset passes the UCX git-clang-format check
  • git diff --check passes

tsg- added 15 commits August 21, 2026 09:02
mem_reg called zeMemGetIpcHandle() but mem_dereg never released it
with zeMemPutIpcHandle(), leaking a driver resource on every dereg.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
Cache hits matched on (pid, address) alone, so a freed-and-reused
remote VA returned a mapping to the wrong allocation. Compare the IPC
handle and length on hit and remap on mismatch. Also purge cached
regions tied to a context before it is destroyed in md_close(), so a
later close doesn't reference a torn-down context.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
Only iov[0] was ever copied, silently dropping data for iovcnt > 1;
reject it like other single-iov transports do. Also validate
remote_addr/length against the rkey's [address, address+length) range
before mapping, closing an OOB/underflow access via a bad rkey.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
The eventfd was armed but never signaled on completion, so a worker
could block indefinitely; drop the EVENT_FD capability instead of
advertising a fd that never fires. Also route iface teardown through
uct_ze_ipc_unmap_memhandle() instead of closing the IPC handle/fd
directly, which double-closed shared cache entries.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
uct_base_iface_t asserts every transport supplies
internal_ops.ep_outstanding_purge; ze_ipc left it NULL, aborting on
iface construction. Stub it out like rocm_ipc/ze_copy/cuda_ipc do.
This only unblocks startup; it does not fix the IPC correctness/
lifetime issues already addressed in prior commits.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
Iface cleanup freed the iface without unregistering its progress
callback, leaving a dangling entry in the worker's callbackq that
would later run against freed memory. Disable progress first, like
tcp/mm ifaces do.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
New test_ze_ipc_rma.cc opens a real ze_ipc iface pair and exercises
put/get_zcopy plus the multi-iov/bounds rejections; this is what
caught the missing ep_outstanding_purge and progress-teardown bugs
fixed in prior commits. test_ze_ipc_cache.cc gains coverage for the
stale-address remap and purge-by-context paths.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
MAX_CMD_LISTS=0 produced num_cmd_lists=0, causing a div-by-zero in the
ep's round-robin cmd-list selection; reject it at iface init instead.
Also stop treating any non-NOT_READY zeEventQueryStatus() result as
success - propagate real ZE errors to the completion callback.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
post_copy always mapped remote memory using the local iface's own
device, ignoring the peer's packed dev_num. On multi-GPU nodes this
could open a GPU-1 allocation against GPU-0. Resolve the device by
the rkey's dev_num instead.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
Applies the exact reformatting Azure build 131309's "format code" job
(git-clang-format) flagged for this PR, resolved by hand against my
prior commits: pointer-cast spacing, brace placement, wrapped-line
alignment. No functional change.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
uct_ze_ipc_purge_cache_by_context() closes mappings regardless of
refcount and frees the cache descriptors, which is only sound if nothing
is still using them. That precondition was never written down.

It is not an extra constraint: the sole caller is md_close(), which
destroys the ZE context on the next line, so an in-flight copy is dead
either way. Honoring refcount would be worse - the mapping would survive
to be closed against a destroyed context. Document that.

The purge test now covers both paths: the quiesced sequence md_close()
actually produces, and a purge with a reference outstanding.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
When a remote VA is reused, map_memhandle() drops the stale region from
the page table and closes its mapping immediately - even if an earlier
operation is still copying through it. The copy then runs against closed
memory, and its unmap tripped the ucs_assert() on the page table lookup
because the entry now belongs to the new allocation.

Move such regions to a per-cache orphan list instead, keyed for lookup by
mapped address since the page table entry is gone, and let the last unmap
close them. uct_ze_ipc_cache_invalidate_regions() had the same problem and
now shares the path.

Factor out uct_ze_ipc_close_mapping() so the fd is closed even when
zeMemCloseIpcHandle() fails, which leaked one dup'd fd per failure.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
map_memhandle() opened the IPC handle and then jumped to err: if the
region allocation or the page table insert failed, so the mapping and its
dup'd fd were leaked and the caller got back a status with *mapped_addr
still set. Add an err_close: path that closes the mapping and clears the
output arguments.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
Comparing the packed IPC handle to decide whether a cached mapping still
refers to the same remote allocation does not work: on Linux the handle
carries an fd number from the exporting process, and fds are reused as
eagerly as virtual addresses, so a freed-and-reallocated peer buffer at
the same address can produce a byte-identical handle and hit in the
cache.

Pack ze_memory_allocation_properties_t.id and compare it with the length
instead. That id is a per-process counter, so it does not survive PID
reuse: once a peer exits and a new process takes over its PID, the new
process's first allocation can report the same pid, address, id and
length as the dead peer's, and the cache hands back the dead mapping.
Pack the peer's process creation time as the generation that separates
the two, as UCT keepalive already does, and fail mkey_pack if it cannot
be read rather than export a key the importer would have to distrust.

Both discriminators live in the cache-hit predicate, so a restarted peer
reuses the existing per-(pid, context) cache and invalidates the stale
region on first touch. Keying the cache hash on the generation instead
would strand every dead peer's mappings until md_close.

Note the id is Intel compute runtime behavior - ZE defines it as an
allocation identifier without specifying its uniqueness scope.

This changes the packed rkey layout, so both peers must run the same
build. That is not a compatibility break yet, since the transport is not
released, but the layout should get a version before it is.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
@yuanwu2017
yuanwu2017 merged commit bab40dc into zhangxiaoli73:cherry/add-ze-ipc-support Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants