Skip to content

UCT/ZE/ZE_IPC: enable level zero ipc support for Intel GPUs - #11218

Merged
brminich merged 35 commits into
openucx:masterfrom
zhangxiaoli73:cherry/add-ze-ipc-support
Oct 5, 2026
Merged

brminich merged 35 commits into
openucx:masterfrom
zhangxiaoli73:cherry/add-ze-ipc-support

Conversation

@zhangxiaoli73

@zhangxiaoli73 zhangxiaoli73 commented Feb 27, 2026 •

Copy link
Copy Markdown
Contributor

What?

Enable level zero IPC support for Intel GPUs, this PR adds:

  • Add level zero IPC component
  • Add level zero IPC iface
  • Add level zero async copy for IPC EP
  • Add IPC cache to reduce IPC exchange overhead in certain cases

Why?

We want to provide IPC transport for users within a single node.

@j-xiong

j-xiong commented Mar 16, 2026 •

Copy link
Copy Markdown
Contributor

@zhangxiaoli73 Please fix the commit titles and do a rebase to resolve the conflicts. Maybe consolidate the two commits at the same time since a forced push is inevitable.

@yosefe I am assuming forced push is allow in this case. Is that right? If not, what is the recommended way to fix the commit titles?

@zhangxiaoli73

Copy link
Copy Markdown
Contributor Author

@zhangxiaoli73 Please fix the commit titles and do a rebase to resolve the conflicts. Maybe consolidate the two commits at the same time since a forced push is inevitable.

@yosefe I am assuming forced push is allow in this case. Is that right? If not, what is the recommended way to fix the commit titles?

Got it. Let me know if I can force push to change the commit.

@zhangxiaoli73
zhangxiaoli73 force-pushed the cherry/add-ze-ipc-support branch from 118b1f6 to 8b1525f Compare April 3, 2026 10:52
@openucx openucx deleted a comment from svc-nixl May 4, 2026
@yuanwu2017

Copy link
Copy Markdown
Contributor

gtest results:
image

Performance bechmark:
image

@yuanwu2017

Copy link
Copy Markdown
Contributor

@yosefe Please help to review.

@yuanwu2017

Copy link
Copy Markdown
Contributor

Hi @yosefe @brminich,

Just a friendly ping on this one 🙂 CI is green now and the branch is up to date with master. Whenever you have a few minutes, could you take a look? Would really love to hear your feedback — happy to tweak anything based on your comments.

Comment thread src/uct/ze/copy/ze_copy_ep.c Resolved
Comment thread src/uct/ze/copy/ze_copy_ep.c Resolved
Comment thread src/uct/ze/copy/ze_copy_ep.c Resolved
@yosefe

yosefe commented May 22, 2026

Copy link
Copy Markdown
Member

@zhangxiaoli73 @yafshar can we add a build with ZE to builds.sh so it will run in CI with at least basic compilation check?
Also, is there a "mock" library for ZE hardware that we can run in CI?

@yafshar

yafshar commented May 22, 2026

Copy link
Copy Markdown
Contributor

can we add a build with ZE to builds.sh so it will run in CI with at least basic compilation check?
Also, is there a "mock" library for ZE hardware that we can run in CI?

Yes, we can add a ZE compile check to builds.sh and run it in CI.

For this to be a meaningful ZE check, the selected Linux job must have Level Zero headers and the ze_loader development package available (or an equivalent module/path setup). We should also force configuration with --with-ze so the job fails when ZE dependencies are missing. If we rely on the default auto-detect mode, ZE may be silently skipped and the build could pass without actually validating ZE compilation.
I plan to send a separate CI-focused PR that:

  • Adds a ZE compile-only check in one x86_64 Linux lane
  • Adds the required container/job setup for Level Zero dependencies
  • Keeps ZE runtime coverage out of scope

In parallel, I will investigate a production-grade ZE mock/simulation option and follow up with another separate PR, as that work is broader in scope.

@yosefe

yosefe commented May 28, 2026

Copy link
Copy Markdown
Member

hi @yafshar ,
I think we can start with log hanging fruit - add container URL just to compile ZE code. It will be good to catch potential regressions caused by infrastructure refactoring.

@yuanwu2017
yuanwu2017 force-pushed the cherry/add-ze-ipc-support branch from 8f6d46f to 5b3f330 Compare June 1, 2026 08:53
Comment thread buildlib/pr/build_job.yml Outdated
Comment on lines +155 to +173
set -eEx
if [ "${install_ze_deps}" != "yes" ]; then
exit 0
fi

# The Azure container runs as a non-root user; use sudo when
# available to install the Level Zero development headers.
SUDO=
if [ "$(id -u)" -ne 0 ]; then
SUDO="sudo -E"
fi

export DEBIAN_FRONTEND=noninteractive
$SUDO apt-get update
$SUDO apt-get install -y --no-install-recommends libze-dev || \
$SUDO apt-get install -y --no-install-recommends level-zero-dev
displayName: Install ZE dev package
env:
install_ze_deps: $(install_ze_deps)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a publicly accessible container with ZE libs installed? it will be more efficient (less code, shorter time) than installing in each CI

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.
The public intel/deep-learning-essentials image is missing autoconf/automake/libtool/m4/doxygen, and Azure forces container jobs to run as non-root so we cannot apt-get install them at runtime. I wrapped it in a small builder image (recipe added at buildlib/dockers/ze-builder.Dockerfile), same pattern as the CUDA/ROCm lanes.

The image is temporarily published to my personal GHCR (ghcr.io/yuanwu2017/ucx-ze-builder:ubuntu24.04) so CI can come up. Could you rebuild this Dockerfile and push it to MLNX Harbor?

Comment thread buildlib/tools/builds.sh Outdated
Comment on lines +536 to +538
ze)
tests=('build_ze')
;;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why need special 'ze' mode, if we already add it to base_tests?
can use test_mode=short - same as done for ROCm container

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. Removed it.

@yuanwu2017

Copy link
Copy Markdown
Contributor

The Codestyle / commit title failure isn't from this PR. Every "Bad commit title" in the log (#11017, #11080, #11053, #11059, …) is an upstream squash-merge, not ours.

@yosefe

yosefe commented Jun 4, 2026

Copy link
Copy Markdown
Member

@yuanwu2017

Copy link
Copy Markdown
Contributor

@yuanwu2017

Copy link
Copy Markdown
Contributor

@yosefe Please help to review and merge the patch.

@yuanwu2017

Copy link
Copy Markdown
Contributor

@yosefe Please help to review.

@yuanwu2017

Copy link
Copy Markdown
Contributor

@yosefe
These bad commits are not from this PR. They are caused by sync the latest code. Please help to review and merge.

Bad commit title: 'Change local/remote handle names (#11145)'
Bad commit title: 'Added Intel Gaudi GDR support for UCX (#11017)'
Bad commit title: 'docs: updated rocm link (#11080)'
Bad commit title: 'Fix peermem availability check (#11143)'

@brminich brminich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm in general

Comment thread AUTHORS Outdated
Michal Shalev <mshalev@nvidia.com>
Mike Dubman <miked@mellanox.com>
Mikhail Brinskii <mikhailb@nvidia.com>
Mikhail Brinskiy <brminich@users.noreply.github.com>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is it needed?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing those lines breaks the AUTHORS file check CI job: contrib/authors_update.sh.

@yuanwu2017

Copy link
Copy Markdown
Contributor

@brminich @yosefe Please help to reivew and merge.

@yuanwu2017

yuanwu2017 commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on latest master. Resolved three merge conflicts, all in CI plumbing that was already reworked by #11569 — took the upstream/master version for each:

  • buildlib/dockers/ze-builder.Dockerfile — upstream is a superset (adds L0 null-driver build for no-GPU CI)
  • buildlib/tools/builds.sh — upstream build_ze extends this PR's compile-only version with real-GPU + null-driver smoke checks
  • buildlib/az-helpers.sh — identical behavior, upstream comment wording

No changes to the feature code under src/uct/ze/ze_ipc/.

Local test on real Intel GPU (intel/deep-learning-essentials:2025.2.0-devel-ubuntu24.04, /dev/dri passthrough):

Configure: --with-ze --enable-gtest --disable-logging --disable-debug --disable-assertions --disable-params-check → HAVE_ZE 1.

All ZE gtests pass:

Suite Tests Result
test_ze_base 5 ✅
test_ze_ipc_md 6 ✅
test_ze_ipc_cache 3 ✅
test_ze_copy_md 7 ✅
ze_copy/test_ze_copy_rma 9 ✅
Total 30 PASSED (473 ms)

Open the IPC handle on the iface device, whose command lists run the
copy. The rkey carried the exporter's device ordinal in dev_num and the
importer used it to pick a local device, which names the wrong GPU
whenever the two processes enumerate GPUs in different orders. Remove
dev_num from the key together with uct_ze_base_get_device_ordinal. That
helper matched only root-device handles, so registering memory allocated
on a sub-device failed with UCS_ERR_INVALID_ADDR.

When the shared event pool is exhausted, give the operation a private
one-event pool rather than failing with UCS_ERR_NO_RESOURCE.
ep_pending_add cannot queue the operation, so UCP would retry the send
in a tight loop. The shared pool size is now configurable with
UCX_ZE_IPC_EVENT_POOL_SIZE (default 1024).

Stop advertising EVENT_SEND_COMP and EVENT_RECV. The iface has no event
fd and its event_arm returns UCS_ERR_UNSUPPORTED, which
ucp_worker_iface_check_events_do treats as fatal ("failed to arm
iface"); ucx_perftest aborted that way at teardown.

Duplicate the fd from a remote IPC handle whenever it is non-negative
and reject a negative one. The old (0, 65536) bound skipped the dup for
fd 0 and for fds of 65536 and above.

Remove the IPC handle dump and checksum helpers and the hand-rolled
timeline profiling. Drop the per-registration and per-query info logs
and move the iface init logs to debug. Report the actual cause in the
ze_copy missing-device error and indent Makefile.am with spaces. Rename
the get_device_by_ordinal_round_trip test and drop the tests of the
removed helper.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
@tsg-
tsg- force-pushed the cherry/add-ze-ipc-support branch from bb11f63 to 0156ae4 Compare October 2, 2026 05:28
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 2, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 3) · commit 0156ae45

TL;DR: The AddressSanitizer/roce worker-3 job did not fail on any test — all 8707 gtests passed — it was killed because the self-hosted Azure agent on swx-rain03 received a shutdown signal mid-step, so Azure marked the operation as canceled. This is CI infrastructure, not a code regression; re-run the job and check the agent/host health.

Full analysis

Summary: AddressSanitizer roce on worker 3 (build 137872) was reported failed, but the gtest run completed with [ PASSED ] 8707 tests and zero failures; the step was terminated by ##[error]The agent has received a shutdown signal followed by ##[error]The operation was canceled.

Root cause: Agent-side infrastructure fault, not a test or product failure. Evidence from the fetched log:

  • 2026-10-02T07:08:15.4802220Z ##[error]The agent has received a shutdown signal. This can happen when the agent service is stopped, or a manually started agent is canceled. — emitted while tcp/test_ucp_peer_failure.basic/0 was in flight.
  • The gtest process kept running and finished normally 4 seconds later: 07:08:19.7912606Z [==========] 8707 tests from 1252 test suites ran. (2606018 ms total) / [ PASSED ] 8707 tests., followed by make: Leaving directory '/__w/1/s/build-test/test/gtest' and the normal unset_test_flags teardown.
  • No ASAN report, no leak summary, no gtest [ FAILED ] block anywhere in the log.
  • This is not a hang/timeout: log timestamps are continuous (sub-second to a few seconds apart) right up to the kill; the largest gap in the whole tail is the 17.5 s poll/test_async_event_mt.multithread_create_destroy/0 test, which is expected and ranks Adding license file for the project #3 in the job's own TOP-20 longest tests list. The job was also far under its 360-minute timeoutInMinutes (buildlib/pr/tests.yml:17).
  • Because the agent daemon on the host (swx-rain03) was stopped/restarted — or the pool machine was taken offline/rebooted — Azure cancelled the remaining steps (coredump/ASAN post-checks were Skipping step due to condition evaluation), and the job surfaced as a red failure.

Implicated commit: none — the failure is unrelated to [REDACTED:Hex High Entropy String] (cherry/add-ze-ipc-support, PR #11218). The ZE IPC changes in that PR never produced an error in this log, and the roce ASAN lane ran to completion cleanly.

File: no product file implicated. Relevant CI config: buildlib/pr/tests.yml:11-51 (the tests_roce job / AddressSanitizer stage instantiation at buildlib/pr/main.yml:538-544).

Suggested fix:

  1. Simply re-run the AddressSanitizer / roce stage — this build carries no actionable code signal.
  2. Ask CI/lab ops to check the MLNX pool agent on swx-rain03 (demand ucx_roce -equals yes) for an agent-service restart, host reboot, or OOM/eviction at 2026-10-02 07:08 UTC; a repeat shutdown on the same host means the agent or the node needs attention, not the test.
  3. Optional hardening so this class of infra abort is less disruptive: add retryCountOnTaskFailure to the Run ./contrib/test_jenkins.sh step in buildlib/pr/tests.yml (the checkout step already has retryCountOnTaskFailure: 5, the test step has none), so an agent-level abort retries instead of failing the PR. Note that Azure does not retry on explicit cancellation, so the primary fix remains agent-host stability.

Related: none found — searches for the agent-shutdown/cancellation signature returned only unrelated PRs (#11932, #11842, #11948, #11866, #11834), no existing issue tracking this flake.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 5eb364c1-dba4-4769-bcf0-e5a3d0ee7c27 in the triage console for the audit trail.

@brminich brminich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls add a cross-process ZE IPC RMA test, or point to existing coverage for that path. The new RMA tests use two endpoints in one process, so they bypass pidfd_getfd, which the intended interprocess use requires.

@tsg-

tsg- commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

@brminich thank you for the catch, and agreed ... the current UCT RMA tests use the same-PID path, so they do not exercise pidfd_getfd. We will add a real exporter/importer test with an exec'ed peer, verify PUT/GET data and cache/FD cleanup on Intel hardware, and then report here.

tsg- added 2 commits October 2, 2026 22:55
The existing ze_ipc RMA tests keep both entities in one process, so the
transport takes its same-pid path and never exercises pidfd_getfd() or
the cross-process reachability check.

Fork and exec the gtest binary as an importer that unpacks the
exporter's rkey and runs put_zcopy and get_zcopy against the exporter's
buffer for several rounds, with and without the IPC cache. The exporter
checks each put and fills the buffer for the following get.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
While a zeMemGetIpcHandle() handle is held, the Level Zero driver (seen
on libze-intel-gpu1 26.27.39122.11) closes the handle's dmabuf fd on the
first command-list operation that a device other than the allocation's
device issues on the buffer, and a later zeMemPutIpcHandle() closes the
same fd number again. ze_ipc packed that fd number into the rkey and
held the handle until dereg, so a peer's pidfd_getfd() failed with EBADF
or duplicated an unrelated file that had reused the number, and the
deferred put could close that file in our process.

Duplicate the fd right after getting the handle, return the handle at
once and pack the duplicate; dereg closes it. The duplicate keeps the
dmabuf alive for importers, and no driver handle stays outstanding for
the driver to close under us.

Add a cross-process test that fills the exported buffer from a second
device before the importer opens it. It is skipped when fewer than two
devices are visible.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ze/ze_ipc/ze_ipc_iface.c Outdated
* Progress all active command list queues
* Similar to CUDA IPC's uct_cuda_base_progress_event_queue
*/
ucs_queue_for_each_extract(q_desc, &iface->active_queue, queue, 1) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker: ucs_queue_for_each_extract pulls the head on every iteration, so pushing q_desc back at the end of the body makes active_queue non-empty again and the loop immediately re-pulls the same descriptor. As long as one event is ZE_RESULT_NOT_READY, count never advances and progress spins forever. pls iterate like uct_cuda_base_iface_progress() does — ucs_queue_for_each_safe plus ucs_queue_del_iter once the event queue drains.

Verified: ucs_queue_for_each_extract (src/ucs/datastruct/queue.h:262) has ucs_queue_pull_non_empty(queue) in its loop condition and re-reads (queue)->head each iteration; the max_poll guard at L277 only triggers after a completion increments count.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ad67476. Progress now iterates the active queue with ucs_queue_for_each_safe and removes a command list with ucs_queue_del_iter once it drains, like uct_cuda_base_iface_progress(). Within a list it completes events in posting order and stops at the first unsignaled one, so a progress call returns instead of spinning until every outstanding copy completes. Each call completes at most MAX_POLL events.

Comment thread src/uct/ze/ze_ipc/ze_ipc_iface.c Outdated
.ep_put_zcopy = uct_ze_ipc_ep_put_zcopy,
.ep_pending_add = (uct_ep_pending_add_func_t)ucs_empty_function_return_busy,
.ep_pending_purge = (uct_ep_pending_purge_func_t)ucs_empty_function,
.ep_flush = uct_base_ep_flush,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker: copies are posted asynchronously on immediate command lists, but uct_base_ep_flush returns UCS_OK unconditionally, so a caller can observe a PUT/GET as complete while the copy is still in flight. cuda_ipc uses uct_cuda_base_ep_flush and rocm_ipc uses uct_rocm_ipc_ep_flush for exactly this reason — pls add a real ep_flush that waits on the outstanding events.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ad67476. uct_ze_ipc_ep_flush returns UCS_OK when nothing is outstanding. Otherwise it queues a marker behind the outstanding events on each active command list and returns UCS_INPROGRESS. Progress completes a marker when it reaches the head of its list, and the last marker invokes comp.

This follows uct_cuda_base_ep_flush, including two of its limits: the flush covers every command list on the iface, not just this endpoint's copies, and UCT_FLUSH_FLAG_CANCEL is not acted on. A flush still pending when the iface is closed is freed without invoking comp. Covered by test_ze_ipc_rma.ep_flush_waits_for_copies and ep_flush_pending_at_close.

uct_ze_ipc_iface_t *iface = ucs_derived_of(tl_iface, uct_ze_ipc_iface_t);
unsigned i;

/* Check if all command list queues are empty */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker: comp is ignored while this returns UCS_INPROGRESS, so the completion is never invoked and the caller waits forever.

Suggested change
/* Check if all command list queues are empty */
if (comp != NULL) {
return UCS_ERR_UNSUPPORTED;
}
/* Check if all command list queues are empty */
for (i = 0; i < iface->num_cmd_lists; i++) {

uct_cuda_base_iface_flush() and uct_rocm_ipc_iface_flush() reject a non-NULL comp the same way.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ad67476. uct_ze_ipc_iface_flush returns UCS_ERR_UNSUPPORTED when given a completion, as uct_cuda_base_iface_flush does. Without one it returns UCS_INPROGRESS while any command list has outstanding events.

.mem_attach = (uct_md_mem_attach_func_t)
ucs_empty_function_return_unsupported,
.detect_memory_type = (uct_md_detect_memory_type_func_t)
ucs_empty_function_return_unsupported,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mem_elem_pack and mem_elem_release are left NULL by this initializer, and uct_md_mem_elem_pack() / uct_md_mem_elem_release() dereference them unconditionally.

Suggested change
ucs_empty_function_return_unsupported,
.detect_memory_type = (uct_md_detect_memory_type_func_t)
ucs_empty_function_return_unsupported,
.mem_elem_pack = (uct_md_mem_elem_pack_func_t)
ucs_empty_function_return_unsupported,
.mem_elem_release = (uct_md_mem_elem_release_func_t)
ucs_empty_function,

Same as ze_copy_md.c.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ad67476. The md ops now set mem_elem_pack to ucs_empty_function_return_unsupported and mem_elem_release to ucs_empty_function.

Comment thread src/uct/ze/ze_ipc/ze_ipc_ep.c Outdated

/* Store information for cache-based cleanup and event tracking */
event_desc->dup_fd = local_fd;
event_desc->pid = ep->remote_pid;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uct_ze_ipc_map_memhandle() keys the remote cache on key->pid, but unmap (here and in iface_progress) passes event_desc->pid. pls store key->pid so map and unmap always resolve to the same cache.

Suggested change
event_desc->pid = ep->remote_pid;
event_desc->pid = key->pid;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ad67476. The event desc stores key->pid, the same pid the map keyed the cache on, and every unmap path in ze_ipc_ep.c and in progress uses it.

Comment thread test/gtest/Makefile.am Outdated
$(ZE_LIBS)
endif

if HAVE_ZE

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls merge this into the existing if HAVE_ZE block above: $(ZE_CPPFLAGS) and $(ZE_LIBS) are added twice, and $(ZE_LDFLAGS) belongs in gtest_LDFLAGS (as in the first block), not gtest_LDADD.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ad67476. There is now a single if HAVE_ZE block, which adds $(ZE_CPPFLAGS) and $(ZE_LIBS) once and puts $(ZE_LDFLAGS) in gtest_LDFLAGS.

Comment thread src/uct/ze/copy/ze_copy_ep.c Outdated

ucs_trace("remote addr %p rkey %p size %zu", (void*)remote_addr,
(void*)rkey, size);
ucs_trace("ze_copy_ep: zcopy %s remote_addr=%p local_buf=%p size=%zu",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: docs/LoggingStyle.md asks for lowercase messages and no = in values, e.g. "zcopy put remote_addr %p local_buf %p size %zu". Same for the new PUT_SHORT/GET_SHORT traces below.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ad67476. The zcopy, put_short and get_short traces are lowercase and use name value without =, per docs/LoggingStyle.md.

Comment thread src/uct/ze/ze_ipc/ze_ipc_cache.c Outdated

typedef struct uct_ze_ipc_cache_hash_key {
pid_t pid;
uint32_t _pad; /* Ensure 8-byte alignment for ze_context */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In src/uct/ze/ze_ipc/ze_ipc_cache.h — minor: _pad and UCS_S_PACKED contradict each other — packing removes the alignment the pad exists for. uct_ze_ipc_cache_hash_equal() compares fields rather than using memcmp, so both can go.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ad67476. Dropped both _pad and UCS_S_PACKED from the cache key.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: No test exercises the progress loop with an outstanding (not-yet-complete) event on a second command list, which is where the ucs_queue_for_each_extract hang reproduces. test_ze_ipc_rma always calls flush() right after posting, and flush() goes through uct_iface_flush, so the broken ep_flush path is never hit either.

Residual coverage gap: ze_ipc is not added to any UCT_TEST_*_TLS list, so the shared uct_test / uct_p2p_test suites (test_p2p_rma, test_flush, test_zcopy_comp, test_uct_iface) do not cover it — those are the suites that would have caught the flush-semantics issues.

@svc-ucx

svc-ucx commented Oct 3, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer gpu on worker 0) · commit 2188836b

TL;DR: The ASan GPU gtest aborted inside rc_mlx5/test_uct_ib_sl.check_ib_sl_config_without_ddp/0, where sending on SL=2 hit "Transport retry count exceeded" (synd 0x15) — a fatal UCT error because the test has no error handler — so the whole gtest process core-dumped; this is an environment/fabric SL-mapping issue in the test, unrelated to the ZE IPC PR, and the test should tolerate/skip blocked SLs (or CI should set GTEST_MAX_IB_SL).

Full analysis

Summary: make -C test/gtest test aborted (core dumped) during rc_mlx5/test_uct_ib_sl.check_ib_sl_config_without_ddp/0, killing the entire AddressSanitizer "gpu on worker 0" job.

Root cause: The test iterates over every SL (0..UCT_IB_SL_NUM) and does a real send_recv_short() on each one (test/gtest/uct/ib/test_ib.cc:728-742). On the test host swx-rdmz-ucx-gpu-01 / mlx5_0:1, SL=2 is not usable (the SM's SL→VL/QoS mapping does not carry it), so the RC QP exhausted its retries:

06:47:14.98  [ RUN ] rc_mlx5/test_uct_ib_sl.check_ib_sl_config_without_ddp/0 <rc_mlx5/mlx5_0:1>
06:47:29.27  ib_mlx5_log.c:179  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
             RC QP 0x3872 wqe[0]: SEND --e [inl len 18] [rqpn 0x386f dlid=16 sl=2 port=1 ...]
==== backtrace ... ucs_fatal_error_message <- ucs_log_default_handler <- uct_ib_mlx5_completion_with_err
make: *** [Makefile:4921: test] Aborted (core dumped)

The 14 s gap is exactly the RC retry/timeout budget (progress was being made, not a hang, and no wall-clock kill occurred). Because the entities created by test_uct_ib::init() have no error handling and the test installs no scoped_log_handler/wrap_errors, the completion error is logged at FATAL level and aborts the whole gtest binary rather than failing/skipping one test. The test's only pre-check is uct_ib_mlx5_devx_query_ooo_sl_mask() (test_ib.cc:671-690), which tells whether adaptive routing is enabled for an SL — it does not tell whether the SL is actually serviced by the fabric. Nothing in the PR's branch (cherry/add-ze-ipc-support, ZE/ZE_IPC transport) touches IB/mlx5, so this is a pre-existing environment-sensitive failure, not a regression from the PR.

Implicated commit: unknown (not a code regression from 2188836; the test loop predates it — relevant test file history: 5959ce74 shasson5 "UCT/IB: Split dp_ordering flag for DV/DevX", 672f0fbc Thomas Vegas "Dump complete packets in trace-data logs")

File: test/gtest/uct/ib/test_ib.cc:728-742 (SL loop in test_check_ib_sl_config), fatal raised from src/uct/ib/mlx5/ib_mlx5_log.c:174-179

Suggested fix:

  1. Short term / unblock the PR: re-run the job — this is a known flaky, host-dependent failure. To stop it recurring on this agent, export GTEST_MAX_IB_SL=1 (or the number of SLs actually provisioned on swx-rdmz-ucx-gpu-01) for the ASan GPU lane in contrib/test_jenkins.sh / buildlib/pr/tests.yml; the test already honours that env var (test_ib.cc:716-724).
  2. Proper fix: make the SL loop non-fatal — wrap the per-SL test_uct_ib::init(); send_recv_short(); in a scoped_log_handler that intercepts UCS_ERR_ENDPOINT_TIMEOUT/"Transport retry count exceeded" and skips that SL instead of letting ucs_log_default_handler abort (and/or create the entities with error handling enabled). This is exactly what the closed PR GTEST/IB: Don't fatal if an SL is blocked #10197 "GTEST/IB: Don't fatal if an SL is blocked" proposed; it was never merged, which is why the failure still occurs.

Related: #10197 (GTEST/IB: Don't fatal if an SL is blocked), #10262 (UCT/IB/TEST: Configure available SLs in gtest — added GTEST_MAX_IB_SL), PR under test: #11218

tsg- added 2 commits October 3, 2026 00:27
uct_base_ep_flush returns UCS_OK while copies are still in flight, so
UCP could complete a flush before the data had landed. ep_flush now
queues a marker behind the events of each active command list, as the
cuda transports do, and invokes the caller's completion once every
marker has reached the head of its queue. Closing the iface with a flush
still pending frees the flush without completing it.

Progress now completes each command list's events in posting order, so a
marker completes only after the copies posted before it. It stops at the
first event that has not signaled instead of extracting the list and
pushing it back. An event leaves its queue only after its completion has
run, so a completion that posts on the same command list no longer adds
the list to the active queue a second time. iface_flush rejects a
completion, like the cuda transports.

Also from review: unmap uses the rkey's pid, matching the map; the md
defines mem_elem_pack and mem_elem_release; the cache hash key drops its
manual pad and packing; ze_copy traces follow the logging style; and the
gtest Makefile has a single HAVE_ZE block.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
compute-runtime 26.27 keeps the fd that zeMemGetIpcHandle returned
inside the opaque IPC handle. zeMemOpenIpcHandle treats a handle whose
fd field equals that stored fd as unmodified, and fetches that fd number
from the exporter with pidfd_getfd instead of using the fd it was given.
The exporter packs a dup and puts the handle right away, which closes
the original fd. So when the importer's pidfd_getfd copy lands on the
same number, the driver fetches a closed fd, and the open fails with
0x78000004, or it fetches whatever file has since reused the number.

The rkey now carries the exporter's original fd. When the importer's
copy has the same number, it moves the copy to another fd. The new xproc
test forces the collision by filling the importer's fd table up to the
exporter's fd, and checks that the importer's copy was moved.

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

kpjeeja added a commit to kpjeeja/sglang that referenced this pull request Oct 4, 2026
Replace hardcoded torch.cuda calls in the Encoder/Prefill/Decode path with
torch.get_device_module equivalents, so staging-buffer KV transfer works on
Intel XPU. Two of the fixes are not XPU-specific: the encoder /send
synchronize was gated on is_cuda and so was skipped on every other
accelerator, and staging init defaulted device_type to "cuda".

Breaking change for out-of-tree backends: init_staging_buffers,
init_staging_allocator and init_dcp_pack_buffers take a required keyword-only
device_type. The "cuda" default was dropped so no backend can silently
allocate staging on CUDA.

The DCP pack kernel is Triton-only, so off CUDA init_dcp_pack_buffers returns
no buffers and DCP transfers fall back to per-token RDMA.

Adds create_device_event and current_device_stream to srt/utils/common.py.

The XPU PD tests run every scenario on both the NIXL and Mooncake backends.
The NIXL classes are skipped: UCX treats SYCL device allocations as host memory,
so NIXL cannot register XPU KV buffers until ai-dynamo/nixl#1536 and
openucx/ucx#11218 land. The Mooncake classes run in CI (nightly-xpu-2-gpu and
nightly-xpu-4-gpu). docker/xpu.Dockerfile builds the Mooncake transfer engine
from a pinned upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON
-DWITH_STORE=OFF) and installs sglang-router. The Mooncake classes skip when
mooncake.engine is not importable, so the nightly stays green on an image
built before that Dockerfile step.

Mooncake's XPU support lives in its TENT engine, so those classes set
MC_USE_TENT=1 and MOONCAKE_PROTOCOL=tcp and need no RDMA NIC. The staging
decode server (TP 2) runs with TORCHDYNAMO_DISABLE=1: at TP > 1 the compiled
vocab-parallel embedding segfaults in inductor's static Triton launcher on XPU.

Verified on 3x Intel Arc Pro B60: xpu/test_disaggregation_staging_xpu.py
(prefill TP 1, decode TP 2, so KV goes through the staging path), 4 passed;
test/registered/unit/disaggregation, 487 passed. With Mooncake over TCP:
disaggregation/test_disaggregation_xpu.py 2 passed, staging 4 passed. The same
Mooncake tests pass against a transfer-engine-only install of upstream Mooncake
16b7ba3c built with the Dockerfile steps.
@brminich

brminich commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

@zhangxiaoli73 can you pls confirm you have CLA signed?

@j-xiong

j-xiong commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

@brminich Yes, Intel has CLA signed. @zhangxiaoli73 is covered by that, so do @yuanwu2017 and @tsg- .

@brminich
brminich merged commit 90c8961 into openucx:master Oct 5, 2026
162 checks passed
kpjeeja added a commit to kpjeeja/sglang that referenced this pull request Oct 7, 2026
Replace hardcoded torch.cuda calls in the Encoder/Prefill/Decode path with
torch.get_device_module equivalents, so staging-buffer KV transfer works on
Intel XPU. Two of the fixes are not XPU-specific: the encoder /send
synchronize was gated on is_cuda and so was skipped on every other
accelerator, and staging init hardcoded a "cuda:{gpu_id}" device string.

Breaking change for out-of-tree backends: init_staging_buffers,
init_staging_allocator and init_dcp_pack_buffers take a required keyword-only
device_type. The "cuda" default was dropped so no backend can silently
allocate staging on CUDA.

The DCP pack kernel is Triton-only, so off CUDA init_dcp_pack_buffers returns
no buffers and DCP transfers fall back to per-token RDMA.

Adds create_device_event and current_device_stream to srt/utils/common.py.

The decode-side scatter now waits on an event recorded on schedule_stream when
the room is registered, right after pre-alloc writes its req_to_token row,
instead of having no ordering edge. Waiting on the stream itself would also
queue each scatter behind any forward running there without overlap.

The XPU PD tests run every scenario on both the NIXL and Mooncake backends.
The NIXL classes are skipped: UCX treats SYCL device allocations as host memory,
so NIXL cannot register XPU KV buffers until ai-dynamo/nixl#1536 and
openucx/ucx#11218 land. The Mooncake classes run in CI (nightly-xpu-2-gpu and
nightly-xpu-4-gpu). docker/xpu.Dockerfile builds the Mooncake transfer engine
from a pinned upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON
-DWITH_STORE=OFF) and installs sglang-router. The Mooncake classes skip when
mooncake.engine is not importable, so the nightly stays green on an image
built before that Dockerfile step.

Mooncake's XPU support lives in its TENT engine, so those classes set
MC_USE_TENT=1 and MOONCAKE_PROTOCOL=tcp and need no RDMA NIC. The staging
decode server (TP 2) runs with TORCHDYNAMO_DISABLE=1: at TP > 1 the compiled
vocab-parallel embedding segfaults in inductor's static Triton launcher on XPU.

Verified on 3x Intel Arc Pro B60: xpu/test_disaggregation_staging_xpu.py
(prefill TP 1, decode TP 2, so KV goes through the staging path), 4 passed;
test/registered/unit/disaggregation, 487 passed. With Mooncake over TCP:
disaggregation/test_disaggregation_xpu.py 2 passed, staging 4 passed. The same
Mooncake tests pass against a transfer-engine-only install of upstream Mooncake
built with the Dockerfile steps.
kpjeeja added a commit to kpjeeja/sglang that referenced this pull request Oct 22, 2026
XPU P/D landed on NIXL (sgl-project#24188), but the XPU CI image does not install NIXL,
so its registered test could not run. This builds both the Mooncake transfer
engine and NIXL into the image and runs the registered XPU P/D test on both
backends.

docker/xpu.Dockerfile builds the Mooncake transfer engine from a pinned
upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the
store does not build with icpx at that commit) and installs a pinned sglang-router,
which the P/D tests launch between prefill and decode. Mooncake's
dependencies.sh is run with libboost-dev in place of libboost-all-dev, which
pulls OpenMPI, libfabric and Ubuntu's UCX 1.16; Mooncake, store and P2P
included, uses only header-only Boost. A grep anchored to the package entry
fails the build if a MOONCAKE_COMMIT bump renames or drops it.

It also builds UCX from a pinned master commit (--with-ze --with-mlx5,
--enable-optimizations), since ZE IPC for Intel GPUs (openucx/ucx#11218) is in
no UCX release yet, into its
own prefix, /opt/ucx (NIXL finds it through its RUNPATH; no ld.so.conf entry), a
static Abseil 20250814.1 into /opt/abseil, and NIXL from a
pinned commit through NIXL's documented source install: the nixl-cu12 package,
the C++ libraries and plugins, then the nixl meta package. Mooncake's
dependencies.sh pulls in Ubuntu's libabsl-dev 20220623, too old for NIXL (VLOG
not declared), so the Abseil build goes first on PKG_CONFIG_PATH. nixl-cu12
keeps a RUNPATH to the meson prefix, so pip and meson share
--prefix=/opt/nixl. Afterwards only the build tools the venv did not already
have are removed, so packages sglang depends on (ninja, pyyaml) stay.
The NIXL pin includes ai-dynamo/nixl#1536: its UCX plugin passes the VRAM memory
type to UCX at registration, and the default auto hint selects ze-device on a
ZE-only UCX, so UCX does not misclassify XPU memory as host (NIXL_ERR_BACKEND)
and no UCX_MEMTYPE_CACHE override is needed.

test/registered/disaggregation/test_disaggregation_xpu.py runs both backends
on nightly-xpu-kernel-main-2-gpu through a shared mixin. Mooncake sets
MC_USE_TENT=1 and MOONCAKE_PROTOCOL=tcp, since CI has no RDMA NIC; TCP is
temporary until kvcache-ai/Mooncake#4379 enables ze_ipc. Both servers use the
documented XPU attention backend, intel_xpu (page size 128), and prefill uses a
256-token chunk, so a ~1.1k-token greedy prompt that must recall a fact from its
first page covers multi-page, multi-chunk KV transfer. Each class fails, not skips, when its backend is not
importable, so the nightly also checks that the image carries both builds. In
CI it also fails, rather than skips, when fewer than two XPUs are visible, so a
runner with a missing card does not leave the nightly green; outside CI it
still skips.
The mixin sets use_rdma = False, a new PDDisaggregationServerBase flag
(default True, so CUDA tests are unchanged). With it, CI mode skips the NIC
lookup, --disaggregation-ib-device and the MC_GID_INDEX/UCX_NET_DEVICES env.
Without it the fixture hands both servers its CUDA-runner default NICs
(mlx5_roce*) on the NIC-less XPU runners, and Mooncake refuses to start.
get_rdma_devices_args() now counts cards with get_gpu_count() instead of
torch.cuda.device_count(), so the card-to-NIC mapping is right off CUDA too;
on CUDA the count is unchanged.
Decode keeps the default host_pool retraction backup now that sgl-project#32503 enables
HiCache L1<->L2 transfer on XPU.

scripts/ci/xpu/xpu_ci_start_container.sh adds --cap-add SYS_PTRACE: UCX's
ze_ipc transport maps the peer process's device memory through pidfd_getfd(),
which needs ptrace rights, so NIXL transfers fail with EPERM in a default
container. pr-test-xpu.yml drops test_disaggregation_xpu.py from its path filter,
since the test now runs only in the nightly.

Server-args validation now rejects SGLANG_MOONCAKE_CUSTOM_MEM_POOL on
--device xpu when it enables a pool: Mooncake's pools use CUDA allocators,
which XPU lacks. Values that leave the pool off are accepted, and CUDA, ROCm,
MUSA and Ascend NPU keep their existing behaviour.

Docs: the XPU page's P/D section adds a Mooncake backend subsection (build or
Docker image, MC_USE_TENT=1 and the temporary MOONCAKE_PROTOCOL=tcp,
host-staged TCP transport, no Mooncake store so no HiCache L3, a full prefill/
decode example) and a backend column with Mooncake and NIXL nightly rows. The
NIXL subsection now requires a UCX built with the Level Zero transport (the
PyPI nixl wheel's bundled UCX has none) and says
containers need --cap-add SYS_PTRACE. Its launch commands no longer set
UCX_POSIX_USE_PROC_LINK=n, matching the nightly; the note now gives it as the fix
if UCX's shared-memory transport cannot open a peer's /proc/<pid>/fd/<fd>.
pd_disaggregation.mdx notes the XPU Mooncake and NIXL builds and that
SGLANG_MOONCAKE_CUSTOM_MEM_POOL is CUDA-only.
kpjeeja added a commit to kpjeeja/sglang that referenced this pull request Oct 23, 2026
XPU P/D landed on NIXL (sgl-project#24188), but the XPU CI image does not install NIXL,
so its registered test could not run. This builds both the Mooncake transfer
engine and NIXL into the image and runs the registered XPU P/D test on both
backends.

docker/xpu.Dockerfile builds the Mooncake transfer engine from a pinned
upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the
store does not build with icpx at that commit). The pin includes
kvcache-ai/Mooncake#4379, so on one node TENT's XPU transport copies KV card to
card through Level Zero IPC and no RDMA NIC is needed. The image also installs
a pinned sglang-router,
which the P/D tests launch between prefill and decode. Mooncake's
dependencies.sh is run with libboost-dev in place of libboost-all-dev, which
pulls OpenMPI, libfabric and Ubuntu's UCX 1.16; Mooncake, store and P2P
included, uses only header-only Boost. A grep anchored to the package entry
fails the build if a MOONCAKE_COMMIT bump renames or drops it.

It also builds UCX from a pinned master commit (--with-ze --with-mlx5,
--enable-optimizations), since ZE IPC for Intel GPUs (openucx/ucx#11218) is in
no UCX release yet, into its
own prefix, /opt/ucx (NIXL finds it through its RUNPATH; no ld.so.conf entry), a
static Abseil 20250814.1 into /opt/abseil, and NIXL from a
pinned commit through NIXL's documented source install: the nixl-cu12 package,
the C++ libraries and plugins, then the nixl meta package. Mooncake's
dependencies.sh pulls in Ubuntu's libabsl-dev 20220623, too old for NIXL (VLOG
not declared), so the Abseil build goes first on PKG_CONFIG_PATH. nixl-cu12
keeps a RUNPATH to the meson prefix, so pip and meson share
--prefix=/opt/nixl. Afterwards only the build tools the venv did not already
have are removed, so packages sglang depends on (ninja, pyyaml) stay.
The NIXL pin includes ai-dynamo/nixl#1536: its UCX plugin passes the VRAM memory
type to UCX at registration, and the default auto hint selects ze-device on a
ZE-only UCX, so UCX does not misclassify XPU memory as host (NIXL_ERR_BACKEND)
and no UCX_MEMTYPE_CACHE override is needed.

test/registered/disaggregation/test_disaggregation_xpu.py runs both backends
on nightly-xpu-kernel-main-2-gpu through a shared mixin. Mooncake sets
only MC_USE_TENT=1, so it copies KV card to card; both servers see both cards
(same ZE_AFFINITY_MASK, decode on --base-gpu-id 1), which TENT needs for that.
Both servers use the
documented XPU attention backend, intel_xpu (page size 128), and prefill uses a
256-token chunk, so a ~1.1k-token greedy prompt that must recall a fact from its
first page covers multi-page, multi-chunk KV transfer. Each class fails, not skips, when its backend is not
importable, so the nightly also checks that the image carries both builds. In
CI it also fails, rather than skips, when fewer than two XPUs are visible, so a
runner with a missing card does not leave the nightly green; outside CI it
still skips.
The mixin sets use_rdma = False, a new PDDisaggregationServerBase flag
(default True, so CUDA tests are unchanged). With it, CI mode skips the NIC
lookup, --disaggregation-ib-device and the MC_GID_INDEX/UCX_NET_DEVICES env.
Without it the fixture hands both servers its CUDA-runner default NICs
(mlx5_roce*) on the NIC-less XPU runners, and Mooncake refuses to start.
get_rdma_devices_args() now counts cards with get_gpu_count() instead of
torch.cuda.device_count(), so the card-to-NIC mapping is right off CUDA too;
on CUDA the count is unchanged.
Decode keeps the default host_pool retraction backup now that sgl-project#32503 enables
HiCache L1<->L2 transfer on XPU.

scripts/ci/xpu/xpu_ci_start_container.sh adds --cap-add SYS_PTRACE: Level
Zero IPC (NIXL's UCX ze_ipc and Mooncake's XPU transport) maps the peer
process's device memory through pidfd_getfd(), which needs ptrace rights. In a
default container NIXL transfers fail with EPERM, and Mooncake stops copying
KV card to card. pr-test-xpu.yml drops test_disaggregation_xpu.py from its path filter,
since the test now runs only in the nightly.

Server-args validation now rejects SGLANG_MOONCAKE_CUSTOM_MEM_POOL on
--device xpu when it enables a pool: Mooncake's pools use CUDA allocators,
which XPU lacks. Values that leave the pool off are accepted, and CUDA, ROCm,
MUSA and Ascend NPU keep their existing behaviour.

Docs: the XPU page's P/D section adds a Mooncake backend subsection (build or
Docker image, MC_USE_TENT=1, card-to-card copies that need both servers to
see both cards and --cap-add SYS_PTRACE in containers, no Mooncake store so no
HiCache L3, a prefill/decode example on ZE_AFFINITY_MASK=0,1) and a backend column with Mooncake and NIXL nightly rows. The
NIXL subsection now requires a UCX built with the Level Zero transport (the
PyPI nixl wheel's bundled UCX has none) and says
containers need --cap-add SYS_PTRACE. Its launch commands no longer set
UCX_POSIX_USE_PROC_LINK=n, matching the nightly; the note now gives it as the fix
if UCX's shared-memory transport cannot open a peer's /proc/<pid>/fd/<fd>.
pd_disaggregation.mdx notes the XPU Mooncake and NIXL builds and that
SGLANG_MOONCAKE_CUSTOM_MEM_POOL is CUDA-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kpjeeja added a commit to kpjeeja/sglang that referenced this pull request Oct 23, 2026
XPU P/D landed on NIXL (sgl-project#24188), but the XPU CI image does not install NIXL,
so its registered test could not run. This builds both the Mooncake transfer
engine and NIXL into the image and runs the registered XPU P/D test on both
backends.

docker/xpu.Dockerfile builds the Mooncake transfer engine from a pinned
upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the
store does not build with icpx at that commit). The pin includes
kvcache-ai/Mooncake#4379, so on one node TENT's XPU transport copies KV card to
card through Level Zero IPC and no RDMA NIC is needed. The image also installs
a pinned sglang-router,
which the P/D tests launch between prefill and decode. Mooncake's
dependencies.sh is run with libboost-dev in place of libboost-all-dev, which
pulls OpenMPI, libfabric and Ubuntu's UCX 1.16; Mooncake, store and P2P
included, uses only header-only Boost. A grep anchored to the package entry
fails the build if a MOONCAKE_COMMIT bump renames or drops it.

It also builds UCX from a pinned master commit (--with-ze --with-mlx5,
--enable-optimizations), since ZE IPC for Intel GPUs (openucx/ucx#11218) is in
no UCX release yet, into its
own prefix, /opt/ucx (NIXL finds it through its RUNPATH; no ld.so.conf entry), a
static Abseil 20250814.1 into /opt/abseil, and NIXL from a
pinned commit through NIXL's documented source install: the nixl-cu12 package,
the C++ libraries and plugins, then the nixl meta package. Mooncake's
dependencies.sh pulls in Ubuntu's libabsl-dev 20220623, too old for NIXL (VLOG
not declared), so the Abseil build goes first on PKG_CONFIG_PATH. nixl-cu12
keeps a RUNPATH to the meson prefix, so pip and meson share
--prefix=/opt/nixl. Afterwards only the build tools the venv did not already
have are removed, so packages sglang depends on (ninja, pyyaml) stay.
The NIXL pin includes ai-dynamo/nixl#1536: its UCX plugin passes the VRAM memory
type to UCX at registration, and the default auto hint selects ze-device on a
ZE-only UCX, so UCX does not misclassify XPU memory as host (NIXL_ERR_BACKEND)
and no UCX_MEMTYPE_CACHE override is needed.

test/registered/disaggregation/test_disaggregation_xpu.py runs both backends
on nightly-xpu-kernel-main-2-gpu through a shared mixin. Mooncake sets
only MC_USE_TENT=1, so it copies KV card to card; both servers see both cards
(same ZE_AFFINITY_MASK, decode on --base-gpu-id 1), which TENT needs for that.
Both servers use the
documented XPU attention backend, intel_xpu (page size 128), and prefill uses a
256-token chunk, so a ~1.1k-token greedy prompt that must recall a fact from its
first page covers multi-page, multi-chunk KV transfer. Each class fails, not skips, when its backend is not
importable, so the nightly also checks that the image carries both builds. In
CI it also fails, rather than skips, when fewer than two XPUs are visible, so a
runner with a missing card does not leave the nightly green; outside CI it
still skips.
The mixin sets use_rdma = False, a new PDDisaggregationServerBase flag
(default True, so CUDA tests are unchanged). With it, CI mode skips the NIC
lookup, --disaggregation-ib-device and the MC_GID_INDEX/UCX_NET_DEVICES env.
Without it the fixture hands both servers its CUDA-runner default NICs
(mlx5_roce*) on the NIC-less XPU runners, and Mooncake refuses to start.
get_rdma_devices_args() now counts cards with get_gpu_count() instead of
torch.cuda.device_count(), so the card-to-NIC mapping is right off CUDA too;
on CUDA the count is unchanged.
Decode keeps the default host_pool retraction backup now that sgl-project#32503 enables
HiCache L1<->L2 transfer on XPU.

scripts/ci/xpu/xpu_ci_start_container.sh adds --cap-add SYS_PTRACE: Level
Zero IPC (NIXL's UCX ze_ipc and Mooncake's XPU transport) maps the peer
process's device memory through pidfd_getfd(), which needs ptrace rights. In a
default container NIXL transfers fail with EPERM, and Mooncake stops copying
KV card to card. pr-test-xpu.yml drops test_disaggregation_xpu.py from its path filter,
since the test now runs only in the nightly.

Server-args validation now rejects SGLANG_MOONCAKE_CUSTOM_MEM_POOL on
--device xpu when it enables a pool: Mooncake's pools use CUDA allocators,
which XPU lacks. Values that leave the pool off are accepted, and CUDA, ROCm,
MUSA and Ascend NPU keep their existing behaviour.

Docs: the XPU page's P/D section adds a Mooncake backend subsection (build or
Docker image, MC_USE_TENT=1, card-to-card copies that need both servers to
see both cards and --cap-add SYS_PTRACE in containers, no Mooncake store so no
HiCache L3, a prefill/decode example on ZE_AFFINITY_MASK=0,1) and a backend column with Mooncake and NIXL nightly rows. The
NIXL subsection now requires a UCX built with the Level Zero transport (the
PyPI nixl wheel's bundled UCX has none) and says
containers need --cap-add SYS_PTRACE. Its launch commands no longer set
UCX_POSIX_USE_PROC_LINK=n, matching the nightly; the note now gives it as the fix
if UCX's shared-memory transport cannot open a peer's /proc/<pid>/fd/<fd>.
pd_disaggregation.mdx notes the XPU Mooncake and NIXL builds and that
SGLANG_MOONCAKE_CUSTOM_MEM_POOL is CUDA-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kpjeeja added a commit to kpjeeja/sglang that referenced this pull request Oct 26, 2026
XPU P/D landed on NIXL (sgl-project#24188), but the XPU CI image does not install NIXL,
so its registered test could not run. This builds both the Mooncake transfer
engine and NIXL into the image and runs the registered XPU P/D test on both
backends.

docker/xpu.Dockerfile builds the Mooncake transfer engine from a pinned
upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the
store does not build with icpx at that commit). The pin includes
kvcache-ai/Mooncake#4379, so on one node TENT's XPU transport copies KV card to
card through Level Zero IPC and no RDMA NIC is needed. The image also installs
a pinned sglang-router,
which the P/D tests launch between prefill and decode. Mooncake's
dependencies.sh is run with libboost-dev in place of libboost-all-dev, which
pulls OpenMPI, libfabric and Ubuntu's UCX 1.16; Mooncake, store and P2P
included, uses only header-only Boost. A grep anchored to the package entry
fails the build if a MOONCAKE_COMMIT bump renames or drops it.

It also builds UCX from a pinned master commit (--with-ze --with-mlx5,
--enable-optimizations), since ZE IPC for Intel GPUs (openucx/ucx#11218) is in
no UCX release yet, into its
own prefix, /opt/ucx (NIXL finds it through its RUNPATH; no ld.so.conf entry), a
static Abseil 20250814.1 into /opt/abseil, and NIXL from a
pinned commit through NIXL's documented source install: the nixl-cu12 package,
the C++ libraries and plugins, then the nixl meta package. Mooncake's
dependencies.sh pulls in Ubuntu's libabsl-dev 20220623, too old for NIXL (VLOG
not declared), so the Abseil build goes first on PKG_CONFIG_PATH. nixl-cu12
keeps a RUNPATH to the meson prefix, so pip and meson share
--prefix=/opt/nixl. Afterwards only the build tools the venv did not already
have are removed, so packages sglang depends on (ninja, pyyaml) stay.
The NIXL pin includes ai-dynamo/nixl#1536: its UCX plugin passes the VRAM memory
type to UCX at registration, and the default auto hint selects ze-device on a
ZE-only UCX, so UCX does not misclassify XPU memory as host (NIXL_ERR_BACKEND)
and no UCX_MEMTYPE_CACHE override is needed.

test/registered/disaggregation/test_disaggregation_xpu.py runs both backends
on nightly-xpu-kernel-main-2-gpu through a shared mixin. Mooncake sets
only MC_USE_TENT=1, so it copies KV card to card; both servers see both cards
(same ZE_AFFINITY_MASK, decode on --base-gpu-id 1), which TENT needs for that.
Both servers use the
documented XPU attention backend, intel_xpu (page size 128), and prefill uses a
256-token chunk, so a ~1.1k-token greedy prompt that must recall a fact from its
first page covers multi-page, multi-chunk KV transfer. Each class fails, not skips, when its backend is not
importable, so the nightly also checks that the image carries both builds. In
CI it also fails, rather than skips, when fewer than two XPUs are visible, so a
runner with a missing card does not leave the nightly green; outside CI it
still skips.
The mixin sets use_rdma = False, a new PDDisaggregationServerBase flag
(default True, so CUDA tests are unchanged). With it, CI mode skips the NIC
lookup, --disaggregation-ib-device and the MC_GID_INDEX/UCX_NET_DEVICES env.
Without it the fixture hands both servers its CUDA-runner default NICs
(mlx5_roce*) on the NIC-less XPU runners, and Mooncake refuses to start.
get_rdma_devices_args() now counts cards with get_gpu_count() instead of
torch.cuda.device_count(), so the card-to-NIC mapping is right off CUDA too;
on CUDA the count is unchanged.
Decode keeps the default host_pool retraction backup now that sgl-project#32503 enables
HiCache L1<->L2 transfer on XPU.

scripts/ci/xpu/xpu_ci_start_container.sh adds --cap-add SYS_PTRACE: Level
Zero IPC (NIXL's UCX ze_ipc and Mooncake's XPU transport) maps the peer
process's device memory through pidfd_getfd(), which needs ptrace rights. In a
default container NIXL transfers fail with EPERM, and Mooncake stops copying
KV card to card. pr-test-xpu.yml drops test_disaggregation_xpu.py from its path filter,
since the test now runs only in the nightly.

Server-args validation now rejects SGLANG_MOONCAKE_CUSTOM_MEM_POOL on
--device xpu when it enables a pool: Mooncake's pools use CUDA allocators,
which XPU lacks. Values that leave the pool off are accepted, and CUDA, ROCm,
MUSA and Ascend NPU keep their existing behaviour.

Docs: the XPU page's P/D section adds a Mooncake backend subsection (build or
Docker image, MC_USE_TENT=1, card-to-card copies that need both servers to
see both cards and --cap-add SYS_PTRACE in containers, no Mooncake store so no
HiCache L3, a prefill/decode example on ZE_AFFINITY_MASK=0,1) and a backend column with Mooncake and NIXL nightly rows. The
NIXL subsection now requires a UCX built with the Level Zero transport (the
PyPI nixl wheel's bundled UCX has none) and says
containers need --cap-add SYS_PTRACE. Its launch commands no longer set
UCX_POSIX_USE_PROC_LINK=n, matching the nightly; the note now gives it as the fix
if UCX's shared-memory transport cannot open a peer's /proc/<pid>/fd/<fd>.
pd_disaggregation.mdx notes the XPU Mooncake and NIXL builds and that
SGLANG_MOONCAKE_CUSTOM_MEM_POOL is CUDA-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kpjeeja added a commit to kpjeeja/sglang that referenced this pull request Oct 26, 2026
XPU P/D landed on NIXL (sgl-project#24188), but the XPU CI image does not install NIXL,
so its registered test could not run. This builds both the Mooncake transfer
engine and NIXL into the image and runs the registered XPU P/D test on both
backends.

docker/xpu.Dockerfile builds the Mooncake transfer engine from a pinned
upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the
store does not build with icpx at that commit). The pin includes
kvcache-ai/Mooncake#4379, so on one node TENT's XPU transport copies KV card to
card through Level Zero IPC and no RDMA NIC is needed. The image also installs
a pinned sglang-router,
which the P/D tests launch between prefill and decode. Mooncake's
dependencies.sh is run with libboost-dev in place of libboost-all-dev, which
pulls OpenMPI, libfabric and Ubuntu's UCX 1.16; Mooncake, store and P2P
included, uses only header-only Boost. A grep anchored to the package entry
fails the build if a MOONCAKE_COMMIT bump renames or drops it.

It also builds UCX from a pinned master commit (--with-ze --with-mlx5,
--enable-optimizations), since ZE IPC for Intel GPUs (openucx/ucx#11218) is in
no UCX release yet, into its
own prefix, /opt/ucx (NIXL finds it through its RUNPATH; no ld.so.conf entry), a
static Abseil 20250814.1 into /opt/abseil, and NIXL from a
pinned commit through NIXL's documented source install: the nixl-cu12 package,
the C++ libraries and plugins, then the nixl meta package. Mooncake's
dependencies.sh pulls in Ubuntu's libabsl-dev 20220623, too old for NIXL (VLOG
not declared), so the Abseil build goes first on PKG_CONFIG_PATH. nixl-cu12
keeps a RUNPATH to the meson prefix, so pip and meson share
--prefix=/opt/nixl. Afterwards only the build tools the venv did not already
have are removed, so packages sglang depends on (ninja, pyyaml) stay.
The NIXL pin includes ai-dynamo/nixl#1536: its UCX plugin passes the VRAM memory
type to UCX at registration, and the default auto hint selects ze-device on a
ZE-only UCX, so UCX does not misclassify XPU memory as host (NIXL_ERR_BACKEND)
and no UCX_MEMTYPE_CACHE override is needed.

test/registered/disaggregation/test_disaggregation_xpu.py runs both backends
on nightly-xpu-kernel-main-2-gpu through a shared mixin. Mooncake sets
only MC_USE_TENT=1, so it copies KV card to card; both servers see both cards
(same ZE_AFFINITY_MASK, decode on --base-gpu-id 1), which TENT needs for that.
Both servers use the
documented XPU attention backend, intel_xpu (page size 128), and prefill uses a
256-token chunk, so a ~1.1k-token greedy prompt that must recall a fact from its
first page covers multi-page, multi-chunk KV transfer. Each class fails, not skips, when its backend is not
importable, so the nightly also checks that the image carries both builds. In
CI it also fails, rather than skips, when fewer than two XPUs are visible, so a
runner with a missing card does not leave the nightly green; outside CI it
still skips.
The mixin sets use_rdma = False, a new PDDisaggregationServerBase flag
(default True, so CUDA tests are unchanged). With it, CI mode skips the NIC
lookup, --disaggregation-ib-device and the MC_GID_INDEX/UCX_NET_DEVICES env.
Without it the fixture hands both servers its CUDA-runner default NICs
(mlx5_roce*) on the NIC-less XPU runners, and Mooncake refuses to start.
get_rdma_devices_args() now counts cards with get_gpu_count() instead of
torch.cuda.device_count(), so the card-to-NIC mapping is right off CUDA too;
on CUDA the count is unchanged.
Decode keeps the default host_pool retraction backup now that sgl-project#32503 enables
HiCache L1<->L2 transfer on XPU.

scripts/ci/xpu/xpu_ci_start_container.sh adds --cap-add SYS_PTRACE: Level
Zero IPC (NIXL's UCX ze_ipc and Mooncake's XPU transport) maps the peer
process's device memory through pidfd_getfd(), which needs ptrace rights. In a
default container NIXL transfers fail with EPERM, and Mooncake stops copying
KV card to card. pr-test-xpu.yml drops test_disaggregation_xpu.py from its path filter,
since the test now runs only in the nightly.

Server-args validation now rejects SGLANG_MOONCAKE_CUSTOM_MEM_POOL on
--device xpu when it enables a pool: Mooncake's pools use CUDA allocators,
which XPU lacks. Values that leave the pool off are accepted, and CUDA, ROCm,
MUSA and Ascend NPU keep their existing behaviour.

Docs: the XPU page's P/D section adds a Mooncake backend subsection (build or
Docker image, MC_USE_TENT=1, card-to-card copies that need both servers to
see both cards and --cap-add SYS_PTRACE in containers, no Mooncake store so no
HiCache L3, a prefill/decode example on ZE_AFFINITY_MASK=0,1) and a backend column with Mooncake and NIXL nightly rows. The
NIXL subsection now requires a UCX built with the Level Zero transport (the
PyPI nixl wheel's bundled UCX has none) and says
containers need --cap-add SYS_PTRACE. Its launch commands no longer set
UCX_POSIX_USE_PROC_LINK=n, matching the nightly; the note now gives it as the fix
if UCX's shared-memory transport cannot open a peer's /proc/<pid>/fd/<fd>.
pd_disaggregation.mdx notes the XPU Mooncake and NIXL builds and that
SGLANG_MOONCAKE_CUSTOM_MEM_POOL is CUDA-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants