Skip to content

[CI] [Intel] disaggregation: Run XPU P/D nightly on Mooncake and NIXL - #45377

Merged
mingfeima merged 6 commits into
sgl-project:mainfrom
kpjeeja:jeeja/xpu_nightly_pd
Nov 2, 2026
Merged

mingfeima merged 6 commits into
sgl-project:mainfrom
kpjeeja:jeeja/xpu_nightly_pd

Conversation

@kpjeeja

@kpjeeja kpjeeja commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

[CI] [Intel] disaggregation: Run XPU P/D nightly on Mooncake and NIXL

Dependencies

  • [Fix] disaggregation: Fix the early-send wait event on non-CUDA devices and in nixl #43299, "[Fix] disaggregation: Fix the early-send wait event on non-CUDA devices and in nixl". Merge it first.
    • Without it, maybe_send_cached_prefix_chunk in disaggregation/prefill.py calls torch.cuda.Event() on XPU and kills the prefill scheduler. That function sends a request's cached prefix early, and with overlap scheduling (the default) it also runs on the later chunks of a chunked prefill.
    • This PR's test_long_prompt_multi_page_transfer prefills in 256-token chunks, so it takes that path even with no prefix shared between requests. Until [Fix] disaggregation: Fix the early-send wait event on non-CUDA devices and in nixl #43299 merges, that test kills prefill and the fixture aborts the rest of the file.
    • test/registered/xpu/test_disaggregation_staging_xpu.py also takes that path (its concurrent case reuses a cached prefix). It now runs in the 4-GPU nightly, because this image installs Mooncake; this PR leaves it on TCP.

Motivation

XPU P/D disaggregation landed on NIXL (#24188), but the XPU CI image does not install NIXL, so its registered test could not run, and Mooncake has no XPU support in its PyPI wheel. This PR builds both transfer backends into docker/xpu.Dockerfile and runs the registered XPU P/D test on both nightly.

Mooncake is pinned to a commit with kvcache-ai/Mooncake#4379, which adds intra-node XPU peer-to-peer copies, so both backends copy KV card to card through Level Zero IPC.

Modifications

docker/xpu.Dockerfile

  • Builds the Mooncake transfer engine from a pinned upstream commit (c67a88f3) with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the store does not build with icpx at that commit).
  • Also installs a pinned sglang-router (0.3.2), which the P/D tests launch between prefill and decode.
  • Rewrites libboost-all-dev to libboost-dev in Mooncake's dependencies.sh before running it. Mooncake uses Boost headers only, and libboost-all-dev pulls OpenMPI, libfabric and a second UCX (Ubuntu's 1.16) into the image.
    • The script itself makes the same swap on SHCA hosts.
    • A grep anchored to the package entry fails the build if a MOONCAKE_COMMIT bump renames or drops it.
    • The rest of the script still runs, so Go, the submodules and the other dependencies (including libcli11-dev, new at this pin) follow upstream once the Mooncake store and P2P store are enabled. Go is removed after the build, along with its PATH line in .bashrc.
  • Builds Mooncake with Ninja as Release, with -DENABLE_DEBUG_SYMBOLS=OFF -DBUILD_BENCHMARK=OFF.
    • Its CMake defaults build the transfer-engine benchmark and use RelWithDebInfo plus -g, and the nightly image is always built with --no-cache.
    • With these, the Mooncake step takes 109 s in a --no-cache build.
  • Builds UCX from a pinned master commit (d76cd54b, --with-ze --with-mlx5 --enable-mt --enable-optimizations --disable-static) into its own prefix, /opt/ucx.
    • It is a master commit because ZE IPC for Intel GPUs (UCT/ZE/ZE_IPC: enable level zero ipc support for Intel GPUs openucx/ucx#11218) is in no UCX release yet. Move to the first release tag that contains it.
    • On x86, --enable-optimizations adds only -mavx on top of UCX's default -O3, so the image still runs on any AVX-capable host.
    • Nothing registers UCX system-wide. NIXL's UCX plugin finds it through its RUNPATH (-Ducx_path), so other UCX users in the image are not affected.
  • Builds a static Abseil 20250814.1 into /opt/abseil. Mooncake's dependencies.sh pulls in Ubuntu's libabsl-dev 20220623, which is too old for NIXL (VLOG not declared), so the new Abseil goes first on PKG_CONFIG_PATH.
  • Builds NIXL from a pinned commit (466ebb1) with NIXL's documented source install: the nixl-cu12 package, the C++ libraries and plugins, then the nixl meta package.
  • After the NIXL build, removes only the build tools the venv did not already have. Packages sglang depends on (ninja, pyyaml) stay.

test/registered/disaggregation/test_disaggregation_xpu.py

  • Runs Mooncake and NIXL on nightly-xpu-kernel-main-2-gpu through a shared mixin. The manual NIXL test (test/manual/kv_transfer/test_disaggregation_nixl_xpu.py) is folded in here.
  • Mooncake sets only MC_USE_TENT=1. With MOONCAKE_PROTOCOL at its default, TENT copies KV card to card through Level Zero IPC. NIXL needs no environment variables.
  • Both servers see both cards: the fixture gives them the same ZE_AFFINITY_MASK and puts decode on --base-gpu-id 1. TENT needs that to copy card to card; with one card per server it logs ... is on a device not visible here ... using another route.
  • Both servers use --attention-backend intel_xpu, the documented XPU attention backend, which has page size 128. Prefill uses --chunked-prefill-size 256.
  • A new long-prompt test sends a ~1k-token greedy prompt that must recall a code from its first KV page. The prompt spans about 8 pages and 4 prefill chunks, so a dropped or misplaced page changes the answer.
  • The Mooncake class checks that KV really goes card to card (separate commit). When TENT cannot map the peer's KV buffer, for example in a container without --cap-add SYS_PTRACE, it logs using another route and the request still succeeds. The class sets capture_per_side_logs = True, and test_kv_copied_card_to_card sends a request and asserts the prefill log has XpuTransport: mapped peer buffer and no using another route.
  • Failing instead of skipping:
    • Each class names its backend module (mooncake.engine, nixl._api), and the mixin imports it first, so a class fails rather than skips when its backend is missing. The nightly therefore also checks that the image carries both builds.
    • In CI, the classes fail rather than skip when fewer than two XPUs are visible, so a runner with a missing card does not leave the nightly green. Outside CI they still skip.
  • The shared mixin sets the fixture's new use_rdma = False (see below). Otherwise, in CI (SGLANG_IS_IN_CI=true) the fixture gives both servers its CUDA-runner default NICs (mlx5_roce*). The XPU runners have no RDMA NIC, so Mooncake refuses to start (No IB devices found in /sys/class/infiniband).
  • Decode keeps the default host_pool retraction backup, now that [Feat] [Intel] hicache: Enable L1<->L2 support on XPU #32503 enables HiCache L1<->L2 transfer on XPU. No --disaggregation-decode-retraction-backup override is needed.

python/sglang/test/server_fixtures/disaggregation_fixture.py

  • Adds a use_rdma class flag to PDDisaggregationServerBase, default True, so existing tests are unchanged.
  • With use_rdma = False, CI mode still defaults to Mooncake but skips the RDMA NIC lookup. It also skips --disaggregation-ib-device and the MC_GID_INDEX and UCX_NET_DEVICES exports, and rdma_devices_for() returns no devices.
  • get_rdma_devices_args() counts cards with get_gpu_count() (torch.accelerator) instead of torch.cuda.device_count().
    • Off CUDA the old count was 0, so the fixture assumed 8 cards and could map cards to the wrong NIC. For example, with 4 XPU cards and 2 NICs, every card mapped to the first NIC.
    • On CUDA the count is unchanged.

scripts/ci/xpu/xpu_ci_start_container.sh

  • Starts the CI container with --cap-add SYS_PTRACE, which XPU P/D needs to copy KV card to card on both backends.
    • Why: to share a KV buffer, a process exports a Level Zero IPC handle. On Linux that handle is a dma-buf file descriptor, valid only inside the owning process.
      • The peer copies the fd into itself with pidfd_open() and pidfd_getfd(). NIXL does this in UCX's ze_ipc transport (uct_ze_ipc_dup_fd_from_pid), and Mooncake in TENT's XPU transport.
      • The kernel gives pidfd_getfd() the same access check as ptrace, and Docker's default seccomp profile only allows it when the container has CAP_SYS_PTRACE.
    • Without it:
      • NIXL transfers fail with pidfd_getfd(...) failed: Operation not permitted and then NIXL_ERR_BACKEND.
      • Mooncake logs cannot map peer buffer ... needs ptrace access ... [Operation not permitted]; using another route. Requests still succeed, but KV no longer goes card to card.
    • Why this option: SYS_PTRACE is the narrowest built-in Docker option that works. --privileged, which the XPU docs' docker run example already uses, grants far more. The capability only lets processes inside the container trace each other; it gives no access to the host.

.github/workflows/pr-test-xpu.yml

  • Drops test_disaggregation_xpu.py from the PR-CI path filter, since the test now runs only in the nightly.

python/sglang/srt/arg_groups/validation_hook.py

  • check_server_args now rejects SGLANG_MOONCAKE_CUSTOM_MEM_POOL at startup on --device xpu when the variable enables a pool (NVLINK, BAREX, INTRA_NODE_NVLINK). Those pools use CUDA allocators, which XPU lacks.
  • Values that leave the pool off are accepted. CUDA, ROCm, MUSA and Ascend NPU keep their existing behaviour.
  • test_server_args.py covers the rejection on XPU, and checks that CUDA, NPU and a disabling value on XPU are still accepted.

Docs

  • The XPU page's P/D section adds a Mooncake subsection and adds Mooncake and NIXL nightly rows to the tested table. The subsection covers:
    • the build or Docker image, and MC_USE_TENT=1, with KV copied card to card through Level Zero IPC;
    • that both servers must see both cards, with a prefill/decode example that gives both ZE_AFFINITY_MASK=0,1 and puts decode on --base-gpu-id 1;
    • that containers need --cap-add SYS_PTRACE (or --privileged) for the card-to-card path, and what TENT logs without it.
  • NIXL subsection:
    • It now requires a UCX built with Level Zero, because the PyPI nixl wheel bundles a UCX without the ze transport.
    • It says containers need --cap-add SYS_PTRACE (or --privileged), and why.
    • Its launch commands no longer set UCX_POSIX_USE_PROC_LINK=n, matching the nightly. The note now gives that variable as the fix when UCX's shared-memory transport cannot open a peer's /proc/<pid>/fd/<fd>.
  • pd_disaggregation.mdx adds XPU notes for Mooncake and NIXL, and notes that SGLANG_MOONCAKE_CUSTOM_MEM_POOL is CUDA-only.

On CUDA and ROCm, every change resolves to the same torch.cuda calls as before.

Accuracy Tests

No model forward or kernel numerics change on CUDA or ROCm. The XPU P/D tests check completions end to end (see below).

Speed Tests and Profiling

N/A. No hot path changes on CUDA or ROCm.

Tests

Full image, built from this branch's docker/xpu.Dockerfile with --no-cache, as the nightly builds it (26.2 GB, 23 min 31 s):

  • The image builds through all 45 steps, including its Mooncake and nixl.nixl_agent('agent1') checks. The test host's clock runs about 7 hours behind, so this build added one line after FROM that turns off apt's Release-date check. Nothing else differs from the branch's Dockerfile.
  • The image sets no UCX environment variables, and NIXL, UCX and Abseil sit in /opt/nixl, /opt/ucx and /opt/abseil.
Build step Time
Mooncake transfer engine (c67a88f3) 109 s
UCX 110 s
Abseil 11 s
NIXL 81 s
NIXL agent check and sglang-router 9 s
All steps this PR adds 320 s
Whole image 23 min 31 s

test_disaggregation_xpu.py was run inside the image on Intel Arc Pro B60 cards 4 and 6 at this branch head, with the branch mounted on PYTHONPATH and #43299's two commits underneath:

Container options Result
Same options as xpu_ci_start_container.sh, including the new --cap-add SYS_PTRACE, with SGLANG_IS_IN_CI=true as the nightly sets it 7/7 passed (Mooncake 4, NIXL 3), 156 s. test_kv_copied_card_to_card found XpuTransport: mapped peer buffer and no using another route in the prefill log, so KV went card to card
Default Docker capabilities (no SYS_PTRACE), Mooncake class, otherwise the same test_kv_copied_card_to_card fails (prefill logged cannot map peer buffer ... using another route); the other 3 Mooncake tests still pass, which is the gap the new test closes
  • NIXL registers XPU memory with no UCX_MEMTYPE_CACHE override.
  • Both servers run intel_xpu at page size 128. The long prompt prefills in chunks of 256+256+256+216 tokens and returns 7342 on both backends.
  • In CI mode no --disaggregation-ib-device is passed and no UCX_NET_DEVICES is exported.
Test Result
Staging test as on main (prefill TP 1, decode TP 2, Mooncake with MOONCAKE_PROTOCOL=tcp) 4/4 passed, 121 s; no --disaggregation-ib-device was passed

The Mooncake prefill/decode example in the XPU docs was also run as written in the same image, with ZE_AFFINITY_MASK=4,6 in place of 0,1 and --cap-add SYS_PTRACE. Both servers and the router started, the documented curl request returned a completion, and prefill mapped all 56 decode KV buffers card to card.

Run time inside the image with the CI options, as pytest reports it:

Scope Time
NIXL class alone (3 tests) 89 s
Whole file (7 tests) 156 s

Each class starts its own prefill server, decode server and router, so server startup takes most of the run time. Each test sends one request with at most 16 new tokens. The file registers est_time=300 on nightly-xpu-kernel-main-2-gpu, about twice the measured time.

Checklist


CI States

Latest PR Test (Base): ✅ Run #40993639146
Latest PR Test (Extra): ❌ Run #40993639062
Latest PR Test (AMD ROCm 10): ✅ Run #40993639056

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-20T15:37:48.136936Z 8a9f2a5 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kpjeeja
kpjeeja marked this pull request as draft October 20, 2026 12:20
@kpjeeja
kpjeeja force-pushed the jeeja/xpu_nightly_pd branch from 4f6aedd to 8a9f2a5 Compare October 20, 2026 15:22
@kpjeeja kpjeeja changed the title [Feat] [Intel] disaggregation: Run XPU P/D nightly on Mooncake and NIXL [CI] [Intel] disaggregation: Run XPU P/D nightly on Mooncake and NIXL Oct 20, 2026
@kpjeeja
kpjeeja marked this pull request as ready for review October 20, 2026 15:34
Comment thread docker/xpu.Dockerfile Resolved
@kpjeeja
kpjeeja force-pushed the jeeja/xpu_nightly_pd branch from 57e35a1 to 69dac58 Compare October 21, 2026 07:12
@mintlify

mintlify Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 44023fe.

Approve preview

@kpjeeja
kpjeeja force-pushed the jeeja/xpu_nightly_pd branch 2 times, most recently from f223310 to 33db419 Compare October 21, 2026 10:56

@siju-samuel siju-samuel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@siju-samuel

Copy link
Copy Markdown
Contributor

/tag-run-ci-label

@arathi-hlab
arathi-hlab requested a review from mingfeima October 22, 2026 04:27
Comment thread docker/xpu.Dockerfile Resolved
Comment thread docs/docs/hardware-platforms/xpu.mdx Resolved
Comment thread docker/xpu.Dockerfile Resolved
@kpjeeja
kpjeeja force-pushed the jeeja/xpu_nightly_pd branch from 9ed54cd to d09e8ec Compare October 22, 2026 12:23
Comment thread docs/docs/hardware-platforms/xpu.mdx Resolved
@kpjeeja
kpjeeja force-pushed the jeeja/xpu_nightly_pd branch 2 times, most recently from 793e3c4 to a50cded Compare October 23, 2026 08:54

@rahulvijayaraghavan rahulvijayaraghavan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
Couple of comments

Comment thread docs/docs/hardware-platforms/xpu.mdx Resolved
Comment thread docker/xpu.Dockerfile Resolved
@kpjeeja
kpjeeja force-pushed the jeeja/xpu_nightly_pd branch 2 times, most recently from 47a4b31 to b3bdd1d Compare October 26, 2026 05:30
@arathi-hlab

arathi-hlab commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

please tests above pr with oneapi 2026.1.3 weather all versions of nixl, ucx, abssl , mooncake
cc:
@mingfeima

XPU P/D landed on NIXL (sgl-project#24188), but the XPU CI image does not install NIXL,
so its registered test could not run. This builds both the Mooncake transfer
engine and NIXL into the image and runs the registered XPU P/D test on both
backends.

docker/xpu.Dockerfile builds the Mooncake transfer engine from a pinned
upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the
store does not build with icpx at that commit). The pin includes
kvcache-ai/Mooncake#4379, so on one node TENT's XPU transport copies KV card to
card through Level Zero IPC and no RDMA NIC is needed. The image also installs
a pinned sglang-router,
which the P/D tests launch between prefill and decode. Mooncake's
dependencies.sh is run with libboost-dev in place of libboost-all-dev, which
pulls OpenMPI, libfabric and Ubuntu's UCX 1.16; Mooncake, store and P2P
included, uses only header-only Boost. A grep anchored to the package entry
fails the build if a MOONCAKE_COMMIT bump renames or drops it.

It also builds UCX from a pinned master commit (--with-ze --with-mlx5,
--enable-optimizations), since ZE IPC for Intel GPUs (openucx/ucx#11218) is in
no UCX release yet, into its
own prefix, /opt/ucx (NIXL finds it through its RUNPATH; no ld.so.conf entry), a
static Abseil 20250814.1 into /opt/abseil, and NIXL from a
pinned commit through NIXL's documented source install: the nixl-cu12 package,
the C++ libraries and plugins, then the nixl meta package. Mooncake's
dependencies.sh pulls in Ubuntu's libabsl-dev 20220623, too old for NIXL (VLOG
not declared), so the Abseil build goes first on PKG_CONFIG_PATH. nixl-cu12
keeps a RUNPATH to the meson prefix, so pip and meson share
--prefix=/opt/nixl. Afterwards only the build tools the venv did not already
have are removed, so packages sglang depends on (ninja, pyyaml) stay.
The NIXL pin includes ai-dynamo/nixl#1536: its UCX plugin passes the VRAM memory
type to UCX at registration, and the default auto hint selects ze-device on a
ZE-only UCX, so UCX does not misclassify XPU memory as host (NIXL_ERR_BACKEND)
and no UCX_MEMTYPE_CACHE override is needed.

test/registered/disaggregation/test_disaggregation_xpu.py runs both backends
on nightly-xpu-kernel-main-2-gpu through a shared mixin. Mooncake sets
only MC_USE_TENT=1, so it copies KV card to card; both servers see both cards
(same ZE_AFFINITY_MASK, decode on --base-gpu-id 1), which TENT needs for that.
Both servers use the
documented XPU attention backend, intel_xpu (page size 128), and prefill uses a
256-token chunk, so a ~1.1k-token greedy prompt that must recall a fact from its
first page covers multi-page, multi-chunk KV transfer. Each class fails, not skips, when its backend is not
importable, so the nightly also checks that the image carries both builds. In
CI it also fails, rather than skips, when fewer than two XPUs are visible, so a
runner with a missing card does not leave the nightly green; outside CI it
still skips.
The mixin sets use_rdma = False, a new PDDisaggregationServerBase flag
(default True, so CUDA tests are unchanged). With it, CI mode skips the NIC
lookup, --disaggregation-ib-device and the MC_GID_INDEX/UCX_NET_DEVICES env.
Without it the fixture hands both servers its CUDA-runner default NICs
(mlx5_roce*) on the NIC-less XPU runners, and Mooncake refuses to start.
get_rdma_devices_args() now counts cards with get_gpu_count() instead of
torch.cuda.device_count(), so the card-to-NIC mapping is right off CUDA too;
on CUDA the count is unchanged.
Decode keeps the default host_pool retraction backup now that sgl-project#32503 enables
HiCache L1<->L2 transfer on XPU.

scripts/ci/xpu/xpu_ci_start_container.sh adds --cap-add SYS_PTRACE: Level
Zero IPC (NIXL's UCX ze_ipc and Mooncake's XPU transport) maps the peer
process's device memory through pidfd_getfd(), which needs ptrace rights. In a
default container NIXL transfers fail with EPERM, and Mooncake stops copying
KV card to card. pr-test-xpu.yml drops test_disaggregation_xpu.py from its path filter,
since the test now runs only in the nightly.

Server-args validation now rejects SGLANG_MOONCAKE_CUSTOM_MEM_POOL on
--device xpu when it enables a pool: Mooncake's pools use CUDA allocators,
which XPU lacks. Values that leave the pool off are accepted, and CUDA, ROCm,
MUSA and Ascend NPU keep their existing behaviour.

Docs: the XPU page's P/D section adds a Mooncake backend subsection (build or
Docker image, MC_USE_TENT=1, card-to-card copies that need both servers to
see both cards and --cap-add SYS_PTRACE in containers, no Mooncake store so no
HiCache L3, a prefill/decode example on ZE_AFFINITY_MASK=0,1) and a backend column with Mooncake and NIXL nightly rows. The
NIXL subsection now requires a UCX built with the Level Zero transport (the
PyPI nixl wheel's bundled UCX has none) and says
containers need --cap-add SYS_PTRACE. Its launch commands no longer set
UCX_POSIX_USE_PROC_LINK=n, matching the nightly; the note now gives it as the fix
if UCX's shared-memory transport cannot open a peer's /proc/<pid>/fd/<fd>.
pd_disaggregation.mdx notes the XPU Mooncake and NIXL builds and that
SGLANG_MOONCAKE_CUSTOM_MEM_POOL is CUDA-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kpjeeja
kpjeeja force-pushed the jeeja/xpu_nightly_pd branch from b3bdd1d to 7bdf33f Compare October 26, 2026 10:32
@kpjeeja

kpjeeja commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

please tests above pr with oneapi 2026.1.3 weather all versions of nixl, ucx, abssl , mooncake cc: @mingfeima

Verified on oneapi 2026.1.3, it works fine.

@siju-samuel siju-samuel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mingfeima mingfeima added intel xpu intel gpu with device `torch.xpu` labels Nov 2, 2026
@mingfeima

Copy link
Copy Markdown
Collaborator

@kpjeeja the pr description generated by ai is usually too long

@mingfeima
mingfeima merged commit 1599b16 into sgl-project:main Nov 2, 2026
210 of 222 checks passed
Yuxingwang-intel pushed a commit to Yuxingwang-intel/sglang that referenced this pull request Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation intel run-ci CI: run the baseline test suite on this PR xpu intel gpu with device `torch.xpu`

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants