Repository navigation
[CI] [Intel] disaggregation: Run XPU P/D nightly on Mooncake and NIXL - #45377
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
4f6aedd to
8a9f2a5
Compare
57e35a1 to
69dac58
Compare
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
f223310 to
33db419
Compare
|
/tag-run-ci-label |
9ed54cd to
d09e8ec
Compare
793e3c4 to
a50cded
Compare
rahulvijayaraghavan
left a comment
There was a problem hiding this comment.
LGTM!
Couple of comments
47a4b31 to
b3bdd1d
Compare
|
please tests above pr with oneapi 2026.1.3 weather all versions of nixl, ucx, abssl , mooncake |
XPU P/D landed on NIXL (sgl-project#24188), but the XPU CI image does not install NIXL, so its registered test could not run. This builds both the Mooncake transfer engine and NIXL into the image and runs the registered XPU P/D test on both backends. docker/xpu.Dockerfile builds the Mooncake transfer engine from a pinned upstream commit with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the store does not build with icpx at that commit). The pin includes kvcache-ai/Mooncake#4379, so on one node TENT's XPU transport copies KV card to card through Level Zero IPC and no RDMA NIC is needed. The image also installs a pinned sglang-router, which the P/D tests launch between prefill and decode. Mooncake's dependencies.sh is run with libboost-dev in place of libboost-all-dev, which pulls OpenMPI, libfabric and Ubuntu's UCX 1.16; Mooncake, store and P2P included, uses only header-only Boost. A grep anchored to the package entry fails the build if a MOONCAKE_COMMIT bump renames or drops it. It also builds UCX from a pinned master commit (--with-ze --with-mlx5, --enable-optimizations), since ZE IPC for Intel GPUs (openucx/ucx#11218) is in no UCX release yet, into its own prefix, /opt/ucx (NIXL finds it through its RUNPATH; no ld.so.conf entry), a static Abseil 20250814.1 into /opt/abseil, and NIXL from a pinned commit through NIXL's documented source install: the nixl-cu12 package, the C++ libraries and plugins, then the nixl meta package. Mooncake's dependencies.sh pulls in Ubuntu's libabsl-dev 20220623, too old for NIXL (VLOG not declared), so the Abseil build goes first on PKG_CONFIG_PATH. nixl-cu12 keeps a RUNPATH to the meson prefix, so pip and meson share --prefix=/opt/nixl. Afterwards only the build tools the venv did not already have are removed, so packages sglang depends on (ninja, pyyaml) stay. The NIXL pin includes ai-dynamo/nixl#1536: its UCX plugin passes the VRAM memory type to UCX at registration, and the default auto hint selects ze-device on a ZE-only UCX, so UCX does not misclassify XPU memory as host (NIXL_ERR_BACKEND) and no UCX_MEMTYPE_CACHE override is needed. test/registered/disaggregation/test_disaggregation_xpu.py runs both backends on nightly-xpu-kernel-main-2-gpu through a shared mixin. Mooncake sets only MC_USE_TENT=1, so it copies KV card to card; both servers see both cards (same ZE_AFFINITY_MASK, decode on --base-gpu-id 1), which TENT needs for that. Both servers use the documented XPU attention backend, intel_xpu (page size 128), and prefill uses a 256-token chunk, so a ~1.1k-token greedy prompt that must recall a fact from its first page covers multi-page, multi-chunk KV transfer. Each class fails, not skips, when its backend is not importable, so the nightly also checks that the image carries both builds. In CI it also fails, rather than skips, when fewer than two XPUs are visible, so a runner with a missing card does not leave the nightly green; outside CI it still skips. The mixin sets use_rdma = False, a new PDDisaggregationServerBase flag (default True, so CUDA tests are unchanged). With it, CI mode skips the NIC lookup, --disaggregation-ib-device and the MC_GID_INDEX/UCX_NET_DEVICES env. Without it the fixture hands both servers its CUDA-runner default NICs (mlx5_roce*) on the NIC-less XPU runners, and Mooncake refuses to start. get_rdma_devices_args() now counts cards with get_gpu_count() instead of torch.cuda.device_count(), so the card-to-NIC mapping is right off CUDA too; on CUDA the count is unchanged. Decode keeps the default host_pool retraction backup now that sgl-project#32503 enables HiCache L1<->L2 transfer on XPU. scripts/ci/xpu/xpu_ci_start_container.sh adds --cap-add SYS_PTRACE: Level Zero IPC (NIXL's UCX ze_ipc and Mooncake's XPU transport) maps the peer process's device memory through pidfd_getfd(), which needs ptrace rights. In a default container NIXL transfers fail with EPERM, and Mooncake stops copying KV card to card. pr-test-xpu.yml drops test_disaggregation_xpu.py from its path filter, since the test now runs only in the nightly. Server-args validation now rejects SGLANG_MOONCAKE_CUSTOM_MEM_POOL on --device xpu when it enables a pool: Mooncake's pools use CUDA allocators, which XPU lacks. Values that leave the pool off are accepted, and CUDA, ROCm, MUSA and Ascend NPU keep their existing behaviour. Docs: the XPU page's P/D section adds a Mooncake backend subsection (build or Docker image, MC_USE_TENT=1, card-to-card copies that need both servers to see both cards and --cap-add SYS_PTRACE in containers, no Mooncake store so no HiCache L3, a prefill/decode example on ZE_AFFINITY_MASK=0,1) and a backend column with Mooncake and NIXL nightly rows. The NIXL subsection now requires a UCX built with the Level Zero transport (the PyPI nixl wheel's bundled UCX has none) and says containers need --cap-add SYS_PTRACE. Its launch commands no longer set UCX_POSIX_USE_PROC_LINK=n, matching the nightly; the note now gives it as the fix if UCX's shared-memory transport cannot open a peer's /proc/<pid>/fd/<fd>. pd_disaggregation.mdx notes the XPU Mooncake and NIXL builds and that SGLANG_MOONCAKE_CUSTOM_MEM_POOL is CUDA-only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b3bdd1d to
7bdf33f
Compare
Verified on oneapi 2026.1.3, it works fine. |
|
@kpjeeja the pr description generated by ai is usually too long |
…sgl-project#45377) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
[CI] [Intel] disaggregation: Run XPU P/D nightly on Mooncake and NIXL
Dependencies
maybe_send_cached_prefix_chunkindisaggregation/prefill.pycallstorch.cuda.Event()on XPU and kills the prefill scheduler. That function sends a request's cached prefix early, and with overlap scheduling (the default) it also runs on the later chunks of a chunked prefill.test_long_prompt_multi_page_transferprefills in 256-token chunks, so it takes that path even with no prefix shared between requests. Until [Fix] disaggregation: Fix the early-send wait event on non-CUDA devices and in nixl #43299 merges, that test kills prefill and the fixture aborts the rest of the file.test/registered/xpu/test_disaggregation_staging_xpu.pyalso takes that path (its concurrent case reuses a cached prefix). It now runs in the 4-GPU nightly, because this image installs Mooncake; this PR leaves it on TCP.Motivation
XPU P/D disaggregation landed on NIXL (#24188), but the XPU CI image does not install NIXL, so its registered test could not run, and Mooncake has no XPU support in its PyPI wheel. This PR builds both transfer backends into
docker/xpu.Dockerfileand runs the registered XPU P/D test on both nightly.Mooncake is pinned to a commit with kvcache-ai/Mooncake#4379, which adds intra-node XPU peer-to-peer copies, so both backends copy KV card to card through Level Zero IPC.
Modifications
docker/xpu.Dockerfilec67a88f3) with icpx (-DUSE_XPU=ON -DUSE_TENT=ON -DWITH_STORE=OFF; the store does not build with icpx at that commit).db4f146e). On one node, TENT's XPU transport copies KV card to card through Level Zero IPC, so no RDMA NIC is needed.sglang-router(0.3.2), which the P/D tests launch between prefill and decode.libboost-all-devtolibboost-devin Mooncake'sdependencies.shbefore running it. Mooncake uses Boost headers only, andlibboost-all-devpulls OpenMPI, libfabric and a second UCX (Ubuntu's 1.16) into the image.MOONCAKE_COMMITbump renames or drops it.libcli11-dev, new at this pin) follow upstream once the Mooncake store and P2P store are enabled. Go is removed after the build, along with itsPATHline in.bashrc.Release, with-DENABLE_DEBUG_SYMBOLS=OFF -DBUILD_BENCHMARK=OFF.RelWithDebInfoplus-g, and the nightly image is always built with--no-cache.--no-cachebuild.d76cd54b,--with-ze --with-mlx5 --enable-mt --enable-optimizations --disable-static) into its own prefix,/opt/ucx.--enable-optimizationsadds only-mavxon top of UCX's default-O3, so the image still runs on any AVX-capable host.-Ducx_path), so other UCX users in the image are not affected./opt/abseil. Mooncake'sdependencies.shpulls in Ubuntu'slibabsl-dev20220623, which is too old for NIXL (VLOGnot declared), so the new Abseil goes first onPKG_CONFIG_PATH.466ebb1) with NIXL's documented source install: thenixl-cu12package, the C++ libraries and plugins, then thenixlmeta package.nixl-cu12keeps a RUNPATH to the meson prefix, so pip and meson share--prefix=/opt/nixl.PYTHONPATH,LD_LIBRARY_PATHorNIXL_PLUGIN_DIRis needed at runtime.UCX_MEMTYPE_CACHE=noverride is needed. Before NIXL/UCX: enforce VRAM memtype hint query behavior and add tests ai-dynamo/nixl#1536, registering the KV cache failed at startup withNIXL_ERR_BACKENDunless the cache was turned off.ninja,pyyaml) stay.test/registered/disaggregation/test_disaggregation_xpu.pynightly-xpu-kernel-main-2-gputhrough a shared mixin. The manual NIXL test (test/manual/kv_transfer/test_disaggregation_nixl_xpu.py) is folded in here.MC_USE_TENT=1. WithMOONCAKE_PROTOCOLat its default, TENT copies KV card to card through Level Zero IPC. NIXL needs no environment variables.ZE_AFFINITY_MASKand puts decode on--base-gpu-id 1. TENT needs that to copy card to card; with one card per server it logs... is on a device not visible here ... using another route.--attention-backend intel_xpu, the documented XPU attention backend, which has page size 128. Prefill uses--chunked-prefill-size 256.--cap-add SYS_PTRACE, it logsusing another routeand the request still succeeds. The class setscapture_per_side_logs = True, andtest_kv_copied_card_to_cardsends a request and asserts the prefill log hasXpuTransport: mapped peer bufferand nousing another route.mooncake.engine,nixl._api), and the mixin imports it first, so a class fails rather than skips when its backend is missing. The nightly therefore also checks that the image carries both builds.use_rdma = False(see below). Otherwise, in CI (SGLANG_IS_IN_CI=true) the fixture gives both servers its CUDA-runner default NICs (mlx5_roce*). The XPU runners have no RDMA NIC, so Mooncake refuses to start (No IB devices found in /sys/class/infiniband).host_poolretraction backup, now that [Feat] [Intel] hicache: Enable L1<->L2 support on XPU #32503 enables HiCache L1<->L2 transfer on XPU. No--disaggregation-decode-retraction-backupoverride is needed.python/sglang/test/server_fixtures/disaggregation_fixture.pyuse_rdmaclass flag toPDDisaggregationServerBase, defaultTrue, so existing tests are unchanged.use_rdma = False, CI mode still defaults to Mooncake but skips the RDMA NIC lookup. It also skips--disaggregation-ib-deviceand theMC_GID_INDEXandUCX_NET_DEVICESexports, andrdma_devices_for()returns no devices.get_rdma_devices_args()counts cards withget_gpu_count()(torch.accelerator) instead oftorch.cuda.device_count().scripts/ci/xpu/xpu_ci_start_container.sh--cap-add SYS_PTRACE, which XPU P/D needs to copy KV card to card on both backends.pidfd_open()andpidfd_getfd(). NIXL does this in UCX's ze_ipc transport (uct_ze_ipc_dup_fd_from_pid), and Mooncake in TENT's XPU transport.pidfd_getfd()the same access check as ptrace, and Docker's default seccomp profile only allows it when the container hasCAP_SYS_PTRACE.pidfd_getfd(...) failed: Operation not permittedand thenNIXL_ERR_BACKEND.cannot map peer buffer ... needs ptrace access ... [Operation not permitted]; using another route. Requests still succeed, but KV no longer goes card to card.SYS_PTRACEis the narrowest built-in Docker option that works.--privileged, which the XPU docs'docker runexample already uses, grants far more. The capability only lets processes inside the container trace each other; it gives no access to the host..github/workflows/pr-test-xpu.ymltest_disaggregation_xpu.pyfrom the PR-CI path filter, since the test now runs only in the nightly.python/sglang/srt/arg_groups/validation_hook.pycheck_server_argsnow rejectsSGLANG_MOONCAKE_CUSTOM_MEM_POOLat startup on--device xpuwhen the variable enables a pool (NVLINK,BAREX,INTRA_NODE_NVLINK). Those pools use CUDA allocators, which XPU lacks.test_server_args.pycovers the rejection on XPU, and checks that CUDA, NPU and a disabling value on XPU are still accepted.Docs
MC_USE_TENT=1, with KV copied card to card through Level Zero IPC;ZE_AFFINITY_MASK=0,1and puts decode on--base-gpu-id 1;--cap-add SYS_PTRACE(or--privileged) for the card-to-card path, and what TENT logs without it.nixlwheel bundles a UCX without the ze transport.--cap-add SYS_PTRACE(or--privileged), and why.UCX_POSIX_USE_PROC_LINK=n, matching the nightly. The note now gives that variable as the fix when UCX's shared-memory transport cannot open a peer's/proc/<pid>/fd/<fd>.pd_disaggregation.mdxadds XPU notes for Mooncake and NIXL, and notes thatSGLANG_MOONCAKE_CUSTOM_MEM_POOLis CUDA-only.On CUDA and ROCm, every change resolves to the same
torch.cudacalls as before.Accuracy Tests
No model forward or kernel numerics change on CUDA or ROCm. The XPU P/D tests check completions end to end (see below).
Speed Tests and Profiling
N/A. No hot path changes on CUDA or ROCm.
Tests
Full image, built from this branch's
docker/xpu.Dockerfilewith--no-cache, as the nightly builds it (26.2 GB, 23 min 31 s):nixl.nixl_agent('agent1')checks. The test host's clock runs about 7 hours behind, so this build added one line afterFROMthat turns off apt's Release-date check. Nothing else differs from the branch's Dockerfile./opt/nixl,/opt/ucxand/opt/abseil.c67a88f3)sglang-routertest_disaggregation_xpu.pywas run inside the image on Intel Arc Pro B60 cards 4 and 6 at this branch head, with the branch mounted onPYTHONPATHand #43299's two commits underneath:xpu_ci_start_container.sh, including the new--cap-add SYS_PTRACE, withSGLANG_IS_IN_CI=trueas the nightly sets ittest_kv_copied_card_to_cardfoundXpuTransport: mapped peer bufferand nousing another routein the prefill log, so KV went card to cardSYS_PTRACE), Mooncake class, otherwise the sametest_kv_copied_card_to_cardfails (prefill loggedcannot map peer buffer ... using another route); the other 3 Mooncake tests still pass, which is the gap the new test closesUCX_MEMTYPE_CACHEoverride.intel_xpuat page size 128. The long prompt prefills in chunks of 256+256+256+216 tokens and returns7342on both backends.--disaggregation-ib-deviceis passed and noUCX_NET_DEVICESis exported.MOONCAKE_PROTOCOL=tcp)--disaggregation-ib-devicewas passedThe Mooncake prefill/decode example in the XPU docs was also run as written in the same image, with
ZE_AFFINITY_MASK=4,6in place of0,1and--cap-add SYS_PTRACE. Both servers and the router started, the documentedcurlrequest returned a completion, and prefill mapped all 56 decode KV buffers card to card.Run time inside the image with the CI options, as pytest reports it:
Each class starts its own prefill server, decode server and router, so server startup takes most of the run time. Each test sends one request with at most 16 new tokens. The file registers
est_time=300onnightly-xpu-kernel-main-2-gpu, about twice the measured time.Checklist
CI States
Latest PR Test (Base): ✅ Run #40993639146
Latest PR Test (Extra): ❌ Run #40993639062
Latest PR Test (AMD ROCm 10): ✅ Run #40993639056
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: