Skip to content

[Fix] disaggregation: Fix the early-send wait event on non-CUDA devices and in nixl - #43299

Merged
mingfeima merged 6 commits into
sgl-project:mainfrom
kpjeeja:jeeja/nixl_early-send_fix
Oct 29, 2026
Merged

mingfeima merged 6 commits into
sgl-project:mainfrom
kpjeeja:jeeja/nixl_early-send_fix

Conversation

@kpjeeja

@kpjeeja kpjeeja commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Under overlap scheduling, maybe_send_cached_prefix_chunk can start the KV transfer for a cached prefix while the previous step's prefill forward is still writing those pages on forward_stream. It records a completion event and attaches it to the KV sender, so the transfer worker can wait for those writes before reading the pages. Two bugs broke this:

  1. The prefill scheduler crashes on non-CUDA devices. The event came from torch.cuda.Event(), which raises Tried to instantiate dummy base class Event on torch builds without CUDA. On Intel XPU with PD + hierarchical cache, the prefill scheduler died on the first request that reused a cached prefix.
  2. nixl reads KV that may still be being written, with no error. [PD] Wait for prefill completion before Mooncake early KV transfer #41395 made the mooncake worker wait on the event, but nixl's add_transfer_request had no wait_event parameter. The event was recorded, attached and then dropped, so the RDMA read raced the writes it was meant to wait for. SGLANG_DISAGG_PREFILL_EARLY_SEND_CACHED_PREFIX defaults to True, so this is the default nixl PD path. Nothing fails visibly: the bytes do arrive, they just may not be final yet, and only an output-quality check catches it.

Modifications

  • disaggregation/prefill.py: create the event with self.device_module.Event(). device_module is set once from self.device in init_overlap(), the same module forward_stream was built from, so the event and the stream are always on the same accelerator. Using the process-default accelerator would bring that mismatch back whenever --device is not the default.
  • disaggregation/nixl/conn.py: add_transfer_request now accepts wait_event and puts it on the queued TransferKVChunk. The transfer worker waits on it the same way mooncake's does:
    • before the failed-room check and before the staging gather, since the gather also reads the source pages;
    • not for a room that has already failed;
    • then clears it, so a chunk re-enqueued on a staging-watermark defer doesn't wait again.
  • disaggregation/common/conn.py: add a CommonKVSender._take_early_send_wait_event() helper (returns the event and clears it), and set _early_send_wait_event in __init__. mooncake, mori and nixl now all read the event through this helper instead of getattr.

Tests

  • test_prefill_abort_result_cleanup.py: injects a stub device module and checks that the event comes from it and is recorded on forward_stream. This fails on the old code, which never touches the stub. The env flag is pinned with override(True).
  • test_nixl_backend_basic.py: the nixl worker waits before reading KV and before the staging gather, skips the wait for a failed room, add_transfer_request puts the event on the queued chunk, and taking the event clears it.
  • test_mooncake_transfer_batching.py: the mooncake worker waits before the staging gather; the existing early-send test uses the new helper.

Validation

Intel XPU, PD with nixl transfer + hierarchical cache, overlap scheduling:

  • Before: the prefill scheduler crashed on the second request of a shared-prefix group.
  • After: a 64-request shared-prefix run finished with 0 KV-transfer failures.

Checklist

  • Format your code according to the Format code with pre-commit.
  • Add unit tests according to the Run and add unit tests.
  • Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ✅ Run #40520157146
Latest PR Test (Extra): ❌ Run #40520157055
Latest PR Test (AMD ROCm 10): ❌ Run #40520156732

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@siju-samuel siju-samuel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@siju-samuel

Copy link
Copy Markdown
Contributor

/tag-run-ci-label

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 12, 2026
@kpjeeja

kpjeeja commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

CI is complete. Failures are not related to this PR:
XPU — Mamba fused-slot kernel segfault

Test: test/registered/layers/mamba/test_mamba_slot_fused.py
Failure: native SIGSEGV / exit code -11.
Cause: XPU/Triton raw pointer handling in the Mamba fused kernel path.
PR relation: No. The PR does not change Mamba, Triton kernels, pointer tables, or this test.
NPU — HiCache MLA test

Test: test/registered/npu/basic_function/HiCache/test_npu_hicache_mla.py
Failure: torch.OutOfMemoryError from Ascend aclrtMallocHostWithCfg during hierarchical-cache allocation; server is killed.
PR relation: No. It fails during server/cache initialization before any early-send event or transfer path runs.
NPU — PD disaggregation with HiCache

Test: test/registered/npu/basic_function/pd_disaggregation/test_npu_pd_disaggregation.py
Failure: same Ascend pinned-host-memory allocation OOM while booting prefill/decode servers.
PR relation: No evidence of a regression. This test is in the PD area, but it uses the ascend transfer backend and fails before the changed event-forwarding/transfer-worker paths execute. The PR’s main backend change is NIXL.

@Lzy17 Lzy17 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. test_add_transfer_request_forwards_the_wait_event_to_the_queued_chunk passes chunk_id positionally (False, 0, wait_event=event). #41140 removes chunk_id and makes wait_event keyword-only, so whichever lands second needs to adjust this call.

@github-actions

This comment has been minimized.

@kpjeeja kpjeeja changed the title [PD] Fix the early-send wait event on non-CUDA devices and in nixl [Fix] disaggregation: Fix the early-send wait event on non-CUDA devices and in nixl Oct 20, 2026
@github-actions

This comment has been minimized.

@kpjeeja

kpjeeja commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

@ShangmingCai can you please help review this fix?

@kpjeeja kpjeeja changed the title [Fix] disaggregation: Fix the early-send wait event on non-CUDA devices and in nixl [Fix] [Intel/AMD/NPU] disaggregation: Fix the early-send wait event on non-CUDA devices and in nixl Oct 20, 2026
@kpjeeja kpjeeja changed the title [Fix] [Intel/AMD/NPU] disaggregation: Fix the early-send wait event on non-CUDA devices and in nixl [Fix] [INTEL/AMD/NPU] disaggregation: Fix the early-send wait event on non-CUDA devices and in nixl Oct 20, 2026
@kpjeeja

kpjeeja commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

@ShangmingCai can you please help review this patch

maybe_send_cached_prefix_chunk records a completion event on forward_stream so
the transfer worker can wait for the prior step's prefill writes. torch.cuda.Event
raises "Tried to instantiate dummy base class Event" wherever torch has no CUDA
build, killing the prefill scheduler as soon as a request reuses a cached prefix
under overlap scheduling. self.device_module is resolved once from self.device in
init_overlap(), and is the same module forward_stream was built from, so the event
and the stream cannot come from different accelerators. Resolving the module from
the process default instead would reinstate that mismatch whenever --device names
something other than the default accelerator.

Reproduced on Intel XPU with PD (nixl transfer) + hierarchical cache: the shared
prefix in the second request of a group crashes the prefill scheduler; with this
fix a 64-request shared-prefix run completes with 0 KV-transfer failures.

The CPU unit test injects a stub device module and asserts the event comes from it
and records against forward_stream, so it fails on the pre-fix source, which never
touches the stub. It pins SGLANG_DISAGG_PREFILL_EARLY_SEND_CACHED_PREFIX with
override() rather than relying on the default, so a deployment that turns early
send off cannot make the case fail for an unrelated reason.

Signed-off-by: KP, Jeeja <jeeja.kp@intel.com>
@kpjeeja
kpjeeja force-pushed the jeeja/nixl_early-send_fix branch from 4f94fa0 to 2ad7e5e Compare October 26, 2026 05:26
kpjeeja and others added 2 commits October 26, 2026 15:33
maybe_send_cached_prefix_chunk records a completion event on forward_stream and
attaches it to the sender so the transfer worker can wait for the prior step's
prefill writes before reading those prefix pages. Since sgl-project#41140 the shared
CommonKVSender.send passes it to add_transfer_request, but nixl's
add_transfer_request accepted wait_event and never put it on the queued chunk,
and its worker never waited, so the RDMA read raced the writes it was meant to
order against. SGLANG_DISAGG_PREFILL_EARLY_SEND_CACHED_PREFIX defaults to True,
so this is the default PD path.

The failure is silent: the transfer did move the bytes, they just were not final,
so decode reads partially written KV and only an output-quality check can catch
it.

nixl's add_transfer_request now carries the event on the TransferKVChunk, and
the worker waits the way mooncake's does: before the failed-room check and
before the staging gather, skipped for a room that already failed, and cleared
so a chunk re-enqueued on a watermark defer does not wait again. The wait is
the worker thread's first device call, so the thread binds to kv_args.gpu_id
before its first wait, as create_backend does; otherwise CUDA opens a context on
device 0 on every prefill rank not on GPU 0. The CommonKVSender.send comment now
lists nixl among the workers that synchronize on the event.

Adds tests that the mooncake worker waits before the staging gather, that the
nixl worker binds its device and waits before reading KV and before the staging
gather and skips the wait for a failed room, and that NixlKVSender.send hands
the event to the next queued chunk only.
…device

sgl-project#45492 records the D2H page-index copy in _send_kv_chunk with
torch.cuda.Event() and torch.cuda.current_stream(). On XPU that raises
"Tried to instantiate dummy base class Event" and kills the prefill
scheduler on every Mooncake P/D request. Use self.device_module, as the
early-send event does; on CUDA and ROCm it is torch.cuda. NIXL does not
set supports_host_tensor_indices, so it never takes this path.

test_dcp_pack.py drives _send_kv_chunk with a sender that sets
supports_host_tensor_indices and a stub device_module, and checks the event
passed to send() comes from that module and is recorded on its current stream,
so reverting to torch.cuda fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kpjeeja
kpjeeja force-pushed the jeeja/nixl_early-send_fix branch from 4590e29 to e6248a2 Compare October 26, 2026 15:44

@siju-samuel siju-samuel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mingfeima mingfeima left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the changes looks fine. some ci failed due to rebase issue.
please rebase to main.

@kpjeeja

kpjeeja commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

the above CI failures are not related to this PR change - Small AMD shard (2): OpenAI Responses API is disabled due to the missing OpenAI type, leading to the 500.
MI35X disaggregation job: Mooncake/RDMA PD evaluation returns effectively zero accuracy.

@kpjeeja

kpjeeja commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

the changes looks fine. some ci failed due to rebase issue. please rebase to main
Rebased and CI is almost complete, the failures are not related to this PR Small AMD shard (2): OpenAI Responses API is disabled due to the missing OpenAI type, leading to the 500.
MI35X disaggregation job: Mooncake/RDMA PD evaluation returns effectively zero accuracy.

@mingfeima
mingfeima merged commit b9407b4 into sgl-project:main Oct 29, 2026
177 of 186 checks passed
jambow0320 added a commit to jambow0320/sglang that referenced this pull request Oct 29, 2026
sgl-project#43299 binds the NIXL transfer worker's device through torch before the
early-send wait synchronizes; the NixlTransport extraction had removed
the import when the agent creation moved to nixl/transport.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mooncake run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants