Repository navigation
[Fix] disaggregation: Fix the early-send wait event on non-CUDA devices and in nixl - #43299
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
/tag-run-ci-label |
|
CI is complete. Failures are not related to this PR: Test: test/registered/layers/mamba/test_mamba_slot_fused.py Test: test/registered/npu/basic_function/HiCache/test_npu_hicache_mla.py Test: test/registered/npu/basic_function/pd_disaggregation/test_npu_pd_disaggregation.py |
Lzy17
left a comment
There was a problem hiding this comment.
LGTM. test_add_transfer_request_forwards_the_wait_event_to_the_queued_chunk passes chunk_id positionally (False, 0, wait_event=event). #41140 removes chunk_id and makes wait_event keyword-only, so whichever lands second needs to adjust this call.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
@ShangmingCai can you please help review this fix? |
60be67c to
65bcacc
Compare
8d9d8dd to
d6e45f2
Compare
|
@ShangmingCai can you please help review this patch |
maybe_send_cached_prefix_chunk records a completion event on forward_stream so the transfer worker can wait for the prior step's prefill writes. torch.cuda.Event raises "Tried to instantiate dummy base class Event" wherever torch has no CUDA build, killing the prefill scheduler as soon as a request reuses a cached prefix under overlap scheduling. self.device_module is resolved once from self.device in init_overlap(), and is the same module forward_stream was built from, so the event and the stream cannot come from different accelerators. Resolving the module from the process default instead would reinstate that mismatch whenever --device names something other than the default accelerator. Reproduced on Intel XPU with PD (nixl transfer) + hierarchical cache: the shared prefix in the second request of a group crashes the prefill scheduler; with this fix a 64-request shared-prefix run completes with 0 KV-transfer failures. The CPU unit test injects a stub device module and asserts the event comes from it and records against forward_stream, so it fails on the pre-fix source, which never touches the stub. It pins SGLANG_DISAGG_PREFILL_EARLY_SEND_CACHED_PREFIX with override() rather than relying on the default, so a deployment that turns early send off cannot make the case fail for an unrelated reason. Signed-off-by: KP, Jeeja <jeeja.kp@intel.com>
4f94fa0 to
2ad7e5e
Compare
maybe_send_cached_prefix_chunk records a completion event on forward_stream and attaches it to the sender so the transfer worker can wait for the prior step's prefill writes before reading those prefix pages. Since sgl-project#41140 the shared CommonKVSender.send passes it to add_transfer_request, but nixl's add_transfer_request accepted wait_event and never put it on the queued chunk, and its worker never waited, so the RDMA read raced the writes it was meant to order against. SGLANG_DISAGG_PREFILL_EARLY_SEND_CACHED_PREFIX defaults to True, so this is the default PD path. The failure is silent: the transfer did move the bytes, they just were not final, so decode reads partially written KV and only an output-quality check can catch it. nixl's add_transfer_request now carries the event on the TransferKVChunk, and the worker waits the way mooncake's does: before the failed-room check and before the staging gather, skipped for a room that already failed, and cleared so a chunk re-enqueued on a watermark defer does not wait again. The wait is the worker thread's first device call, so the thread binds to kv_args.gpu_id before its first wait, as create_backend does; otherwise CUDA opens a context on device 0 on every prefill rank not on GPU 0. The CommonKVSender.send comment now lists nixl among the workers that synchronize on the event. Adds tests that the mooncake worker waits before the staging gather, that the nixl worker binds its device and waits before reading KV and before the staging gather and skips the wait for a failed room, and that NixlKVSender.send hands the event to the next queued chunk only.
…device sgl-project#45492 records the D2H page-index copy in _send_kv_chunk with torch.cuda.Event() and torch.cuda.current_stream(). On XPU that raises "Tried to instantiate dummy base class Event" and kills the prefill scheduler on every Mooncake P/D request. Use self.device_module, as the early-send event does; on CUDA and ROCm it is torch.cuda. NIXL does not set supports_host_tensor_indices, so it never takes this path. test_dcp_pack.py drives _send_kv_chunk with a sender that sets supports_host_tensor_indices and a stub device_module, and checks the event passed to send() comes from that module and is recorded on its current stream, so reverting to torch.cuda fails it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4590e29 to
e6248a2
Compare
mingfeima
left a comment
There was a problem hiding this comment.
the changes looks fine. some ci failed due to rebase issue.
please rebase to main.
|
the above CI failures are not related to this PR change - Small AMD shard (2): OpenAI Responses API is disabled due to the missing OpenAI type, leading to the 500. |
|
sgl-project#43299 binds the NIXL transfer worker's device through torch before the early-send wait synchronizes; the NixlTransport extraction had removed the import when the agent creation moved to nixl/transport.py.
Motivation
Under overlap scheduling,
maybe_send_cached_prefix_chunkcan start the KV transfer for a cached prefix while the previous step's prefill forward is still writing those pages onforward_stream. It records a completion event and attaches it to the KV sender, so the transfer worker can wait for those writes before reading the pages. Two bugs broke this:torch.cuda.Event(), which raisesTried to instantiate dummy base class Eventon torch builds without CUDA. On Intel XPU with PD + hierarchical cache, the prefill scheduler died on the first request that reused a cached prefix.add_transfer_requesthad nowait_eventparameter. The event was recorded, attached and then dropped, so the RDMA read raced the writes it was meant to wait for.SGLANG_DISAGG_PREFILL_EARLY_SEND_CACHED_PREFIXdefaults toTrue, so this is the default nixl PD path. Nothing fails visibly: the bytes do arrive, they just may not be final yet, and only an output-quality check catches it.Modifications
disaggregation/prefill.py: create the event withself.device_module.Event().device_moduleis set once fromself.deviceininit_overlap(), the same moduleforward_streamwas built from, so the event and the stream are always on the same accelerator. Using the process-default accelerator would bring that mismatch back whenever--deviceis not the default.disaggregation/nixl/conn.py:add_transfer_requestnow acceptswait_eventand puts it on the queuedTransferKVChunk. The transfer worker waits on it the same way mooncake's does:disaggregation/common/conn.py: add aCommonKVSender._take_early_send_wait_event()helper (returns the event and clears it), and set_early_send_wait_eventin__init__. mooncake, mori and nixl now all read the event through this helper instead ofgetattr.Tests
test_prefill_abort_result_cleanup.py: injects a stub device module and checks that the event comes from it and is recorded onforward_stream. This fails on the old code, which never touches the stub. The env flag is pinned withoverride(True).test_nixl_backend_basic.py: the nixl worker waits before reading KV and before the staging gather, skips the wait for a failed room,add_transfer_requestputs the event on the queued chunk, and taking the event clears it.test_mooncake_transfer_batching.py: the mooncake worker waits before the staging gather; the existing early-send test uses the new helper.Validation
Intel XPU, PD with nixl transfer + hierarchical cache, overlap scheduling:
Checklist
CI States
Latest PR Test (Base): ✅ Run #40520157146
Latest PR Test (Extra): ❌ Run #40520157055
Latest PR Test (AMD ROCm 10): ❌ Run #40520156732