Repository navigation
[Fix] PD: retraction restores racing the in-flight decode forward - #42702
Merged
xiezhq-hermann merged 3 commits intoOct 19, 2026
Merged
xiezhq-hermann merged 3 commits into
xiezhq-hermann merged 3 commits into
Conversation
The PD decode event loop processes step N-1's results only after launching step N. A request that finished in step N-1 is still in batch N, whose forward writes its last token's KV into a slot the request owned; that slot goes back to the allocator when step N-1's result is processed. In the next iteration resume_retracted_reqs can hand it to a resumed request and restore_kv_cache copies the backup into it: the host_pool backend on the L2 transfer stream after an event on the scheduler stream, the cpu_tensor backend on the scheduler stream. The scheduler stream only waits for the forward's shared reads, not its KV writes, so the restore can land first and the in-flight forward then overwrites a restored token. Make the scheduler stream wait for the forward stream before each restore under the overlap scheduler, as get_new_prebuilt_batch already does before seeding a possibly reused row. The wait runs only in an iteration that resumes a retracted request. The new test reproduces the interleaving on a GPU with real pools and a host pool: a forward stream sleeps and then writes the slots the resumed request is given, and the restored KV must survive. It fails without the wait. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
jumanzii
requested review from
ByronHsu,
Duyi-Wang,
HaiShaw,
ShangmingCai,
hnyls2002 and
sogalin
as code owners
October 6, 2026 02:34
4 of 5 tasks
3 tasks done
Contributor
Author
|
@ShangmingCai hi, mind taking a look when you have time? its a small fence in the PD resume path, the repro test is in the pr. ci only needs the run-ci label |
xiezhq-hermann
approved these changes
Oct 16, 2026
This comment has been minimized.
This comment has been minimized.
3 of 4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On a PD decode server with the overlap scheduler, a retracted request's KV restored on resume can be overwritten by a forward that is still running. This holds for both retraction backups (
--disaggregation-decode-retraction-backup host_poolorcpu_tensor; one of them is picked by default).The decode event loop processes step N-1's results only after launching step N. A request that finished in step N-1 is still in batch N, so that forward writes the KV of its last token into a slot the request owned. Its slots, that one included, go back to the allocator when step N-1's result is processed (the decode radix cache is off by default). In the next iteration,
process_decode_queue->DecodePreallocQueue.resume_retracted_reqscan allocate those slots to a resumed request, andrestore_kv_cachecopies the backup into them:host_poolbackend issues a host-to-device copy on the L2 transfer stream, ordered only after an event on the scheduler stream;cpu_tensorbackend copies on the scheduler stream.The scheduler stream does not wait for forward N's KV writes.
_apply_war_barrierwaits only for the forward's shared reads, unless it falls back to the coarse barrier. So the restore can land first, and forward N then overwrites a restored token with the finished request's KV. The resumed request continues decoding with one corrupted token in its context.The prebuilt path in
get_new_prebuilt_batchalready drains the in-flight forward for the same reason, from #35748:self.schedule_stream.wait_stream(self.forward_stream)before a prebuilt request seeds a possibly reused row and before its host-received KV is restored.resume_retracted_reqshad no such wait.Modifications
DecodePreallocQueue.resume_retracted_reqs: under the overlap scheduler, make the scheduler stream wait for the forward stream before eachrestore_kv_cache. Both backends issue their copy after work on the scheduler stream (thehost_poolcopy waits on an event recorded there), so both are ordered after the forward that may still write the reused slots. As with the prebuilt path, this gives up overlap for the step in which a retracted request resumes; resuming is rare (after retraction under memory pressure).test_decode_queue_cleanup.py: the hand-built scheduler oftest_paged_swa_retraction_resume_uses_physical_page_budgetsetsenable_overlap=False.test_resumed_restore_lands_after_the_inflight_forwardintest/registered/unit/mem_cache/test_decode_retraction_backup.py(CUDA). It uses real pools and a realUnifiedRadixCachewith a host pool, and reproduces the interleaving deterministically:torch.cuda._sleep) and then writes the slots that the resumed request is given;resume_retracted_reqsthen runs on the scheduler stream;On
mainthe stale write lands after the restore and the test fails. A first pass with a short sleep loads every kernel first: CUDA lazy loading synchronizes the device on a kernel's first launch, which would otherwise order the race by accident.Default behavior, not an option: the alternative is silent KV corruption, and the wait applies only in the iteration that resumes a retracted request.
This came out of auditing the retraction backup (
UnifiedRadixCache.backup_kv_cache) for the overlap write-through race fixed in #42701. The backup is not exposed: it copies onlyseqlen - 1tokens, and the forward in flight at retraction writes only positionseqlen - 1, so every copied position was written by a forward whose result the scheduler has already processed. The restore side, here, is exposed.Out of scope: KV that the prefill server writes over RDMA into slots preallocated in the same iteration (
pop_preallocated) is not ordered after the in-flight forward either. Transfer latency likely hides it, and I have not verified it.Related: #42701 fences HiCache write-through copies for the same kind of race. The two PRs are independent and can merge in either order.
Accuracy Tests
Not run end to end: reproducing this needs a PD deployment under enough memory pressure to retract and resume decode requests, and I did not set one up. The unit test above reproduces the interleaving on a GPU against the real pool, cache and restore code.
Speed Tests and Profiling
Not measured. The wait runs only in a scheduler iteration that resumes a retracted request.
Checklist
Test results (RTX 5090, CUDA 13, torch 2.14.1, rebased on f2f0be7):
main: fails, the restored KV is replaced by the in-flight write. With this PR: passes. Reproduced on two machines.test_decode_retraction_backup.pyandtest/registered/unit/disaggregation: 662 passed.test/registered/unit/managersandtest/registered/unit/mem_cachematchedmainplus the new case; the one failure there (test_mm_process_config.py::TestStreamOrderedMmFeaturePool::test_consumer_slot_uses_global_tp_rank) also fails onmainin the same environment.🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #39211951759
Latest PR Test (Extra): ❌ Run #39211951672
Latest PR Test (AMD ROCm 10): ❌ Run #39211951712