Skip to content

[Fix] PD: retraction restores racing the in-flight decode forward - #42702

Merged
xiezhq-hermann merged 3 commits into
sgl-project:mainfrom
fractalyze:fix/pd-decode-retraction-restore-fence
Oct 19, 2026
Merged

xiezhq-hermann merged 3 commits into
sgl-project:mainfrom
fractalyze:fix/pd-decode-retraction-restore-fence

Conversation

@jumanzii

@jumanzii jumanzii commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

On a PD decode server with the overlap scheduler, a retracted request's KV restored on resume can be overwritten by a forward that is still running. This holds for both retraction backups (--disaggregation-decode-retraction-backup host_pool or cpu_tensor; one of them is picked by default).

The decode event loop processes step N-1's results only after launching step N. A request that finished in step N-1 is still in batch N, so that forward writes the KV of its last token into a slot the request owned. Its slots, that one included, go back to the allocator when step N-1's result is processed (the decode radix cache is off by default). In the next iteration, process_decode_queue -> DecodePreallocQueue.resume_retracted_reqs can allocate those slots to a resumed request, and restore_kv_cache copies the backup into them:

  • the host_pool backend issues a host-to-device copy on the L2 transfer stream, ordered only after an event on the scheduler stream;
  • the cpu_tensor backend copies on the scheduler stream.

The scheduler stream does not wait for forward N's KV writes. _apply_war_barrier waits only for the forward's shared reads, unless it falls back to the coarse barrier. So the restore can land first, and forward N then overwrites a restored token with the finished request's KV. The resumed request continues decoding with one corrupted token in its context.

The prebuilt path in get_new_prebuilt_batch already drains the in-flight forward for the same reason, from #35748: self.schedule_stream.wait_stream(self.forward_stream) before a prebuilt request seeds a possibly reused row and before its host-received KV is restored. resume_retracted_reqs had no such wait.

Modifications

  • DecodePreallocQueue.resume_retracted_reqs: under the overlap scheduler, make the scheduler stream wait for the forward stream before each restore_kv_cache. Both backends issue their copy after work on the scheduler stream (the host_pool copy waits on an event recorded there), so both are ordered after the forward that may still write the reused slots. As with the prebuilt path, this gives up overlap for the step in which a retracted request resumes; resuming is rare (after retraction under memory pressure).
  • test_decode_queue_cleanup.py: the hand-built scheduler of test_paged_swa_retraction_resume_uses_physical_page_budget sets enable_overlap=False.
  • Test test_resumed_restore_lands_after_the_inflight_forward in test/registered/unit/mem_cache/test_decode_retraction_backup.py (CUDA). It uses real pools and a real UnifiedRadixCache with a host pool, and reproduces the interleaving deterministically:
    • a "forward" stream sleeps (torch.cuda._sleep) and then writes the slots that the resumed request is given;
    • resume_retracted_reqs then runs on the scheduler stream;
    • the restored KV must survive.
      On main the stale write lands after the restore and the test fails. A first pass with a short sleep loads every kernel first: CUDA lazy loading synchronizes the device on a kernel's first launch, which would otherwise order the race by accident.

Default behavior, not an option: the alternative is silent KV corruption, and the wait applies only in the iteration that resumes a retracted request.

This came out of auditing the retraction backup (UnifiedRadixCache.backup_kv_cache) for the overlap write-through race fixed in #42701. The backup is not exposed: it copies only seqlen - 1 tokens, and the forward in flight at retraction writes only position seqlen - 1, so every copied position was written by a forward whose result the scheduler has already processed. The restore side, here, is exposed.

Out of scope: KV that the prefill server writes over RDMA into slots preallocated in the same iteration (pop_preallocated) is not ordered after the in-flight forward either. Transfer latency likely hides it, and I have not verified it.

Related: #42701 fences HiCache write-through copies for the same kind of race. The two PRs are independent and can merge in either order.

Accuracy Tests

Not run end to end: reproducing this needs a PD deployment under enough memory pressure to retract and resume decode requests, and I did not set one up. The unit test above reproduces the interleaving on a GPU against the real pool, cache and restore code.

Speed Tests and Profiling

Not measured. The wait runs only in a scheduler iteration that resumes a retracted request.

Checklist

Test results (RTX 5090, CUDA 13, torch 2.14.1, rebased on f2f0be7):

  • New test on main: fails, the restored KV is replaced by the in-flight write. With this PR: passes. Reproduced on two machines.
  • test_decode_retraction_backup.py and test/registered/unit/disaggregation: 662 passed.
  • Before the rebase, test/registered/unit/managers and test/registered/unit/mem_cache matched main plus the new case; the one failure there (test_mm_process_config.py::TestStreamOrderedMmFeaturePool::test_consumer_slot_uses_global_tp_rank) also fails on main in the same environment.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #39211951759
Latest PR Test (Extra): ❌ Run #39211951672
Latest PR Test (AMD ROCm 10): ❌ Run #39211951712

The PD decode event loop processes step N-1's results only after launching
step N. A request that finished in step N-1 is still in batch N, whose forward
writes its last token's KV into a slot the request owned; that slot goes back
to the allocator when step N-1's result is processed. In the next iteration
resume_retracted_reqs can hand it to a resumed request and restore_kv_cache
copies the backup into it: the host_pool backend on the L2 transfer stream
after an event on the scheduler stream, the cpu_tensor backend on the
scheduler stream. The scheduler stream only waits for the forward's shared
reads, not its KV writes, so the restore can land first and the in-flight
forward then overwrites a restored token.

Make the scheduler stream wait for the forward stream before each restore
under the overlap scheduler, as get_new_prebuilt_batch already does before
seeding a possibly reused row. The wait runs only in an iteration that
resumes a retracted request.

The new test reproduces the interleaving on a GPU with real pools and a host
pool: a forward stream sleeps and then writes the slots the resumed request
is given, and the restored KV must survive. It fails without the wait.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jumanzii jumanzii changed the title [PD] Order retraction restores after the in-flight decode forward [PD] Fix retraction restores racing the in-flight decode forward Oct 6, 2026
@jumanzii

Copy link
Copy Markdown
Contributor Author

@ShangmingCai hi, mind taking a look when you have time? its a small fence in the PD resume path, the repro test is in the pr. ci only needs the run-ci label

@xiezhq-hermann xiezhq-hermann self-assigned this Oct 16, 2026
@xiezhq-hermann xiezhq-hermann added the run-ci CI: run the baseline test suite on this PR label Oct 16, 2026
@xiezhq-hermann xiezhq-hermann changed the title [PD] Fix retraction restores racing the in-flight decode forward [Fix] PD retraction restores racing the in-flight decode forward Oct 16, 2026
@github-actions

This comment has been minimized.

@xiezhq-hermann xiezhq-hermann changed the title [Fix] PD retraction restores racing the in-flight decode forward [Fix] PD: retraction restores racing the in-flight decode forward Oct 16, 2026
@xiezhq-hermann xiezhq-hermann added the highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed label Oct 16, 2026
@xiezhq-hermann
xiezhq-hermann merged commit 0ce41c5 into sgl-project:main Oct 19, 2026
249 of 265 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants