Skip to content

[Fix] umbp: Order direct-linker offload copies after the forward that produced the KV - #45479

Closed
lixiufei-leo wants to merge 1 commit into
sgl-project:mainfrom
lixiufei-leo:fix/umbp-offload-wait-forward
Closed

lixiufei-leo wants to merge 1 commit into
sgl-project:mainfrom
lixiufei-leo:fix/umbp-offload-wait-forward

Conversation

@lixiufei-leo

@lixiufei-leo lixiufei-leo commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Under overlap scheduling, a prefill chunk is checkpointed into the radix tree, and may be offloaded, while the forward that writes its KV is still running on forward_stream. UMBPDirectLinker.offload() records its ready event on the calling (scheduler) stream only, so the offload thread can start copying pages before that forward has written them, and the store keeps an incomplete copy that later load-backs serve to other requests.

On CUDA the coarse WAR barrier (schedule_stream.wait_stream(forward_stream) after every launch) happens to order the two. On ROCm the barrier is off by default (SGLANG_ENABLE_WAR_BARRIER), and the race shows up whenever intermediate chunks are offloaded. #41520's checkpoint hit counting makes that happen on every chunk (fixed separately in #44893), but any offload of a node whose forward is still in flight can hit it.

This is the same class of bug that #44800 fixed for the HiCache write-back path.

Modifications

  • UnifiedCacheLinker.set_kv_producer_stream(stream): optional hook, no-op by default.
  • UMBPDirectLinker: offload() records a second event on the producer stream, and the offload thread waits on both before copying. The scheduler-stream event is kept because it still guards the index tensors produced on that stream.
  • Scheduler: passes forward_stream to the linker when overlap scheduling is on. With overlap off, nothing is set and offload() behaves as before.
  • Test: test_umbp_direct_linker_offload_ordering.py runs a forward that is still writing a page on a separate stream; the offload must read the written values. It passes on MI355X together with the existing UMBP linker unit tests; with the producer stream unset (the previous behaviour) the copy reads the pre-forward contents.

Loads are not changed here.

Accuracy Tests

No numerics change: the existing copy is only delayed until its producer finishes.

Copy staleness measured with a diagnostic that flags offloaded rows whose last layer is still mostly zero (unwritten) at copy time, same workload as below:

offloaded rows mostly unwritten at copy
main (1e86143) 4,531,710 2,773,485 (61.2%)
+ this PR 4,558,524 0

Speed Tests and Profiling

DeepSeek-V4-Pro FP4, 1P1D on MI355X: prefill TP8 / DP8 / EP8 with the UMBP linker (write_through), decode TP8 with DP attention, DSPARK, agentic workload at concurrency 256, 3600 s. Same node pair for every row, one run each. Decode step is regressed to 16 running req/rank and 220k context.

decode step (ms) P90 interactivity (tok/s/user) TTFT mean / p50 / p90 (s) total tok/s/GPU
main (1e86143) 49.69 60.35 8.14 / 4.94 / 16.23 60.1k
+ this PR 47.71 63.90 8.74 / 5.05 / 17.35 61.0k
+ #44893 47.48 64.39 8.71 / 5.13 / 16.68 61.0k
SGLANG_ENABLE_WAR_BARRIER=1 47.79 62.97 10.27 / 6.44 / 20.18 58.9k

The barrier gives the same decode speed but serializes every scheduler step behind the forward (TTFT +26%); this PR only makes the offload thread wait.

Checklist

  • Added a unit test for the ordering and ran it on MI355X.
  • Ran pre-commit on the changed files.

CI States

Latest PR Test (Base): ❌ Run #39531330120
Latest PR Test (Extra): ❌ Run #39531329965
Latest PR Test (AMD ROCm 10): ❌ Run #39531330094

@lixiufei-leo

Copy link
Copy Markdown
Contributor Author

Closing in favour of #45559, which applies the same fix to both the UMBP and Mooncake linkers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants