Repository navigation
[Fix] umbp: Order direct-linker offload copies after the forward that produced the KV - #45479
Closed
lixiufei-leo wants to merge 1 commit into
Closed
lixiufei-leo wants to merge 1 commit into
lixiufei-leo wants to merge 1 commit into
Conversation
lixiufei-leo
requested review from
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
October 21, 2026 02:08
Contributor
Author
|
Closing in favour of #45559, which applies the same fix to both the UMBP and Mooncake linkers. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Under overlap scheduling, a prefill chunk is checkpointed into the radix tree, and may be offloaded, while the forward that writes its KV is still running on
forward_stream.UMBPDirectLinker.offload()records its ready event on the calling (scheduler) stream only, so the offload thread can start copying pages before that forward has written them, and the store keeps an incomplete copy that later load-backs serve to other requests.On CUDA the coarse WAR barrier (
schedule_stream.wait_stream(forward_stream)after every launch) happens to order the two. On ROCm the barrier is off by default (SGLANG_ENABLE_WAR_BARRIER), and the race shows up whenever intermediate chunks are offloaded. #41520's checkpoint hit counting makes that happen on every chunk (fixed separately in #44893), but any offload of a node whose forward is still in flight can hit it.This is the same class of bug that #44800 fixed for the HiCache write-back path.
Modifications
UnifiedCacheLinker.set_kv_producer_stream(stream): optional hook, no-op by default.UMBPDirectLinker:offload()records a second event on the producer stream, and the offload thread waits on both before copying. The scheduler-stream event is kept because it still guards the index tensors produced on that stream.Scheduler: passesforward_streamto the linker when overlap scheduling is on. With overlap off, nothing is set andoffload()behaves as before.test_umbp_direct_linker_offload_ordering.pyruns a forward that is still writing a page on a separate stream; the offload must read the written values. It passes on MI355X together with the existing UMBP linker unit tests; with the producer stream unset (the previous behaviour) the copy reads the pre-forward contents.Loads are not changed here.
Accuracy Tests
No numerics change: the existing copy is only delayed until its producer finishes.
Copy staleness measured with a diagnostic that flags offloaded rows whose last layer is still mostly zero (unwritten) at copy time, same workload as below:
Speed Tests and Profiling
DeepSeek-V4-Pro FP4, 1P1D on MI355X: prefill TP8 / DP8 / EP8 with the UMBP linker (
write_through), decode TP8 with DP attention, DSPARK, agentic workload at concurrency 256, 3600 s. Same node pair for every row, one run each. Decode step is regressed to 16 running req/rank and 220k context.SGLANG_ENABLE_WAR_BARRIER=1The barrier gives the same decode speed but serializes every scheduler step behind the forward (TTFT +26%); this PR only makes the offload thread wait.
Checklist
CI States
Latest PR Test (Base): ❌ Run #39531330120
Latest PR Test (Extra): ❌ Run #39531329965
Latest PR Test (AMD ROCm 10): ❌ Run #39531330094