Repository navigation
[BugFix] Resolve multiple async kv load deadlock - #44560
Merged
Merged
Conversation
Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
requested review from
ApostaC,
WoosukKwon,
alexm-redhat,
heheda12345,
orozery,
robertgshaw2-redhat and
ywang96
as code owners
June 4, 2026 17:19
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| @@ -386,7 +392,8 @@ def allocate_slots( | |||
| num_tokens_main_model=num_tokens_main_model, | |||
| ) | |||
|
|
|||
| if num_blocks_to_allocate > self.block_pool.get_num_free_blocks(): | |||
| available_blocks = self.block_pool.get_num_free_blocks() - reserved_blocks | |||
Contributor
There was a problem hiding this comment.
This estimation is pessimistic, so it might reduce utilization? I'm not sure of the trade-off vs preemption (which only happens at the peak usage).
Contributor
There was a problem hiding this comment.
Discussed offline. This is correct for PD case and might affect throughput for non-PD on some edge cases. But those cases are rare and should run in PD anyway.
WoosukKwon
approved these changes
Jun 5, 2026
njhill
enabled auto-merge (squash)
June 6, 2026 22:37
njhill
added a commit
to njhill/vllm
that referenced
this pull request
Jun 7, 2026
Resolve conflicts with the async KV-load deadlock fix (vllm-project#44560) in the KV cache admission path: - allocate_slots() now takes both num_running_reqs (watermark sizing) and reserved_blocks (in-flight prefill reservation). - The free-block admission check stacks both: a request is admitted only if num_blocks_to_allocate + watermark_blocks <= free_blocks - reserved_blocks. - scheduler passes len(self.running) and the in-flight-prefill reservation. Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
nkzhenhua
pushed a commit
to nkzhenhua/vllm
that referenced
this pull request
Jun 24, 2026
Signed-off-by: Nick Hill <nickhill123@gmail.com>
MingqiWang-coder
added a commit
to intellistream/vllm-hust-legacy-20260831
that referenced
this pull request
Jul 2, 2026
Cherry-pick 62 bugfix/security PRs from upstream vllm-project/vllm main (2026-05-03 to 2026-06-17), covering scheduler, engine core, model runner, worker, attention, KV cache, compilation, and structured output fixes. Security (4): vllm-project#43286 vllm-project#44744 vllm-project#45118 vllm-project#45252 Bugfix (56): vllm-project#35536 vllm-project#36616 vllm-project#38895 vllm-project#39155 vllm-project#39324 vllm-project#39562 vllm-project#39805 vllm-project#40398 vllm-project#40726 Runner fix (2): vllm-project#44568 vllm-project#44603 Skipped: vllm-project#43781 (ROCm-specific, not applicable to Ascend NPU) Conflict resolutions: - Manual merge: vllm-project#43286 vllm-project#45118 vllm-project#42112 vllm-project#43160 vllm-project#43719 vllm-project#44560 - Upstream-preferred (-X theirs): vllm-project#43808 vllm-project#43988 vllm-project#42967 vllm-project#35536 vllm-project#45195 - Test files (--theirs): vllm-project#44744 vllm-project#41895 vllm-project#42040 vllm-project#41233 vllm-project#45345 vllm-project#43982 Co-authored-by: GitHub Copilot Signed-off-by: MingqiWang-coder <mingqiwang@hust.edu.cn>
philippesic
pushed a commit
to philippesic/vllm-semantic-cache
that referenced
this pull request
Jul 19, 2026
Signed-off-by: Nick Hill <nickhill123@gmail.com>
xmdhb
pushed a commit
to xmdhb/vllm-hust
that referenced
this pull request
Jul 24, 2026
…ulable queue head Requests parked in WAITING_FOR_REMOTE_KVS hold their allocated blocks and are only promoted back to WAITING when the waiting-queue traversal reaches them and consumes their finished_recving signal. When the queue head fails block allocation, the traversal stops with a bare break; if nothing is running at that point, no future event can free blocks and the scheduler freezes permanently (Running: 0, Waiting: N, forever). The exact FCFS scenario of vllm-project#45388 was already fixed by vllm-project#44560's admission gate, but the gate only protects an FCFS admission-time invariant: under priority scheduling a later, higher-priority, unschedulable request can still strand a parked async load that holds the very blocks it needs (deterministic regression test included), and post-admission events can invalidate the reserved-fit invariant the same way. Fix: keep the bare break only while something is running (completion will free blocks; queue-order admission is preserved). With nothing running, skip the unschedulable request via the existing step_skipped_waiting mechanism and keep scanning so parked requests can be promoted, scheduled, and eventually release the blocks the head needs. Fixes the permanent-deadlock family of vllm-project#45388, vllm-project#42371. Co-authored-by: Claude Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Wayne Chiu <waynehacking8@gmail.com>
1 task done
wzhao18
pushed a commit
to wzhao18/vllm
that referenced
this pull request
Sep 14, 2026
…ulable queue head Requests parked in WAITING_FOR_REMOTE_KVS hold their allocated blocks and are only promoted back to WAITING when the waiting-queue traversal reaches them and consumes their finished_recving signal. When the queue head fails block allocation, the traversal stops with a bare break; if nothing is running at that point, no future event can free blocks and the scheduler freezes permanently (Running: 0, Waiting: N, forever). The exact FCFS scenario of vllm-project#45388 was already fixed by vllm-project#44560's admission gate, but the gate only protects an FCFS admission-time invariant: under priority scheduling a later, higher-priority, unschedulable request can still strand a parked async load that holds the very blocks it needs (deterministic regression test included), and post-admission events can invalidate the reserved-fit invariant the same way. Fix: keep the bare break only while something is running (completion will free blocks; queue-order admission is preserved). With nothing running, skip the unschedulable request via the existing step_skipped_waiting mechanism and keep scanning so parked requests can be promoted, scheduled, and eventually release the blocks the head needs. Fixes the permanent-deadlock family of vllm-project#45388, vllm-project#42371. Co-authored-by: Claude Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Wayne Chiu <waynehacking8@gmail.com>
6 tasks done
6 tasks done
2 of 4 tasks
6 tasks done
2 of 6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a different fix for the issue targeted by #40968.
The changes are simpler and the resulting behavior should also be better - we avoid async-loading new kv if it would occupy blocks required for other in-flight chunked prefills or async-loading requests.
Note: we apply this throttle to async kv loads only, rather than chunked-prefill requests in general, since the latter are naturally throttled by batch capacity and are also already eligible for preemption.
cc @Dao007forever