Repository navigation
Conversation
… in-flight requests The scheduler requests eviction of GPU-resident encoder outputs based on its own bookkeeping of request progress, which can be stale by the time the worker acts on it: - Under async scheduling, num_computed_tokens is advanced speculatively and rolled back when draft tokens are rejected, so an entry freed as "fully consumed" can be needed again on the next step. - The spec-decode drafter reads a +1-shifted window and can need an entry one step longer than the target model. - Entries are keyed by content hash and shared across concurrent requests, so one request's completion can evict an entry another request needs. Any of these races crashes the engine with "AssertionError: Encoder cache miss" in _gather_mm_embeddings, killing all in-flight requests (issue vllm-project#38551). Fix: EncoderCache now tracks which in-flight requests reference each mm_hash. A scheduler-requested eviction of a still-referenced entry is deferred until the last referencing request is removed; unreferenced entries are evicted eagerly as before. Memory overhead is bounded by the encoder outputs of in-flight requests (unlike full retention, which is unbounded). Encoder-decoder models (e.g. Whisper) keep eager eviction since the scheduler only frees their entries once provably dead. The same policy is applied to both the new gpu/ worker architecture (EncoderCache was already request-aware) and the legacy gpu_model_runner (via a tracker sharing the existing encoder_cache dict). FIX vllm-project#38551 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Roger Wang <hey@rogerw.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.