Repository navigation
Rewrite worker block tables after SWA eviction - #42276
AbhiOnGithub wants to merge 2 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements a mechanism to handle block table rewrites for cached requests, specifically addressing scenarios where Sliding Window Attention (SWA) eviction replaces earlier logical slots with null blocks. The KVCacheManager and KVCacheCoordinator now track when a block table is updated, and the Scheduler communicates this via a new block_table_rewrite_req_ids field in CachedRequestData. Correspondingly, the GPU model runners have been updated to overwrite the block table instead of appending when a rewrite is signaled. I have no feedback to provide as no review comments were submitted.
ec0e963 to
4aafaa6
Compare
Track when sliding-window eviction rewrites existing logical block slots and send the rewritten row to workers so stale entries are replaced instead of appended past. Signed-off-by: Abhishek Gupta (AbhiOnGithub) <12324923+AbhiOnGithub@users.noreply.github.com>
4aafaa6 to
2ebc56a
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
This pull request has merge conflicts that must be resolved before it can be |
…refix caching is off vllm-project#48017 was described as a pure no-op that skips the pointless LRU hash-split when prefix caching is disabled, but the merged condition (block.block_hash is None and self.enable_caching) routes every freed block through append_n instead of prepend_n on that path. Since allocation pops from the head of FreeKVCacheBlockQueue, this silently flipped block reuse from LIFO (immediate reuse of a small hot set of ids) to FIFO (cycling the entire block-id space) - contradicting both the PR description and the in-code cache-locality rationale. This change implements the optimization as described: with caching off, skip the partition entirely, decrement refcounts, and prepend_n the freed blocks, preserving the pre-vllm-project#48017 reuse order. Behavior with prefix caching enabled is untouched. The reuse order is observable: on TPU v7x the accidental FIFO order deterministically triggers E0200 RuntimeUnexpectedCoreHalt (SparseCore UserFatal) minutes into sustained serving, bisected to vllm-project#48017 across 20+ isolation runs (evidence: https://github.com/vllm-project/vllm-torchtpu/blob/mhhua/sc2-sparsecore-halt-evidence/docs/sc2-sparsecore-halt-2026-08.md). Reuse order also interacts with the stale block-table entries of vllm-project#42273 (see discussion in that issue) - the proper decoupling is vllm-project#42276-style worker-side slot rewrites, after which reuse order is functionally neutral on all platforms. Signed-off-by: theminghuang <theminghuang@gmail.com> Signed-off-by: Ming Huang <mhhua@google.com>
|
This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you! |
Fixes #42273
Summary:
Tests:
.venv/bin/python -m pytest tests/v1/core/test_scheduler.py::test_schedule tests/v1/core/test_scheduler.py::test_swa_eviction_sends_block_table_rewrite_for_cached_request tests/v1/core/test_single_type_kv_cache_manager.py tests/v1/simple_kv_offload/test_scheduler.py::test_eager_store_and_load_roundtrip tests/v1/worker/test_gpu_model_runner.py::test_update_states_rewrites_cached_block_table -v(13 passed).venv/bin/python -m ruff check vllm/v1/core/kv_cache_coordinator.py vllm/v1/core/kv_cache_manager.py vllm/v1/core/sched/output.py vllm/v1/core/sched/scheduler.py vllm/v1/core/single_type_kv_cache_manager.py vllm/v1/worker/gpu/model_runner.py vllm/v1/worker/gpu_model_runner.py tests/v1/core/test_scheduler.py tests/v1/core/test_single_type_kv_cache_manager.py tests/v1/core/utils.py tests/v1/simple_kv_offload/test_scheduler.py tests/v1/worker/test_gpu_model_runner.py(passed)