Repository navigation
feat(scheduler): reserve KV for projected decode growth - #2469
amd-ruitang3 wants to merge 1 commit into
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
|
The engineering here is careful — zero effect when unset, the probe-vs-admission distinction in My question is about what the gate actually does. 1. This is a static concurrency cap, not a KV reservationSimplifying the predicate: (len(running) + 1) * R <= T ⟺ max concurrent running = floor(total_blocks / reserve_blocks) Both terms on the right are constants after startup — More importantly it never consults occupancy. A sequence holding 100 blocks of prompt KV weighs the same as one holding 1. An empty pool and a 95%-full pool give the same answer. And preemption — the thing the description wants to avoid — is driven by occupancy, not by sequence count. The number needed is one method up: free = total_blocks - self.block_manager.kv.num_used
return free >= (len(self.running) + 1) * self._reserve_decode_blocksThat version loosens on an empty pool and tightens as prompts fill it, which is what "reserve for projected decode growth" reads as. Was the capacity-based form intended? If so, the name, the doc row and the PR title should say "concurrency cap derived from KV capacity" — and it would help to say why that beats setting --max-num-seqs directly (deriving it from pool size so it survives a hardware change is a real reason, it just isn't written down). 2. The gate silently disables max_queue_msdef _oldest_waiting_prefill_age_ms(self) -> float:
if not self._kv_reservation_allows_new_prefill():
return 0.0That value feeds PrefillDelayer's max_queue_ms, the only wall-clock force-release in the repo. Reporting 0 tells the delayer nothing has been queued long. Self-consistent — work that cannot be admitted shouldn't force a release — but per #1 the gate stays shut until a sequence finishes, which at steady-state high concurrency is the normal condition rather than the exception. An operator who sets ATOM_PREFILL_DELAYER_MAX_QUEUE_MS finds it never fires, with nothing in the log explaining why. At minimum an info line the first time the age signal is suppressed; better, pass the real age plus a "gated" flag so the decision stays in one place. Smaller
|
Summary
ATOM_RESERVE_DECODE_TOKENSadmission control for fresh prefillsceil(tokens / block_size)KV blocks per running sequence before admitting another requestThis is intended for high-concurrency offline workloads where front-loading every prefill can leave too little KV headroom for projected decode growth.
Tests
python -m pytest tests/test_scheduler.py -q— 193 passedruff check atom/model_engine/scheduler.py atom/utils/envs.py tests/test_scheduler.pyblack --check --fast atom/model_engine/scheduler.py atom/utils/envs.py tests/test_scheduler.pygit diff --check