Skip to content

feat(scheduler): reserve KV for projected decode growth - #2469

Open
amd-ruitang3 wants to merge 1 commit into
mainfrom
amd-ruitang3/reserve-decode-kv-admission
Open

amd-ruitang3 wants to merge 1 commit into
mainfrom
amd-ruitang3/reserve-decode-kv-admission

Conversation

@amd-ruitang3

Copy link
Copy Markdown
Contributor

Summary

  • add opt-in ATOM_RESERVE_DECODE_TOKENS admission control for fresh prefills
  • reserve ceil(tokens / block_size) KV blocks per running sequence before admitting another request
  • keep partial-prefill resumes available and align PrefillDelayer queue signals with the same gate
  • preserve current scheduling when the variable is unset, invalid, zero, or negative

This is intended for high-concurrency offline workloads where front-loading every prefill can leave too little KV headroom for projected decode growth.

Tests

  • python -m pytest tests/test_scheduler.py -q — 193 passed
  • ruff check atom/model_engine/scheduler.py atom/utils/envs.py tests/test_scheduler.py
  • black --check --fast atom/model_engine/scheduler.py atom/utils/envs.py tests/test_scheduler.py
  • git diff --check

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2469 --add-label <label>

@zufayu
zufayu requested a review from yitingw1 October 5, 2026 00:31
@yitingw1

yitingw1 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

The engineering here is careful — zero effect when unset, the probe-vs-admission distinction in hypothetical_new_prefills is right in both call sites and documented, partials are correctly exempt with a test pinning it, and all five delayer signal surfaces are gated. Missing any one of those would have the delayer reporting work that Phase 2 then refuses, which is the failure mode #2238 spent two rounds on. None missed.

My question is about what the gate actually does.

1. This is a static concurrency cap, not a KV reservation

Simplifying the predicate:

(len(running) + 1) * R <= T ⟺ max concurrent running = floor(total_blocks / reserve_blocks)

Both terms on the right are constants after startup — T is pool capacity, R comes from the env var. So the gate is equivalent to --max-num-seqs min(current, num_kvcache_blocks // ceil(reserve_tokens / block_size)), reachable today with no new code.

More importantly it never consults occupancy. A sequence holding 100 blocks of prompt KV weighs the same as one holding 1. An empty pool and a 95%-full pool give the same answer. And preemption — the thing the description wants to avoid — is driven by occupancy, not by sequence count.

The number needed is one method up: _kv_usage (:946) already reads bm.kv.num_used. A reservation that tracks load would be:

free = total_blocks - self.block_manager.kv.num_used
return free >= (len(self.running) + 1) * self._reserve_decode_blocks

That version loosens on an empty pool and tightens as prompts fill it, which is what "reserve for projected decode growth" reads as. Was the capacity-based form intended? If so, the name, the doc row and the PR title should say "concurrency cap derived from KV capacity" — and it would help to say why that beats setting --max-num-seqs directly (deriving it from pool size so it survives a hardware change is a real reason, it just isn't written down).

2. The gate silently disables max_queue_ms

def _oldest_waiting_prefill_age_ms(self) -> float:
    if not self._kv_reservation_allows_new_prefill():
        return 0.0

That value feeds PrefillDelayer's max_queue_ms, the only wall-clock force-release in the repo. Reporting 0 tells the delayer nothing has been queued long.

Self-consistent — work that cannot be admitted shouldn't force a release — but per #1 the gate stays shut until a sequence finishes, which at steady-state high concurrency is the normal condition rather than the exception. An operator who sets ATOM_PREFILL_DELAYER_MAX_QUEUE_MS finds it never fires, with nothing in the log explaining why. At minimum an info line the first time the age signal is suppressed; better, pass the real age plus a "gated" flag so the decision stays in one place.

Smaller

  • The two probes count hypotheticals differently: _local_prefill_pending_work uses the enumerate index (candidates examined), _waiting_new_token_count increments manually only once a candidate is counted. They agree today only because that loop body has no continue — perf(scheduler): coalesce prefills on TP #2238 just added break branches to it, and the next continue breaks the equivalence. Worth using the manual counter in both.
  • max(0, _int_env_or_default(...)) swallows negatives silently: -1 parses fine, becomes 0, feature quietly off, no warning. feat(observability): add decode-step engine stats #2471 (same day, same author) adds _positive_int_env, which does warn on non-positive. If both land, the repo gets two helpers for one job with different behaviour on the same typo — worth converging on one.
  • The new entry in envs.py sits between the DSpark comment block and the # --- PrefillDelayer --- header, belonging to neither. The docs got a proper "Scheduler admission" section; envs.py should get the matching divider.
  • scheduler.py:2534 appends to running on the PD-transition path, bypassing the gate. Almost certainly correct (the request was admitted on the prefill node), but it means a decode node's len(running) can exceed the local cap and then keep the gate shut for its own waiting queue. One line in the docs would cover it.
  • For a change to scheduling admission there are no numbers — no throughput, no preemption count, no tail latency, no named workload. This is unusually easy to show: two arms, on and off, reporting preemption count and p99. The description's premise ("front-loading every prefill can leave too little KV headroom") is exactly what such a table would demonstrate.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants