[RFC]: Control-plane orchestration of vLLM sleep mode and kvcached for warm standby and multi-model GPU pooling

[Jeffwan]

Summary

AIBrix scales replicas as 0-or-N and only observes engine memory state; it never drives it. Two engine-side mechanisms now make GPU memory reclaimable at runtime — vLLM sleep mode (park a whole idle model's weights+KV) and kvcached (elastically shrink/grow a model's KV so several models share one GPU) — but both are mechanism-only, with no control plane. This RFC proposes AIBrix own the policy: a control-plane-driven "warm/asleep" tier between one replica and zero, plus KV-budget arbitration across co-resident models. The result: multi-minute cold starts become second-scale wakes, and one GPU can densely host many long-tail models.

Motivation

Scaling is binary today. To reclaim a GPU we either keep a replica running (burning HBM while idle) or delete the pod and pay a multi-minute cold start (weight load + CUDA graph capture) when traffic returns. Neither fits bursty or long-tail traffic.

The engine now exposes the levers, but not the policy:

	vLLM sleep mode	kvcached (github.com/ovg-project/kvcached)
Scope	whole model (weights + KV)	KV cache only (weights allocated outside it)
Granularity	all-or-nothing, single process	page-level elastic, cross-process co-residency
Target	a fully idle model	several active models sharing a GPU
Mechanism	CUDA VMM; /sleep /wake_up; level 1 weights→CPU, level 2 discard	CUDA VMM; per-process cudaMemGetInfo headroom + /dev/shm budget via kvctl
Control	manual API only; VLLM_SERVER_DEV_MODE-gated; no autoscaling	decentralized by design; no scheduler/admission/cross-node; controller/ is a demo

They are complementary: kvcached handles elastic KV among active co-resident models; sleep parks idle models' weights. Both are VMM actuators that enforce/observe — neither decides. The missing piece is a control plane that decides which model sleeps/wakes and when, how much KV budget each co-resident model gets, admission when a GPU is oversubscribed, and routing/readiness as models move between awake / warm / asleep. AIBrix already scrapes vllm:engine_sleep_state but never acts on it, and has no notion of co-resident KV budget — so it is the natural owner.

Proposed Change

(Sketch only; full design — CRD, state machine, budget algorithm, agent protocol — to follow in a complete doc.)

1. A warm/asleep lifecycle state between minReplicas and zero: low traffic → POST /sleep?level=1&mode=wait (graceful drain); traffic returns → /wake_up (seconds). Pod/IP/endpoint persist.
2. A KV-budget arbiter across co-resident models, driving kvctl limits by policy (per-model min/max/share, priority) with cost-asymmetric reclaim — shrink KV first (cheap), sleep weights only when protected floors don't fit (expensive).
3. A decide/execute split: a Go controller (placement, admission, sleep/wake & budget policy, status) over a pod-side agent (kvcached engine, kvctl, sleep).
4. Required couplings: invalidate the prefix index on sleep (KV is wiped — see the sleep/prefix-index bug), exclude asleep pods from routing/readiness, and document the VLLM_SERVER_DEV_MODE requirement.

Alternatives Considered

• HPA scale-to-zero (minReplicas=0): already supported, but pays the full cold start and has no warm tier.
• Leave it to kvcached's decentralized scheme: works on a single node but has no admission/priority/cross-node policy and no sleep orchestration — precisely the demo controller/ AIBrix would replace.
• Token-level time-sharing (swap at request granularity): higher utilization ceiling but far more complex/intrusive; this RFC targets model-granularity pooling.

[jiangxiaobin96]

We can follow Aegaeon grouping the same model request and use some algorithms to determine when to switch model by sleep mode to guarantee TTFT.

[newhans]

A few things I'm not clear on after reading this:

Who provides GPU co-location? AIBrix uses nvidia.com/gpu today (whole-GPU). This RFC assumes pods are already co-resident, but that requires fractional GPU scheduling (HAMi, Volcano, time-slicing, or custom). Is AIBrix going to own that layer, or is it an external dep? If external, does the control plane need to know which sharing mechanism is in use to calculate headroom correctly?

Wake-up under memory pressure. When a sleeping model wakes, its neighbors may have expanded into the freed space. What happens if they can't shrink fast enough (in-flight requests holding pages)? What if multiple models wake at once? The RFC says "cost-asymmetric reclaim" but doesn't say what happens when reclaim fails — does the wake get rejected, or does a neighbor get killed? Feels like this needs an explicit overcommit policy.

Host memory as the second bottleneck. Level 1 sleep offloads weights to host RAM. With multiple models sleeping on the same node (say 8 GPUs × 2-3 models each), offloaded weights alone can eat hundreds of GB. The control plane needs to account for node-level memory headroom when deciding sleep levels — otherwise you trade GPU OOM for host OOM (or the Linux OOM-killer starts evicting pods). Is the sleep-level decision (L1 vs L2) going to be node-aware?

Also — #2287 (prefix index not invalidated on sleep) is a good example of why the control plane needs to own the full sleep lifecycle.

[Jeffwan]

@newhans Great questions — let me address them one by one.

Who provides GPU co-location?

I don't think we need a GPU resource claim for this abstraction. The goal here isn't to deliver yet another GPU-sharing solution — the isolation should be transparent to the GPU. In other words, only the GPU worker claims the GPU; the rest of the model registration stays resource-agnostic.

Wake-up under memory pressure.

This is a good point. Technically, we can surface the response status from vLLM: if a model can't be woken up on its original pod, it can be re-registered on a different one. This is analogous to VPA's in-place update failure falling back to the out-of-place path.

Host memory as the second bottleneck.

Absolutely. Both DRAM and HBM need to be monitored and accounted for by the controller when making placement decisions, this ultimately becomes a multi-dimensional resource management problem.

That said, I'd personally defer this to a second phase. Building a full resource-management layer on top of Kubernetes adds significant complexity, and that complexity is exactly the gap between a prototype and production-ready usage.

You clearly have a lot of context in this space. :D I'd love to keep this conversation going. Happy to hear your thoughts on the phasing

[Jeffwan]

@jiangxiaobin96 we can build the interface abstraction for that strategy, technically, I personally feel aegaeon's granularity is too fine-grain which limits its landing. We'd like to build a more generic multiplexing system which is engine agnostic or just reply on few engine features.