Prefix cache index not invalidated when a vLLM engine clears its prefix cache (reset/OOM/sleep)

[Jeffwan]

Describe the bug

When an engine clears its prefix cache — /reset_prefix_cache, OOM-driven reset, or /sleep level>=1 — it wipes the cache and enqueues AllBlocksCleared. We don't act on it: handleAllBlocksCleared is a no-op (pkg/kvevent/handler.go), and the only caller of RemovePrefix is pod unsubscribe, which doesn't fire here (the pod stays Running, /health still returns 200). So stale entries linger, and because the prefix router prefers the pod with the most matching blocks, it actively routes prefix traffic to the pod whose cache was just wiped — a cold miss.

The event alone isn't a reliable fix: vLLM publishes queued KV events only from inside a scheduler step (update_from_output), so an idle/sleeping engine may never flush AllBlocksCleared, and ZMQ delivery is lossy.

Steps to Reproduce

1. Two replicas with the kv-event prefix router; warm a shared prefix on pod A.
2. POST /reset_prefix_cache on pod A (or /sleep?level=1).
3. Send the same prefix again → still routed to A, now a cold miss.

Expected behavior

• handleAllBlocksCleared calls RemovePrefix(model, loraID, podKey) (already on the SyncIndexer interface).
• A reconcile against the scraped vllm:engine_sleep_state metric backstops dropped events (purge entries for pods reporting non-awake).
• When AIBrix drives sleep itself (separate sleep-mode RFC), it should also purge synchronously on /sleep — tracked there.

Environment

AIBrix: main · vLLM: main (kv-events enabled).

[Jeffwan]

wait until #2290 is implemented

[pjdurden]

Opened #2385 for part 1: handleAllBlocksCleared now calls RemovePrefix(model, loraID, podKey) to purge the pod from the prefix index, mirroring handleBlockRemoved. Tests cover the purge call, temporary-error swallow, and error propagation.

Left the metric-driven reconcile backstop (purge against scraped vllm:engine_sleep_state) for a follow-up since it is a larger change and, as you noted, the event alone is not fully reliable. Happy to take that next if the approach in #2385 looks right.