Skip to content

[Fix] hicache: Log and label buffer-only backup drops - #44339

Draft
xiezhq-hermann wants to merge 2 commits into
mainfrom
hicache/backup-drop-visibility
Draft

xiezhq-hermann wants to merge 2 commits into
mainfrom
hicache/backup-drop-visibility

Conversation

@xiezhq-hermann

@xiezhq-hermann xiezhq-hermann commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

With --hicache-host-memory-mode buffer_only and write_through, a store that drains slower than prefill produces KV silently loses whole prompts from L3. Mechanism (python/sglang/srt/mem_cache/buffer_mode/pipeline.py): a finished node's backup intent is queued unpinned (the device lock is only taken at the D2H launch in _stage_backup_intent), host staging is held until the storage ack (finish_storage_write_ack), and flush_pending_writes stops launching once write_staged_tokens_ >= live_cap. The FULL eviction walk does not consult the queue, so a device pool turning over faster than the window drains evicts exactly the queued leaves; the next _refresh_pending_backup_intents sweeps the span and the only trace was an unlabeled increment of sglang:hicache_backup_dropped_tokens_total. On a TP2 lane with the stock file backend (~1.6 GB/s/rank single-threaded writes vs ~4.3 GB/s of prefill KV) this loses 20-28% of a 40-prompt cold seed, always whole prompts, with zero log lines; the user-visible symptom is an unexplained L3 miss rate on prefixes seen seconds ago. The same pipeline runs on every tree core and backend.

This PR makes the loss visible and attributable. It does not change what is dropped (the follow-up PR adds an opt-in throttle).

Modifications

Metric label change: sglang:hicache_backup_dropped_tokens_total gains a reason label (five values, pre-seeded so the zero-valued series exist immediately). Aggregates such as sum by (model_name) (rate(...)) are unchanged; a dashboard that selected the bare series now gets one series per reason and needs sum by (...) or a reason filter. No unlabeled twin is kept (no precedent in this collector; a twin double-counts in name-regex queries).

  • BufferModePipeline._log_backup_dropped(num_tokens, reason, node_id): every drop site passes its reason (source_evicted for the stale sweep, parent_uncovered/oversize at admission, parent_cascade/oversize at the queue head, backlog_cap for the leak backstop) and tallies it; once per tick (_log_pending_backup_drops, after the stale sweep so the line reports the post-sweep queue) each reason with new spans gets at most one line per 10 s interval (WARNING; ERROR for backlog_cap, replacing its count-based log) with the swept node ids, tokens since the last line, running total, queue length / backlog / backlog cap, staged tokens and live cap. A tally whose interval has elapsed is reported on the next tick even without a new drop, so a burst's tail is not held back. Volume bound: one line per reason per 10 s per rank (a TP8 deployment under a sustained deficit emits up to ~48 WARNING lines/min in aggregate). _drop_queue_head(..., reason=) replaces the dropped flag.
  • StorageMetricsCollector: sglang:hicache_backup_dropped_tokens_total gains a reason label (all five values pre-seeded, like storage_prefetch_unfulfilled_tokens_total); new gauges sglang:hicache_backup_backlog_tokens (admitted, D2H not launched), sglang:hicache_backup_staged_tokens (host staging held until the storage ack) and sglang:hicache_backup_staging_cap_tokens (the live window), published every flush_pending_writes tick so staged-at-cap shows the write path is store-paced before drops start. RayStorageMetricsCollector gets the matching _gauge_cls.
  • Tests: test/registered/unit/mem_cache/test_buffer_mode_backup_drops.py (CPU, mock pipeline: each site emits exactly one labeled increment per span, the per-reason rate limit plus interval tail flush, gauges on every flush including an empty queue, collector pre-seeds every pipeline reason); test_unified_radix_cache_unittest.py::test_buffer_only_evicted_queued_write_is_labeled_and_warned (real cache + file backend across the suite's configs: evict a node whose write is still queued, flush, assert source_evicted + WARNING and an idle pipeline).

Compatibility: aggregate queries (sum(rate(sglang:hicache_backup_dropped_tokens_total[5m]))) are unchanged; a selector on the bare series now returns one series per reason. benchmark/hicache/bench_buffer_mode.py sums by metric name and keeps working.

Accuracy Tests

Not applicable (observability only; no change to what is written or evicted). The GPU runs below verify byte-identical greedy output cold vs L3 on the verify prompts (0 mismatches).

Speed Tests and Profiling

Lane: Qwen2.5-1.5B-Instruct TP2 (B200), page 64, --max-total-tokens 65536 --hicache-ratio 2 (host 131,136 tokens, live cap 118,023), python tree core, HybridCacheController, --hicache-storage-backend file, buffer_only, write_through, prefetch policy timeout; 40 cold prompts x 6-12k tokens, flush, verify, replay with a 64-token extension; per-rank counters (ranks identical).

tree conc seed tokens dropped / rank drop % reason WARN lines / rank replay full / zero / partial seed TTFT p50 / p90 / max (s)
base 12dc623102 4 381,464 102,720 26.9% (unlabeled) 0 29 / 11 / 0 0.086 / 0.117 / 0.251
this PR 4 364,504 90,112 24.7% source_evicted 100% 1 30 / 10 / 0 0.084 / 0.107 / 0.147
base 8 373,528 81,792 21.9% (unlabeled) 0 29 / 6 / 5 0.223 / 0.271 / 0.387
this PR 8 386,008 75,712 19.6% source_evicted 100% 1 29 / 5 / 6 0.250 / 0.277 / 0.382
this PR, store throttled to 0.3 GB/s (fault-injecting HiCacheFile subclass), c8 + 256-token canaries every 0.5 s 8 357,976 160,768 44.9% source_evicted 100% 3 (first drop, then "22 span(s), 154,432 tokens since the last report" 10 s later) 22 / 16 / 2 0.213 / 0.257 / 0.339; canary TTFT p50 14 ms, p99 189 ms, 0 errors

Drop percentages sit inside the run-to-run band of the base commit (16-33% across seeds), i.e. no behavior change; the difference is the label and the log line. parent_uncovered, parent_cascade, oversize and backlog_cap stayed at 0 in every run. Gauge timeline of the 0.3 GB/s run (0.25 s sampler): staged pegged at 123-124k against the 118,023 cap from +4 s to +7 s while backlog went 42k -> 14k -> 0 as the device LRU evicted the queued spans, then staging drained to 0 over ~8 s.

Overhead: one dict lookup and an integer compare per dropped span, three gauge sets per scheduler tick when storage metrics are enabled.

Validation run:

  • pytest test/registered/unit/mem_cache/test_buffer_mode_backup_drops.py test/registered/unit/mem_cache/test_buffer_mode_sidecar.py test/registered/unit/observability/test_stat_loggers_di.py test/registered/unit/managers/test_prefill_adder.py -> 78 passed.
  • pytest test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py -k "buffer or backup or staged" (1 GPU) -> all passed; the new case runs on 13 configs (6 Mamba configs skipped by the buffer fixture).
  • pre-commit run --files <changed files> -> passed.
  • Not run: the full CI suites, AMD/NPU lanes, the Rust tree core (SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=rust) under the new regression test.

Checklist

Buffer-only write_through loses whole prompts from L3 whenever the store
drains slower than prefill produces KV: a queued backup intent is unpinned
until its D2H launches, the staging window is held until the storage ack,
and the device LRU evicts exactly the queued leaves. The stale sweep then
retired the span with nothing but an unlabeled counter increment, so the
symptom was an unexplained L3 miss rate on prompts seen seconds ago.

Every drop site now labels sglang:hicache_backup_dropped_tokens_total with
a reason (source_evicted, parent_uncovered, parent_cascade, oversize,
backlog_cap) and logs a per-reason rate-limited WARNING (ERROR for the
leak-backstop cap) naming the spans and the queue/staging state. The
pipeline also publishes the write window every tick as gauges
(hicache_backup_backlog_tokens, hicache_backup_staged_tokens,
hicache_backup_staging_cap_tokens) so drain-vs-production headroom is
visible before drops start. No behavior change.
…eue snapshot

The sweep-time line was emitted from inside the stale sweep, where the
backlog had already been released but the queue was still the pre-sweep
deque, so it mixed the two states. Drop sites now only tally; the per-tick
reporter emits each reason's line after the sweep, at most once per
interval, and includes the backlog cap the cap reason refers to.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant