Repository navigation
[Fix] hicache: Log and label buffer-only backup drops - #44339
Draft
xiezhq-hermann wants to merge 2 commits into
Draft
xiezhq-hermann wants to merge 2 commits into
xiezhq-hermann wants to merge 2 commits into
Conversation
Buffer-only write_through loses whole prompts from L3 whenever the store drains slower than prefill produces KV: a queued backup intent is unpinned until its D2H launches, the staging window is held until the storage ack, and the device LRU evicts exactly the queued leaves. The stale sweep then retired the span with nothing but an unlabeled counter increment, so the symptom was an unexplained L3 miss rate on prompts seen seconds ago. Every drop site now labels sglang:hicache_backup_dropped_tokens_total with a reason (source_evicted, parent_uncovered, parent_cascade, oversize, backlog_cap) and logs a per-reason rate-limited WARNING (ERROR for the leak-backstop cap) naming the spans and the queue/staging state. The pipeline also publishes the write window every tick as gauges (hicache_backup_backlog_tokens, hicache_backup_staged_tokens, hicache_backup_staging_cap_tokens) so drain-vs-production headroom is visible before drops start. No behavior change.
5 tasks done
…eue snapshot The sweep-time line was emitted from inside the stale sweep, where the backlog had already been released but the queue was still the pre-sweep deque, so it mixed the two states. Drop sites now only tally; the per-tick reporter emits each reason's line after the sweep, at most once per interval, and includes the backlog cap the cap reason refers to.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
With
--hicache-host-memory-mode buffer_onlyandwrite_through, a store that drains slower than prefill produces KV silently loses whole prompts from L3. Mechanism (python/sglang/srt/mem_cache/buffer_mode/pipeline.py): a finished node's backup intent is queued unpinned (the device lock is only taken at the D2H launch in_stage_backup_intent), host staging is held until the storage ack (finish_storage_write_ack), andflush_pending_writesstops launching oncewrite_staged_tokens_ >= live_cap. The FULL eviction walk does not consult the queue, so a device pool turning over faster than the window drains evicts exactly the queued leaves; the next_refresh_pending_backup_intentssweeps the span and the only trace was an unlabeled increment ofsglang:hicache_backup_dropped_tokens_total. On a TP2 lane with the stockfilebackend (~1.6 GB/s/rank single-threaded writes vs ~4.3 GB/s of prefill KV) this loses 20-28% of a 40-prompt cold seed, always whole prompts, with zero log lines; the user-visible symptom is an unexplained L3 miss rate on prefixes seen seconds ago. The same pipeline runs on every tree core and backend.This PR makes the loss visible and attributable. It does not change what is dropped (the follow-up PR adds an opt-in throttle).
Modifications
Metric label change:
sglang:hicache_backup_dropped_tokens_totalgains areasonlabel (five values, pre-seeded so the zero-valued series exist immediately). Aggregates such assum by (model_name) (rate(...))are unchanged; a dashboard that selected the bare series now gets one series per reason and needssum by (...)or areasonfilter. No unlabeled twin is kept (no precedent in this collector; a twin double-counts in name-regex queries).BufferModePipeline._log_backup_dropped(num_tokens, reason, node_id): every drop site passes its reason (source_evictedfor the stale sweep,parent_uncovered/oversizeat admission,parent_cascade/oversizeat the queue head,backlog_capfor the leak backstop) and tallies it; once per tick (_log_pending_backup_drops, after the stale sweep so the line reports the post-sweep queue) each reason with new spans gets at most one line per 10 s interval (WARNING; ERROR forbacklog_cap, replacing its count-based log) with the swept node ids, tokens since the last line, running total, queue length / backlog / backlog cap, staged tokens and live cap. A tally whose interval has elapsed is reported on the next tick even without a new drop, so a burst's tail is not held back. Volume bound: one line per reason per 10 s per rank (a TP8 deployment under a sustained deficit emits up to ~48 WARNING lines/min in aggregate)._drop_queue_head(..., reason=)replaces thedroppedflag.StorageMetricsCollector:sglang:hicache_backup_dropped_tokens_totalgains areasonlabel (all five values pre-seeded, likestorage_prefetch_unfulfilled_tokens_total); new gaugessglang:hicache_backup_backlog_tokens(admitted, D2H not launched),sglang:hicache_backup_staged_tokens(host staging held until the storage ack) andsglang:hicache_backup_staging_cap_tokens(the live window), published everyflush_pending_writestick so staged-at-cap shows the write path is store-paced before drops start.RayStorageMetricsCollectorgets the matching_gauge_cls.test/registered/unit/mem_cache/test_buffer_mode_backup_drops.py(CPU, mock pipeline: each site emits exactly one labeled increment per span, the per-reason rate limit plus interval tail flush, gauges on every flush including an empty queue, collector pre-seeds every pipeline reason);test_unified_radix_cache_unittest.py::test_buffer_only_evicted_queued_write_is_labeled_and_warned(real cache + file backend across the suite's configs: evict a node whose write is still queued, flush, assertsource_evicted+ WARNING and an idle pipeline).Compatibility: aggregate queries (
sum(rate(sglang:hicache_backup_dropped_tokens_total[5m]))) are unchanged; a selector on the bare series now returns one series perreason.benchmark/hicache/bench_buffer_mode.pysums by metric name and keeps working.Accuracy Tests
Not applicable (observability only; no change to what is written or evicted). The GPU runs below verify byte-identical greedy output cold vs L3 on the verify prompts (0 mismatches).
Speed Tests and Profiling
Lane: Qwen2.5-1.5B-Instruct TP2 (B200), page 64,
--max-total-tokens 65536 --hicache-ratio 2(host 131,136 tokens, live cap 118,023), python tree core,HybridCacheController,--hicache-storage-backend file,buffer_only,write_through, prefetch policytimeout; 40 cold prompts x 6-12k tokens, flush, verify, replay with a 64-token extension; per-rank counters (ranks identical).reason12dc623102source_evicted100%source_evicted100%HiCacheFilesubclass), c8 + 256-token canaries every 0.5 ssource_evicted100%Drop percentages sit inside the run-to-run band of the base commit (16-33% across seeds), i.e. no behavior change; the difference is the label and the log line.
parent_uncovered,parent_cascade,oversizeandbacklog_capstayed at 0 in every run. Gauge timeline of the 0.3 GB/s run (0.25 s sampler):stagedpegged at 123-124k against the 118,023 cap from +4 s to +7 s whilebacklogwent 42k -> 14k -> 0 as the device LRU evicted the queued spans, then staging drained to 0 over ~8 s.Overhead: one dict lookup and an integer compare per dropped span, three gauge sets per scheduler tick when storage metrics are enabled.
Validation run:
pytest test/registered/unit/mem_cache/test_buffer_mode_backup_drops.py test/registered/unit/mem_cache/test_buffer_mode_sidecar.py test/registered/unit/observability/test_stat_loggers_di.py test/registered/unit/managers/test_prefill_adder.py-> 78 passed.pytest test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py -k "buffer or backup or staged"(1 GPU) -> all passed; the new case runs on 13 configs (6 Mamba configs skipped by the buffer fixture).pre-commit run --files <changed files>-> passed.SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=rust) under the new regression test.Checklist