Repository navigation
Conversation
BlockRemoved says which blocks left a tier but not why, so a consumer cannot tell capacity eviction from a demotion that left the content readable in a lower tier -- the difference between a cache miss that means pressure and one that is still servable. The field is optional and omitted when unset, so an emitter that states no reason produces exactly the map consumers read today. The coalescing predicate now includes it: merging removals with different causes would report demoted blocks as evicted.
Of the removal call sites, all but three are capacity eviction and take the default. These three do not: _detach_backuped and _demote free a device copy that stays readable on host, and _release_full_host_duplicate reclaims a host copy while the device copy remains.
Covers both directions of compatibility, the coalescing boundary the field exists to protect, and that an unset reason stays off the wire.
HiRadixCache was removed in sgl-project#40787, so its removal-reason call sites go with it. test_kv_events.py keeps both the new TestReplay class from main and TestBlockRemovedReason.
|
Merged |
sohom-cs
left a comment
There was a problem hiding this comment.
The reason is only set in the Python tree core. On main, with the default tree core (SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND is rust; Python is the fallback), a removal goes through record_remove_event_ (rust/sglang-radix-tree/src/unified_tree_core.rs:3713), which carries the hashes and the medium only, and the adapter turns that tuple into BlockRemoved(block_hashes=..., medium=...) (python/sglang/srt/mem_cache/rust_tree_core/adapter.py:145). So on that path every event has reason=None, including the Rust counterparts of the two paths labelled here, demote_ and release_coexisting_host_value_. The new tests exercise the recorder and the wire format, not either core's eviction paths.
Should the reason go through the Rust KvCacheEvent::BlockRemoved and the binding tuple in this PR, or should the description name the Rust core as a follow-up?
The Rust core is the default tree core backend, and its BlockRemoved had no reason, so every event from it reached Python with reason=None. KvCacheEvent::BlockRemoved now carries a RemovalReason (evicted, demoted, duplicate_reclaimed) set at the four record_remove_event_ sites like their Python counterparts, coalescing keeps removals with different reasons apart, the binding appends the reason to the block_removed tuple, and the adapter passes it on.
|
@sohom-cs Good catch, thank you. With the Rust core as the default, leaving it out would have made the reason
|
This comment has been minimized.
This comment has been minimized.
BlockRemoved
Motivation
Refs #39604.
BlockRemovedsays which blocks left a tier but not why, so a consumer cannot tell capacity eviction from a demotion that left the content readable in a lower tier. That is the difference between a cache miss meaning pressure and a miss that is still servable, and today it has to be inferred from timing and unrelated scheduler state.The issue asks for the event semantics to be settled before implementation, so the first half of this description is what the call sites actually say. The vocabulary below is derived from them, not proposed from outside, and I am happy to rename any of it.
What the removal paths actually are
#40313 removed the SWA and Mamba radix caches and #40787 removed HiRadixCache while this PR was open, so after merging
mainthere are 5kv_events.record_remove(...)call sites, inradix_cacheandunified_tree_core. Grouped by what is true of the content afterwards:radix_cache.evict,unified_tree_core._release_all_component_layers,unified_tree_core._evict_host_leafunified_tree_core._demoteunified_tree_core._release_full_host_duplicateThree of the five collapse to one reason. The Rust tree core, the default backend, has the same paths and emits the same reasons:
release_all_component_layers_andevict_host_leaf_(evicted),demote_(demoted),release_coexisting_host_value_(duplicate reclaim).Two of the four candidate values in the issue have no emitting call site.
request_finishedandaborteddo not appear, because finishing or aborting a request only drops a lock reference; the blocks stay in the tree until one of the causes above removes them. Emitting them would mean inventing removals that do not happen, so this PR leaves them out.AllBlocksClearedstays the representation of an explicit global clear, as the issue suggests.Compatibility
KVCacheEventis a tagged msgpack map withomit_defaults=True. OnlyEventBatchisarray_like, and it is untouched. So an optional field left unset is absent from the payload, and a consumer that does not know the key never sees it.BlockStoredalready gainedcache_saltandsession_idthis way.The in-repo Rust consumer confirms it rather than assumes it.
sgl-kv-indexer's bridge decodesBlockRemovedby looking up the keys it routes on and ignores the rest, and theKvCacheEventdoc comment in the router'swiremodule already says exactly that aboutBlockStored: "Keys the gateway does not route on (cache_salt,session_id) are ignored." Nothing in the KV-event path usesdeny_unknown_fields; the only uses of it in the router are on config and tokenize types. No Rust change is needed, and none is included.Modifications
RemovalReason(str, enum.Enum)next toStorageMedium, mirroring howmediumis already typed: a documented vocabulary, carried on the wire as a plain string. Values:evicted,demoted,duplicate_reclaimed.None, notevicted. "Not stated" and "evicted" are different claims, and an unset field keeps today's byte shape.KVCacheEventRecorder.record_removetakesreasonas a keyword defaulting toRemovalReason.EVICTED, the waymediumalready defaults toStorageMedium.GPU. Only the three non-eviction sites pass anything, which keeps the diff to the paths whose meaning actually differs.reasonas well asmedium(point 4 of the issue). Merging removals with different causes would report demoted blocks as evicted.KvCacheEvent::BlockRemovedcarries aRemovalReason(as_str()gives the Python values), set at the fourrecord_remove_event_sites.enqueue_kv_event_coalesces removals only when the reason matches as well. The binding appends the reason to theblock_removedtuple, so the existing indices stay where they are, andrust_tree_core/adapter.pypasses it toBlockRemoved.Deliberately not included:
cache_saltonBlockRemoved(question 3 in the issue). It is a separate change with its own symmetry argument, and folding it in here would blur what this PR is asking to agree.Accuracy Tests
Not applicable; no model outputs or kernels are touched.
Speed Tests and Profiling
Not applicable. The added work per removal is one keyword argument and one extra
==in the coalescing predicate.Unit tests
Extended the two files that already own this surface,
test/registered/unit/disaggregation/test_kv_events.py(wire contract) andtest/registered/unit/mem_cache/test_radix_cache_unit.py(recorder and coalescing).Noneevicted{type, block_hashes, medium}reasonis a named field, and survives a mixed batchrecord_removedefaults toevictedand carries an explicit reasonRun on CPU, no accelerator:
On current
main,test_kv_events.pyis 17 passed / 22 subtests (numbers after mergingmain, which addedTestReplay).Rust tree core (
cargo test --no-default-featuresforsglang-radix-tree, as inpr-test-rust-workspace.yml): 945 passed, 1 ignored. New:demotion_emits_a_demoted_gpu_block_removed,reclaiming_a_host_duplicate_emits_a_duplicate_reclaimed_cpu_block_removedandblock_removed_events_with_different_reasons_stay_separate; all three fail when the sites passEvictedagain and the coalescing ignores the reason. The existing event tests now stateRemovalReason::Evictedfor their eviction paths. Intest_rust_tree_core_integration.pythe demote expectation now hasreason=RemovalReason.DEMOTEDand the two eviction expectationsRemovalReason.EVICTED; that suite needs the built extension, whichtorch_build.pyonly supports on Linux, so CI runs it and I could not here.Mutation results, 5 of 5 caught:
demoted"evicted"instead of unsetdemotedandevictedcollapse to one valueThe rest of
test_radix_cache_unit.pyfails the same 9 tests onmainand on this branch, all withHiCache native hash is only supported on little-endian Linux. The unified and rust-core suites had the same pass, fail and skip counts on both sides when I last compared them. Their failures are environmental on this machine (the native hash, and CUDA-only paths); I have no GPU here, so the cache-level tests that would exercise the two named call sites end to end are skipped rather than passing. Those two classifications come from reading the call sites, and are the part of this PR most worth a maintainer's eye.Checklist
Documentation is unticked because I could not find an existing consumer-facing page describing the KV event schema; the vocabulary is documented on
RemovalReasonitself. Point me at the right page and I will add it.File and symbol references are against
main@47a157f257f13311a4408dbde3506f6a9cedaa74.CI States
Latest PR Test (Base): ❌ Run #40150885933
Latest PR Test (Extra): ❌ Run #40150885867
Latest PR Test (AMD ROCm 10): ❌ Run #40150885861