Skip to content

[Feat] kv-events: Include the removal reason in BlockRemoved - #39620

Open
L4XB wants to merge 6 commits into
sgl-project:mainfrom
L4XB:fix/39604-blockremoved-reason
Open

L4XB wants to merge 6 commits into
sgl-project:mainfrom
L4XB:fix/39604-blockremoved-reason

Conversation

@L4XB

@L4XB L4XB commented Sep 15, 2026 •

Copy link
Copy Markdown

Motivation

Refs #39604. BlockRemoved says which blocks left a tier but not why, so a consumer cannot tell capacity eviction from a demotion that left the content readable in a lower tier. That is the difference between a cache miss meaning pressure and a miss that is still servable, and today it has to be inferred from timing and unrelated scheduler state.

The issue asks for the event semantics to be settled before implementation, so the first half of this description is what the call sites actually say. The vocabulary below is derived from them, not proposed from outside, and I am happy to rename any of it.

What the removal paths actually are

#40313 removed the SWA and Mamba radix caches and #40787 removed HiRadixCache while this PR was open, so after merging main there are 5 kv_events.record_remove(...) call sites, in radix_cache and unified_tree_core. Grouped by what is true of the content afterwards:

cause call sites after the event, the content is
capacity eviction radix_cache.evict, unified_tree_core._release_all_component_layers, unified_tree_core._evict_host_leaf gone
demotion unified_tree_core._demote on host
duplicate reclaim unified_tree_core._release_full_host_duplicate still on device

Three of the five collapse to one reason. The Rust tree core, the default backend, has the same paths and emits the same reasons: release_all_component_layers_ and evict_host_leaf_ (evicted), demote_ (demoted), release_coexisting_host_value_ (duplicate reclaim).

Two of the four candidate values in the issue have no emitting call site. request_finished and aborted do not appear, because finishing or aborting a request only drops a lock reference; the blocks stay in the tree until one of the causes above removes them. Emitting them would mean inventing removals that do not happen, so this PR leaves them out. AllBlocksCleared stays the representation of an explicit global clear, as the issue suggests.

Compatibility

KVCacheEvent is a tagged msgpack map with omit_defaults=True. Only EventBatch is array_like, and it is untouched. So an optional field left unset is absent from the payload, and a consumer that does not know the key never sees it. BlockStored already gained cache_salt and session_id this way.

The in-repo Rust consumer confirms it rather than assumes it. sgl-kv-indexer's bridge decodes BlockRemoved by looking up the keys it routes on and ignores the rest, and the KvCacheEvent doc comment in the router's wire module already says exactly that about BlockStored: "Keys the gateway does not route on (cache_salt, session_id) are ignored." Nothing in the KV-event path uses deny_unknown_fields; the only uses of it in the router are on config and tokenize types. No Rust change is needed, and none is included.

Modifications

class BlockRemoved(KVCacheEvent):
    block_hashes: list[int]
    medium: Optional[str] = None
    reason: Optional[str] = None
  • RemovalReason(str, enum.Enum) next to StorageMedium, mirroring how medium is already typed: a documented vocabulary, carried on the wire as a plain string. Values: evicted, demoted, duplicate_reclaimed.
  • The wire default is None, not evicted. "Not stated" and "evicted" are different claims, and an unset field keeps today's byte shape.
  • KVCacheEventRecorder.record_remove takes reason as a keyword defaulting to RemovalReason.EVICTED, the way medium already defaults to StorageMedium.GPU. Only the three non-eviction sites pass anything, which keeps the diff to the paths whose meaning actually differs.
  • The coalescing predicate now compares reason as well as medium (point 4 of the issue). Merging removals with different causes would report demoted blocks as evicted.
  • Rust tree core: KvCacheEvent::BlockRemoved carries a RemovalReason (as_str() gives the Python values), set at the four record_remove_event_ sites. enqueue_kv_event_ coalesces removals only when the reason matches as well. The binding appends the reason to the block_removed tuple, so the existing indices stay where they are, and rust_tree_core/adapter.py passes it to BlockRemoved.

Deliberately not included: cache_salt on BlockRemoved (question 3 in the issue). It is a separate change with its own symmetry argument, and folding it in here would blur what this PR is asking to agree.

Accuracy Tests

Not applicable; no model outputs or kernels are touched.

Speed Tests and Profiling

Not applicable. The added work per removal is one keyword argument and one extra == in the coalescing predicate.

Unit tests

Extended the two files that already own this surface, test/registered/unit/disaggregation/test_kv_events.py (wire contract) and test/registered/unit/mem_cache/test_radix_cache_unit.py (recorder and coalescing).

test what it stops
an old consumer decodes an event carrying each reason the field breaking existing subscribers
a new consumer reads an event emitted without one, and sees None a missing reason being read as evicted
omitting the reason leaves exactly {type, block_hashes, medium} the wire shape changing for emitters that state nothing
reason is a named field, and survives a mixed batch the value being dropped in transit
removals with different reasons do not coalesce the merge that would relabel demotions as evictions
removals with the same reason still coalesce the guard defeating the batching it sits in
record_remove defaults to evicted and carries an explicit reason the recorder silently dropping the argument

Run on CPU, no accelerator:

test/registered/unit/disaggregation/test_kv_events.py              22 passed, 25 subtests passed
test/registered/unit/mem_cache/test_radix_cache_unit.py            5 passed (TestKVCacheEventQueue)

On current main, test_kv_events.py is 17 passed / 22 subtests (numbers after merging main, which added TestReplay).

Rust tree core (cargo test --no-default-features for sglang-radix-tree, as in pr-test-rust-workspace.yml): 945 passed, 1 ignored. New: demotion_emits_a_demoted_gpu_block_removed, reclaiming_a_host_duplicate_emits_a_duplicate_reclaimed_cpu_block_removed and block_removed_events_with_different_reasons_stay_separate; all three fail when the sites pass Evicted again and the coalescing ignores the reason. The existing event tests now state RemovalReason::Evicted for their eviction paths. In test_rust_tree_core_integration.py the demote expectation now has reason=RemovalReason.DEMOTED and the two eviction expectations RemovalReason.EVICTED; that suite needs the built extension, which torch_build.py only supports on Linux, so CI runs it and I could not here.

Mutation results, 5 of 5 caught:

mutation result
coalescing ignores the reason again caught
the recorder stops forwarding the reason caught
the recorder default flips to demoted caught
the wire field defaults to "evicted" instead of unset caught
demoted and evicted collapse to one value caught

The rest of test_radix_cache_unit.py fails the same 9 tests on main and on this branch, all with HiCache native hash is only supported on little-endian Linux. The unified and rust-core suites had the same pass, fail and skip counts on both sides when I last compared them. Their failures are environmental on this machine (the native hash, and CUDA-only paths); I have no GPU here, so the cache-level tests that would exercise the two named call sites end to end are skipped rather than passing. Those two classifications come from reading the call sites, and are the part of this PR most worth a maintainer's eye.

Checklist

  • Format your code according to Format code with pre-commit.
  • Add unit tests according to Run and add unit tests.
  • Update documentation according to Write documentations.
  • Provide accuracy and speed benchmark results.
  • Follow the SGLang code style guidance.

Documentation is unticked because I could not find an existing consumer-facing page describing the KV event schema; the vocabulary is documented on RemovalReason itself. Point me at the right page and I will add it.

File and symbol references are against main @ 47a157f257f13311a4408dbde3506f6a9cedaa74.


CI States

Latest PR Test (Base): ❌ Run #40150885933
Latest PR Test (Extra): ❌ Run #40150885867
Latest PR Test (AMD ROCm 10): ❌ Run #40150885861

BlockRemoved says which blocks left a tier but not why, so a consumer
cannot tell capacity eviction from a demotion that left the content
readable in a lower tier -- the difference between a cache miss that
means pressure and one that is still servable.

The field is optional and omitted when unset, so an emitter that states
no reason produces exactly the map consumers read today. The coalescing
predicate now includes it: merging removals with different causes would
report demoted blocks as evicted.
Of the removal call sites, all but three are capacity eviction and take
the default. These three do not: _detach_backuped and _demote free a
device copy that stays readable on host, and _release_full_host_duplicate
reclaims a host copy while the device copy remains.
Covers both directions of compatibility, the coalescing boundary the
field exists to protect, and that an unset reason stays off the wire.
HiRadixCache was removed in sgl-project#40787, so its removal-reason call sites go
with it. test_kv_events.py keeps both the new TestReplay class from main
and TestBlockRemovedReason.
@L4XB

L4XB commented Oct 5, 2026

Copy link
Copy Markdown
Author

Merged main. HiRadixCache (#40787) and the SWA and Mamba radix caches (#40313) were removed in the meantime, so their call sites leave this PR with them; test_kv_events.py keeps both TestReplay from main and the removal-reason tests (22 passed, 25 subtests on CPU). The call-site table in the description is updated: 5 sites remain, in radix_cache and unified_tree_core.

@sohom-cs sohom-cs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason is only set in the Python tree core. On main, with the default tree core (SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND is rust; Python is the fallback), a removal goes through record_remove_event_ (rust/sglang-radix-tree/src/unified_tree_core.rs:3713), which carries the hashes and the medium only, and the adapter turns that tuple into BlockRemoved(block_hashes=..., medium=...) (python/sglang/srt/mem_cache/rust_tree_core/adapter.py:145). So on that path every event has reason=None, including the Rust counterparts of the two paths labelled here, demote_ and release_coexisting_host_value_. The new tests exercise the recorder and the wire format, not either core's eviction paths.

Should the reason go through the Rust KvCacheEvent::BlockRemoved and the binding tuple in this PR, or should the description name the Rust core as a follow-up?

L4XB added 2 commits October 25, 2026 02:18
The Rust core is the default tree core backend, and its BlockRemoved had no reason, so every event from it reached Python with reason=None. KvCacheEvent::BlockRemoved now carries a RemovalReason (evicted, demoted, duplicate_reclaimed) set at the four record_remove_event_ sites like their Python counterparts, coalescing keeps removals with different reasons apart, the binding appends the reason to the block_removed tuple, and the adapter passes it on.
@L4XB
L4XB requested a review from Jialin as a code owner October 25, 2026 01:30
@L4XB

L4XB commented Oct 25, 2026

Copy link
Copy Markdown
Author

@sohom-cs Good catch, thank you. With the Rust core as the default, leaving it out would have made the reason None for almost everyone, so I carried it through in this PR (9cd5cb6, after merging main):

  • KvCacheEvent::BlockRemoved has a reason: RemovalReason (evicted, demoted, duplicate_reclaimed, same strings as the Python enum).
  • The four record_remove_event_ sites pass it the way their Python counterparts do: release_all_component_layers_ and evict_host_leaf_ evicted, demote_ demoted, release_coexisting_host_value_ duplicate reclaimed.
  • enqueue_kv_event_ only coalesces removals whose reason matches too.
  • The binding appends the reason to the block_removed tuple (the existing indices stay put), and adapter.py builds BlockRemoved(..., reason=RemovalReason(event[4])).

cargo test --no-default-features for sglang-radix-tree: 945 passed, 1 ignored, including three new tests (demotion, duplicate reclaim, no coalescing across reasons) that fail without the change. I updated the three BlockRemoved expectations in test_rust_tree_core_integration.py (the demote one to DEMOTED, the two evictions to EVICTED). That suite needs the built extension, which torch_build.py only supports on Linux, so it will run in CI. The description is updated as well.

@github-actions

This comment has been minimized.

@L4XB L4XB changed the title [Feature] [KV events] Include the removal reason in BlockRemoved [Feat] kv-events: Include the removal reason in BlockRemoved Oct 25, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants