Repository navigation
Add incremental radix-cache observations for routing - #85
Conversation
Export residency metadata and token bytes only for segments the caller has not acknowledged, and let the scheduler hand that delta to a CPU publisher without touching cache state. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Required checks failed on 12256ca; this PR cannot be merged until they pass. PR title (run)
Lint (run) |
|
This merge affects Spindle What breaks Spindle's rollout image ( With Spindle Suggested fixes
Smaller items
|
|
A follow-up on where this export could go long term. Upstream SGLang now publishes most of what this patch adds, through its KV-event stream. Already upstream, in v0.5.22
The branch here is based on v0.5.21, so it has none of this. modal-labs/spindle#132 bumps the rollout image to v0.5.22. Suggested direction after that bump The sidecar could subscribe to the engine's KV events and load publisher over localhost, in the same container. It would build the prefix set and load fields from those and keep serving The gap that remains KV events report only KV pages. For hybrid models with linear-attention or Mamba layers, a cached prefix is only reusable where the recurrent state was also kept, and events don't say where that is. This export handles it with the per-node |
Summary
routing_snapshot_deltaon the unified radix tree and its Python binding. It returns residency metadata for every resident segment and token bytes only for segments missing from the caller's acknowledged inventory.Scheduler.get_next_batch_to_runand publish the result from a separate CPU process. The scheduler does not wait on hashing or file I/O, and the capture does not mutate the tree.The branch is based on
79e38a5605a576cae7034d9ab70047fe478e26ee, the revision Spindle pins.spindle-sglang-v0.5.21is 10 commits ahead of that pin; the scheduler and radix-tree files this change edits are identical at both revisions, so those commits are not part of the pinned tree.Observations stay off unless
SGLANG_SPINDLE_ROUTING_OBSERVATIONS=1.Test plan
to_le_bytestoken packing,get_parallel()rank guard, one in-flight delta).cargo check --features python-extensionwas not rerun here; local libtorch is 2.9/2.10 and this crate expects PyTorch 2.11.Made with Cursor
CI States
Latest PR Test (Base): ❌ Run #40384562250
Latest PR Test (Extra): ❌ Run #40384562149
Latest PR Test (AMD ROCm 10): ❌ Run #40384562185