Skip to content

Add incremental radix-cache observations for routing - #85

Merged
kailash109 merged 1 commit into
spindle-sglang-v0.5.21from
codex/routing-snapshot-delta
Oct 26, 2026
Merged

kailash109 merged 1 commit into
spindle-sglang-v0.5.21from
codex/routing-snapshot-delta

Conversation

@kailash109

@kailash109 kailash109 commented Oct 26, 2026 •

Copy link
Copy Markdown

Summary

  • Add routing_snapshot_delta on the unified radix tree and its Python binding. It returns residency metadata for every resident segment and token bytes only for segments missing from the caller's acknowledged inventory.
  • Call that capture from Scheduler.get_next_batch_to_run and publish the result from a separate CPU process. The scheduler does not wait on hashing or file I/O, and the capture does not mutate the tree.
  • This is the SGLang half of spindle#121. Spindle can check out this commit instead of string-patching the engine during the image build.

The branch is based on 79e38a5605a576cae7034d9ab70047fe478e26ee, the revision Spindle pins. spindle-sglang-v0.5.21 is 10 commits ahead of that pin; the scheduler and radix-tree files this change edits are identical at both revisions, so those commits are not part of the pinned tree.

Observations stay off unless SGLANG_SPINDLE_ROUTING_OBSERVATIONS=1.

Test plan

  • Confirmed the landed Rust method, binding, scheduler hook, and observer module match the spindle image installer (to_le_bytes token packing, get_parallel() rank guard, one in-flight delta).
  • Existing CPU validation on this same pin, app ap-UuO93Lc7Auu8r78eh5gQLc: 12 exporter tests passed, including splits, eviction, reset, stale publication suppression, and publisher restart. Unchanged-cache capture at 8,388,608 tokens dropped from 591 ms on the scheduler to 0.41 ms. No end-to-end model throughput claim.
  • cargo check --features python-extension was not rerun here; local libtorch is 2.9/2.10 and this crate expects PyTorch 2.11.

Made with Cursor


CI States

Latest PR Test (Base): ❌ Run #40384562250
Latest PR Test (Extra): ❌ Run #40384562149
Latest PR Test (AMD ROCm 10): ❌ Run #40384562185

Export residency metadata and token bytes only for segments the caller has not acknowledged, and let the scheduler hand that delta to a CPU publisher without touching cache state.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

Copy link
Copy Markdown

Required checks failed on 12256ca; this PR cannot be merged until they pass.

PR title (run)

PR title must use "[Type] scope: title" or "[Type] [Hardware] scope: title". Type must be one of: Feat, Fix, Refactor, Perf, Doc, Test, CI, Chore. Scope is required and may be any module or model name, with unrestricted letter case (letters, numbers, dots, underscores, slashes, or hyphens). Hardware is optional; use a single tag without spaces. Use uppercase hardware tags (AMD, NVIDIA, NPU, MI300X), except for exactly these four spellings: Intel, Apple, Huawei, Ascend. Reverts must use GitHub's default format: Revert "original title". Use one space between fields and after the colon, with a non-empty title and no leading or trailing whitespace. Examples: "[Fix] attention: Fix KV cache indexing", "[Feat] qwen3: Add model support", "[Perf] [AMD] flash-attn: Optimize decode performance".
Edit the title; the check reruns on its own.

Lint (run)
Run pre-commit run --all-files locally and push the result. The run log shows any other failing step.

@kailash109
kailash109 merged commit b6a84dd into spindle-sglang-v0.5.21 Oct 26, 2026
68 of 79 checks passed
@micahtyong

Copy link
Copy Markdown

This merge affects Spindle main now, not only the routing stack.

What breaks

Spindle's rollout image (src/spindle/providers/modal/rollout_image.py) clones the head of spindle-sglang-v0.5.21 with no commit pin. It copies only python/ over the base image. So any fresh image build now picks up spindle_routing_observer.py and the new scheduler hook. The Rust binding is still the one from lmsysorg/sglang:v0.5.21, though, and that build has no routing_snapshot_delta.

With SGLANG_SPINDLE_ROUTING_OBSERVATIONS=1 and LoRA on, the default cache (UnifiedRadixCache with the Rust tree core) takes the incremental path. About a second after startup, snapshot() calls cache.tree_core._binding.routing_snapshot_delta(...) inside get_next_batch_to_run. That raises AttributeError and the scheduler exits.

Spindle main already sets that env var in examples/dapo-27b-4h200-12b200/deploy.py, so that example fails on its next uncached image build. Deployments without the env var are not affected, because snapshot() returns early.

Suggested fixes

  1. In Spindle, pin the clone to a commit and copy rust/ as well, the way [routing 4/4] Integrate native routing and route LoRA pools with cache_wait by default modal-labs/spindle#123 does. This should land on its own, ahead of the rest of the routing stack.
  2. Here, make the observer fail safe. Catch exceptions in snapshot(), log once, and turn it off. Observations should never be able to stop the scheduler. Checking hasattr(binding, "routing_snapshot_delta") before the incremental path would also cover a stale binary.

Smaller items

  • The new method was inserted below root_node_handle's doc comment ("The NodeId anchoring matches…"). Rustdoc now attaches that line to routing_snapshot_delta, and root_node_handle loses it.
  • The Python-tree fallback (capture_python_cache) copies every token on the scheduler thread once a second and ignores the acknowledged inventory. That is the 591 ms case at 8M tokens. Consider dropping it, or only enabling observations when the Rust core is in use.

@micahtyong

Copy link
Copy Markdown

A follow-up on where this export could go long term. Upstream SGLang now publishes most of what this patch adds, through its KV-event stream.

Already upstream, in v0.5.22

The branch here is based on v0.5.21, so it has none of this. modal-labs/spindle#132 bumps the rollout image to v0.5.22.

Suggested direction after that bump

The sidecar could subscribe to the engine's KV events and load publisher over localhost, in the same container. It would build the prefix set and load fields from those and keep serving /routing-state in the same format. Then the router doesn't change, and this branch drops the scheduler hook, the observer process, and routing_snapshot_delta. That removes a fork patch we would otherwise have to port on every SGLang upgrade.

The gap that remains

KV events report only KV pages. For hybrid models with linear-attention or Mamba layers, a cached prefix is only reusable where the recurrent state was also kept, and events don't say where that is. This export handles it with the per-node mamba flag. I didn't find upstream work on this. A small upstream PR to add state availability to the events, or a flag on BlockStored, would close it. Until then, this export could stay as a fallback for hybrid models only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants