Skip to content
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,5 +39,7 @@ jobs:
esac
test -n "$requirement"
uv pip install --python .venv/bin/python "$requirement" --upgrade-package tinker
# The router extension runs only in the Modal image; its tests need it locally.
- run: uv pip install --python .venv/bin/python ./rust/router
- run: uv run --no-sync pytest -q
- run: uv build
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ build/
dist/
.env
scripts/results/
*.so
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Notable changes to Spindle, following [Keep a Changelog](https://keepachangelog.
### Added

- Qwen3.8-27B LoRA recipe with B200 samplers, `qwen38_27b_lora_16k_b200`, which serves with the TensorRT-LLM attention backend and 64-token pages (#145).
- LoRA samplers can route each request to the replica with its longest cached prefix with `inference_lora_routing = "cache_wait"`, while keeping request load and pinned KV memory within bounds. The router runs in Rust inside each sampling container. It needs `inference_routing_prefill_tokens_per_second`, and `inference_routing_balance_absolute`, `inference_routing_balance_relative` and `inference_routing_cache_threshold` tune it. Pools without routable replicas, including pools scaled to zero, fall back to Modal's gateway routing. Modal session affinity remains the default (#121, #122, #123).

## 0.1.2 - 2026-10-27

Expand Down
2 changes: 2 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
include rust/router/Cargo.toml rust/router/Cargo.lock rust/router/pyproject.toml
recursive-include rust/router/src *.rs
9 changes: 1 addition & 8 deletions examples/dapo-27b-4h200-12b200/deploy.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,19 +38,12 @@ class Config(Parent):
# Blog routing policy. The GPU split and sglang_cfg limits are what deploy
# applies; cache_wait is the policy used for that run.
inference_lora_routing = "cache_wait"
inference_router_runtime = "rust"
inference_routing_balance_absolute = 32
inference_routing_balance_relative = 2.0
inference_routing_cache_threshold = 0.5
inference_routing_work_rates = {
"prefill_tokens_per_second": 10463.96139191663,
"decode_tokens_per_second": 1290.4652700698725,
"gpu_lora_seconds": 0.07488066750002531,
"registration_seconds": 0.6767439374999924,
}
inference_routing_prefill_tokens_per_second = 10463.96139191663
inference_env = {
"SGLANG_ENABLE_LORA_CSGMV_SPLIT_K": "1",
"SGLANG_SPINDLE_ROUTING_OBSERVATIONS": "1",
"SGLANG_SPINDLE_MODEL_ROUTING": "1",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: nothing reads SGLANG_SPINDLE_MODEL_ROUTING, in Spindle or in the pinned SGLang (a3e0e17), so we can drop it. The routing observer is now turned on by inference_lora_routing = "cache_wait" itself.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped in 50c689b, along with the example's SGLANG_ENABLE_LORA_CSGMV_SPLIT_K: split-K is on by default for LoRA pools now, and it never fires on the 27B's 5,120-wide layers anyway.

}
session_idle_timeout_s = 43200
Expand Down
4 changes: 4 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ where = ["src"]
[tool.pytest.ini_options]
testpaths = ["tests"]

[tool.setuptools.data-files]
"share/spindle/router" = ["rust/router/Cargo.toml", "rust/router/Cargo.lock", "rust/router/pyproject.toml"]
"share/spindle/router/src" = ["rust/router/src/lib.rs", "rust/router/src/native.rs", "rust/router/src/cache_wait.rs"]

[tool.ruff]
extend-exclude = ["*.md", "src/spindle/proto/*_pb2.py"]

Expand Down
Loading
Loading