Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,5 +39,7 @@ jobs:
esac
test -n "$requirement"
uv pip install --python .venv/bin/python "$requirement" --upgrade-package tinker
# The router extension runs only in the Modal image; its tests need it locally.
- run: uv pip install --python .venv/bin/python ./rust/router
- run: uv run --no-sync pytest -q
- run: uv build
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ build/
dist/
.env
scripts/results/
*.so
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ Notable changes to Spindle, following [Keep a Changelog](https://keepachangelog.
### Added

- Qwen3.8-27B LoRA recipe with B200 samplers, `qwen38_27b_lora_16k_b200`, which serves with the TensorRT-LLM attention backend and 64-token pages (#145).
- LoRA samplers can route each request to the replica with its longest cached prefix with `inference_lora_routing = "cache_wait"`, while keeping request load and pinned KV memory within bounds. The router runs in Rust inside each sampling container. It measures the pool's prefill rate from completed requests unless `inference_routing_prefill_tokens_per_second` sets it, and `inference_routing_balance_absolute`, `inference_routing_balance_relative` and `inference_routing_cache_threshold` tune it. Pools without routable replicas, including pools scaled to zero and pools whose SGLang cache publishes no observations, fall back to Modal's gateway routing (#121, #122, #123).

### Changed

- LoRA recipes use `cache_wait` routing by default when SGLang runs one data-parallel rank, and Modal session affinity otherwise. `inference_lora_routing = "affinity"` keeps session affinity. `inference_routing_balance_absolute` defaults to 40% of `max_running_requests`, and at least 4 (#123).
- LoRA samplers enable SGLang's CSGMV split-K kernel. For rank-32 adapters on models with 4,096-wide hidden layers, such as Qwen3.5-9B and Qwen3-8B, it speeds up the LoRA kernels for decode-sized batches by up to 2.9× on H200 and 2.6× on B200. Set `SGLANG_ENABLE_LORA_CSGMV_SPLIT_K=0` in `inference_env` to disable it (#123).

## 0.1.2 - 2026-10-27

Expand Down
2 changes: 2 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
include rust/router/Cargo.toml rust/router/Cargo.lock rust/router/pyproject.toml
recursive-include rust/router/src *.rs
13 changes: 1 addition & 12 deletions examples/dapo-27b-4h200-12b200/deploy.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,21 +38,10 @@ class Config(Parent):
# Blog routing policy. The GPU split and sglang_cfg limits are what deploy
# applies; cache_wait is the policy used for that run.
inference_lora_routing = "cache_wait"
inference_router_runtime = "rust"
inference_routing_balance_absolute = 32
inference_routing_balance_relative = 2.0
inference_routing_cache_threshold = 0.5
inference_routing_work_rates = {
"prefill_tokens_per_second": 10463.96139191663,
"decode_tokens_per_second": 1290.4652700698725,
"gpu_lora_seconds": 0.07488066750002531,
"registration_seconds": 0.6767439374999924,
}
inference_env = {
"SGLANG_ENABLE_LORA_CSGMV_SPLIT_K": "1",
"SGLANG_SPINDLE_ROUTING_OBSERVATIONS": "1",
"SGLANG_SPINDLE_MODEL_ROUTING": "1",
}
inference_routing_prefill_tokens_per_second = 10463.96139191663
session_idle_timeout_s = 43200
pool_idle_timeout_s = 43200
overrides = {
Expand Down
4 changes: 4 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ where = ["src"]
[tool.pytest.ini_options]
testpaths = ["tests"]

[tool.setuptools.data-files]
"share/spindle/router" = ["rust/router/Cargo.toml", "rust/router/Cargo.lock", "rust/router/pyproject.toml"]
"share/spindle/router/src" = ["rust/router/src/lib.rs", "rust/router/src/native.rs", "rust/router/src/cache_wait.rs"]

[tool.ruff]
extend-exclude = ["*.md", "src/spindle/proto/*_pb2.py"]

Expand Down
Loading
Loading