Skip to content

[routing 4/4] Integrate native routing and route LoRA pools with cache_wait by default - #123

Merged
Kailash Ranganathan (kailash109) merged 15 commits into
mainfrom
codex/routing-4-native
Oct 28, 2026
Merged

Kailash Ranganathan (kailash109) merged 15 commits into
mainfrom
codex/routing-4-native

Conversation

@kailash109

@kailash109 Kailash Ranganathan (kailash109) commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Run observation polling, admission, worker HTTP and bounded retries in Rust, inside each sampling container. Python owns the runtime's lifecycle and awaits each request in-process through PyO3; there is no local HTTP hop.

Defaults. LoRA recipes whose SGLang server has one data-parallel rank now route with cache_wait; other recipes keep Modal session affinity, and inference_lora_routing = "affinity" opts out. The router measures the prefill rate from its own completed requests (SGLang meta_info timings) unless inference_routing_prefill_tokens_per_second is set, and inference_routing_balance_absolute defaults to 40% of max_running_requests (at least 4). Pools whose SGLang cache publishes no observations (sliding-window or pipeline-parallel) fall back to gateway routing immediately. LoRA samplers also enable CSGMV split-K (SGLANG_ENABLE_LORA_CSGMV_SPLIT_K=0 disables it).

Rebased onto main after #122. The sampling-group wire is gone with that policy: same-affinity samples still share a replica through the affinity tie-break, and SGLang's LPM scheduler holds queued prefix-sharing requests. Only the Modal image compiles the Rust extension; installing Spindle and running spindle deploy need no Rust toolchain.

About half of the diff is Cargo.lock, required by cargo test --locked. The rest is the in-process runtime, its HTTP tests, and the sampling/Modal wiring.

Validation.

  • The router's 7 Rust tests and 21 native-router cases pass; the latter run real HTTP against a fake worker and cover redelivery with the same RID, bounded transport retries, deadlines and cancellation, stale and unpublished observations, and the measured prefill rate.
  • GPU runs with untuned defaults, cache_wait vs affinity output tok/s:
    • SWE-Gym, Qwen3.6-27B (balance_absolute 4): +17.2% and +10.2% over two repeats.
    • Qwen3.6-35B-A3B on 4 replicas (balance_absolute 51), mean of 3 repeats: agent-6turn +5.9%, few-long +27.6%, single-turn +4.3%, skew +8.9%; 11 of 12 runs positive.
  • Split-K kernel, rank 32 on 4,096-wide layers: 1.1–2.9× faster at ≤256 tokens on H200 and B200, 6–11% slower at 8,192 tokens. Details in the PR comments.

This stack supersedes #88; its historical benchmark plots and qualifications remain available there.

Depends on #122 (merged).

Stack (merge bottom-up):

  1. Replay duplicate generations and bound cancellation (merged)
  2. Publish incremental cache observations off the scheduler (merged)
  3. Select LoRA workers by KV reuse and memory pressure (merged)
  4. Integrate native routing and route LoRA pools with cache_wait by default

@kailash109
Kailash Ranganathan (kailash109) force-pushed the codex/routing-4-native branch 4 times, most recently from 99b4d64 to 5084b54 Compare October 27, 2026 20:18
Base automatically changed from codex/routing-3-policy to main October 28, 2026 00:39

@micahtyong Micah Yong (micahtyong) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are three ways I can see for every cache_wait sample to hang until its 3000 s deadline (inline below), plus a few dangerous edges.

Nothing has run on Modal since the router moved in-process (fca0882 → bd249f8). Before merge, could we get one stock tinker-cookbook RL job against a cache_wait recipe with the default inference_min_replicas = 0, as an E2E section in the PR? That would also confirm the gateway accepts the duplicated Modal-Key/Modal-Secret headers. reqwest appends rather than replaces, and both s.headers and the per-request headers carry them (native.rs:159-167).

recipe.inference_env
| (
{
"SGLANG_SPINDLE_ROUTING_OBSERVATIONS": "1",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switching an existing deployment to cache_wait leaves its old pool in place, and every sample hangs. The observer env var is added here from inference_lora_routing, but inference_identity() (deployments.py:146-169) doesn't include inference_lora_routing. So after a user flips the policy, spindle deploy sees no inference change and keeps the old pool, which has no observer. Its /routing-state returns work: null, every replica is dropped, and each sample waits out the full deadline.

Could we add inference_lora_routing to the identity?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6c3d725. inference_identity() now includes inference_lora_routing, so switching the policy changes the identity and spindle deploy republishes the pool with the observer. The cache_wait deploy test asserts that its identity differs from affinity's.

Comment thread rust/router/src/native.rs
.unwrap()
.select_destination(&request, id.clone(), &mut admission)
.map_err(|_| error(500, "Scheduler failed"))?;
if d.status == "route" {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A pool that has scaled to zero never starts again. inference_min_replicas defaults to 0. With no containers, list_flash_containers returns [], so this loop keeps getting "unavailable" and never sends a /generate to the gateway. Modal therefore never cold-starts a replica, and the sample times out. The DAPO example pins min_replicas = 3, which is why this never showed up.

When no workers are discovered, could we forward to the gateway without Modal-Flash-Upstream so Modal cold-starts one? Or reject cache_wait with min_replicas == 0 at deploy.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6c3d725. Once discovery has run and found no workers, the router forwards to the gateway without Modal-Flash-Upstream, so Modal cold-starts a replica. It does the same after 5 s of continuous "unavailable", for example when every replica is stale, so a pool that never publishes observations falls back to gateway routing instead of hanging. test_unroutable_pools_fall_back_to_the_gateway covers both cases.

I also ran this end to end: stock tinker-cookbook rl_basic on a Qwen3.5-4B LoRA pool with cache_wait and min_replicas=0 cold-started from zero and completed 3 steps.

Comment thread rust/router/src/native.rs Outdated
});
}
let mut rows = Vec::new();
while let Some(result) = tasks.join_next().await {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One slow replica makes every replica stale. This waits for every worker, up to the 2 s timeout, before installing anything, and the wait is added to each row's age_seconds. Normal ages are about 0.5–1.5 s, so one replica that is draining, starting, or has a busy sidecar pushes the healthy ones past the 3 s limit. Then nothing is routable for most of each round. test_native_router.py:270-279 asserts this as intended.

Since install_snapshot now merges with previous observations, could we install each row as it arrives, or run one polling loop per worker?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6c3d725. Each replica is polled in its own task, and its row is installed as soon as that poll returns. The age counts only that poll's round trip. A replica whose poll is still in flight isn't polled again until it returns, so a slow one only ages out its own row. The test that asserted the old wait is replaced by test_a_slow_peer_does_not_stale_a_fast_replica.

Comment thread src/spindle/deployments.py Outdated
if routing not in {"affinity", "cache_wait"}:
raise ValueError(f"unknown inference_lora_routing: {routing}")
prefill_rate = pinned.inference_routing_prefill_tokens_per_second
if routing == "cache_wait" and (prefill_rate is None or prefill_rate <= 0):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we also reject the configurations that never publish observations? With them every sample hangs, the same way as with a pool that has no observer.

  • In the pinned SGLang, spindle_routing_observer.snapshot returns early when dp_size != 1.
  • Sliding-window-attention caches publish nothing.
  • inference_min_replicas == 0, unless the router falls back to the gateway.

The other routing settings (balance_relative < 1, a threshold outside [0, 1]) currently fail on every sample instead of at deploy, so they'd fit here too.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In 6c3d725, DeploymentConfig.create rejects cache_wait at deploy when:

  • SGLang runs more than one data-parallel rank: dp_size != 1, or more than one engine per node;
  • balance_relative isn't finite or is below 1;
  • balance_absolute isn't a non-negative integer;
  • cache_threshold is outside [0, 1].

min_replicas == 0 is handled by the gateway fallback instead (see the scale-to-zero thread).

I didn't add a sliding-window check. The recipe doesn't know the model's attention layout, so the check would have to read the Hugging Face config at deploy time. Such a pool no longer hangs, but each request waits the 5 s fallback before going to the gateway. I can add a recipe flag or a config check as a follow-up if you'd rather reject it.

Comment thread rust/router/src/native.rs
reservation.outcome = if busy {
Outcome::Busy
} else if retry {
Outcome::Failed

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An identified 5xx is treated as a replica failure. A 5xx that carries x-spindle-replica comes from a live sidecar, but it gets Outcome::Failed, which excludes that replica for 2 s, and then retries up to 8 times. When the budget runs out, the worker's real error is replaced by "Worker retry budget exhausted", and Python keeps retrying that 502 until its deadline. So one adapter whose registration returns 500 can cycle every replica through cooldown and stall the other tenants' samples in this container.

Could we treat identified 5xx as Released and return the response, keep Failed for unidentified or transport errors, and return the last worker response when the budget is exhausted?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6c3d725. A 5xx that carries x-spindle-replica is returned to the caller unchanged. It releases the reservation without cooling the replica. Only a 429, or a 503 from an identified replica, counts as busy and is retried. An unidentified 408 or 5xx still counts as a lost upstream and is retried. If the retry budget runs out, the router returns the last upstream response instead of a synthetic 502. test_a_worker_error_is_returned_without_cooling_the_replica and the transport-limit test cover this.

Comment thread src/spindle/inference/sampling.py Outdated
headers=headers,
)
)
if model_id is None or latest:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

latest sessions skip the router. Sessions from save_weights_and_get_sampling_client are latest=True, so they never reach the router, and that's the call the stock tinker-cookbook RL loop uses on most steps. We hit this in a real run: the first smoke routed nothing until the client switched to save_weights_for_sampler + create_sampling_client(model_path=...).

Could we route latest sessions using the currently published version as the adapter hint? If not, the CHANGELOG and docs should say cache_wait only applies to exact-version sampling clients.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6c3d725. Latest sessions now route by the version they were created at. That is the version the trainer just published for that sampling client, and the cookbook creates a new sampling client each step. test_versioned_sessions_route_by_their_session_version covers both latest and pinned sessions. The stock cookbook RL run in the PR comment used save_weights_and_get_sampling_client unmodified.

Comment thread src/spindle/providers/modal/app.py Outdated
image = (
modal.Image.debian_slim(python_version="3.12")
.apt_install("git")
router_image()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes the image for every frontend function in every deployment, even with the default affinity policy. The API server, sweeps and everything else move from debian_slim to rust:1.90-slim with the full toolchain kept in the image. The Rust build also comes before pip_install, so any Rust edit rebuilds every later layer.

Could we build the wheel and delete the toolchain in one run_commands layer on debian_slim? Ideally that would be the last layer, used only by _execute_sample.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6c3d725. Frontend functions keep the debian_slim image, and only _execute_sample uses with_router(base_image).

The router build is one run_commands layer on debian_slim. It installs a minimal rustup 1.90.0 toolchain and maturin, builds and installs the wheel, then deletes the toolchain, sources and build packages in the same command. That layer sits below the configs and Spindle source layers, which change on every deploy. So a deploy reuses the cached router layer, and a Rust edit rebuilds only the sample image's router layer and the layers above it.

The runtime was receiving four floats in a fixed order. The recipe already names those rates, and the retry and header rules are easier to follow as named limits.

Co-authored-by: Cursor <cursoragent@cursor.com>
Saturated waits now count toward the affinity budget the policy bounds
them with. The built extension module is ignored in src/.

Co-authored-by: Cursor <cursoragent@cursor.com>
Python awaits a PyO3 future that runs on the Rust runtime, so every
concurrent request reaches Rust admission rather than queueing behind a
256-connection client pool. Cancelling the await drops the request and
releases its reservation. The local axum server and bearer token are gone.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
… arguments

Co-authored-by: Cursor <cursoragent@cursor.com>
…in the changelog

Co-authored-by: Cursor <cursoragent@cursor.com>
The merged policy no longer has prepare_group, so the native runtime, the Python routing key, and the sibling-wait test were still calling a mechanism that does not exist.

Co-authored-by: Cursor <cursoragent@cursor.com>
Deadline, conflict and failure-limit cases that exercised the same code path are merged, and the concurrency tests for the removed localhost HTTP hop are gone. A shared helper checks that every test releases its reservations.

Co-authored-by: Cursor <cursoragent@cursor.com>
The per-sequence seconds, replica, output and cached-token stats and the attempt's version error have no reader. The router retries 408 and 429 itself until the deadline, so neither reaches sampling.

Co-authored-by: Cursor <cursoragent@cursor.com>
modal deploy imports the app module locally, so importing the router there made every Spindle install compile Rust. The sampler imports it when it first routes, and CI installs the extension for the router tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ker errors.

Address review on routing 4/4: a pool with no routable replicas (scaled to zero or without an observer) routes through Modal's gateway; each replica's observation is installed on arrival; identified worker 5xx responses return without cooling the replica; caller headers replace the runtime's; latest sessions route by their session version; cache_wait rejects data parallelism and invalid balance settings at deploy; switching routing redeploys the pool; the extension builds in one debian_slim layer used only by execute_sample.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@kailash109

Copy link
Copy Markdown
Collaborator Author

SWE-Gym rerun on the current router

This reruns the pre-refactor SWE-Gym 27B router experiment (router-swewait1018) on this PR's code, with the same serving shape and the calibrated prefill rate from that run.

Setup: Qwen3.8-27B LoRA on H200, 4 sampler replicas at TP4 with max_running_requests=4 and max_queued_requests=8. Each arm has 8 clients × 8 trajectories (64 trajectories), up to 20 turns of up to 8,192 tokens with thinking on, against Modal sandboxes built from the SWE-Gym images. Both arms run at the same time, and the sweep repeats twice. This is rollout-only: 0 training steps, matching the original run.

Repeat Arm Output tok/s Makespan Output tokens Cache hit rate p95 request latency
0 affinity 986 856 s 844k 0.846 68.2 s
0 cache_wait 1,246 (+26.4%) 611 s 761k 0.839 64.3 s
1 affinity 1,119 674 s 754k 0.836 69.4 s
1 cache_wait 1,302 (+16.4%) 645 s 839k 0.832 82.5 s

cache_wait keeps the gain from the original run, which measured +16.5% over the default router (1,101 vs 945 tok/s). In turns 1–5 it hits the cache more often (0.54 and 0.57, against 0.51 and 0.53 for affinity), because early turns land on the replica that already holds the prefix. Repeat 0 completed 127 of 128 rollouts; one sandbox vanished mid-run.

Caveat: the gain depends on replicas queueing. On Qwen3.6-35B-A3B with up to 128 running requests per replica, nothing queued and cache_wait showed no consistent gain; scenarios ranged from −9.8% to +25% within run-to-run noise.

Stock tinker-cookbook RL end to end

The unmodified tinker-cookbook rl_basic recipe (GSM8K, batch 16 × group 8, save_weights_and_get_sampling_client each step) ran against a Qwen3.5-4B LoRA deployment with inference_lora_routing="cache_wait" and inference_min_replicas=0. The pool cold-started from zero through the gateway fallback, all 3 steps completed, and the sampler logs show the native router polling /routing-state.

Step Sampling Total
0 (cold start) 178 s 351 s
1 26 s 64 s
2 17 s 26 s

Review fixes

6c3d725 addresses the inline review comments; replies are on each thread. It also makes caller headers replace runtime headers, instead of sending both, and strips the session-affinity headers only when a replica is pinned. The branch is rebased on main and CI passes.

…e prefill rate

LoRA recipes with one SGLang data-parallel rank resolve to cache_wait at deploy. The router measures each pool's prefill rate from SGLang's per-request timing unless a recipe sets one, and balance_absolute defaults to 40% of max_running_requests. Pools whose workers publish no observations use the gateway at once. LoRA samplers enable the CSGMV split-K kernel.

Co-authored-by: Cursor <cursoragent@cursor.com>
@kailash109 Kailash Ranganathan (kailash109) changed the title [routing 4/4] Integrate native routing with exact-version sampling [routing 4/4] Integrate native routing and route LoRA pools with cache_wait by default Oct 28, 2026
@kailash109

Copy link
Copy Markdown
Collaborator Author

5161b5e makes the routing work on by default; the description has the new defaults.

Both GPU studies reran with untuned defaults (measured prefill rate, scaled balance_absolute), cache_wait vs affinity output tok/s:

Study Result
SWE-Gym, Qwen3.6-27B +17.2%, +10.2% (calibrated rate earlier: +26.4%, +16.4%)
Qwen3.6-35B-A3B agent-6turn +5.9% (old defaults −4.5%)
few-long +27.6% (old +5.7%)
single-turn, no reuse +4.3%
skew +8.9% (old −0.1%)

Caveats: agent-6turn's first repeat was −7%. Requests over 100 s (about 127 s, on both arms) were excluded from the Qwen3.6 means. Neither model has 4,096-wide layers, so these runs don't exercise split-K; its evidence is the kernel benchmark (1.1–2.9× faster at ≤256 tokens, 6–11% slower at 8,192). Gating split-K on token count belongs in the SGLang fork.

inference_env = {
"SGLANG_ENABLE_LORA_CSGMV_SPLIT_K": "1",
"SGLANG_SPINDLE_ROUTING_OBSERVATIONS": "1",
"SGLANG_SPINDLE_MODEL_ROUTING": "1",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: nothing reads SGLANG_SPINDLE_MODEL_ROUTING, in Spindle or in the pinned SGLang (a3e0e17), so we can drop it. The routing observer is now turned on by inference_lora_routing = "cache_wait" itself.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped in 50c689b, along with the example's SGLANG_ENABLE_LORA_CSGMV_SPLIT_K: split-K is on by default for LoRA pools now, and it never fires on the 27B's 5,120-wide layers anyway.

Comment thread rust/router/src/native.rs Outdated
if !g.timeout_seconds.is_finite() || g.timeout_seconds <= 0.0 || g.timeout_seconds > 86400.0 {
return Err(error(400, "Invalid deadline"));
}
if !s.requests.lock().unwrap().insert(g.request_id.clone()) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trim: I think this duplicate-ID guard can't trigger. The requests set, ActiveRequest and this 409 add about 20 lines. Python creates a fresh SampleAttempt ID for each attempt (sampling.py:320), forward appends :{attempt} for its own retries, and a cancelled attempt drops its future before Python retries. So two in-flight calls never share a request_id. Could we drop it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, dropped in 50c689b: the requests set, ActiveRequest and the 409 are gone. health() became pending(), which the tests use to check that every reservation is released.

Comment thread src/spindle/inference/native_router.py Outdated
)
self.poller = None

def prepare_request(self, adapter, affinity):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trim: this wrapper can be thinner, about 15–20 lines across this file and sampling.py.

  • RoutingKey and prepare_request only bundle (adapter, affinity). Passing the two strings to dispatch would drop the routing_request plumbing in sampling.py.
  • The balance and threshold defaults in __init__ repeat the recipe defaults, and app.py always passes them explicitly.
  • The per-call gateway argument and its "different inference pool" check in dispatch (L83-84) are redundant, because each router is built for one pool's gateway and keeps it as self.gateway.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 50c689b. dispatch(adapter, affinity, ...) takes the two strings, and _sample_one builds the version key itself, so RoutingKey, prepare_request and routing_request are gone. The threshold defaults are removed, and so is the per-call gateway with its check; the router uses self.gateway. Also dropped app.py's model_id check, which repeated sample_task's, and its max_queued_requests or 0 fallback, since every recipe resolves an integer.

Comment thread rust/router/src/native.rs
#[derive(Deserialize)]
struct Config {
gateway: String,
headers: HashMap<String, String>,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trim: policy only goes into the runtime config to be echoed back in routing telemetry (L169). Only cache_wait ever reaches the runtime, and Python already knows the policy, so sampling.py or otlp.py can set spindle.routing.policy directly. That removes the field from the Rust config, Service, the JSON echo and NativeRouter.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 50c689b. policy is out of the Rust config, Service, the JSON echo and NativeRouter; otlp.py sets spindle.routing.policy = cache_wait on attempts that went through the router.

…ive router

Co-authored-by: Cursor <cursoragent@cursor.com>
@kailash109

Copy link
Copy Markdown
Collaborator Author

Split-K end to end: the stock tinker-cookbook RL loop (4 steps) on default qwen35_9b_lora_16k, whose 4,096-wide layers trigger split-K, with cache_wait and cold-starting pools. With split-K on vs off, optim/kl_sample_train_v1 was 0.00015–0.00039 vs 0.00018–0.00060, so sampler/trainer agreement is unchanged, and summed sampling time was 194 s vs 202 s. Both runs completed and stopped.

@kailash109
Kailash Ranganathan (kailash109) merged commit 7e2c9fb into main Oct 28, 2026
4 checks passed
@kailash109
Kailash Ranganathan (kailash109) deleted the codex/routing-4-native branch October 28, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants