Repository navigation
[routing 4/4] Integrate native routing and route LoRA pools with cache_wait by default - #123
Conversation
f79060b to
f6124ee
Compare
7ec9e7a to
91149a1
Compare
f6124ee to
eeacf47
Compare
91149a1 to
afe0cb0
Compare
eeacf47 to
d17ba9b
Compare
afe0cb0 to
2e838fa
Compare
7932273 to
15b06a2
Compare
99b4d64 to
5084b54
Compare
fc0b7c9 to
02ea77a
Compare
5084b54 to
20e92b6
Compare
20e92b6 to
5fe9426
Compare
Micah Yong (micahtyong)
left a comment
There was a problem hiding this comment.
There are three ways I can see for every cache_wait sample to hang until its 3000 s deadline (inline below), plus a few dangerous edges.
Nothing has run on Modal since the router moved in-process (fca0882 → bd249f8). Before merge, could we get one stock tinker-cookbook RL job against a cache_wait recipe with the default inference_min_replicas = 0, as an E2E section in the PR? That would also confirm the gateway accepts the duplicated Modal-Key/Modal-Secret headers. reqwest appends rather than replaces, and both s.headers and the per-request headers carry them (native.rs:159-167).
| recipe.inference_env | ||
| | ( | ||
| { | ||
| "SGLANG_SPINDLE_ROUTING_OBSERVATIONS": "1", |
There was a problem hiding this comment.
Switching an existing deployment to cache_wait leaves its old pool in place, and every sample hangs. The observer env var is added here from inference_lora_routing, but inference_identity() (deployments.py:146-169) doesn't include inference_lora_routing. So after a user flips the policy, spindle deploy sees no inference change and keeps the old pool, which has no observer. Its /routing-state returns work: null, every replica is dropped, and each sample waits out the full deadline.
Could we add inference_lora_routing to the identity?
There was a problem hiding this comment.
Fixed in 6c3d725. inference_identity() now includes inference_lora_routing, so switching the policy changes the identity and spindle deploy republishes the pool with the observer. The cache_wait deploy test asserts that its identity differs from affinity's.
| .unwrap() | ||
| .select_destination(&request, id.clone(), &mut admission) | ||
| .map_err(|_| error(500, "Scheduler failed"))?; | ||
| if d.status == "route" { |
There was a problem hiding this comment.
A pool that has scaled to zero never starts again. inference_min_replicas defaults to 0. With no containers, list_flash_containers returns [], so this loop keeps getting "unavailable" and never sends a /generate to the gateway. Modal therefore never cold-starts a replica, and the sample times out. The DAPO example pins min_replicas = 3, which is why this never showed up.
When no workers are discovered, could we forward to the gateway without Modal-Flash-Upstream so Modal cold-starts one? Or reject cache_wait with min_replicas == 0 at deploy.
There was a problem hiding this comment.
Fixed in 6c3d725. Once discovery has run and found no workers, the router forwards to the gateway without Modal-Flash-Upstream, so Modal cold-starts a replica. It does the same after 5 s of continuous "unavailable", for example when every replica is stale, so a pool that never publishes observations falls back to gateway routing instead of hanging. test_unroutable_pools_fall_back_to_the_gateway covers both cases.
I also ran this end to end: stock tinker-cookbook rl_basic on a Qwen3.5-4B LoRA pool with cache_wait and min_replicas=0 cold-started from zero and completed 3 steps.
| }); | ||
| } | ||
| let mut rows = Vec::new(); | ||
| while let Some(result) = tasks.join_next().await { |
There was a problem hiding this comment.
One slow replica makes every replica stale. This waits for every worker, up to the 2 s timeout, before installing anything, and the wait is added to each row's age_seconds. Normal ages are about 0.5–1.5 s, so one replica that is draining, starting, or has a busy sidecar pushes the healthy ones past the 3 s limit. Then nothing is routable for most of each round. test_native_router.py:270-279 asserts this as intended.
Since install_snapshot now merges with previous observations, could we install each row as it arrives, or run one polling loop per worker?
There was a problem hiding this comment.
Fixed in 6c3d725. Each replica is polled in its own task, and its row is installed as soon as that poll returns. The age counts only that poll's round trip. A replica whose poll is still in flight isn't polled again until it returns, so a slow one only ages out its own row. The test that asserted the old wait is replaced by test_a_slow_peer_does_not_stale_a_fast_replica.
| if routing not in {"affinity", "cache_wait"}: | ||
| raise ValueError(f"unknown inference_lora_routing: {routing}") | ||
| prefill_rate = pinned.inference_routing_prefill_tokens_per_second | ||
| if routing == "cache_wait" and (prefill_rate is None or prefill_rate <= 0): |
There was a problem hiding this comment.
Could we also reject the configurations that never publish observations? With them every sample hangs, the same way as with a pool that has no observer.
- In the pinned SGLang,
spindle_routing_observer.snapshotreturns early whendp_size != 1. - Sliding-window-attention caches publish nothing.
inference_min_replicas == 0, unless the router falls back to the gateway.
The other routing settings (balance_relative < 1, a threshold outside [0, 1]) currently fail on every sample instead of at deploy, so they'd fit here too.
There was a problem hiding this comment.
In 6c3d725, DeploymentConfig.create rejects cache_wait at deploy when:
- SGLang runs more than one data-parallel rank:
dp_size != 1, or more than one engine per node; balance_relativeisn't finite or is below 1;balance_absoluteisn't a non-negative integer;cache_thresholdis outside [0, 1].
min_replicas == 0 is handled by the gateway fallback instead (see the scale-to-zero thread).
I didn't add a sliding-window check. The recipe doesn't know the model's attention layout, so the check would have to read the Hugging Face config at deploy time. Such a pool no longer hangs, but each request waits the 5 s fallback before going to the gateway. I can add a recipe flag or a config check as a follow-up if you'd rather reject it.
| reservation.outcome = if busy { | ||
| Outcome::Busy | ||
| } else if retry { | ||
| Outcome::Failed |
There was a problem hiding this comment.
An identified 5xx is treated as a replica failure. A 5xx that carries x-spindle-replica comes from a live sidecar, but it gets Outcome::Failed, which excludes that replica for 2 s, and then retries up to 8 times. When the budget runs out, the worker's real error is replaced by "Worker retry budget exhausted", and Python keeps retrying that 502 until its deadline. So one adapter whose registration returns 500 can cycle every replica through cooldown and stall the other tenants' samples in this container.
Could we treat identified 5xx as Released and return the response, keep Failed for unidentified or transport errors, and return the last worker response when the budget is exhausted?
There was a problem hiding this comment.
Fixed in 6c3d725. A 5xx that carries x-spindle-replica is returned to the caller unchanged. It releases the reservation without cooling the replica. Only a 429, or a 503 from an identified replica, counts as busy and is retried. An unidentified 408 or 5xx still counts as a lost upstream and is retried. If the retry budget runs out, the router returns the last upstream response instead of a synthetic 502. test_a_worker_error_is_returned_without_cooling_the_replica and the transport-limit test cover this.
| headers=headers, | ||
| ) | ||
| ) | ||
| if model_id is None or latest: |
There was a problem hiding this comment.
latest sessions skip the router. Sessions from save_weights_and_get_sampling_client are latest=True, so they never reach the router, and that's the call the stock tinker-cookbook RL loop uses on most steps. We hit this in a real run: the first smoke routed nothing until the client switched to save_weights_for_sampler + create_sampling_client(model_path=...).
Could we route latest sessions using the currently published version as the adapter hint? If not, the CHANGELOG and docs should say cache_wait only applies to exact-version sampling clients.
There was a problem hiding this comment.
Fixed in 6c3d725. Latest sessions now route by the version they were created at. That is the version the trainer just published for that sampling client, and the cookbook creates a new sampling client each step. test_versioned_sessions_route_by_their_session_version covers both latest and pinned sessions. The stock cookbook RL run in the PR comment used save_weights_and_get_sampling_client unmodified.
| image = ( | ||
| modal.Image.debian_slim(python_version="3.12") | ||
| .apt_install("git") | ||
| router_image() |
There was a problem hiding this comment.
This changes the image for every frontend function in every deployment, even with the default affinity policy. The API server, sweeps and everything else move from debian_slim to rust:1.90-slim with the full toolchain kept in the image. The Rust build also comes before pip_install, so any Rust edit rebuilds every later layer.
Could we build the wheel and delete the toolchain in one run_commands layer on debian_slim? Ideally that would be the last layer, used only by _execute_sample.
There was a problem hiding this comment.
Fixed in 6c3d725. Frontend functions keep the debian_slim image, and only _execute_sample uses with_router(base_image).
The router build is one run_commands layer on debian_slim. It installs a minimal rustup 1.90.0 toolchain and maturin, builds and installs the wheel, then deletes the toolchain, sources and build packages in the same command. That layer sits below the configs and Spindle source layers, which change on every deploy. So a deploy reuses the cached router layer, and a Rust edit rebuilds only the sample image's router layer and the layers above it.
The runtime was receiving four floats in a fixed order. The recipe already names those rates, and the retry and header rules are easier to follow as named limits. Co-authored-by: Cursor <cursoragent@cursor.com>
Saturated waits now count toward the affinity budget the policy bounds them with. The built extension module is ignored in src/. Co-authored-by: Cursor <cursoragent@cursor.com>
Python awaits a PyO3 future that runs on the Rust runtime, so every concurrent request reaches Rust admission rather than queueing behind a 256-connection client pool. Cancelling the await drops the request and releases its reservation. The local axum server and bearer token are gone. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
… arguments Co-authored-by: Cursor <cursoragent@cursor.com>
…in the changelog Co-authored-by: Cursor <cursoragent@cursor.com>
The merged policy no longer has prepare_group, so the native runtime, the Python routing key, and the sibling-wait test were still calling a mechanism that does not exist. Co-authored-by: Cursor <cursoragent@cursor.com>
Deadline, conflict and failure-limit cases that exercised the same code path are merged, and the concurrency tests for the removed localhost HTTP hop are gone. A shared helper checks that every test releases its reservations. Co-authored-by: Cursor <cursoragent@cursor.com>
The per-sequence seconds, replica, output and cached-token stats and the attempt's version error have no reader. The router retries 408 and 429 itself until the deadline, so neither reaches sampling. Co-authored-by: Cursor <cursoragent@cursor.com>
modal deploy imports the app module locally, so importing the router there made every Spindle install compile Rust. The sampler imports it when it first routes, and CI installs the extension for the router tests. Co-authored-by: Cursor <cursoragent@cursor.com>
…ker errors. Address review on routing 4/4: a pool with no routable replicas (scaled to zero or without an observer) routes through Modal's gateway; each replica's observation is installed on arrival; identified worker 5xx responses return without cooling the replica; caller headers replace the runtime's; latest sessions route by their session version; cache_wait rejects data parallelism and invalid balance settings at deploy; switching routing redeploys the pool; the extension builds in one debian_slim layer used only by execute_sample. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
70eb08f to
c4cbcca
Compare
SWE-Gym rerun on the current routerThis reruns the pre-refactor SWE-Gym 27B router experiment ( Setup: Qwen3.8-27B LoRA on H200, 4 sampler replicas at TP4 with
Caveat: the gain depends on replicas queueing. On Qwen3.6-35B-A3B with up to 128 running requests per replica, nothing queued and Stock tinker-cookbook RL end to endThe unmodified tinker-cookbook
Review fixes
|
…e prefill rate LoRA recipes with one SGLang data-parallel rank resolve to cache_wait at deploy. The router measures each pool's prefill rate from SGLang's per-request timing unless a recipe sets one, and balance_absolute defaults to 40% of max_running_requests. Pools whose workers publish no observations use the gateway at once. LoRA samplers enable the CSGMV split-K kernel. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Both GPU studies reran with untuned defaults (measured prefill rate, scaled
Caveats: agent-6turn's first repeat was −7%. Requests over 100 s (about 127 s, on both arms) were excluded from the Qwen3.6 means. Neither model has 4,096-wide layers, so these runs don't exercise split-K; its evidence is the kernel benchmark (1.1–2.9× faster at ≤256 tokens, 6–11% slower at 8,192). Gating split-K on token count belongs in the SGLang fork. |
| inference_env = { | ||
| "SGLANG_ENABLE_LORA_CSGMV_SPLIT_K": "1", | ||
| "SGLANG_SPINDLE_ROUTING_OBSERVATIONS": "1", | ||
| "SGLANG_SPINDLE_MODEL_ROUTING": "1", |
There was a problem hiding this comment.
Nit: nothing reads SGLANG_SPINDLE_MODEL_ROUTING, in Spindle or in the pinned SGLang (a3e0e17), so we can drop it. The routing observer is now turned on by inference_lora_routing = "cache_wait" itself.
There was a problem hiding this comment.
Dropped in 50c689b, along with the example's SGLANG_ENABLE_LORA_CSGMV_SPLIT_K: split-K is on by default for LoRA pools now, and it never fires on the 27B's 5,120-wide layers anyway.
| if !g.timeout_seconds.is_finite() || g.timeout_seconds <= 0.0 || g.timeout_seconds > 86400.0 { | ||
| return Err(error(400, "Invalid deadline")); | ||
| } | ||
| if !s.requests.lock().unwrap().insert(g.request_id.clone()) { |
There was a problem hiding this comment.
Trim: I think this duplicate-ID guard can't trigger. The requests set, ActiveRequest and this 409 add about 20 lines. Python creates a fresh SampleAttempt ID for each attempt (sampling.py:320), forward appends :{attempt} for its own retries, and a cancelled attempt drops its future before Python retries. So two in-flight calls never share a request_id. Could we drop it?
There was a problem hiding this comment.
Agreed, dropped in 50c689b: the requests set, ActiveRequest and the 409 are gone. health() became pending(), which the tests use to check that every reservation is released.
| ) | ||
| self.poller = None | ||
|
|
||
| def prepare_request(self, adapter, affinity): |
There was a problem hiding this comment.
Trim: this wrapper can be thinner, about 15–20 lines across this file and sampling.py.
RoutingKeyandprepare_requestonly bundle(adapter, affinity). Passing the two strings todispatchwould drop therouting_requestplumbing insampling.py.- The balance and threshold defaults in
__init__repeat the recipe defaults, andapp.pyalways passes them explicitly. - The per-call
gatewayargument and its "different inference pool" check indispatch(L83-84) are redundant, because each router is built for one pool's gateway and keeps it asself.gateway.
There was a problem hiding this comment.
Done in 50c689b. dispatch(adapter, affinity, ...) takes the two strings, and _sample_one builds the version key itself, so RoutingKey, prepare_request and routing_request are gone. The threshold defaults are removed, and so is the per-call gateway with its check; the router uses self.gateway. Also dropped app.py's model_id check, which repeated sample_task's, and its max_queued_requests or 0 fallback, since every recipe resolves an integer.
| #[derive(Deserialize)] | ||
| struct Config { | ||
| gateway: String, | ||
| headers: HashMap<String, String>, |
There was a problem hiding this comment.
Trim: policy only goes into the runtime config to be echoed back in routing telemetry (L169). Only cache_wait ever reaches the runtime, and Python already knows the policy, so sampling.py or otlp.py can set spindle.routing.policy directly. That removes the field from the Rust config, Service, the JSON echo and NativeRouter.
There was a problem hiding this comment.
Done in 50c689b. policy is out of the Rust config, Service, the JSON echo and NativeRouter; otlp.py sets spindle.routing.policy = cache_wait on attempts that went through the router.
…ive router Co-authored-by: Cursor <cursoragent@cursor.com>
|
Split-K end to end: the stock tinker-cookbook RL loop (4 steps) on default |
Run observation polling, admission, worker HTTP and bounded retries in Rust, inside each sampling container. Python owns the runtime's lifecycle and awaits each request in-process through PyO3; there is no local HTTP hop.
Defaults. LoRA recipes whose SGLang server has one data-parallel rank now route with
cache_wait; other recipes keep Modal session affinity, andinference_lora_routing = "affinity"opts out. The router measures the prefill rate from its own completed requests (SGLangmeta_infotimings) unlessinference_routing_prefill_tokens_per_secondis set, andinference_routing_balance_absolutedefaults to 40% ofmax_running_requests(at least 4). Pools whose SGLang cache publishes no observations (sliding-window or pipeline-parallel) fall back to gateway routing immediately. LoRA samplers also enable CSGMV split-K (SGLANG_ENABLE_LORA_CSGMV_SPLIT_K=0disables it).Rebased onto main after #122. The sampling-group wire is gone with that policy: same-affinity samples still share a replica through the affinity tie-break, and SGLang's LPM scheduler holds queued prefix-sharing requests. Only the Modal image compiles the Rust extension; installing Spindle and running
spindle deployneed no Rust toolchain.About half of the diff is
Cargo.lock, required bycargo test --locked. The rest is the in-process runtime, its HTTP tests, and the sampling/Modal wiring.Validation.
cache_waitvs affinity output tok/s:balance_absolute4): +17.2% and +10.2% over two repeats.balance_absolute51), mean of 3 repeats: agent-6turn +5.9%, few-long +27.6%, single-turn +4.3%, skew +8.9%; 11 of 12 runs positive.This stack supersedes #88; its historical benchmark plots and qualifications remain available there.
Depends on #122 (merged).
Stack (merge bottom-up):