Repository navigation
feat(router): add sticky min-load worker-selection policy - #15600
Merged
Merged
Conversation
Contributor
Add the builtin dynamo-sticky-min-load policy. A session's first request binds to the eligible worker with the fewest active requests; later requests return to that worker until it leaves the candidate set. Requests without a session ID take the least-loaded worker. Idle bindings expire after max_idle_secs (default four hours). Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
force-pushed
the
idhanani/sticky-min-load-policy
branch
from
October 3, 2026 20:32
0d47334 to
a466ae1
Compare
ishandhanani
marked this pull request as ready for review
October 3, 2026 20:32
Contributor
Contributor
✅ Dynamo PR CI passed — run 37426311088 (attempt 1) on
|
Apply the session-affinity table's limits to the sticky min-load policy: session IDs over 256 bytes are not bound and the table holds at most 65,536 bindings. Requests that cannot be bound take the least-loaded worker. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
force-pushed
the
idhanani/sticky-min-load-policy
branch
from
October 3, 2026 22:22
690f16f to
2919323
Compare
jthomson04
approved these changes
Oct 3, 2026
jthomson04
left a comment
Contributor
There was a problem hiding this comment.
Reviewed 2919323. Approving with two P2 follow-up comments: expiry checks between sweeps and hard-affinity enforcement in standalone selection. Source review only; no tests or CI checks run.
There was a problem hiding this comment.
Previously reported defects still present:
- Original discussion: Idle expiry is still skipped between periodic sweeps: lookup refreshes
last_accessbefore checking the binding's age, so a binding older thanmax_idle_secsremains sticky whenever the sweep interval has not elapsed. Check expiry on lookup before refreshing the binding. - Original discussion: Standalone selection still passes an existing hard session-affinity target to this custom picker without restricting its candidates, and the picker ignores that target. It can select a different worker and make
commit_sessionreject an otherwise booked request. Enforce the eligible hard target in the shared standalone host before invoking the custom picker. - Original discussion: The expiry defect is still present:
selectonly evicts when the sweep interval has elapsed, then refreshes an existing binding before checking its age, so an expired binding can be reused and renewed wheneviction_interval_secsis larger thanmax_idle_secs. - Original discussion: The hard affinity concern is still present:
pickreads only the session ID and ignorescontext.affinity_target(), so a valid hard affinity target is not enforced before the sticky-min-load fallback path can select another worker. - Original discussion: The private test-helper doc comment at line 226 still only restates the
(worker_id, active_requests)tuple type and helper implementation; the existing instruction to remove it remains applicable. - Original discussion: Verified still present for standalone selection with hard session affinity: the custom picker ignores context.affinity_target(), while the standalone host does not restrict its candidate set for this policy. After the policy-local binding expires or diverges, it can choose a different worker and commit_session rejects the booked request. Enforce the eligible hard-affinity target in the standalone host before invoking the custom picker.
- Original discussion: Verified still present in standalone select-and-reserve: the picker ignores
context.affinity_target(), and the policy does not enable exclusive affinity. When its local binding differs from a valid hard session-affinity target, selection can book another worker andcommit_sessionrejects the request. Enforce the eligible hard target in the standalone host before invoking the custom picker. - Original discussion: Verified still present: a binding is refreshed before its age is checked. When max_idle_secs is shorter than eviction_interval_secs, requests arriving after the idle TTL but before the next sweep keep the stale worker indefinitely. Check and expire the looked-up binding before updating last_access.
- Original discussion: Verified still present: a binding older than
max_idle_secsremains valid until a sweep is due, and lookup refresheslast_accessbefore checking its age. Sessions accessed less frequently than the configured TTL but more frequently than the sweep interval can therefore remain pinned indefinitely. Check expiry during lookup before refreshing the binding, while retaining periodic sweeps for cleanup.
Questions for the author:
- lib/router-plugins/builtin/src/sticky_min_load.rs: At the documented 65,536-binding capacity, the request that crosses each eviction interval synchronously scans the entire
HashMapwhile worker selection is serialized. Please provide a calculation or p99 selection-latency measurement for a full table, including mostly-live and mostly-expired bindings, to establish whether this periodic inference-path stall is acceptable.
A binding older than max_idle_secs could survive until the next periodic sweep and keep routing its session. Check the binding's age on lookup and rebind to the least-loaded worker when it has expired. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
athreesh
approved these changes
Oct 5, 2026
There was a problem hiding this comment.
Previously reported defects still present:
- Original discussion: Still present:
StickyMinLoadPicker::pickderives only the session ID from the context and callsselectover the full candidate set, with no handling of an eligiblecontext.affinity_target(). The previously raised hard-affinity/custom-policy conflict remains the active record. - Original discussion: The hard session-affinity issue remains present:
StickyMinLoadPicker::pickreads onlysession_context()and ignorescontext.affinity_target(), while custom policies do not request exclusive affinity, so standalone select-and-reserve can select a worker different from an existing hard affinity target and then failcommit_session. - Original discussion: The hard-affinity concern is still present:
pickreads onlysession_context().session_id()and passes the unrestricted candidate set intoselect, so an eligiblecontext.affinity_target()can still be ignored by this custom policy before standalone commit validation. - Original discussion: The previously reported hard session-affinity issue is still present: the standalone selection path passes an existing hard affinity binding to custom policies as
context.affinity_target(), butdynamo-sticky-min-loadignores that target and can choose its own least-loaded or policy-local sticky worker instead. When that differs from the active hard binding,commit_sessionrejects the already selected/booked request. The host-level fix described in the thread has not landed in this PR. - Original discussion: Verified still present for standalone select-and-reserve with hard session affinity: this policy ignores
context.affinity_target(), and the custom-policy host does not restrict candidates unless the policy opts into exclusive affinity. If the sticky-min-load local binding expires, diverges, or is absent while the session-affinity table still has a valid hard target, the policy can book another worker andcommit_sessionrejects the otherwise accepted request. Enforce the eligible hard-affinity target in the shared standalone host before invoking custom worker selection, as discussed in the original thread.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…oad-policy Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
This was referenced Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
dynamo-sticky-min-load, a built-in worker-selection policy that pins each session to a least-loaded worker.X-Dynamo-Session-IDreturn to that worker regardless of load or cache overlap, until the worker leaves the candidate set; only then is the session rebound.max_idle_secs(default 4 h), swept at most everyeviction_interval_secs(default 60 s).It is selected through
--router-policy-configwith no rebuild:Why
On continuously rolling multi-turn sessions, where each turn appends to the previous context, the default KV cost lets small differences in active decode load outweigh the cache credit. In the benchmark below, about 40% of turns move to the other worker and re-prefill their context, which costs ~22% output throughput against SMG's sticky
min_loadplacement. This policy keeps each session on its first worker.Benchmark
Harness: public rolling benchmark gist. Qwen3-8B BF16, two GB300 GPUs with one TP1 SGLang 0.5.16 worker each, SMG 1.7.0. 256 session slots, 8 branches per group, 12 turns, context growing 1K to 16K tokens, 256 output tokens, 2 s gap between turns. Each trial runs 180 s and scores 30 to 150 s; arms run in forward then reverse order across two repeats. Output tok/s is the sum of SGLang decode counter deltas; client and backend token counts match exactly in every trial.
manual/min_loaddynamo-sticky-min-load(this branch)Other Dynamo routing on the same harness, each measured against its own run's SMG arm:
dynamo-two-tier-cost-fndefaultsdynamo-two-tier-cost-fn+--router-session-affinity-ttl-secs 3600Two-tier's load tier (spread > 32 active requests and > 1.1x) fires on normal queue swings at this concurrency (~100 active requests per worker against 64 running slots) and moves 7 to 9% of turns. Two repeats do not establish a confidence interval.
Running the gist against current Dynamo needed three harness changes: the SGLang sidecar takes
--grpc-endpointand requires SGLang--incremental-streaming-output; the SMG client accepts incremental output; and the "router sees a directly warmed cache" probe is skipped for this policy, as it is for the SMG arm, because the policy does not use cache overlap.Validation
cargo test -p dynamo-custom-policy-builtin --lib: 55 passed, 2 ignored. Includes 6 new policy tests (least-loaded first placement, stickiness under load and row reordering, rebinding only when the worker leaves, no binding without a session, idle expiry, parameter validation) and a YAML resolution test for all worker types.cargo clippy -p dynamo-custom-policy-builtin --all-targets -- -D warningsandcargo fmt --check: clean.Summary by CodeRabbit
dynamo-sticky-min-loadworker-selection policy. Sessions are initially assigned to a least-loaded eligible worker and remain there while it stays eligible; sessions are reassigned if it becomes unavailable.