Skip to content

feat(router): add sticky min-load worker-selection policy - #15600

Merged
ishandhanani merged 5 commits into
mainfrom
idhanani/sticky-min-load-policy
Nov 2, 2026
Merged

ishandhanani merged 5 commits into
mainfrom
idhanani/sticky-min-load-policy

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds dynamo-sticky-min-load, a built-in worker-selection policy that pins each session to a least-loaded worker.

  • A session's first request binds to the eligible worker with the fewest active requests (ties broken at random).
  • Later requests with the same X-Dynamo-Session-ID return to that worker regardless of load or cache overlap, until the worker leaves the candidate set; only then is the session rebound.
  • Requests without a session ID go to the least-loaded worker and create no binding.
  • Idle bindings expire after max_idle_secs (default 4 h), swept at most every eviction_interval_secs (default 60 s).
  • Each worker pool keeps its own bindings, so prefill and decode stages bind independently.

It is selected through --router-policy-config with no rebuild:

worker_selection:
  aggregated: sticky-min-load
  instances:
    - name: sticky-min-load
      type: dynamo-sticky-min-load

Why

On continuously rolling multi-turn sessions, where each turn appends to the previous context, the default KV cost lets small differences in active decode load outweigh the cache credit. In the benchmark below, about 40% of turns move to the other worker and re-prefill their context, which costs ~22% output throughput against SMG's sticky min_load placement. This policy keeps each session on its first worker.

Benchmark

Harness: public rolling benchmark gist. Qwen3-8B BF16, two GB300 GPUs with one TP1 SGLang 0.5.16 worker each, SMG 1.7.0. 256 session slots, 8 branches per group, 12 turns, context growing 1K to 16K tokens, 256 output tokens, 2 s gap between turns. Each trial runs 180 s and scores 30 to 150 s; arms run in forward then reverse order across two repeats. Output tok/s is the sum of SGLang decode counter deltas; client and backend token counts match exactly in every trial.

Routing Repeat A tok/s Repeat B tok/s vs SMG A / B Cache reuse Session moves
SMG manual/min_load 7,144 7,020 82.5% 0
dynamo-sticky-min-load (this branch) 7,130 7,090 -0.2% / +1.0% 82.6% 0

Other Dynamo routing on the same harness, each measured against its own run's SMG arm:

Routing Dynamo vs SMG
KV default 1.4.2 -21.8% / -21.5%
KV default nightly 1.6.0.dev20261002 -21.2%
dynamo-two-tier-cost-fn defaults nightly -6.2% / -5.4%
dynamo-two-tier-cost-fn + --router-session-affinity-ttl-secs 3600 nightly -2.2% / -0.1%

Two-tier's load tier (spread > 32 active requests and > 1.1x) fires on normal queue swings at this concurrency (~100 active requests per worker against 64 running slots) and moves 7 to 9% of turns. Two repeats do not establish a confidence interval.

Running the gist against current Dynamo needed three harness changes: the SGLang sidecar takes --grpc-endpoint and requires SGLang --incremental-streaming-output; the SMG client accepts incremental output; and the "router sees a directly warmed cache" probe is skipped for this policy, as it is for the SMG arm, because the policy does not use cache overlap.

Validation

  • cargo test -p dynamo-custom-policy-builtin --lib: 55 passed, 2 ignored. Includes 6 new policy tests (least-loaded first placement, stickiness under load and row reordering, rebinding only when the worker leaves, no binding without a session, idle expiry, parameter validation) and a YAML resolution test for all worker types.
  • cargo clippy -p dynamo-custom-policy-builtin --all-targets -- -D warnings and cargo fmt --check: clean.
  • Built the aarch64 runtime wheel from this branch and ran the benchmark above with it.
  • pre-commit hooks were not run locally.

Summary by CodeRabbit

  • New Features
    • Added the dynamo-sticky-min-load worker-selection policy. Sessions are initially assigned to a least-loaded eligible worker and remain there while it stays eligible; sessions are reassigned if it becomes unavailable.
    • Requests without a session ID use least-loaded placement without a persistent assignment.
    • Session assignments expire after a configurable idle period. Prefill and decode pools maintain separate assignments.

@github-actions github-actions Bot added feat documentation Improvements or additions to documentation labels Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Add the builtin dynamo-sticky-min-load policy. A session's first request
binds to the eligible worker with the fewest active requests; later
requests return to that worker until it leaves the candidate set.
Requests without a session ID take the least-loaded worker. Idle
bindings expire after max_idle_secs (default four hours).

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/sticky-min-load-policy branch from 0d47334 to a466ae1 Compare October 3, 2026 20:32
@ishandhanani
ishandhanani marked this pull request as ready for review October 3, 2026 20:32
@ishandhanani
ishandhanani requested review from a team as code owners October 3, 2026 20:32
@coderabbitai

coderabbitai Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 65348902-d732-4768-b550-3d9a91ec2fde

📥 Commits

Reviewing files that changed from the base of the PR and between 7ffd86f and a466ae1.


📒 Files selected for processing (3)
  • docs/fern/pages/developer-guide/knowledge-base/modular-components/router/configuration-and-tuning.md
  • lib/router-plugins/builtin/src/lib.rs
  • lib/router-plugins/builtin/src/sticky_min_load.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.



Walkthrough

The builtin router adds the dynamo-sticky-min-load policy. It selects among least-loaded eligible workers, retains session bindings while workers remain eligible, and expires idle bindings using configurable timing parameters.

Changes

Sticky Minimum-Load Routing

Layer / File(s) Summary
Binding configuration and expiration
lib/router-plugins/builtin/src/sticky_min_load.rs, docs/fern/pages/developer-guide/knowledge-base/modular-components/router/configuration-and-tuning.md
Adds defaults and validation for binding expiration settings. Removes bindings that exceed the configured idle duration during periodic sweeps. Documents selection behavior, tuning parameters, and separate prefill and decode bindings.
Worker selection and session bindings
lib/router-plugins/builtin/src/sticky_min_load.rs
Selects uniformly among least-loaded workers. Session requests retain an eligible binding or bind to a least-loaded worker; requests without a session ID remain unbound. Adds picker input validation and tests for selection, binding, expiration, and validation.
Builtin plugin registration
lib/router-plugins/builtin/src/lib.rs, lib/router-plugins/builtin/src/sticky_min_load.rs
Registers the policy provider in the builtin plugin catalog. Adds a configuration-resolution test for aggregated, prefill, and decode stages.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to a466a

The new routing policy has no identified issue that needs resolution before merge.

Pre-merge checks | Passed 3 | Failed 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 54.55% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 22 functions across 2 files. (1 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check Warning The description provides detailed overview, implementation details, benchmark results, validation, and reviewer-relevant context. However, it omits the required Related Issues reference and does not i… Add a Related Issues entry with the applicable issue reference, such as Closes #1234 or Closes DYN-1234. Add a Where should the reviewer start section that identifies the primary files for review.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly and concisely identifies the main change: adding the sticky min-load worker-selection policy.

Full details: Docstring Coverage

Explanation

Docstring coverage is 54.55% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 22 functions across 2 files. (1 skipped: 1 unsupported.)


Full details: Description check

Explanation

The description provides detailed overview, implementation details, benchmark results, validation, and reviewer-relevant context. However, it omits the required Related Issues reference and does not include a Where should the reviewer start section.


  • Fix all pre-merge checks with AI
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration

devin-ai-integration Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 37426311088 (attempt 1) on b8f3229a73

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 37426311088. Updated on every full-CI run of this PR.

Apply the session-affinity table's limits to the sticky min-load policy:
session IDs over 256 bytes are not bound and the table holds at most
65,536 bindings. Requests that cannot be bound take the least-loaded
worker.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/sticky-min-load-policy branch from 690f16f to 2919323 Compare October 3, 2026 22:22

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2919323. Approving with two P2 follow-up comments: expiry checks between sweeps and hard-affinity enforcement in standalone selection. Source review only; no tests or CI checks run.

@dynamo-review-agent dynamo-review-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously reported defects still present:

  • Original discussion: Idle expiry is still skipped between periodic sweeps: lookup refreshes last_access before checking the binding's age, so a binding older than max_idle_secs remains sticky whenever the sweep interval has not elapsed. Check expiry on lookup before refreshing the binding.
  • Original discussion: Standalone selection still passes an existing hard session-affinity target to this custom picker without restricting its candidates, and the picker ignores that target. It can select a different worker and make commit_session reject an otherwise booked request. Enforce the eligible hard target in the shared standalone host before invoking the custom picker.
  • Original discussion: The expiry defect is still present: select only evicts when the sweep interval has elapsed, then refreshes an existing binding before checking its age, so an expired binding can be reused and renewed when eviction_interval_secs is larger than max_idle_secs.
  • Original discussion: The hard affinity concern is still present: pick reads only the session ID and ignores context.affinity_target(), so a valid hard affinity target is not enforced before the sticky-min-load fallback path can select another worker.
  • Original discussion: The private test-helper doc comment at line 226 still only restates the (worker_id, active_requests) tuple type and helper implementation; the existing instruction to remove it remains applicable.
  • Original discussion: Verified still present for standalone selection with hard session affinity: the custom picker ignores context.affinity_target(), while the standalone host does not restrict its candidate set for this policy. After the policy-local binding expires or diverges, it can choose a different worker and commit_session rejects the booked request. Enforce the eligible hard-affinity target in the standalone host before invoking the custom picker.
  • Original discussion: Verified still present in standalone select-and-reserve: the picker ignores context.affinity_target(), and the policy does not enable exclusive affinity. When its local binding differs from a valid hard session-affinity target, selection can book another worker and commit_session rejects the request. Enforce the eligible hard target in the standalone host before invoking the custom picker.
  • Original discussion: Verified still present: a binding is refreshed before its age is checked. When max_idle_secs is shorter than eviction_interval_secs, requests arriving after the idle TTL but before the next sweep keep the stale worker indefinitely. Check and expire the looked-up binding before updating last_access.
  • Original discussion: Verified still present: a binding older than max_idle_secs remains valid until a sweep is due, and lookup refreshes last_access before checking its age. Sessions accessed less frequently than the configured TTL but more frequently than the sweep interval can therefore remain pinned indefinitely. Check expiry during lookup before refreshing the binding, while retaining periodic sweeps for cleanup.

Questions for the author:

  • lib/router-plugins/builtin/src/sticky_min_load.rs: At the documented 65,536-binding capacity, the request that crosses each eviction interval synchronously scans the entire HashMap while worker selection is serialized. Please provide a calculation or p99 selection-latency measurement for a full table, including mostly-live and mostly-expired bindings, to establish whether this periodic inference-path stall is acceptable.

A binding older than max_idle_secs could survive until the next periodic
sweep and keep routing its session. Check the binding's age on lookup and
rebind to the least-loaded worker when it has expired.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

@dynamo-review-agent dynamo-review-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously reported defects still present:

  • Original discussion: Still present: StickyMinLoadPicker::pick derives only the session ID from the context and calls select over the full candidate set, with no handling of an eligible context.affinity_target(). The previously raised hard-affinity/custom-policy conflict remains the active record.
  • Original discussion: The hard session-affinity issue remains present: StickyMinLoadPicker::pick reads only session_context() and ignores context.affinity_target(), while custom policies do not request exclusive affinity, so standalone select-and-reserve can select a worker different from an existing hard affinity target and then fail commit_session.
  • Original discussion: The hard-affinity concern is still present: pick reads only session_context().session_id() and passes the unrestricted candidate set into select, so an eligible context.affinity_target() can still be ignored by this custom policy before standalone commit validation.
  • Original discussion: The previously reported hard session-affinity issue is still present: the standalone selection path passes an existing hard affinity binding to custom policies as context.affinity_target(), but dynamo-sticky-min-load ignores that target and can choose its own least-loaded or policy-local sticky worker instead. When that differs from the active hard binding, commit_session rejects the already selected/booked request. The host-level fix described in the thread has not landed in this PR.
  • Original discussion: Verified still present for standalone select-and-reserve with hard session affinity: this policy ignores context.affinity_target(), and the custom-policy host does not restrict candidates unless the policy opts into exclusive affinity. If the sticky-min-load local binding expires, diverges, or is absent while the session-affinity table still has a valid hard target, the policy can book another worker and commit_session rejects the otherwise accepted request. Enforce the eligible hard-affinity target in the shared standalone host before invoking custom worker selection, as discussed in the original thread.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…oad-policy

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants