Repository navigation
feat(router): add native scorer and picker policies - #12526
Merged
ishandhanani merged 22 commits intoAug 7, 2026
Merged
Conversation
ishandhanani
force-pushed
the
idhanani/dyn-3712-native-worker-selection-api
branch
from
August 1, 2026 01:38
f6e5bf7 to
4a77505
Compare
|
🔗 Commit SHA: 344d95c | Docs | Datadog PR Page | Give us feedback! |
ishandhanani
marked this pull request as ready for review
August 3, 2026 18:43
ishandhanani
force-pushed
the
idhanani/dyn-3711-selection-service-worker-selector-injection
branch
from
August 3, 2026 18:43
5d0e148 to
b0121a5
Compare
ishandhanani
force-pushed
the
idhanani/dyn-3712-native-worker-selection-api
branch
from
August 3, 2026 18:43
e1487e5 to
90e3b80
Compare
ishandhanani
force-pushed
the
idhanani/dyn-3711-selection-service-worker-selector-injection
branch
2 times, most recently
from
August 3, 2026 23:35
d71bdf7 to
b83a0dc
Compare
ishandhanani
force-pushed
the
idhanani/dyn-3712-native-worker-selection-api
branch
from
August 3, 2026 23:43
90e3b80 to
4abe217
Compare
Base automatically changed from
idhanani/dyn-3711-selection-service-worker-selector-injection
to
idhanani/dyn-3709-default-score-pick
August 3, 2026 23:59
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
PeaBrane
approved these changes
Aug 5, 2026
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
MatejKosec
approved these changes
Aug 5, 2026
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Contributor
jthomson04
reviewed
Aug 5, 2026
jthomson04
left a comment
Contributor
There was a problem hiding this comment.
Two P2 findings from a source-only review.
jthomson04
approved these changes
Aug 5, 2026
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
enabled auto-merge (squash)
August 6, 2026 17:57
ishandhanani
added a commit
to ishandhanani/dynamo
that referenced
this pull request
Oct 14, 2026
Restore the per-candidate debug rows that left the production selector when scoring moved into the builtin policy plugin (ai-dynamo#12526). The host now logs one `Scored worker candidate` row per eligible worker and rank at DEBUG, stamped with request_id, worker_type, cost, and whether the row was selected, so `DYN_LOG=debug` again shows the full routing decision. The rows are cost-only because custom policies own their formula. An `enabled!` check keeps the loop off the hot path when DEBUG is off. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
added a commit
to ishandhanani/dynamo
that referenced
this pull request
Oct 15, 2026
Restore the per-candidate debug rows that left the production selector when scoring moved into the builtin policy plugin (ai-dynamo#12526). The host now logs one `Scored worker candidate` row per eligible worker and rank at DEBUG, stamped with request_id, worker_type, cost, and whether the row was selected, so `DYN_LOG=debug` again shows the full routing decision. The rows are cost-only because custom policies own their formula. An `enabled!` check keeps the loop off the hot path when DEBUG is off. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
added a commit
to ishandhanani/dynamo
that referenced
this pull request
Oct 15, 2026
Restore the per-candidate debug rows that left the production selector when scoring moved into the builtin policy plugin (ai-dynamo#12526). The host now logs one `Scored worker candidate` row per eligible worker and rank at DEBUG, stamped with request_id, worker_type, cost, and whether the row was selected, so `DYN_LOG=debug` again shows the full routing decision. The rows are cost-only because custom policies own their formula. An `enabled!` check keeps the loop off the hot path when DEBUG is off. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
added a commit
to ishandhanani/dynamo
that referenced
this pull request
Oct 15, 2026
Restore the per-candidate debug rows that left the production selector when scoring moved into the builtin policy plugin (ai-dynamo#12526). The host now logs one `Scored worker candidate` row per eligible worker and rank at DEBUG, stamped with request_id, worker_type, cost, and whether the row was selected, so `DYN_LOG=debug` again shows the full routing decision. The rows are cost-only because custom policies own their formula. An `enabled!` check keeps the loop off the hot path when DEBUG is off. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
added a commit
to ishandhanani/dynamo
that referenced
this pull request
Oct 16, 2026
Restore the per-candidate log rows that left the production selector when scoring moved into the builtin policy plugin (ai-dynamo#12526). The host now logs one `Scored worker candidate` row per eligible worker and rank at TRACE, stamped with request_id, worker_type, cost, and whether the row was selected, so a targeted filter such as `DYN_LOG='info,dynamo_kv_router::scheduling::selector=trace'` shows the whole routing decision again. TRACE rather than DEBUG because the rows are emitted on the scheduler actor and scale with the worker count. The rows are cost-only because custom policies own their formula. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds native scorer/picker policies and the construction seams needed to use them from Dynamo's HTTP frontend or EPP. Custom Rust strategies now plug into the same SelectionService path as the concrete default while Dynamo retains discovery, eligibility, scheduling, validation, accounting, reservations, metrics, and dispatch.
CLOSES: DYN-3712
CLOSES: DYN-3725
How This Was Implemented
WorkerScorer,WorkerPicker, compact borrowed worker inputs, and host-owned reusable scratch; custom policies declare only the inputs they consume.SelectionServiceBuilder; one policy instance is created perSelectionEntry, never per request.SelectionService, preserving its existing Kubernetes discovery, reconciliation, validation, and reservation lifecycle.Walkthrough
Mental model
The entrypoint chooses a selector implementation at startup, then all requests use Dynamo's existing selection pipeline. The default entrypoint constructs
DefaultWorkerSelector; a rebuilt custom image supplies normal Rust trait implementations instead.flowchart LR H["HTTP frontend or EPP"] --> S["SelectionService and policy-class scheduler"] S --> E["Host eligibility"] E --> P{"Worker policy"} P -->|default| D["Concrete scorer and picker"] P -->|custom| C["Linked Rust scorer and picker traits"] D --> V["Host index validation"] C --> V V --> A["Accounting, reservation, and dispatch"]State and ordering
Filters and host eligibility are unchanged and run before worker selection. Scorers add lower-is-better costs for eligible rows, the picker returns one row index, and host code validates that index before constructing the result or booking accounting state. Temperature-zero default selection keeps its existing streaming reservoir loop; nonzero temperature and opted-in custom policies reuse entry-owned scratch.
The policy factory runs once when each decode or prefill
SelectionEntryis built. Its policy state is therefore isolated per scheduling partition and reused across requests.Request lifecycle
Discovery creates the existing worker set and SelectionService entry. A request enters the existing policy-class queue, the scheduler computes host eligibility, the selected policy scores and picks, and Dynamo validates and reserves the worker. Completion, cancellation, and drop continue through the existing accounting cleanup paths.
For the HTTP frontend,
HttpFrontendis the common construction surface for both default and custom images. For EPP,EppRouter::from_selection_serviceaccepts a caller-built service and validates queue capacity and replica-sync requirements before Kubernetes discovery starts.Boundaries and limitations
This PR changes worker scoring and picking only; it does not publish a filter/eligibility API. It adds no policy registry, configuration graph, macros, runtime loading, C ABI dependency, or example policy. The standalone SGLang cache/load policy remains in #12536.
Validation
cargo test -p dynamo-kv-router --features standalone-selection services::selection::(53 passed)cargo test -p dynamo-ext-proc --lib(117 passed)cargo check -p dynamo-llm --testscargo clippy -p dynamo-kv-router --features standalone-selection --lib --tests -- -D warnings -A dead-codecargo clippy -p dynamo-llm --tests -- -D warningscargo clippy -p dynamo-ext-proc --all-targets -- -D warningscargo fmt --all -- --checkcargo metadata --locked --offline --no-depspre-commithooksBenchmark Results
Criterion A/B and B/A runs compare the concrete default before and after the native policy refactor. Deltas are this PR versus the prior implementation; lower is better.
The 2-worker, zero-temperature case is order-sensitive at nanosecond scale; its paired-run average was 1.84% faster. All larger zero-temperature cases and every nonzero-temperature case improved in both orders.
Summary by CodeRabbit