Skip to content

feat(router): add native scorer and picker policies - #12526

Merged
ishandhanani merged 22 commits into
mainfrom
idhanani/dyn-3712-native-worker-selection-api
Aug 7, 2026
Merged

ishandhanani merged 22 commits into
mainfrom
idhanani/dyn-3712-native-worker-selection-api

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Aug 1, 2026 •

Copy link
Copy Markdown
Contributor

Adds native scorer/picker policies and the construction seams needed to use them from Dynamo's HTTP frontend or EPP. Custom Rust strategies now plug into the same SelectionService path as the concrete default while Dynamo retains discovery, eligibility, scheduling, validation, accounting, reservations, metrics, and dispatch.

CLOSES: DYN-3712
CLOSES: DYN-3725

How This Was Implemented

  • Publishes WorkerScorer, WorkerPicker, compact borrowed worker inputs, and host-owned reusable scratch; custom policies declare only the inputs they consume.
  • Runs default and custom policies through the same host selection function, with concrete static dispatch and no candidate-vector construction retained for the default path.
  • Adds a startup-time policy factory to SelectionServiceBuilder; one policy instance is created per SelectionEntry, never per request.
  • Threads that factory through the existing HTTP watcher, decode/prefill routers, and scheduler without adding a registry, runtime loading, or request-path type erasure.
  • Lets EPP callers supply a prebuilt SelectionService, preserving its existing Kubernetes discovery, reconciliation, validation, and reservation lifecycle.
Walkthrough

Mental model

The entrypoint chooses a selector implementation at startup, then all requests use Dynamo's existing selection pipeline. The default entrypoint constructs DefaultWorkerSelector; a rebuilt custom image supplies normal Rust trait implementations instead.

flowchart LR
    H["HTTP frontend or EPP"] --> S["SelectionService and policy-class scheduler"]
    S --> E["Host eligibility"]
    E --> P{"Worker policy"}
    P -->|default| D["Concrete scorer and picker"]
    P -->|custom| C["Linked Rust scorer and picker traits"]
    D --> V["Host index validation"]
    C --> V
    V --> A["Accounting, reservation, and dispatch"]
Loading

State and ordering

Filters and host eligibility are unchanged and run before worker selection. Scorers add lower-is-better costs for eligible rows, the picker returns one row index, and host code validates that index before constructing the result or booking accounting state. Temperature-zero default selection keeps its existing streaming reservoir loop; nonzero temperature and opted-in custom policies reuse entry-owned scratch.

The policy factory runs once when each decode or prefill SelectionEntry is built. Its policy state is therefore isolated per scheduling partition and reused across requests.

Request lifecycle

Discovery creates the existing worker set and SelectionService entry. A request enters the existing policy-class queue, the scheduler computes host eligibility, the selected policy scores and picks, and Dynamo validates and reserves the worker. Completion, cancellation, and drop continue through the existing accounting cleanup paths.

For the HTTP frontend, HttpFrontend is the common construction surface for both default and custom images. For EPP, EppRouter::from_selection_service accepts a caller-built service and validates queue capacity and replica-sync requirements before Kubernetes discovery starts.

Boundaries and limitations

This PR changes worker scoring and picking only; it does not publish a filter/eligibility API. It adds no policy registry, configuration graph, macros, runtime loading, C ABI dependency, or example policy. The standalone SGLang cache/load policy remains in #12536.

Validation

  • cargo test -p dynamo-kv-router --features standalone-selection services::selection:: (53 passed)
  • cargo test -p dynamo-ext-proc --lib (117 passed)
  • cargo check -p dynamo-llm --tests
  • cargo clippy -p dynamo-kv-router --features standalone-selection --lib --tests -- -D warnings -A dead-code
  • cargo clippy -p dynamo-llm --tests -- -D warnings
  • cargo clippy -p dynamo-ext-proc --all-targets -- -D warnings
  • cargo fmt --all -- --check
  • cargo metadata --locked --offline --no-deps
  • Changed-file pre-commit hooks

Benchmark Results

Criterion A/B and B/A runs compare the concrete default before and after the native policy refactor. Deltas are this PR versus the prior implementation; lower is better.

Workers Temperature A/B delta B/A delta
2 0.0 +2.82% -6.15%
32 0.0 -3.85% -5.67%
1,024 0.0 -5.68% -6.38%
10,000 0.0 -5.35% -5.13%
2 0.7 -17.79% -21.60%
32 0.7 -29.41% -29.20%
1,024 0.7 -21.23% -21.61%
10,000 0.7 -19.05% -18.72%

The 2-worker, zero-temperature case is order-sensitive at nanosecond scale; its paired-run average was 1.84% faster. All larger zero-temperature cases and every nonzero-temperature case improved in both orders.

Summary by CodeRabbit

  • New Features
    • Added configurable worker-selection policies for routing requests based on cache, load, and routing information.
    • Added support for custom routing policies through the HTTP frontend.
    • Expanded scheduling interfaces for advanced worker selection and scoring.
    • Added service status checks for queueing and replica synchronization.
  • Bug Fixes
    • Improved validation and error handling for invalid worker-selection results, overloaded workers, and unsupported routing configurations.
    • Improved sampling behavior for extreme costs and zero-temperature settings.

@github-actions github-actions Bot added the feat label Aug 1, 2026
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3712-native-worker-selection-api branch from f6e5bf7 to 4a77505 Compare August 1, 2026 01:38
@datadog-official

datadog-official Bot commented Aug 1, 2026 •

Copy link
Copy Markdown

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 344d95c | Docs | Datadog PR Page | Give us feedback!

@ishandhanani
ishandhanani marked this pull request as ready for review August 3, 2026 18:43
@ishandhanani
ishandhanani requested a review from a team as a code owner August 3, 2026 18:43
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3711-selection-service-worker-selector-injection branch from 5d0e148 to b0121a5 Compare August 3, 2026 18:43
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3712-native-worker-selection-api branch from e1487e5 to 90e3b80 Compare August 3, 2026 18:43
@copy-pr-bot

copy-pr-bot Bot commented Aug 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

devin-ai-integration[bot]

This comment was marked as resolved.

@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3711-selection-service-worker-selector-injection branch 2 times, most recently from d71bdf7 to b83a0dc Compare August 3, 2026 23:35
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3712-native-worker-selection-api branch from 90e3b80 to 4abe217 Compare August 3, 2026 23:43
devin-ai-integration[bot]

This comment was marked as resolved.

Base automatically changed from idhanani/dyn-3711-selection-service-worker-selector-injection to idhanani/dyn-3709-default-score-pick August 3, 2026 23:59
@ishandhanani
ishandhanani requested review from a team as code owners August 3, 2026 23:59
devin-ai-integration[bot]

This comment was marked as resolved.

Base automatically changed from idhanani/dyn-3709-default-score-pick to main August 4, 2026 20:46
@nv-tusharma
nv-tusharma requested a review from a team as a code owner August 4, 2026 20:46
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

@PeaBrane PeaBrane left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅

devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani requested a review from a team as a code owner August 5, 2026 18:16
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 5, 2026
@github-actions

github-actions Bot commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

devin-ai-integration[bot]

This comment was marked as resolved.

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two P2 findings from a source-only review.

@ishandhanani
ishandhanani enabled auto-merge (squash) August 6, 2026 17:57
@ishandhanani
ishandhanani merged commit 3d37771 into main Aug 7, 2026
136 of 165 checks passed
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Oct 14, 2026
Restore the per-candidate debug rows that left the production selector
when scoring moved into the builtin policy plugin (ai-dynamo#12526). The host now
logs one `Scored worker candidate` row per eligible worker and rank at
DEBUG, stamped with request_id, worker_type, cost, and whether the row
was selected, so `DYN_LOG=debug` again shows the full routing decision.
The rows are cost-only because custom policies own their formula. An
`enabled!` check keeps the loop off the hot path when DEBUG is off.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Oct 15, 2026
Restore the per-candidate debug rows that left the production selector
when scoring moved into the builtin policy plugin (ai-dynamo#12526). The host now
logs one `Scored worker candidate` row per eligible worker and rank at
DEBUG, stamped with request_id, worker_type, cost, and whether the row
was selected, so `DYN_LOG=debug` again shows the full routing decision.
The rows are cost-only because custom policies own their formula. An
`enabled!` check keeps the loop off the hot path when DEBUG is off.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Oct 15, 2026
Restore the per-candidate debug rows that left the production selector
when scoring moved into the builtin policy plugin (ai-dynamo#12526). The host now
logs one `Scored worker candidate` row per eligible worker and rank at
DEBUG, stamped with request_id, worker_type, cost, and whether the row
was selected, so `DYN_LOG=debug` again shows the full routing decision.
The rows are cost-only because custom policies own their formula. An
`enabled!` check keeps the loop off the hot path when DEBUG is off.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Oct 15, 2026
Restore the per-candidate debug rows that left the production selector
when scoring moved into the builtin policy plugin (ai-dynamo#12526). The host now
logs one `Scored worker candidate` row per eligible worker and rank at
DEBUG, stamped with request_id, worker_type, cost, and whether the row
was selected, so `DYN_LOG=debug` again shows the full routing decision.
The rows are cost-only because custom policies own their formula. An
`enabled!` check keeps the loop off the hot path when DEBUG is off.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Oct 16, 2026
Restore the per-candidate log rows that left the production selector
when scoring moved into the builtin policy plugin (ai-dynamo#12526). The host now
logs one `Scored worker candidate` row per eligible worker and rank at
TRACE, stamped with request_id, worker_type, cost, and whether the row
was selected, so a targeted filter such as
`DYN_LOG='info,dynamo_kv_router::scheduling::selector=trace'` shows the
whole routing decision again. TRACE rather than DEBUG because the rows
are emitted on the scheduler actor and scale with the worker count. The
rows are cost-only because custom policies own their formula.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat router Relates to routing, KV-aware routing, etc. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants