Repository navigation
feat(router): add custom worker-selection policy catalog - #12536
Merged
Merged
Conversation
Contributor
|
ishandhanani
force-pushed
the
idhanani/dyn-3713-sglang-cache-load-policy
branch
2 times, most recently
from
August 3, 2026 02:13
d8813f5 to
0250fe8
Compare
ishandhanani
changed the base branch from
idhanani/dyn-3712-native-worker-selection-api
to
idhanani/dyn-3713-custom-policy-frontend
August 3, 2026 02:45
ishandhanani
force-pushed
the
idhanani/dyn-3713-custom-policy-frontend
branch
from
August 3, 2026 20:26
8a1adbc to
a0bbf8d
Compare
ishandhanani
force-pushed
the
idhanani/dyn-3713-sglang-cache-load-policy
branch
from
August 3, 2026 20:35
0250fe8 to
68e0e6a
Compare
ishandhanani
force-pushed
the
idhanani/dyn-3713-sglang-cache-load-policy
branch
2 times, most recently
from
August 4, 2026 21:16
6ecc6a2 to
8fa12bc
Compare
ishandhanani
enabled auto-merge (squash)
August 9, 2026 04:14
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Contributor
Author
|
/ok to test ef1980c |
nv-tusharma
reviewed
Aug 10, 2026
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
nv-tusharma
approved these changes
Aug 10, 2026
Contributor
Author
|
/ok-to-test db2f22f |
rmccorm4
approved these changes
Aug 10, 2026
rmccorm4
left a comment
Contributor
There was a problem hiding this comment.
Approving for @ai-dynamo/dynamo-process-codeowners
alec-flowers
approved these changes
Aug 10, 2026
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Contributor
Author
|
/ok-to-test 5574fc3 |
Contributor
Author
|
/ok-to-test 55f5bfc |
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Closed
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an optional build-time catalog for native Rust worker-selection policies, selected through named YAML instances by the KV HTTP frontend, standalone selection service, and standalone EPP. An organization replaces one fixed dependency alias in its own build, while stock Dynamo artifacts retain the built-in selector.
CLOSES: DYN-3713
How This Was Implemented
WorkerScorerandWorkerPickercomposition separate from Dynamo-owned eligibility, validation, accounting, reservations, and metrics.worker_selectioninstances to router-policy YAML, with an environment override for the selected instance.HttpFrontend,SelectionServiceBuilder, and standalone EPP paths.Walkthrough
Mental model
Policy implementations are linked into a private build; YAML configures and selects only those linked implementations. One catalog crate can depend on policy crates from separate repositories and register all their policy types.
flowchart LR C["Catalog crate"] --> R["Policy type registry"] Y["Router-policy YAML"] --> K["KvRouterConfig"] E["Environment override"] --> K K --> R R --> F["Selected partition factory"] F --> S["Frontend, selection service, or standalone EPP"] S --> H["Host eligibility, scheduling, validation, accounting"] H --> P["Linked scorer and picker"]State and ordering
worker_selection.instancesmaps a user-chosen instance name to a registered type and a YAML parameter mapping. The YAMLdefaultselects the process-wide instance;DYN_ROUTER_WORKER_SELECTION_POLICYoverrides it; unset ordefaultretains Dynamo's built-in selector. The provider parses and validates parameters once at startup, and the resulting factory creates one policy per worker-type/routing partition, not per request.Request lifecycle
Only KV HTTP frontend startup applies a linked factory; a selected instance makes non-HTTP Python inputs fail before startup because they do not build
HttpFrontendorSelectionService.dynamo.select_servicepasses the same resolved factory toSelectionServiceBuilder; a selected EPP policy is supported only in standalone mode, where health and metrics start before the policy-backed selector completes discovery. Stock Python bindings and stock EPP reject a selected linked instance rather than silently routing with the built-in selector.Boundaries and limitations
This does not dynamically load Rust code, offer a runtime registry, or make selection policy names part of the request path. One process selects one worker-selection instance; a custom EPP build without a selected linked instance can still run Dynamo-runtime EPP mode. A policy-enabled EPP image must currently be packaged by the organization's own build; Dynamo's EPP Dockerfile has no catalog build surface.
Validation
cargo fmt --all -- --checkcargo check --manifest-path deploy/inference-gateway/ext-proc/Cargo.tomlcargo test --locked -p dynamo-ext-proc --lib(119 passed)cargo check --manifest-path lib/bindings/python/Cargo.toml --features custom-policy,select-servicecargo test -p dynamo-kv-router(836 passed)fern check && fern docs broken-linksfromdocs/fernBenchmark Results
The selector control uses three interleaved Criterion runs per arm; the catalog build has no measurable regression at scale. The EPP comparison uses an external
DefaultWorkerScorerplusDefaultWorkerPickerpolicy, two ABBA phases per arm, and 64 clean requests per arm; it establishes no apparent catalog overhead, not a performance-improvement claim.Summary by CodeRabbit
New Features
Documentation