Skip to content

feat(router): add custom worker-selection policy catalog - #12536

Merged
ishandhanani merged 18 commits into
mainfrom
idhanani/dyn-3713-sglang-cache-load-policy
Aug 10, 2026
Merged

ishandhanani merged 18 commits into
mainfrom
idhanani/dyn-3713-sglang-cache-load-policy

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Aug 1, 2026 •

Copy link
Copy Markdown
Contributor

Adds an optional build-time catalog for native Rust worker-selection policies, selected through named YAML instances by the KV HTTP frontend, standalone selection service, and standalone EPP. An organization replaces one fixed dependency alias in its own build, while stock Dynamo artifacts retain the built-in selector.

CLOSES: DYN-3713

How This Was Implemented

  • Keeps WorkerScorer and WorkerPicker composition separate from Dynamo-owned eligibility, validation, accounting, reservations, and metrics.
  • Adds one optional catalog dependency alias, which a private build replaces with a crate registering policy types from one or more policy crates.
  • Adds worker_selection instances to router-policy YAML, with an environment override for the selected instance.
  • Resolves the factory once at startup through the existing HttpFrontend, SelectionServiceBuilder, and standalone EPP paths.
  • Rejects a selected linked instance in stock artifacts; local standalone EPP can use the same Cargo alias, but Dynamo does not provide a configurable EPP Docker build yet.
Walkthrough

Mental model

Policy implementations are linked into a private build; YAML configures and selects only those linked implementations. One catalog crate can depend on policy crates from separate repositories and register all their policy types.

flowchart LR
    C["Catalog crate"] --> R["Policy type registry"]
    Y["Router-policy YAML"] --> K["KvRouterConfig"]
    E["Environment override"] --> K
    K --> R
    R --> F["Selected partition factory"]
    F --> S["Frontend, selection service, or standalone EPP"]
    S --> H["Host eligibility, scheduling, validation, accounting"]
    H --> P["Linked scorer and picker"]
Loading

State and ordering

worker_selection.instances maps a user-chosen instance name to a registered type and a YAML parameter mapping. The YAML default selects the process-wide instance; DYN_ROUTER_WORKER_SELECTION_POLICY overrides it; unset or default retains Dynamo's built-in selector. The provider parses and validates parameters once at startup, and the resulting factory creates one policy per worker-type/routing partition, not per request.

Request lifecycle

Only KV HTTP frontend startup applies a linked factory; a selected instance makes non-HTTP Python inputs fail before startup because they do not build HttpFrontend or SelectionService. dynamo.select_service passes the same resolved factory to SelectionServiceBuilder; a selected EPP policy is supported only in standalone mode, where health and metrics start before the policy-backed selector completes discovery. Stock Python bindings and stock EPP reject a selected linked instance rather than silently routing with the built-in selector.

Boundaries and limitations

This does not dynamically load Rust code, offer a runtime registry, or make selection policy names part of the request path. One process selects one worker-selection instance; a custom EPP build without a selected linked instance can still run Dynamo-runtime EPP mode. A policy-enabled EPP image must currently be packaged by the organization's own build; Dynamo's EPP Dockerfile has no catalog build surface.

Validation

  • cargo fmt --all -- --check
  • cargo check --manifest-path deploy/inference-gateway/ext-proc/Cargo.toml
  • cargo test --locked -p dynamo-ext-proc --lib (119 passed)
  • cargo check --manifest-path lib/bindings/python/Cargo.toml --features custom-policy,select-service
  • cargo test -p dynamo-kv-router (836 passed)
  • fern check && fern docs broken-links from docs/fern

Benchmark Results

Scenario Stock/main Catalog build Delta
Default selector, 10,000 workers, temperature 0.0 567.229 us 565.161 us -0.37%
Default selector, 10,000 workers, temperature 0.7 638.875 us 638.541 us -0.05%
EPP TTFT, two Qwen3-0.6B workers 87.34 ms 79.02 ms -9.53%
EPP ITL, two Qwen3-0.6B workers 3.437 ms 2.830 ms -17.66%
EPP request throughput, two Qwen3-0.6B workers 36.16 RPS 39.22 RPS +8.48%

The selector control uses three interleaved Criterion runs per arm; the catalog build has no measurable regression at scale. The EPP comparison uses an external DefaultWorkerScorer plus DefaultWorkerPicker policy, two ABBA phases per arm, and 64 clean requests per arm; it establishes no apparent catalog overhead, not a performance-improvement claim.

Summary by CodeRabbit

  • New Features

    • Added support for compile-time custom worker-selection policies in Python runtimes and standalone inference gateways.
    • Added YAML configuration for named worker-selection instances, policy parameters, and default selections.
    • Added environment-variable overrides for selecting worker policies at startup.
    • Added policy validation and clearer errors for unsupported, missing, or invalid configurations.
  • Documentation

    • Added comprehensive guidance and examples for creating, configuring, and deploying custom worker-selection policies.
    • Updated router configuration references and navigation links.

@github-actions github-actions Bot added feat documentation Improvements or additions to documentation labels Aug 1, 2026
@github-actions

github-actions Bot commented Aug 1, 2026 •

Copy link
Copy Markdown
Contributor

@datadog-official

datadog-official Bot commented Aug 1, 2026 •

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 2 Pipeline jobs failed

PR | backend-status-check   View in Datadog   GitHub Actions

PR | sglang-runtime / Test cuda13.0, amd64   View in Datadog   GitHub Actions

See error Connection refused while trying to reach 172.20.0.1:443.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a5edbd4 | Docs | Datadog PR Page | Give us feedback!

@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3713-sglang-cache-load-policy branch 2 times, most recently from d8813f5 to 0250fe8 Compare August 3, 2026 02:13
@github-actions github-actions Bot added the router Relates to routing, KV-aware routing, etc. label Aug 3, 2026
@ishandhanani ishandhanani changed the title feat(router): add external native selection policy example feat(router): add SGLang cache-load policy example Aug 3, 2026
@ishandhanani
ishandhanani changed the base branch from idhanani/dyn-3712-native-worker-selection-api to idhanani/dyn-3713-custom-policy-frontend August 3, 2026 02:45
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3713-custom-policy-frontend branch from 8a1adbc to a0bbf8d Compare August 3, 2026 20:26
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3713-sglang-cache-load-policy branch from 0250fe8 to 68e0e6a Compare August 3, 2026 20:35
@pull-request-size pull-request-size Bot added size/L and removed size/XL labels Aug 3, 2026
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3713-sglang-cache-load-policy branch 2 times, most recently from 6ecc6a2 to 8fa12bc Compare August 4, 2026 21:16
@copy-pr-bot

copy-pr-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added backend::vllm Relates to the vllm backend deployment::k8s Relates to dynamo deployment in kubernetes backend::sglang Relates to the sglang backend backend::trtllm Relates to the trtllm backend planner frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal container labels Aug 4, 2026
@ishandhanani
ishandhanani enabled auto-merge (squash) August 9, 2026 04:14
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Contributor Author

/ok to test ef1980c

Comment thread lib/router-plugins/catalog/Cargo.toml Resolved
Comment thread .github/workflows/pre-merge.yml Resolved
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Contributor Author

/ok-to-test db2f22f

@rmccorm4 rmccorm4 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving for @ai-dynamo/dynamo-process-codeowners

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Contributor Author

/ok-to-test 5574fc3

@ishandhanani

Copy link
Copy Markdown
Contributor Author

/ok-to-test 55f5bfc

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions backend::sglang Relates to the sglang backend backend::trtllm Relates to the trtllm backend backend::vllm Relates to the vllm backend container deployment::k8s Relates to dynamo deployment in kubernetes documentation Improvements or additions to documentation feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal planner router Relates to routing, KV-aware routing, etc. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants