Skip to content

refactor(router): narrow worker policy API - #13008

Merged
ishandhanani merged 16 commits into
mainfrom
idhanani/dyn-3712-worker-policy-api-hardening
Aug 11, 2026
Merged

ishandhanani merged 16 commits into
mainfrom
idhanani/dyn-3712-worker-policy-api-hardening

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

This change narrows the Rust contract for custom worker scorers and pickers and adds strict rules for future API additions. The smaller contract lets Dynamo change its built-in routing formula without preserving intermediate values or default-policy components.

CLOSES: DYN-3712

How This Was Implemented

  • Kept DefaultWorkerScorer, DefaultWorkerPicker, WorkerInputs::ALL, and bitset inspection internal to the selector.
  • Removed public access to request IDs, raw prefill blocks, and effective overlap blocks from custom policies.
  • Kept raw device and shared-cache overlap separate from the fallback values used only by the built-in scorer.
  • Exposed a policy-facing SessionContext with the complete current session metadata. The internal AgentContext adapter is exhaustive, so a new protocol field requires an explicit API decision.
  • Added selector-local rules for public fields, optional data, input cost, host ownership, tests, documentation, and protocol-field mapping.
  • Reworked the Fern guide and examples with the routing pipeline, role-specific scoring, additive scorers, a session-aware picker, registration, YAML, Python frontend, and EPP workflows.
Walkthrough

Mental model

A custom policy owns scoring and picking. Dynamo continues to own worker eligibility, score validation, accounting, and reservation.

flowchart LR
    A["Request"] --> B["Host eligibility"]
    B --> C["Custom scorer(s)"]
    C --> D["Host sums finite costs"]
    D --> E["Custom picker"]
    E --> F["Host validates the row"]
    F --> G["Accounting and reservation"]
Loading

State and ordering

Each scorer requests only the CACHE, LOAD, or ROUTING groups that it reads. Dynamo calls every scorer for each eligible worker and adds their finite, lower-is-better costs. The picker receives the scored rows and the columns that it explicitly requested, then returns one row index.

The built-in scorer and picker remain private implementation details. An external policy supplies both components through WorkerSelectionPolicy::new.

Request lifecycle

The host constructs request context and candidate rows after eligibility filtering. The policy cannot restore an ineligible worker. The host rejects non-finite scores and invalid picker rows before accounting or reservation.

Boundaries and limitations

This change does not alter the default routing calculation or the low-level WorkerSelector path used across Dynamo crates.

Validation

  • cargo fmt --all -- --check
  • git diff --check
  • CARGO_TARGET_DIR=/ephemeral/dynamo/target cargo test -p dynamo-kv-router --features standalone-selection scheduling::selector (27 passed)
  • CARGO_TARGET_DIR=/ephemeral/dynamo/target cargo test -p dynamo-mocker session_identity_reaches_scheduling_request --lib
  • CARGO_TARGET_DIR=/ephemeral/dynamo/target cargo check -p dynamo-llm --lib
  • CARGO_TARGET_DIR=/ephemeral/dynamo/target cargo check -p dynamo-llm --tests
  • CARGO_TARGET_DIR=/ephemeral/dynamo/target cargo test -p dynamo-custom-policy-example-basic -p dynamo-custom-policy-example-disaggregated -p dynamo-custom-policy-example-catalog
  • CARGO_TARGET_DIR=/ephemeral/dynamo/target cargo build -p dynamo-custom-policy-example-epp
  • pre-commit run --files <changed files>
  • cd docs && fern check --warnings
  • cd docs && fern docs broken-links

Summary by CodeRabbit

  • Documentation

    • Reworked custom worker-selection guides with comprehensive implementation, configuration, validation, testing, and integration instructions.
    • Added guidance for designing and extending worker-selection policies, including supported inputs and required safeguards.
  • Refactor

    • Restricted internal worker-selection implementations and APIs to reduce unintended public exposure.
    • Retained the default worker selector as the supported public entry point.

@github-actions github-actions Bot added refactor documentation Improvements or additions to documentation labels Aug 11, 2026
@github-actions

github-actions Bot commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

@ishandhanani ishandhanani left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass

@datadog-official

This comment has been minimized.

@ishandhanani
ishandhanani marked this pull request as ready for review August 11, 2026 07:09
@ishandhanani
ishandhanani requested review from a team as code owners August 11, 2026 07:09
devin-ai-integration[bot]

This comment was marked as resolved.

@coderabbitai

coderabbitai Bot commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The PR narrows public selector internals and expands documentation for custom scorer and picker implementation, registration, configuration, validation, Python frontend integration, and EPP execution.

Changes

Custom worker-selection policies

Layer / File(s) Summary
Selector API boundary
lib/kv-router/src/lib.rs, lib/kv-router/src/scheduling/CLAUDE.md, lib/kv-router/src/scheduling/selector/*
Default scorer and picker types are restricted to internal visibility. Selected policy inputs and accessors are no longer public. Tests use device overlap data.
Policy authoring and registration
docs/fern/pages/developer-guide/advanced-customizations/custom-worker-selection.mdx, examples/router/custom-policy-example/README.md
The guides describe scorer and picker implementations, signal declarations, parameter parsing, factories, registration, YAML configuration, and startup validation.
Frontend and EPP integration
docs/fern/pages/developer-guide/advanced-customizations/custom-worker-selection.mdx, examples/router/custom-policy-example/README.md
The guides document linked Python frontend builds, embedded selection behavior, EPP catalog registration, standalone execution, policy contracts, testing, and benchmarking.

Estimated code review effort: 3 (Moderate) | ~25 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ⚠️ Warning The description explains the implementation and validation, but it omits the required Related Issues template and reviewer-start section. Add the required Related Issues section with a valid issue reference or no-issue confirmation, and identify where reviewers should start.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the primary change: narrowing the router worker policy API.

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]

This comment was marked as resolved.

@ishandhanani
ishandhanani requested a review from a team as a code owner August 11, 2026 07:33
@github-actions github-actions Bot added the router Relates to routing, KV-aware routing, etc. label Aug 11, 2026
dynamo-review-agent[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

dynamo-review-agent[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@PeaBrane PeaBrane left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ AI reviewed

@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3712-worker-policy-api-hardening branch from d8c7d06 to 76b0bd1 Compare August 11, 2026 18:24
devin-ai-integration[bot]

This comment was marked as resolved.

@ishandhanani
ishandhanani requested a review from a team as a code owner August 11, 2026 18:57
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3712-worker-policy-api-hardening branch from 889e730 to da253b8 Compare August 11, 2026 20:47
@ishandhanani
ishandhanani merged commit 0824598 into main Aug 11, 2026
122 checks passed
@ishandhanani
ishandhanani deleted the idhanani/dyn-3712-worker-policy-api-hardening branch August 11, 2026 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation refactor router Relates to routing, KV-aware routing, etc. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants