Skip to content

refactor(router): clarify disaggregated policy dispatch - #13155

Merged
ishandhanani merged 30 commits into
mainfrom
codex/clarify-disagg-policy-stages
Aug 14, 2026
Merged

ishandhanani merged 30 commits into
mainfrom
codex/clarify-disagg-policy-stages

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

TDLR

Worker role Before this PR Current
Prefill "prefill" Prefill → "prefill"
Decode "decode" Decode → "decode"
Aggregated Hard-coded "decode" Aggregated → "decode"
Encode Effectively "decode" Encode → "decode"

The stock path still uses DefaultWorkerSelector directly. The new match runs once when the worker pool is constructed, not per request.

Custom worker-selection policies can select explicit aggregated, prefill, decode, and encode instances using Dynamo's canonical typed WorkerType. Policy resolution and typed dispatch happen during startup or worker-pool construction, so the stock request path does not gain role matching or policy lookup work.

How This Was Implemented

  • Move the canonical WorkerType to dynamo-kv-router and re-export it from dynamo-llm for source compatibility.
  • Add independent worker_selection.aggregated, worker_selection.prefill, worker_selection.decode, and worker_selection.encode YAML selections for every current worker type.
  • Add prefill/decode CLI and environment overrides plus the existing global override; unspecified roles use Dynamo's built-in selector.
  • Resolve named providers once at startup, cache shared factories, and dispatch factories through an exhaustive typed match when each worker pool is constructed.
  • Preserve the built-in selector's historical scoring labels and the existing scheduler, discovery, validation, LoRA, and accounting behavior.
Walkthrough

Mental model

Configuration selects a named policy instance for each worker role. The host supplies a typed role when it constructs a routing pool, and the registry returns either that role's custom policy or Dynamo's built-in selector.

flowchart LR
    Config["YAML / environment / CLI"] --> Resolve["Resolve named instances at startup"]
    Card["Structured model card"] --> Role["Derive typed WorkerType"]
    Discovery["Discovery constructs worker pool"] --> Role
    Resolve --> Factory["Cached policy factories"]
    Role --> Dispatch["Typed role dispatch at pool construction"]
    Factory --> Dispatch
    Dispatch --> Selector["Existing selector and scheduler"]
Loading

State and ordering

For prefill and decode, precedence is the role-specific CLI or environment override, DYN_ROUTER_WORKER_SELECTION_POLICY, the matching YAML role, then the built-in selector. For aggregated and encode pools, precedence is DYN_ROUTER_WORKER_SELECTION_POLICY, the matching YAML role, then the built-in selector.

The reserved selection value default selects Dynamo's built-in policy for that role. A role without an explicit selection also uses the built-in selector. Because this configuration has not shipped, there is no legacy worker_selection.default YAML alias or compatibility fallback; that key is rejected by schema validation.

When multiple roles select the same named instance, the registry resolves its provider once and shares the resulting factory. The exhaustive WorkerType::{Aggregated, Prefill, Decode, Encode} match gives every current worker type an independent YAML choice and forces new enum variants to define their selection behavior.

Request lifecycle

The linked registry parses and validates configuration before serving traffic. The embedded frontend receives typed roles from discovery. python -m dynamo.router waits for a structured worker model card only when remote/served indexing needs the model name or a linked custom policy needs the typed role; DYN_ROUTER_MODEL_CARD_WAIT_SECS bounds that startup wait and defaults to 600 seconds. The stock standalone router retains its previous nonblocking discovery snapshot.

The example policy matches directly on WorkerType::Prefill and WorkerType::Decode to build separate filter/scorer/picker compositions. Strings do not decide which custom policy is selected.

After pool construction, requests use the existing selector, scheduling, reservation, and accounting paths. No role parsing, provider lookup, registry lookup, allocation, lock, callback registration, or new background task was added to request processing.

Boundaries and limitations

Standalone EPP has one full-request worker pool, supplies WorkerType::Aggregated, and uses worker_selection.aggregated. It warns when explicit prefill, decode, or encode selections are ignored; a policy designed only for disaggregated prefill/decode pools can reject Aggregated with a typed selection error.

worker_selection.encode applies to surface-carrying encode worker sets that construct the standard KV chooser. The surface-less multimodal encoder hop exits earlier and uses EncoderRouter round-robin routing, outside worker-selection policy dispatch.

The built-in selector historically distinguishes prefill from all other typed pools. WorkerType::default_selector_label keeps prefill on the prefill scorer and decode, aggregated, and encode on the decode scorer. The Python standalone router separately preserves its pre-existing endpoint identity and router_track_active_blocks inference for metric labels and LoRA load-feed behavior; that path does not select custom policies.

The stock/default-only frontend remains monomorphized on DefaultWorkerSelector. New parsing, custom-policy model-card role discovery, provider resolution, and typed dispatch are startup or pool-construction work, not steady-state request work.

Validation

  • cargo fmt -- --check in ., lib/bindings/python, lib/runtime/examples, and lib/bindings/kvbm
  • cargo test --manifest-path examples/router/custom-policy-example/disagg-filter-score-pick/Cargo.toml (3 passed)
  • cargo test -p dynamo-kv-router worker_selection (6 passed)
  • cargo test -p dynamo-kv-router worker_type (8 passed)
  • cargo test -p dynamo-kv-router --features standalone-selection --lib services::selection::policy_registry::tests (3 passed)
  • cargo test -p dynamo-ext-proc runner::tests (3 passed)
  • git diff --check

Benchmark Results

Paired Criterion measurements compare direct built-in selection with the wrapped built-in policy used by a configured custom-policy process. Negative deltas favor the wrapped path; the results show no systematic steady-state regression.

Workers Direct built-in Wrapped policy Delta
2 291.31 ns 289.86 ns -1.45 ns (-0.50%)
32 2.9329 us 2.9205 us -0.0124 us (-0.42%)
1,024 93.651 us 93.611 us -0.040 us (-0.04%)

Summary by CodeRabbit

  • New Features

    • Added separate worker-selection policies for prefill and decode stages.
    • Configure stage-specific policies through YAML, command-line options, or environment variables.
    • Added clear precedence rules: command-line settings override environment variables and configuration values.
    • Policies now support stage-specific selection with fallback to the default policy.
    • Custom policy examples and Python startup configuration now demonstrate separate prefill and decode policies.
  • Documentation

    • Updated advanced customization and custom policy guides with configuration, override behavior, and standalone EPP limitations.

CLOSES: DYN-3883

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@github-actions github-actions Bot added refactor documentation Improvements or additions to documentation labels Aug 13, 2026
@github-actions

github-actions Bot commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

ishandhanani and others added 2 commits August 13, 2026 04:12
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani marked this pull request as ready for review August 13, 2026 05:12
@ishandhanani
ishandhanani requested review from a team as code owners August 13, 2026 05:12
@coderabbitai

coderabbitai Bot commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 105459fc-4f4d-4395-8979-76cc2355443d

📥 Commits

Reviewing files that changed from the base of the PR and between cdc400c and 84588fa.

📒 Files selected for processing (11)
  • components/src/dynamo/common/configuration/groups/kv_router_args.py
  • components/src/dynamo/common/tests/configuration/test_kv_router_args.py
  • docs/fern/pages/developer-guide/advanced-customizations/custom-worker-selection.mdx
  • examples/router/custom-policy-example/README.md
  • examples/router/custom-policy-example/disagg-filter-score-pick/src/lib.rs
  • lib/bindings/python/rust/llm/entrypoint.rs
  • lib/kv-router/src/scheduling/config.rs
  • lib/kv-router/src/scheduling/policy_config.rs
  • lib/kv-router/src/scheduling/worker_selection_config.rs
  • lib/kv-router/src/services/selection/mod.rs
  • lib/kv-router/src/services/selection/policy_registry.rs

Walkthrough

The KV router now supports independent prefill and decode worker-selection policies. Configuration precedence, Rust resolution, registry dispatch, Python bindings, CLI options, tests, examples, and documentation were updated.

Changes

Stage-specific worker-selection policies

Layer / File(s) Summary
Configuration and precedence resolution
lib/kv-router/src/scheduling/...
YAML, environment, and frontend settings now resolve fallback, prefill, and decode policy instances with validation and serialization coverage.
Policy registry dispatch
lib/kv-router/src/services/selection/...
The registry caches selected instances and dispatches policies by worker type.
Frontend configuration wiring
components/src/dynamo/common/configuration/..., components/src/dynamo/common/tests/..., lib/bindings/python/rust/llm/entrypoint.rs
CLI and Python configuration now accept and propagate separate prefill and decode policy overrides.
Custom policy example and documentation
examples/router/custom-policy-example/..., docs/fern/pages/developer-guide/advanced-customizations/custom-worker-selection.mdx
The example policy factory, launch commands, YAML, and documentation now use separate prefill and decode policy instances.

Estimated code review effort: 4 (Complex) | ~45 minutes

Mergeability Score: ⚪ Minimal · up to 84588

The change makes prefill/decode policy selection explicit and rejects unsupported stages instead of silently falling back; no actionable merge-blocking risk remains, so it is merge-ready after normal checks and review.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 78.85% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly describes the main change: clarifying dispatch for disaggregated custom policies.
Description check ✅ Passed The description thoroughly explains the implementation, behavior, boundaries, validation, and related issue, although it does not follow every template heading.

Comment @coderabbitai help to get the list of available commands.

devin-ai-integration[bot]

This comment was marked as resolved.

@datadog-official

This comment has been minimized.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani requested a review from a team as a code owner August 13, 2026 08:23
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

@jthomson04 jthomson04 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: withdrawn. Commit 670ec7ac08 documents the intended boundary: worker_selection.encode applies to surface-carrying Encode worker sets that construct the standard KV chooser, while the surface-less multimodal encoder hop intentionally remains on EncoderRouter round-robin selection. That contract matches the implementation, so this is not a remaining P1. The related selector-semantics and standalone prefill-classification threads were also addressed on the current branch.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current head addresses the review concerns. The Encode policy boundary is now explicit and matches the implementation.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

@GuanLuo GuanLuo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you ask agent to add function-level documentation?

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Contributor Author

/ok to test c7a20b3

@ishandhanani
ishandhanani merged commit c7aa0bf into main Aug 14, 2026
122 checks passed
@ishandhanani
ishandhanani deleted the codex/clarify-disagg-policy-stages branch August 14, 2026 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation refactor router Relates to routing, KV-aware routing, etc. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants