Skip to content

feat(router): configure separate prefill and decode policies - #13160

Merged
ishandhanani merged 1 commit into
codex/clarify-disagg-policy-stagesfrom
codex/configure-disagg-stage-policies
Aug 13, 2026
Merged

ishandhanani merged 1 commit into
codex/clarify-disagg-policy-stagesfrom
codex/configure-disagg-stage-policies

Conversation

@ishandhanani

Copy link
Copy Markdown
Contributor

Adds separate prefill and decode worker-selection policy choices to the embedded Dynamo frontend through YAML, environment variables, and CLI flags. This makes disaggregated routing policies independently configurable without changing worker discovery or per-request scheduling.

How This Was Implemented

  • Add worker_selection.prefill and worker_selection.decode selectors with validation and fallback to the existing YAML default.
  • Add --router-prefill-policy and --router-decode-policy, their stage-specific environment variables, and Python-to-Rust configuration wiring.
  • Resolve named policy instances once at startup, reuse duplicate instance factories, and dispatch the resolved factory when each prefill or decode worker pool is constructed.
  • Preserve the built-in selector, the existing global override, stock-build rejection of unsupported custom instances, and standalone EPP select behavior.
Walkthrough

Mental model

The YAML file defines named policy instances. The frontend resolves one choice for each worker role at startup, then the existing discovery path supplies prefill or decode when it constructs that role's worker pool.

flowchart LR
    Config["YAML / env / CLI"] --> Resolve["Resolve stage selections at startup"]
    Resolve --> Factory["Role-dispatching policy factory"]
    Discovery["Worker discovery"] -->|"prefill or decode"| Factory
    Factory --> Prefill["Prefill policy instance"]
    Factory --> Decode["Decode policy instance"]
    Prefill --> Scheduler["Existing scheduler and accounting"]
    Decode --> Scheduler
Loading

State and ordering

Each stage uses this precedence, from highest to lowest: stage CLI flag, matching stage environment variable, DYN_ROUTER_WORKER_SELECTION_POLICY, YAML stage selection, YAML default, and the built-in selector. The reserved value default selects the built-in policy for that scope.

Named instances are parsed and validated with the existing startup policy configuration. When prefill, decode, and fallback select the same instance, the registry resolves its provider once and shares the resulting factory.

Request lifecycle

The Python frontend passes the two optional stage selections into KvRouterConfig. The linked policy registry resolves the fallback, prefill, and decode factories before the HTTP frontend starts. Discovery later constructs each routing partition with its existing worker_type; the wrapper selects the matching factory once for that partition, after which filtering, scoring, picking, scheduling, and accounting follow the existing paths.

Invalid instance names and policy types continue to fail during startup. A stock build without a linked custom-policy catalog rejects any custom stage selection instead of silently ignoring it.

Boundaries and limitations

This change does not alter how workers are classified, discovered, or selected per request. The role dispatch and all new allocation occur during startup or worker-pool construction, not on the request hot path.

Standalone EPP still supplies select because it has one worker pool and therefore uses worker_selection.default; the prefill and decode selectors apply to the embedded frontend. Existing configurations that only use worker_selection.default or DYN_ROUTER_WORKER_SELECTION_POLICY retain their previous behavior.

Validation

  • cargo test -p dynamo-kv-router (867 passed, 2 ignored)
  • cargo test -p dynamo-kv-router --features standalone-selection (960 passed, 2 ignored; 6 integration tests passed)
  • cargo test -p dynamo-custom-policy-example-catalog -p dynamo-custom-policy-example-disagg-filter-score-pick (5 passed)
  • .venv/bin/python -m pytest -q components/src/dynamo/common/tests/configuration/test_kv_router_args.py (49 passed)
  • cargo clippy -p dynamo-kv-router --features standalone-selection --all-targets (passed with one pre-existing dead-code warning)
  • cargo check --manifest-path lib/bindings/python/Cargo.toml --features custom-policy and a custom-policy maturin develop build
  • Live Mocker smoke test with 2 prefill workers, 2 decode workers, different policy types for each stage, and an HTTP 200 chat-completions response

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@github-actions github-actions Bot added feat documentation Improvements or additions to documentation labels Aug 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@ishandhanani
ishandhanani marked this pull request as ready for review August 13, 2026 05:12
@ishandhanani
ishandhanani requested review from a team as code owners August 13, 2026 05:12
@ishandhanani
ishandhanani merged commit 0f7318a into codex/clarify-disagg-policy-stages Aug 13, 2026
64 of 65 checks passed
@ishandhanani
ishandhanani deleted the codex/configure-disagg-stage-policies branch August 13, 2026 05:12

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Open in Devin Review

Comment on lines +1094 to +1095
let selected = self.selected_worker_selection_policy_instances_from(selected)?;
Ok(selected.fallback.or(selected.prefill).or(selected.decode))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Single-instance accessor now reports stage selections, widening startup rejection for stock builds and dynamo.router

selected_worker_selection_policy_instance() now returns fallback.or(prefill).or(decode) instead of only the global/YAML-default selection. Three callers use it purely as a "is any custom policy configured?" gate: deploy/inference-gateway/ext-proc/src/runner.rs:143 (stock EPP), lib/bindings/python/rust/lib.rs:275 (build without custom-policy), and lib/bindings/python/rust/llm/kv.rs:1882 (python -m dynamo.router). As a result, a policy YAML that only sets worker_selection.prefill/decode, or a process that inherits DYN_ROUTER_PREFILL_POLICY/DYN_ROUTER_DECODE_POLICY from a shared environment, will now hard-fail at startup in those processes even though the docs state stage selections apply only to the embedded frontend (see docs/fern/pages/developer-guide/advanced-customizations/custom-worker-selection.mdx:440). The PR describes this fail-closed behavior as intentional; the follow-up worth confirming is deployment configurations where a single YAML/env is shared by a frontend and a dynamo.router/stock EPP process.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant