Repository navigation
feat(router): configure separate prefill and decode policies - #13160
ishandhanani merged 1 commit into
Conversation
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
0f7318a
into
codex/clarify-disagg-policy-stages
| let selected = self.selected_worker_selection_policy_instances_from(selected)?; | ||
| Ok(selected.fallback.or(selected.prefill).or(selected.decode)) |
There was a problem hiding this comment.
🔍 Single-instance accessor now reports stage selections, widening startup rejection for stock builds and dynamo.router
selected_worker_selection_policy_instance() now returns fallback.or(prefill).or(decode) instead of only the global/YAML-default selection. Three callers use it purely as a "is any custom policy configured?" gate: deploy/inference-gateway/ext-proc/src/runner.rs:143 (stock EPP), lib/bindings/python/rust/lib.rs:275 (build without custom-policy), and lib/bindings/python/rust/llm/kv.rs:1882 (python -m dynamo.router). As a result, a policy YAML that only sets worker_selection.prefill/decode, or a process that inherits DYN_ROUTER_PREFILL_POLICY/DYN_ROUTER_DECODE_POLICY from a shared environment, will now hard-fail at startup in those processes even though the docs state stage selections apply only to the embedded frontend (see docs/fern/pages/developer-guide/advanced-customizations/custom-worker-selection.mdx:440). The PR describes this fail-closed behavior as intentional; the follow-up worth confirming is deployment configurations where a single YAML/env is shared by a frontend and a dynamo.router/stock EPP process.
Was this helpful? React with 👍 or 👎 to provide feedback.
Adds separate prefill and decode worker-selection policy choices to the embedded Dynamo frontend through YAML, environment variables, and CLI flags. This makes disaggregated routing policies independently configurable without changing worker discovery or per-request scheduling.
How This Was Implemented
worker_selection.prefillandworker_selection.decodeselectors with validation and fallback to the existing YAML default.--router-prefill-policyand--router-decode-policy, their stage-specific environment variables, and Python-to-Rust configuration wiring.selectbehavior.Walkthrough
Mental model
The YAML file defines named policy instances. The frontend resolves one choice for each worker role at startup, then the existing discovery path supplies
prefillordecodewhen it constructs that role's worker pool.flowchart LR Config["YAML / env / CLI"] --> Resolve["Resolve stage selections at startup"] Resolve --> Factory["Role-dispatching policy factory"] Discovery["Worker discovery"] -->|"prefill or decode"| Factory Factory --> Prefill["Prefill policy instance"] Factory --> Decode["Decode policy instance"] Prefill --> Scheduler["Existing scheduler and accounting"] Decode --> SchedulerState and ordering
Each stage uses this precedence, from highest to lowest: stage CLI flag, matching stage environment variable,
DYN_ROUTER_WORKER_SELECTION_POLICY, YAML stage selection, YAML default, and the built-in selector. The reserved valuedefaultselects the built-in policy for that scope.Named instances are parsed and validated with the existing startup policy configuration. When prefill, decode, and fallback select the same instance, the registry resolves its provider once and shares the resulting factory.
Request lifecycle
The Python frontend passes the two optional stage selections into
KvRouterConfig. The linked policy registry resolves the fallback, prefill, and decode factories before the HTTP frontend starts. Discovery later constructs each routing partition with its existingworker_type; the wrapper selects the matching factory once for that partition, after which filtering, scoring, picking, scheduling, and accounting follow the existing paths.Invalid instance names and policy types continue to fail during startup. A stock build without a linked custom-policy catalog rejects any custom stage selection instead of silently ignoring it.
Boundaries and limitations
This change does not alter how workers are classified, discovered, or selected per request. The role dispatch and all new allocation occur during startup or worker-pool construction, not on the request hot path.
Standalone EPP still supplies
selectbecause it has one worker pool and therefore usesworker_selection.default; the prefill and decode selectors apply to the embedded frontend. Existing configurations that only useworker_selection.defaultorDYN_ROUTER_WORKER_SELECTION_POLICYretain their previous behavior.Validation
cargo test -p dynamo-kv-router(867 passed, 2 ignored)cargo test -p dynamo-kv-router --features standalone-selection(960 passed, 2 ignored; 6 integration tests passed)cargo test -p dynamo-custom-policy-example-catalog -p dynamo-custom-policy-example-disagg-filter-score-pick(5 passed).venv/bin/python -m pytest -q components/src/dynamo/common/tests/configuration/test_kv_router_args.py(49 passed)cargo clippy -p dynamo-kv-router --features standalone-selection --all-targets(passed with one pre-existing dead-code warning)cargo check --manifest-path lib/bindings/python/Cargo.toml --features custom-policyand a custom-policymaturin developbuild