Repository navigation
refactor(router): clarify disaggregated policy dispatch - #13155
Conversation
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (11)
WalkthroughThe KV router now supports independent prefill and decode worker-selection policies. Configuration precedence, Rust resolution, registry dispatch, Python bindings, CLI options, tests, examples, and documentation were updated. ChangesStage-specific worker-selection policies
Estimated code review effort: 4 (Complex) | ~45 minutes Mergeability Score: ⚪ Minimal · up to The change makes prefill/decode policy selection explicit and rejects unsupported stages instead of silently falling back; no actionable merge-blocking risk remains, so it is merge-ready after normal checks and review. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
This comment has been minimized.
This comment has been minimized.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
There was a problem hiding this comment.
Update: withdrawn. Commit 670ec7ac08 documents the intended boundary: worker_selection.encode applies to surface-carrying Encode worker sets that construct the standard KV chooser, while the surface-less multimodal encoder hop intentionally remains on EncoderRouter round-robin selection. That contract matches the implementation, so this is not a remaining P1. The related selector-semantics and standalone prefill-classification threads were also addressed on the current branch.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
jthomson04
left a comment
There was a problem hiding this comment.
The current head addresses the review concerns. The Encode policy boundary is now explicit and matches the implementation.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
GuanLuo
left a comment
There was a problem hiding this comment.
Can you ask agent to add function-level documentation?
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
|
/ok to test c7a20b3 |
TDLR
The stock path still uses
DefaultWorkerSelectordirectly. The new match runs once when the worker pool is constructed, not per request.Custom worker-selection policies can select explicit aggregated, prefill, decode, and encode instances using Dynamo's canonical typed
WorkerType. Policy resolution and typed dispatch happen during startup or worker-pool construction, so the stock request path does not gain role matching or policy lookup work.How This Was Implemented
WorkerTypetodynamo-kv-routerand re-export it fromdynamo-llmfor source compatibility.worker_selection.aggregated,worker_selection.prefill,worker_selection.decode, andworker_selection.encodeYAML selections for every current worker type.Walkthrough
Mental model
Configuration selects a named policy instance for each worker role. The host supplies a typed role when it constructs a routing pool, and the registry returns either that role's custom policy or Dynamo's built-in selector.
flowchart LR Config["YAML / environment / CLI"] --> Resolve["Resolve named instances at startup"] Card["Structured model card"] --> Role["Derive typed WorkerType"] Discovery["Discovery constructs worker pool"] --> Role Resolve --> Factory["Cached policy factories"] Role --> Dispatch["Typed role dispatch at pool construction"] Factory --> Dispatch Dispatch --> Selector["Existing selector and scheduler"]State and ordering
For prefill and decode, precedence is the role-specific CLI or environment override,
DYN_ROUTER_WORKER_SELECTION_POLICY, the matching YAML role, then the built-in selector. For aggregated and encode pools, precedence isDYN_ROUTER_WORKER_SELECTION_POLICY, the matching YAML role, then the built-in selector.The reserved selection value
defaultselects Dynamo's built-in policy for that role. A role without an explicit selection also uses the built-in selector. Because this configuration has not shipped, there is no legacyworker_selection.defaultYAML alias or compatibility fallback; that key is rejected by schema validation.When multiple roles select the same named instance, the registry resolves its provider once and shares the resulting factory. The exhaustive
WorkerType::{Aggregated, Prefill, Decode, Encode}match gives every current worker type an independent YAML choice and forces new enum variants to define their selection behavior.Request lifecycle
The linked registry parses and validates configuration before serving traffic. The embedded frontend receives typed roles from discovery.
python -m dynamo.routerwaits for a structured worker model card only when remote/served indexing needs the model name or a linked custom policy needs the typed role;DYN_ROUTER_MODEL_CARD_WAIT_SECSbounds that startup wait and defaults to 600 seconds. The stock standalone router retains its previous nonblocking discovery snapshot.The example policy matches directly on
WorkerType::PrefillandWorkerType::Decodeto build separate filter/scorer/picker compositions. Strings do not decide which custom policy is selected.After pool construction, requests use the existing selector, scheduling, reservation, and accounting paths. No role parsing, provider lookup, registry lookup, allocation, lock, callback registration, or new background task was added to request processing.
Boundaries and limitations
Standalone EPP has one full-request worker pool, supplies
WorkerType::Aggregated, and usesworker_selection.aggregated. It warns when explicit prefill, decode, or encode selections are ignored; a policy designed only for disaggregated prefill/decode pools can rejectAggregatedwith a typed selection error.worker_selection.encodeapplies to surface-carrying encode worker sets that construct the standard KV chooser. The surface-less multimodal encoder hop exits earlier and usesEncoderRouterround-robin routing, outside worker-selection policy dispatch.The built-in selector historically distinguishes prefill from all other typed pools.
WorkerType::default_selector_labelkeeps prefill on the prefill scorer and decode, aggregated, and encode on the decode scorer. The Python standalone router separately preserves its pre-existing endpoint identity androuter_track_active_blocksinference for metric labels and LoRA load-feed behavior; that path does not select custom policies.The stock/default-only frontend remains monomorphized on
DefaultWorkerSelector. New parsing, custom-policy model-card role discovery, provider resolution, and typed dispatch are startup or pool-construction work, not steady-state request work.Validation
cargo fmt -- --checkin.,lib/bindings/python,lib/runtime/examples, andlib/bindings/kvbmcargo test --manifest-path examples/router/custom-policy-example/disagg-filter-score-pick/Cargo.toml(3 passed)cargo test -p dynamo-kv-router worker_selection(6 passed)cargo test -p dynamo-kv-router worker_type(8 passed)cargo test -p dynamo-kv-router --features standalone-selection --lib services::selection::policy_registry::tests(3 passed)cargo test -p dynamo-ext-proc runner::tests(3 passed)git diff --checkBenchmark Results
Paired Criterion measurements compare direct built-in selection with the wrapped built-in policy used by a configured custom-policy process. Negative deltas favor the wrapped path; the results show no systematic steady-state regression.
Summary by CodeRabbit
New Features
Documentation
CLOSES: DYN-3883