Repository navigation
chore(router): remove power-of-two and plugin-API paths marked for 1.7.0 removal - #17899
Conversation
… on the hosted path Plugins can now be shared across request threads without a lock, and a picker-only policy can run outside the KV scheduler through a new hosted entry point. This is the foundation for running the non-KV router modes as ordinary plugins. - WorkerFilter::keep, WorkerScorer::score and WorkerPicker::pick take &self, and the traits require Send + Sync. The default policy keeps its per-call state in a local and its sampling scratch in a thread-local buffer, so selection still allocates nothing. - WorkerSelectionPolicy::into_hosted_picker returns a HostedWorkerPicker for a picker-only policy that declares at most ACTIVE_REQUESTS and has no runtime config; HostedWorkerPicker::select picks over a lazy view of the host's worker IDs with no candidate rows and no lock. - WorkerInputs::ACTIVE_REQUESTS replaces OCCUPANCY (deprecated alias until v1.9), with WorkerInputView::active_requests(row), len() and is_empty(). - HostedSelectionRequest carries a hosted selection's request metadata. BREAKING CHANGE: worker-selection plugin methods take &self instead of &mut self, and the plugin traits require Send + Sync. A plugin that mutates state between calls must use atomics or a lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Karen Chung <karenc@nvidia.com>
…er-policies The crate holds every router policy Dynamo ships, including the routing modes from the next commit; "custom" no longer described it. Mechanical: the package name, its dependents' manifests and use paths, and both lockfiles. The directory stays lib/router-plugins/builtin, and the custom-catalog crate keeps its documented name. BREAKING CHANGE: Rust dependents import dynamo_router_policies instead of dynamo_custom_policy_builtin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Karen Chung <karenc@nvidia.com>
--router-mode random, round-robin, least-loaded and power-of-two now resolve by type name in the router plugin registry and run as ordinary worker-selection plugins on the hosted path. Every selection algorithm lives once, in the new dynamo-routing-algorithms crate. - dynamo-routing-algorithms (lib/router-plugins/algorithms): random, round-robin, least-loaded, power-of-two and device-aware selection, index-based, depending only on fastrand. - dynamo-router-policies: four &self pickers in router_modes, thin adapters over the algorithms, registered in default_registry(). - dynamo-runtime: RoutePicker dispatches to the shared algorithms; BuiltinRoutePicker is removed. PushRouter's API is unchanged. - dynamo-kv-router: WorkerSelectionContext::is_preview, HostedSelectionRequest::with_preview and RouterPluginRegistry::resolve_type; the old WorkerSelectionInput::Hosted variant is removed. - dynamo-llm: RoutingPolicy::Hosted(HostedWorkerPicker) replaces BuiltinWorkerSelector for the main and LoRA selectors; previews do not advance round-robin. Device-aware weighted routing keeps its input plumbing in PushRouter for now; only its algorithm moves. BREAKING CHANGE: dynamo_runtime::pipeline::BuiltinRoutePicker and dynamo_kv_router's WorkerSelectionInput::Hosted, into_hosted, hosted() and HostedSelectionInputs are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Karen Chung <karenc@nvidia.com>
Split router_modes.rs into router_modes/random.rs, round_robin.rs, least_loaded.rs and power_of_two.rs. Each file holds its mode's type name and picker. router_modes.rs keeps the shared helpers and the registration, and re-exports the four type names, so callers are unchanged. Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
Build the hosted selection request only when a picker runs, drop the narrating round-robin comment, return into_configured's tuple without a Result, and rename routing_host/builtin.rs to hosted.rs. Signed-off-by: Karen Chung <karenc@nvidia.com>
Store the hosted picker as the policy's Box instead of converting it to an Arc nothing clones, and drop HostedSelectionRequest's three builders and its block_size argument, which no host calls. Hosted pickers see the same values as before: the prompt length, and the KV router's defaults for the rest. Signed-off-by: Karen Chung <karenc@nvidia.com>
…ance selectors Delete dynamo-routing-algorithms: the round-robin, random, least-loaded and power-of-two algorithms now live only in their router-mode plugins. PushRouter picks through an InstanceSelector passed in PushRouterOptions::selectors and resolved when the router is built. HostedWorkerPicker implements it, and dynamo-llm passes the plugins at every PushRouter it builds. round_robin(), random(), power_of_two_choices() and least_loaded() become aliases over one select path. Device-aware weighted stays in the runtime. Also regenerate lib/bindings/kvbm/Cargo.lock for the earlier crate rename. Signed-off-by: Karen Chung <karenc@nvidia.com>
install_router_plugin_registry and the standalone EPP selector apply dynamo_router_policies::with_defaults to the catalog they are given, which adds Dynamo's default and each routing mode the catalog lacks, so --router-mode and a policy file naming a mode type resolve however the catalog was built. The routing-mode providers reject a parameters block, like the other builtin providers, and the into_hosted_picker rustdoc says the hosts call it only for the four router-mode types for now. Signed-off-by: Karen Chung <karenc@nvidia.com>
…worker ID WorkerSelectionContext now holds the request values it exposes instead of borrowing the whole SchedulingRequest, so HostedSelectionRequest carries only the prompt length and SchedulingRequest::hosted() is gone. Hosted select returns the chosen WorkerId rather than a WorkerSelectionResult whose cache and load fields were always zero. Signed-off-by: Karen Chung <karenc@nvidia.com>
…I paths Deprecate the power-of-two router mode for removal in v1.8 (DYN-5265). --router-mode power-of-two and a policy file naming the power-of-two type still work and log a deprecation warning, and PushRouter::power_of_two_choices() is marked #[deprecated]. The RouterMode::PowerOfTwoChoices model-card value stays decodable until v1.9 for N-2 workers. Remove the plugin-API compatibility paths marked for v1.7 (DYN-5297): the kv-router re-exports of the worker-selection types outside plugins::, WorkerSelectionPolicyRegistry, RouterPluginRegistry::register, and the dynamo-llm install_worker_selection_policy_registry, worker_selection_policy_registry and HttpFrontend::worker_selection_policy_factory aliases. In-tree callers, the custom-policy examples and the docs now use the plugins:: paths, RouterPluginRegistry and register_worker_selection. Signed-off-by: Karen Chung <karenc@nvidia.com>
WalkthroughThe changes deprecate power-of-two routing while retaining it, and remove legacy worker-selection and request-classifier API paths. Router integrations, tests, benchmarks, custom-policy examples, and documentation are updated to use the plugin APIs. ChangesPower-of-two deprecation
Router plugin API migration
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk:
|
Keep both v1.7.0 deprecations.mdx entries. Port main's new dynamo-sticky-min-load picker to the &self WorkerPicker trait by holding its session bindings behind a lock. Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
#17483 added live-retune test cases that pass --router-mode power-of-two to the frontend and a worker. #17899 removed that mode, and the two merged 29 minutes apart, so main's planner unit tests fail on both cases. Use least-loaded for the frontend case and drop the worker case, which the random, least-loaded and --router-mode=kv cases already cover. Refs DYN-5265 Signed-off-by: Karen Chung <karenc@nvidia.com> Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Absorbs the 95 upstream commits since 464ff29. Each upstream change lands in the crate that owns that code at the stack head; the stack's moves stay intact. Router-touching PRs and where they landed: - ai-dynamo#17712 host-path routing modes for worker-selection plugins: &self + Send + Sync plugin traits and WorkerInputs::ACTIVE_REQUESTS (deprecated OCCUPANCY alias) in dynamo-worker-selection and dynamo-router-protocol; HostedWorkerPicker and HostedSelectionRequest in dynamo-worker-selection (hosted.rs), reporting SelectionFailure, which the kernel converts into KvSchedulerError; the InstanceSelector adapter in lib/llm (HostedInstanceSelector); the compat test's hosted coverage in kv-router's prelude test. - ai-dynamo#17713 random/round-robin/least-loaded as builtin plugins: router_modes in dynamo-router-policies (renamed from dynamo-custom-policy-builtin) importing the prelude; RouterPluginRegistry::resolve_type in kv-router; the frontend's hosted path through dynamo_worker_selection::host. - ai-dynamo#17899 1.7.0 removals: taken whole; the alias and the setter were gone at the head; power-of-two never arrives. - ai-dynamo#17657 single-source fetch continuations: KvTransferSourceInterval and source_intervals in dynamo-kv-indexer, the bucketed summary in dynamo-kv-hints, the interval filter in the kernel's resolver; tests in the kernel. - ai-dynamo#17856 KVCR hints in offline replay: mocker and lib/llm bodies; supports_kv_transfer_chain_retention on the flat config; its indexer-vs-config test in kv-router's core host tests. - ai-dynamo#17794 vLLM partial-tail filter and ai-dynamo#17126 SGLang engine-owned multimodal pads: dynamo-kv-events zmq_wire (mm_exclusion.rs); the codec-plus-indexer tests in dynamo-kv-indexer; the engine_owned_mm_identity worker flag through the kernel's WorkerRequest, kv-router's indexer registry and run config, and the bindings. - ai-dynamo#17653 SGLang 0.5.23: pad constant docs in dynamo-multimodal. - ai-dynamo#15333, ai-dynamo#17874, ai-dynamo#17484, ai-dynamo#17688, ai-dynamo#17237, ai-dynamo#17894, ai-dynamo#17958: host-side changes merged in place (bindings, lib/llm, CI). Lockfiles regenerated; no shim reintroduced. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…dency-signal Picks up the test_live_retune_frontend_config.py update from ai-dynamo#14775. The previous merge took main inside the window between ai-dynamo#17899 (which removed the power-of-two router mode) and that update, so the planner CPU and runtime sequential suites failed on two power-of-two cases. Signed-off-by: Akira Lonske <akiralonske@gmail.com>
Drop the power-of-two router-mode spelling: main removed it (#17899). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jain-ria <riajain@NVIDIA.com>
Overview
This removes the
power-of-tworouter mode, a built-in policy we don't use (DYN-5265), and the plugin-API compatibility paths marked for removal in v1.7 (DYN-5297). Router modes and custom plugins each keep one supported surface:--router-modetakesround-robin,random,least-loaded,kv,directordevice-aware-weighted, and plugins import fromdynamo_kv_router::plugins.Key Changes
Core logic: +122 / −208 lines, net −86 (excluding tests, docs and comments). +41 / −35 of it is the custom-policy examples moving to the
plugins::paths. Most need review:push_router.rs:RouterMode::PowerOfTwoChoicesstays only as a wire value,resolve_selectorsrejects it with an error naming the removal, andpower_of_two_choices()is removed.routing_host/builder.rs:hosted_pickerrejects the same value for the routing host.kv-router/src/lib.rsandplugins/registry.rs: the compatibility re-exports,WorkerSelectionPolicyRegistryandRouterPluginRegistry::registerare removed.Breaking:
--router-mode power-of-two,DYN_ROUTER_MODE=power-of-twoand a policy file that names thepower-of-twotype fail at startup.PushRouter::power_of_two_choices(), the PythonRouterMode.PowerOfTwoChoices, and the plugin-API paths outsideplugins::are removed.deprecations.mdxlists each replacement.lib/llm/AGENTS.mdasks for. The wire value goes in v1.9, when v1.6 leaves the window.Validation
--router-mode power-of-twoDYN_ROUTER_MODE=power-of-twounknown router mode 'power-of-two'type: power-of-twopower_of_two_choicesthe power-of-two router mode was removed in Dynamo v1.7; use another router modedeprecations.mdxnames the replacementThe first three rows weren't run locally.
Issues and Ongoing Work
main.dynamo-power-of-two-choicesbuilt-in policy; this PR removes power-of-two instead.PushRouter::power_of_two_choices(), which this PR removes.PowerOfTwoChoicesamong the load-aware modes.Pull Request Checklist
/pr-checklistskill.Summary by CodeRabbit
power-of-tworouting is deprecated and now emits a warning when used. Switch toleast-loaded; removal is planned for v1.8.RouterPluginRegistry, andregister_worker_selection.