Skip to content

refactor(router): move the worker-selection stage to lib/router/worker-selection - #128

Closed
ishandhanani wants to merge 1 commit into
idhanani/dyn-177-stage-surfacefrom
idhanani/dyn-177-worker-selection
Closed

ishandhanani wants to merge 1 commit into
idhanani/dyn-177-stage-surfacefrom
idhanani/dyn-177-worker-selection

Conversation

@ishandhanani

Copy link
Copy Markdown
Owner

Step 5d-4 of the router split v2 (DYNO-177). Stacked on #127. The crate move: dynamo-worker-selection at lib/router/worker-selection, depending on dynamo-router-protocol and nothing router-side beyond it.

Summary

  • A move. plugins/worker_selection/{context, inputs, runtime, selection_config, error, trace, shared_cache, reference_picker}.rs, the WorkerFilter/WorkerScorer/WorkerPicker traits from the facade, scheduling/selector/policy.rs (the composed policy, CandidateRowSource, RankVisitor, CandidateRank, CandidatePreparer, PreparedRows, the trace builder), kv_hints/sources.rs (as kv_transfer.rs), conditional_disagg.rs, and scheduling/lora_filter.rs move as they are. lib/kv-router re-exports every name at the paths plugins and hosts already use: plugins::worker_selection::*, scheduling::selector::*, kv_hints::*, conditional_disagg::*, scheduling::lora_filter, protocols::{SelectionTrace*, SharedCacheHits}, scheduling::types::WorkerSelectionPolicyError. No downstream crate changes.
  • Stays in lib/kv-router. The WorkerSelector trait and its adapter over select_rows, the request-to-row SelectionRowSource, the derived columns (selector/derive.rs), the reference DefaultWorkerSelector, WorkerSelectionPolicyFactory (names RoutingPartitionRef), the registry, the host instance config plugins/worker_selection/config.rs (registry input; a deviation from the inventory's §2, recorded in §4.2), and kv_hints::resolve_transfer_sources with WorkerKvTransferKeysExt (they read worker configs and the indexer's routing snapshot).
  • Tests split by what they drive. The policy tests that build a SchedulingRequest and call WorkerSelector stay as selector/policy_tests.rs; the kv-transfer summary tests that resolve candidates against a residency snapshot stay in kv_hints.rs (summary_tests); the stage keeps row_source_tests, the trace tests, the picker tests, the KvTransferScope coverage test (it reads a private method), and the conditional-disagg and LoRA tests.
  • Visibility the boundary required. The selection config slice's serde defaults and validators, which KvRouterConfig shares, are pub and the selection_config module is public; the conditional-disagg defaults likewise; WorkerInputs::ALL (the reference row); WorkerSelectionPolicy::reference_parts (gated testing/bench). The reference selector reads the context through its accessors instead of fields. DEFAULT_MAX_BATCHED_TOKENS is taken from dynamo_router_protocol::load directly (its scheduling::queue re-export for lib/llm is unchanged).
  • Features. testing and bench each add the reference picker (parking_lot); kv-router's testing/bench forward to them and its dev-dependency enables testing. fastrand is a plain dependency (the random-bypass conditional-disagg policy draws from it).
  • Gate and ownership. scripts/router_layering_check.sh gains the dynamo-worker-selection row (22 checks, 0 failures: router deps equal {dynamo-router-protocol}, no sockets, HTTP, Prometheus, or other router crates under --all-features). .github/codeowners/areas.yaml claims lib/router/worker-selection/ under the router area and CODEOWNERS is regenerated. lib/router/README.md status updated.

Validation

  • cargo test -p dynamo-worker-selection --all-features: 41 passed (row-source, trace, picker, scope, conditional-disagg, LoRA, runtime, config).

  • cargo test -p dynamo-kv-router --features standalone-selection,standalone-slot-tracker,metrics,testing: 554 + 2 + 7 passed (554 + 41 moved = the 595 before the move).

  • cargo test -p dynamo-custom-policy-builtin --all-features: 56 + 5 + 1 + 3 + 20 + 3 + 4 passed.

  • cargo test -p dynamo-llm kv_router: 485 passed.

  • cargo clippy --all-features --all-targets -D warnings on protocol, worker-selection, kv-router, builtin: clean; cargo clippy --tests -D warnings on lib/llm (default features), mocker, ext-proc, bench: clean; cargo check --tests on lib/llm, mocker, bench, sglang-sidecar, ext-proc: clean.

  • scripts/router_layering_check.sh: 22 checks, 0 failures.

  • scripts/router_move_diff.py 5deeea6e10 WORKTREE from the kv-router paths to the new crate plus the kv-router remainder: 13748 common / 102 new / 35 dropped. The new lines are the #[inline] hints (75) and the visibility promotions above; the dropped lines are the six accessor calls in reference.rs, the protocol constant path, mod tests renamed mod summary_tests, and rustfmt re-wrapping four lines of the dedented test module.

  • Bench: interleaved same-session A/B on an idle pinned core, the 5d-3b head's bench binary against this branch's, two rounds (the leftover SGLang scheduler was on cores 3 and 11; an earlier round pinned to core 3 ran both binaries 1.7× slower and was discarded):

bench (median) 5d-3b round 1 / 2 this branch round 1 / 2
default_worker_selection/direct/temperature_0/10000 (reference) 791.3 / 783.3 µs 789.3 / 785.7 µs
custom_worker_selection/10_000/custom_no_filter 695.7 / 685.4 µs 672.2 / 673.8 µs
custom_worker_selection/10_000/custom_keep_all_filter 942.3 / 944.9 µs 933.4 / 935.2 µs

Parity or better. Before the #[inline] hints on the stage's small accessors (WorkerInputs::contains, the context getters, the row and view getters, WorkerCandidate::new, with_inputs_from) the reference path measured +2% in two interleaved rounds: those are non-generic functions the kernel now calls across a crate boundary per rank, and thin LTO alone did not inline all of them.

…r-selection

Step 5d-4 of the router split (DYNO-177): dynamo-worker-selection, a
stage crate that depends on dynamo-router-protocol and nothing router-
side beyond it. A move: the files that made up the stage inside
lib/kv-router move as they are, and lib/kv-router re-exports every name
at the path plugins and hosts already use.

Moved: plugins/worker_selection/{context, inputs, runtime,
selection_config, error, trace, shared_cache, reference_picker}.rs, the
plugin traits from the plugins/worker_selection.rs facade,
scheduling/selector/policy.rs (the composed policy, CandidateRowSource,
RankVisitor, CandidateRank, CandidatePreparer, PreparedRows, the trace
builder), kv_hints/sources.rs as kv_transfer.rs, conditional_disagg.rs,
and scheduling/lora_filter.rs.

Stays in lib/kv-router: the WorkerSelector trait and its adapter, the
request-to-row source, the derived columns (selector/derive.rs), the
reference selector, the policy factory type, the registry, the host
instance config (plugins/worker_selection/config.rs), and the resolver
for kv-transfer sources. The policy tests that drive WorkerSelector with
a SchedulingRequest stay as selector/policy_tests.rs; the kv-transfer
summary tests that resolve against a routing snapshot stay in
kv_hints.rs; the stage keeps its own row-source, trace, picker, scope,
and conditional-disagg tests.

Visibility the crate boundary required: the selection config slice's
defaults and validators, which the flat router config shares, are pub,
as are the conditional-disagg defaults, WorkerInputs::ALL, and
WorkerSelectionPolicy::reference_parts; the reference selector reads the
context through its accessors. DEFAULT_MAX_BATCHED_TOKENS is taken from
the protocol crate directly.

Layering gate: a row for dynamo-worker-selection (22 checks). CODEOWNERS
gains lib/router/worker-selection/ under the router area.

Validation: stage 41 tests; kv-router 554 + 2 + 7 (554 + 41
moved = the 595 before); builtin plugin and lib/llm kv_router 485 tests;
clippy clean on the stage crates and lib/llm, mocker, ext-proc, bench;
layering gate 22 checks; move-diff 13748 common / 102 new / 35 dropped (the new lines are #[inline] hints and visibility).
Interleaved A/B bench against the 5d-3b head at 10k workers: parity or
better once the stage's small accessors carry #[inline] (non-generic
functions the kernel calls per rank across the crate boundary; without
the hints the reference path measured +2%).

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Owner Author

Landed upstream as ai-dynamo#18055 (squash commit 79692cf on main, 2026-11-11). Closing this fork review unit.

This branch had an error being deployed

1 failed deployment
external_collaborator — 7598a008 Deployed Nov 2, 2026 by ishandhanani via ok-to-test #520
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant