Skip to content

refactor(router): open the worker-selection stage and move it to lib/router/worker-selection - #181

Closed
ishandhanani wants to merge 6 commits into
idhanani/rs2-07-session-affinityfrom
idhanani/rs2-08-worker-selection
Closed

ishandhanani wants to merge 6 commits into
idhanani/rs2-07-session-affinityfrom
idhanani/rs2-08-worker-selection

Conversation

@ishandhanani

Copy link
Copy Markdown
Owner

Router split v2, unit 08 of 17. Stacked on #180 (unit 07).

Summary

dynamo-worker-selection (lib/router/worker-selection) picks one rank from a candidate table under a policy: the candidate rows and views over protocol's columns, the filter, scorer, picker API, the composed policy and runtime updates, CandidatePreparer, the KV-transfer source choice, the shipped default policy, the LoRA filter, conditional disaggregation, SelectionConfig, WorkerSelectionPolicyError; it depends on protocol only, with testing and bench features, and its plugin API is tier 1 through kv-router's prelude. Five behavior-preserving boundary changes open the stage first (#123 the selection context holds fields, not the request; #124 the stage's own config slices; #125 KV transfer sources resolved before ranking; #126 selection over rows the kernel yields; #127 the surface a kernel in another crate needs), then #128 moves the files. Rule applied: the producer owns its output, and the stage never names the request or the flat router config. kv-router re-exports every name at the paths plugins and hosts already use. The rebase placed two upstream changes inside this range: ai-dynamo#15600's sticky_min_load.rs and its resolves_sticky_min_load_yaml test take &SelectionConfig at #124's commit, and ai-dynamo#17700's SelectionLogFields import, log_selection(request_id, &SchedulingResponse) head, and log_fields construction are carried inside the stack's finish_selection in scheduling/selector/mod.rs at #126's commit. Replaces fork PRs #123, #124, #125, #126, #127, #128.

What moves

  • refactor(router): make WorkerSelectionContext hold fields, not the request #123 566285b68b refactor(router): make WorkerSelectionContext hold fields, not the request: thirteen value fields including the new request_id and shared_cache_hits; a behavior-preserving refactor, not a move; no move-diff.
  • refactor(router): give worker selection its own config slices #124 b90543338e refactor(router): give worker selection its own config slices: SelectionConfig and ConditionalDisaggConfig; the factory, the policy, and the reference policy take the slice; behavior-preserving, not a move; no move-diff.
  • refactor(router): resolve KV transfer sources before worker selection ranks them #125 290307639c refactor(router): resolve KV transfer sources before worker selection ranks them: KvTransferCandidateSource to protocol kv_hints, the resolver in the router, the summary over resolved records. Move-diff (its body): 5,671 common / 88 new / 33 dropped (the ResolvedTransferSource record, the resolver, the summary's new signature and loop over records, from_parts and the three accessors, the extension trait and the cache's call, the call-site spelling).
  • refactor(router): select over rows the kernel yields, not the request #126 915e48ea8f refactor(router): select over rows the kernel yields, not the request: WorkerSelectionPolicy::select_rows(context, source, ...) replaces the request-reading path. Move-diff (its body): 10,748 common / 835 new / 314 dropped; the dropped lines are MaterializedSelectionInput, collect_policy_candidates, collect_prepared_candidates, push_transfer_target, WorkerSelectionPolicyStateRef, and the deferred fetch fill.
  • refactor(router): open the worker-selection stage to a kernel in another crate #127 a7525ecde9 refactor(router): open the worker-selection stage to a kernel in another crate: WorkerSelectionContext::new and with_*, public row fields, stage-owned WorkerSelectionPolicyError, SelectionTrace*, and SharedCacheHits, re-exported where they were. Move-diff (its body): 16,744 common / 238 new / 125 dropped; the dropped lines are the picker internals (from_parts, softmax_sample_entries, the two inline pick loops), the from_request field copies, and the pub(crate) field declarations.
  • refactor(router): move the worker-selection stage to lib/router/worker-selection #128 7598a008de refactor(router): move the worker-selection stage to lib/router/worker-selection: plugins/worker_selection/{context, inputs, runtime, selection_config, error, trace, shared_cache, reference_picker}.rs, the WorkerFilter/WorkerScorer/WorkerPicker traits, scheduling/selector/policy.rs, kv_hints/sources.rs (as kv_transfer.rs), conditional_disagg.rs, scheduling/lora_filter.rs move as they are. Move-diff (its body): 13,748 common / 102 new / 35 dropped; the new lines are the #[inline] hints (75) and the visibility promotions, the dropped lines the six accessor calls in reference.rs, the protocol constant path, mod tests renamed mod summary_tests, and rustfmt re-wrapping four lines of the dedented test module.

Dependency edges

  • dynamo-worker-selection -> dynamo-router-protocol (router deps; dynamo-runtime, dynamo-kv-router, dynamo-kv-indexer, dynamo-slot-tracker, dynamo-session-affinity, dynamo-kv-events, dynamo-kv-hashing, dynamo-zmq, axum, hyper, reqwest, zmq, prometheus forbidden in its tree; direct column pinned in unit 14)

Validation

Unit-level move-diff, run from the worktree at the unit head 7598a008de: python3 -I scripts/router_move_diff.py 182e257876 7598a008de --from <22 paths> --to <32 paths>, the union of the replaced fork PRs' --from/--to lists filtered to the paths present at each end (full command in the artifact /tmp/rr-split/unit-prs/08-move-diff.txt).

summary from_normalized=21064 to_normalized=22042 common=20501 new=1541 dropped=563
result DIFFERS

1,541 new / 563 dropped. The range includes two boundary PRs with no move of their own (#123, #124) whose rewrites of context.rs, inputs.rs, selection_config.rs, selector/policy.rs, and reference.rs appear here because those files are in the moved set; the four per-PR runs (#125 to #128) sum to 1,263 / 507. By category: the braces and punctuation of the new row API and its tests (206); visibility promotions for the kernel, pub constructors, row fields, WorkerSelectionContext::new and with_* (156 / 60); the 81 #[inline] hints at the crate boundary (#128); the select_rows(context, source, ...) API with CandidateRowSource, RankVisitor, CandidateRank, CandidatePreparer, PreparedRows, RowPick, SelectionFailure, and the trace builder, replacing MaterializedSelectionInput, collect_policy_candidates, collect_prepared_candidates, push_transfer_target, WorkerSelectionPolicyStateRef, and the deferred fetch fill (#126, #127); SelectionConfig/ConditionalDisaggConfig with their FromStr parsers and error strings replacing kv_router_config reads (#124); the thirteen context fields and tracks_prefill_tokens()-style accessors replacing the request reads (#123); ResolvedTransferSource, the resolver, from_parts, the three accessors (#125); the picker internals (from_parts, softmax_sample_entries, the inline pick loops) and the from_request field copies dropped (#127); mod tests renamed mod summary_tests and the rustfmt re-wraps (#128). Each fork PR body explains its own lines.

New and dropped normalized lines (1541 new, 563 dropped). The listing below is the 80 most repeated distinct lines of each kind (1038 distinct new, 446 distinct dropped in the artifact).
new 164x }
new 75x #[inline]
new 19x SelectionConfig::default(),
new 12x Some(SelectionConfig {
new 10x self
new 8x &mut self,
new 8x );
new 7x let config = SelectionConfig {
new 6x #[inline(always)]
new 6x })
new 6x },
new 5x "test",
new 5x #[test]
new 5x Self {
new 5x _context: &WorkerSelectionContext<'_>,
new 5x context,
new 5x context: &WorkerSelectionContext<'_>,
new 5x let selector = DefaultWorkerSelector::new(Some(SelectionConfig::default()), "test");
new 5x source: &S,
new 4x max_raw_cached_tokens,
new 4x source
new 4x source: &SelectionRowSource<'_, C>,
new 4x trace,
new 4x } else {
new 4x });
new 3x )
new 3x ) -> Result<usize, WorkerSelectionPolicyError> {
new 3x )),
new 3x ));
new 3x ..Default::default()
new 3x .eligibility
new 3x .for_each_eligible_worker_rank(source.workers, |worker, config| {
new 3x .iter()
new 3x .unwrap();
new 3x PickMode::Lowest {
new 3x SelectionPolicySource::Factory(Arc::new(move |config: &SelectionConfig, _, _| {
new 3x assert!(matches!(
new 3x cache_snapshot: &CacheSnapshot<'_>,
new 3x fn default() -> Self {
new 3x fn pick(
new 3x fn visit<R: CandidateRank>(&mut self, rank: &R) -> bool {
new 3x input: WorkerInputView<'_>,
new 3x let context = source.context();
new 3x let source = SelectionRowSource::new(&request, &workers, request.eligibility(), 16);
new 3x pub fn from_config(config: &ConditionalDisaggConfig) -> Self {
new 3x request.transfer_local_prefix_blocks_for(worker),
new 3x request_id = context.request_id().unwrap_or("-"),
new 3x return Ok(());
new 3x scope,
new 3x source,
new 3x source.track_kept(&row);
new 3x vec![Box::new(DeviceOverlapScorer)],
new 3x };
new 2x #[default]
new 2x #[derive(Debug)]
new 2x #[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
new 2x #[derive(Debug, Clone, PartialEq)]
new 2x &SelectionConfig::from(&config),
new 2x ) -> Result<(), SelectionFailure> {
new 2x ) -> Result<RowPick, SelectionFailure> {
new 2x ) -> Self {
new 2x )?;
new 2x *best_worker = Some(worker);
new 2x ..
new 2x .enumerate()
new 2x .map(|(row, _)| row)
new 2x .workers
new 2x 0.0
new 2x Box::new(LeastLoaded),
new 2x Box::new(LowestCostPicker),
new 2x KvTransferSourceSummary::new(
new 2x Ok(())
new 2x Ok(RowPick {
new 2x PickMode::Softmax {
new 2x Vec::new(),
new 2x WorkerInfo::new(worker, source.capacity(worker))
new 2x WorkerLoadInput {
new 2x WorkerType::Decode,
new 2x _ => Err(format!(
new 2x available: true,
dropped 14x KvRouterConfig::default(),
dropped 12x Some(KvRouterConfig {
dropped 8x workers: &HashMap<WorkerId, C>,
dropped 7x let config = KvRouterConfig {
dropped 6x if defer_transfer {
dropped 6x input.track_kept_candidate(worker);
dropped 5x input: &MaterializedSelectionInput<'_>,
dropped 5x let selector = DefaultWorkerSelector::new(Some(KvRouterConfig::default()), "test");
dropped 4x best_worker = Some(worker);
dropped 4x let input = MaterializedSelectionInput::new(&request, 16);
dropped 4x request: &SchedulingRequest,
dropped 3x &input.context,
dropped 3x ) {
dropped 3x .preferred_taint_multiplier(config.taints())
dropped 3x .routing_constraints
dropped 3x None
dropped 3x SelectionPolicySource::Factory(Arc::new(move |config: &KvRouterConfig, _, _| {
dropped 3x pub fn from_config(config: &KvRouterConfig) -> Self {
dropped 3x request
dropped 3x request_id = context.request.mode.request_id().unwrap_or("-"),
dropped 3x state.score_candidates(&input.context, &input.cache_snapshot, workers)?;
dropped 2x && state
dropped 2x &mut state.unscored_candidates,
dropped 2x &state.transfer_targets,
dropped 2x ) -> Result<bool, KvSchedulerError> {
dropped 2x .request
dropped 2x .scorer_picker_inputs
dropped 2x HashMap::from([("preferred".to_string(), 0.5)]);
dropped 2x best_cost = cost;
dropped 2x eligibility.any_eligible_worker_rank(workers, |worker, config| {
dropped 2x eligibility: RoutingEligibility<'_>,
dropped 2x enabled: config.conditional_disagg_enabled,
dropped 2x if cost < best_cost {
dropped 2x if temperature == 0.0 {
dropped 2x input,
dropped 2x input.fill_kv_transfer_sources(
dropped 2x kv_router_config: KvRouterConfig,
dropped 2x let ComposedPolicyState {
dropped 2x let defer_transfer = request.kv_transfer_candidates.is_some()
dropped 2x let input = MaterializedSelectionInput::new(request, block_size);
dropped 2x let mut best_cost = f64::INFINITY;
dropped 2x let mut best_worker = None;
dropped 2x let mut tie_count = 0;
dropped 2x let preferred_taint_multiplier = if materialize_preferred_taint {
dropped 2x max_raw_cached_tokens: Option<usize>,
dropped 2x push_transfer_target(&mut state.transfer_targets, request, worker, config);
dropped 2x request,
dropped 2x request.routing_constraints.preferred_taints =
dropped 2x return best_worker.map(|worker| (worker, best_cost));
dropped 2x state.push_candidate(candidate);
dropped 2x state: &mut ComposedPolicyState,
dropped 2x tie_count += 1;
dropped 2x tie_count = 1;
dropped 2x } else if cost == best_cost {
dropped 1x !picker_inputs.contains(WorkerInputs::CACHE)
dropped 1x !picker_inputs.contains(WorkerInputs::LOAD)
dropped 1x #[cfg(any(test, feature = "bench"))] deterministic_rng: Option<Arc<Mutex<fastrand::Rng>>>,
dropped 1x #[inline(never)]
dropped 1x && !context.track_prefill_tokens
dropped 1x && additional_inputs.contains(WorkerInputs::PREFERRED_TAINT);
dropped 1x && self.inputs.contains(WorkerInputs::KV_TRANSFER);
dropped 1x && state.filter_inputs.contains(WorkerInputs::PREFERRED_TAINT);
dropped 1x &entries,
dropped 1x &input,
dropped 1x &mut probabilities,
dropped 1x (crate::plugins::worker_selection::inputs::DEFAULT_MAX_BATCHED_TOKENS / 2) as usize;
dropped 1x (worker_type, Some(worker), worker_keys.topology.as_slice())
dropped 1x ) -> (T, f64) {
dropped 1x ) -> CandidateData {
dropped 1x ) -> Result<(), KvSchedulerError> {
dropped 1x ) -> WorkerInfo {
dropped 1x .and_then(|(target, local)| sources.source_for(row.worker, target, local))
dropped 1x .as_deref()
dropped 1x .as_ref()
dropped 1x .collect();
dropped 1x .contains(WorkerInputs::KV_TRANSFER);
dropped 1x .contains(WorkerInputs::PREFERRED_TAINT);
dropped 1x .contains(replacement.input_requirements())
dropped 1x .context
dropped 1x .get_or_init(|| KvTransferSourceSummary::new(candidates, workers, scope())),

Chain v2, full mode, on the rebased stack head ebd31cdf4d (every unit's commits are ancestors of it; the chain ran once on the head, not per unit), log /tmp/rr-split/srebase-validation.log: stage 1, the 13 router crates (clippy under default and all features, cargo test 1,247 passed / 5 ignored, rustdoc with all features), 1 min 50 s; stage 2, kv-router (clippy, bench check, 229 tests including the prelude test), 1 min 20 s; stage 3, workspace clippy, 4 min 58 s; stage 4, hosts (dynamo-llm lib alone, its kv_router 492 / discovery 275 / pool_relay 261 tests, 13 host crates clippy, bindings clippy; the pre-existing kvbm-consolidator manifest failure is the one non-zero exit), 29 min 21 s; stage 5, gates (every kv-router feature alone and together, 6 public-API snapshots unchanged with kv-router at 918 items, layering gate 47 checks and --host 66 checks with 0 failures, codeowners 7590/7590, fmt, 3 lockfiles --locked), 3 min 06 s; 40 min 35 s total, exit 0.

Each replaced fork PR carried its own validation chain; see its body.

Tickets

Part of DYNO-177

…quest

WorkerSelectionContext, the view filters, scorers, and pickers read, held
a reference to the kernel's SchedulingRequest and answered its accessors
by reaching into it. The worker-selection stage must not depend on the
kernel's request type, so the context now carries the values a policy may
read: request id, ISL tokens, request blocks, block size, prefill-token
tracking, pinned worker, temperature override, session context, affinity
target, expected output tokens, the two priorities, and the shared-cache
hits. The kernel builds it once per selection with from_request. Two
accessors are new (request_id, shared_cache_hits); the existing accessors
return the same values from the copied fields.

The reference selector was the only reader of the request through the
context: it now uses the accessors, and the per-worker local transfer
prefix it needs for the fetchable-blocks term is passed in by the caller,
which already computes it from the request. Plugin crates read the context
through accessors only and do not change.

Step 5d-0 of the router split v2 plan (worker-selection boundary). DYNO-177.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Worker-selection code read the flat KvRouterConfig: the policy factory type
took it, WorkerSelectionPolicy::new carried it, the reference selector
scored with its weights, and ConditionalDisaggPolicy read six of its
fields. The stage must not depend on the router's flat configuration, so
two slices now own those fields.

SelectionConfig (plugins::worker_selection::selection_config) holds the
scoring weights, the sampling temperature, the shared-cache multiplier and
type, and conditional_disagg_enabled, with Default equal to the flat
defaults, From<&KvRouterConfig> next to the flat type, and validate() using
the validators that moved with it. SharedCacheType moves into the slice
module. WorkerSelectionPolicyFactory takes &SelectionConfig;
WorkerSelectionPolicy::new, new_with_filters, and reference, and the
reference DefaultWorkerSelector, take SelectionConfig. The registry, the
selection core, lib/llm's prepared-policy wrapper, the mocker, and the
in-tree plugins convert once at the boundary.

ConditionalDisaggConfig (conditional_disagg) holds the six
conditional_disagg_* fields with Default and From<&KvRouterConfig>;
ConditionalDisaggPolicyKind moves beside it; make_conditional_disagg_policy
and the policies' from_config take the slice, and lib/llm's prefill leg
converts once. config:: re-exports every moved name and the root exports
the two slices, so existing paths resolve.

The builtin plugin's default policy, its parameters, and its selector take
SelectionConfig; the four example policies and the builtin, kv-router, and
lib/llm tests convert where they built a flat config for selection.

Step 5d-1 of the router split v2 plan (worker-selection boundary). DYNO-177.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
… ranks them

The transfer-source summary that credits a candidate worker for the KV
prefix it could fetch read two things the worker-selection stage must not
see: each worker's config (through WorkerConfigLike::kv_transfer_keys) and
the indexer's residency routing snapshot (through
KvTransferCandidates.routing_snapshot). Split the step in two.

The router resolves first: kv_hints::resolve_transfer_sources walks the
request's candidates with today's skips (sources no longer than every
target's local prefix or longer than the retained chain, worker ranks that
publish no source endpoint, cache owners missing from the snapshot, the
attached worker's topology for an owner that publishes none) and yields
ResolvedTransferSource records: who holds the prefix, how long it is, the
hashed hint worker type, the attached worker, and the topology labels.
KvTransferSourceSummary::new takes those records and the scope and ranks
them exactly as before; source_for is unchanged.

Worker keys move the same way. WorkerKvTransferKeys keeps its data,
target(), and a from_parts constructor over already-hashed labels, with
worker_type() and topology() accessors; deriving the keys from a config
becomes the WorkerKvTransferKeysExt extension trait beside
WorkerConfigLike, and WorkerKvTransferKeysCache moves next to it.
KvTransferCandidateSource moves to dynamo_router_protocol::kv_hints as
shared vocabulary; the indexer and kv-router re-export it at the old
paths. The label hashing (label_key, topology_keys) stays with the indexer,
which builds the snapshot keys; the router calls it for worker keys.

Step 5d-2 of the router split v2 plan (worker-selection boundary). DYNO-177.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 5d-3 of the router split (DYNO-177). The composed worker-selection
policy no longer reads SchedulingRequest, RoutingEligibility, or worker
configs. The kernel implements a CandidateRowSource (walk the eligible
ranks, each a CandidateRank that yields its row at the requested
columns and its capacity; a rank's capacity and booked load by worker)
and drives the policy through WorkerSelectionPolicy::select_rows(
context, source, prepared, trace), which returns a RowPick or a
SelectionFailure. The walk takes a RankVisitor whose visit method is
generic over the rank, so the walk, the rank, and the row build are
monomorphized together; a dyn visitor cost 11 ns per rank. The kernel maps the pick
to WorkerSelectionResult, logs the selection, and turns NoCandidates
into overloaded-vs-no-endpoints from eligibility; the stage reports
AllFiltered only when the source yielded a row.

Rows still materialize lazily: a policy with filters asks a rank for
the filters' columns and for the remaining columns only once it is
kept, and the no-filter path stays one pass. CandidateData gains a u32
raw_cached_tokens column (packed beside inputs; the row stays 128
bytes) that the kernel fills for tracked requests, so the stage folds
the best eligible cache reuse over kept rows without the request.

The fetch column is filled as each row is built. Its summary's scope is
every eligible rank's target and local prefix (KvTransferScope::empty +
add_target), computed once; this admits only sources no kept rank can
use, which source_for already rejects, so kept rows see the same
sources as the kept-rows scope did. The selected worker's fetch source
keeps its single-target scope when no scorer declared KV_TRANSFER.

CandidatePreparer::prepare_with owns the pooled buffer and the kernel's
CandidatePreparerExt::prepare builds rows into it; PreparedCandidates
is now the kernel wrapper carrying the fetch summary beside the stage's
PreparedRows. Selection-trace assembly and the per-candidate TRACE log
move into the stage with the scratch rows they read. The reference
selector keeps reading the request through the kernel's row source.

Validation: kv-router 594 + 2 + 7 (four new stage-only tests drive
select_rows over a table with no request in sight), builtin plugin
tests, lib/llm kv_router tests, clippy clean, layering gate 20 checks.
criterion worker_selection against the parent commit at 10k workers:
custom_no_filter +1.6%, custom_keep_all_filter +1.2%,
custom_capacity_filter +2.4%, unused_preferred_taint_metadata no
change, reference direct -1.6%; default_policy_wrapper -2.5% to -8% at
32 and 1024 workers, +9-11 ns per selection at 2 workers.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…her crate

Step 5d-3b of the router split (DYNO-177): the last boundary changes
before the stage becomes its own crate, each one something the kernel
could do only because it shared a crate with the stage.

- WorkerSelectionContext gains a public constructor: new(isl_tokens,
  block_size) with every other field at its default plus with_* setters;
  the kernel's from_request becomes selection_context in the selector.
- The row types the kernel fills have public fields: CandidateData,
  WorkerCacheData, WorkerLoadInput, WorkerTransferInput, CacheSnapshot,
  and WorkerInputRows with new/empty/read.
- What the stage derived from kernel-only types leaves it: the config-
  and projection-derived constructors are now the WorkerCapacityExt,
  WorkerInfoExt, and WorkerLoadInputExt traits in selector::derive, and
  the facade re-exports them at their old paths. WorkerInfo::of becomes
  WorkerInfoExt::of.
- RankLoad moves to the protocol crate's load module so both the slot
  tracker's projection and the stage's load column implement one trait
  without either stage naming the other; the slot tracker re-exports it.
- WorkerSelectionPolicyError, the selection trace types, and
  SharedCacheHits move into the stage's modules; scheduling::types and
  protocols re-export them, so every existing path still resolves.
- The reference picker moves into the stage behind the bench and test
  features as a push-based pick: begin(temperature), offer(worker,
  cost), finish(). The kernel's pick_default_worker offers scores in
  eligibility order, or sorted order when the picker is seeded, and
  never touches the RNG or the softmax scratch. Same draws, same ties.

Validation: protocol 10, slot-tracker 130, kv-router 595 + 2 + 7,
builtin plugin, lib/llm kv_router 485 tests; clippy clean on stage
crates and lib/llm, mocker, ext-proc, bench; layering gate 20 checks;
move-diff 16,744 common / 238 new / 125 dropped. Interleaved A/B bench
against the 5d-3 head at 10k workers: composed paths at parity, the
test-only reference path within about 1 ns per rank once the pick
state mirrors the old loop's locals (its first cut measured +3.8%).

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…r-selection

Step 5d-4 of the router split (DYNO-177): dynamo-worker-selection, a
stage crate that depends on dynamo-router-protocol and nothing router-
side beyond it. A move: the files that made up the stage inside
lib/kv-router move as they are, and lib/kv-router re-exports every name
at the path plugins and hosts already use.

Moved: plugins/worker_selection/{context, inputs, runtime,
selection_config, error, trace, shared_cache, reference_picker}.rs, the
plugin traits from the plugins/worker_selection.rs facade,
scheduling/selector/policy.rs (the composed policy, CandidateRowSource,
RankVisitor, CandidateRank, CandidatePreparer, PreparedRows, the trace
builder), kv_hints/sources.rs as kv_transfer.rs, conditional_disagg.rs,
and scheduling/lora_filter.rs.

Stays in lib/kv-router: the WorkerSelector trait and its adapter, the
request-to-row source, the derived columns (selector/derive.rs), the
reference selector, the policy factory type, the registry, the host
instance config (plugins/worker_selection/config.rs), and the resolver
for kv-transfer sources. The policy tests that drive WorkerSelector with
a SchedulingRequest stay as selector/policy_tests.rs; the kv-transfer
summary tests that resolve against a routing snapshot stay in
kv_hints.rs; the stage keeps its own row-source, trace, picker, scope,
and conditional-disagg tests.

Visibility the crate boundary required: the selection config slice's
defaults and validators, which the flat router config shares, are pub,
as are the conditional-disagg defaults, WorkerInputs::ALL, and
WorkerSelectionPolicy::reference_parts; the reference selector reads the
context through its accessors. DEFAULT_MAX_BATCHED_TOKENS is taken from
the protocol crate directly.

Layering gate: a row for dynamo-worker-selection (22 checks). CODEOWNERS
gains lib/router/worker-selection/ under the router area.

Validation: stage 41 tests; kv-router 554 + 2 + 7 (554 + 41
moved = the 595 before); builtin plugin and lib/llm kv_router 485 tests;
clippy clean on the stage crates and lib/llm, mocker, ext-proc, bench;
layering gate 22 checks; move-diff 13748 common / 102 new / 35 dropped (the new lines are #[inline] hints and visibility).
Interleaved A/B bench against the 5d-3b head at 10k workers: parity or
better once the stage's small accessors carry #[inline] (non-generic
functions the kernel calls per rank across the crate boundary; without
the hints the reference path measured +2%).

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Owner Author

Landed upstream as ai-dynamo#18055 (squash commit 79692cf on main, 2026-11-11). Closing this fork review unit.

This branch had an error being deployed

1 failed deployment
external_collaborator — 7598a008 Deployed Nov 2, 2026 by ishandhanani via ok-to-test #520
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation refactor router

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant