Repository navigation
refactor(router): open the worker-selection stage and move it to lib/router/worker-selection - #181
Closed
ishandhanani wants to merge 6 commits into
Closed
ishandhanani wants to merge 6 commits into
ishandhanani wants to merge 6 commits into
Conversation
…quest WorkerSelectionContext, the view filters, scorers, and pickers read, held a reference to the kernel's SchedulingRequest and answered its accessors by reaching into it. The worker-selection stage must not depend on the kernel's request type, so the context now carries the values a policy may read: request id, ISL tokens, request blocks, block size, prefill-token tracking, pinned worker, temperature override, session context, affinity target, expected output tokens, the two priorities, and the shared-cache hits. The kernel builds it once per selection with from_request. Two accessors are new (request_id, shared_cache_hits); the existing accessors return the same values from the copied fields. The reference selector was the only reader of the request through the context: it now uses the accessors, and the per-worker local transfer prefix it needs for the fetchable-blocks term is passed in by the caller, which already computes it from the request. Plugin crates read the context through accessors only and do not change. Step 5d-0 of the router split v2 plan (worker-selection boundary). DYNO-177. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Worker-selection code read the flat KvRouterConfig: the policy factory type took it, WorkerSelectionPolicy::new carried it, the reference selector scored with its weights, and ConditionalDisaggPolicy read six of its fields. The stage must not depend on the router's flat configuration, so two slices now own those fields. SelectionConfig (plugins::worker_selection::selection_config) holds the scoring weights, the sampling temperature, the shared-cache multiplier and type, and conditional_disagg_enabled, with Default equal to the flat defaults, From<&KvRouterConfig> next to the flat type, and validate() using the validators that moved with it. SharedCacheType moves into the slice module. WorkerSelectionPolicyFactory takes &SelectionConfig; WorkerSelectionPolicy::new, new_with_filters, and reference, and the reference DefaultWorkerSelector, take SelectionConfig. The registry, the selection core, lib/llm's prepared-policy wrapper, the mocker, and the in-tree plugins convert once at the boundary. ConditionalDisaggConfig (conditional_disagg) holds the six conditional_disagg_* fields with Default and From<&KvRouterConfig>; ConditionalDisaggPolicyKind moves beside it; make_conditional_disagg_policy and the policies' from_config take the slice, and lib/llm's prefill leg converts once. config:: re-exports every moved name and the root exports the two slices, so existing paths resolve. The builtin plugin's default policy, its parameters, and its selector take SelectionConfig; the four example policies and the builtin, kv-router, and lib/llm tests convert where they built a flat config for selection. Step 5d-1 of the router split v2 plan (worker-selection boundary). DYNO-177. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
… ranks them The transfer-source summary that credits a candidate worker for the KV prefix it could fetch read two things the worker-selection stage must not see: each worker's config (through WorkerConfigLike::kv_transfer_keys) and the indexer's residency routing snapshot (through KvTransferCandidates.routing_snapshot). Split the step in two. The router resolves first: kv_hints::resolve_transfer_sources walks the request's candidates with today's skips (sources no longer than every target's local prefix or longer than the retained chain, worker ranks that publish no source endpoint, cache owners missing from the snapshot, the attached worker's topology for an owner that publishes none) and yields ResolvedTransferSource records: who holds the prefix, how long it is, the hashed hint worker type, the attached worker, and the topology labels. KvTransferSourceSummary::new takes those records and the scope and ranks them exactly as before; source_for is unchanged. Worker keys move the same way. WorkerKvTransferKeys keeps its data, target(), and a from_parts constructor over already-hashed labels, with worker_type() and topology() accessors; deriving the keys from a config becomes the WorkerKvTransferKeysExt extension trait beside WorkerConfigLike, and WorkerKvTransferKeysCache moves next to it. KvTransferCandidateSource moves to dynamo_router_protocol::kv_hints as shared vocabulary; the indexer and kv-router re-export it at the old paths. The label hashing (label_key, topology_keys) stays with the indexer, which builds the snapshot keys; the router calls it for worker keys. Step 5d-2 of the router split v2 plan (worker-selection boundary). DYNO-177. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 5d-3 of the router split (DYNO-177). The composed worker-selection policy no longer reads SchedulingRequest, RoutingEligibility, or worker configs. The kernel implements a CandidateRowSource (walk the eligible ranks, each a CandidateRank that yields its row at the requested columns and its capacity; a rank's capacity and booked load by worker) and drives the policy through WorkerSelectionPolicy::select_rows( context, source, prepared, trace), which returns a RowPick or a SelectionFailure. The walk takes a RankVisitor whose visit method is generic over the rank, so the walk, the rank, and the row build are monomorphized together; a dyn visitor cost 11 ns per rank. The kernel maps the pick to WorkerSelectionResult, logs the selection, and turns NoCandidates into overloaded-vs-no-endpoints from eligibility; the stage reports AllFiltered only when the source yielded a row. Rows still materialize lazily: a policy with filters asks a rank for the filters' columns and for the remaining columns only once it is kept, and the no-filter path stays one pass. CandidateData gains a u32 raw_cached_tokens column (packed beside inputs; the row stays 128 bytes) that the kernel fills for tracked requests, so the stage folds the best eligible cache reuse over kept rows without the request. The fetch column is filled as each row is built. Its summary's scope is every eligible rank's target and local prefix (KvTransferScope::empty + add_target), computed once; this admits only sources no kept rank can use, which source_for already rejects, so kept rows see the same sources as the kept-rows scope did. The selected worker's fetch source keeps its single-target scope when no scorer declared KV_TRANSFER. CandidatePreparer::prepare_with owns the pooled buffer and the kernel's CandidatePreparerExt::prepare builds rows into it; PreparedCandidates is now the kernel wrapper carrying the fetch summary beside the stage's PreparedRows. Selection-trace assembly and the per-candidate TRACE log move into the stage with the scratch rows they read. The reference selector keeps reading the request through the kernel's row source. Validation: kv-router 594 + 2 + 7 (four new stage-only tests drive select_rows over a table with no request in sight), builtin plugin tests, lib/llm kv_router tests, clippy clean, layering gate 20 checks. criterion worker_selection against the parent commit at 10k workers: custom_no_filter +1.6%, custom_keep_all_filter +1.2%, custom_capacity_filter +2.4%, unused_preferred_taint_metadata no change, reference direct -1.6%; default_policy_wrapper -2.5% to -8% at 32 and 1024 workers, +9-11 ns per selection at 2 workers. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…her crate Step 5d-3b of the router split (DYNO-177): the last boundary changes before the stage becomes its own crate, each one something the kernel could do only because it shared a crate with the stage. - WorkerSelectionContext gains a public constructor: new(isl_tokens, block_size) with every other field at its default plus with_* setters; the kernel's from_request becomes selection_context in the selector. - The row types the kernel fills have public fields: CandidateData, WorkerCacheData, WorkerLoadInput, WorkerTransferInput, CacheSnapshot, and WorkerInputRows with new/empty/read. - What the stage derived from kernel-only types leaves it: the config- and projection-derived constructors are now the WorkerCapacityExt, WorkerInfoExt, and WorkerLoadInputExt traits in selector::derive, and the facade re-exports them at their old paths. WorkerInfo::of becomes WorkerInfoExt::of. - RankLoad moves to the protocol crate's load module so both the slot tracker's projection and the stage's load column implement one trait without either stage naming the other; the slot tracker re-exports it. - WorkerSelectionPolicyError, the selection trace types, and SharedCacheHits move into the stage's modules; scheduling::types and protocols re-export them, so every existing path still resolves. - The reference picker moves into the stage behind the bench and test features as a push-based pick: begin(temperature), offer(worker, cost), finish(). The kernel's pick_default_worker offers scores in eligibility order, or sorted order when the picker is seeded, and never touches the RNG or the softmax scratch. Same draws, same ties. Validation: protocol 10, slot-tracker 130, kv-router 595 + 2 + 7, builtin plugin, lib/llm kv_router 485 tests; clippy clean on stage crates and lib/llm, mocker, ext-proc, bench; layering gate 20 checks; move-diff 16,744 common / 238 new / 125 dropped. Interleaved A/B bench against the 5d-3 head at 10k workers: composed paths at parity, the test-only reference path within about 1 ns per rank once the pick state mirrors the old loop's locals (its first cut measured +3.8%). Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…r-selection
Step 5d-4 of the router split (DYNO-177): dynamo-worker-selection, a
stage crate that depends on dynamo-router-protocol and nothing router-
side beyond it. A move: the files that made up the stage inside
lib/kv-router move as they are, and lib/kv-router re-exports every name
at the path plugins and hosts already use.
Moved: plugins/worker_selection/{context, inputs, runtime,
selection_config, error, trace, shared_cache, reference_picker}.rs, the
plugin traits from the plugins/worker_selection.rs facade,
scheduling/selector/policy.rs (the composed policy, CandidateRowSource,
RankVisitor, CandidateRank, CandidatePreparer, PreparedRows, the trace
builder), kv_hints/sources.rs as kv_transfer.rs, conditional_disagg.rs,
and scheduling/lora_filter.rs.
Stays in lib/kv-router: the WorkerSelector trait and its adapter, the
request-to-row source, the derived columns (selector/derive.rs), the
reference selector, the policy factory type, the registry, the host
instance config (plugins/worker_selection/config.rs), and the resolver
for kv-transfer sources. The policy tests that drive WorkerSelector with
a SchedulingRequest stay as selector/policy_tests.rs; the kv-transfer
summary tests that resolve against a routing snapshot stay in
kv_hints.rs; the stage keeps its own row-source, trace, picker, scope,
and conditional-disagg tests.
Visibility the crate boundary required: the selection config slice's
defaults and validators, which the flat router config shares, are pub,
as are the conditional-disagg defaults, WorkerInputs::ALL, and
WorkerSelectionPolicy::reference_parts; the reference selector reads the
context through its accessors. DEFAULT_MAX_BATCHED_TOKENS is taken from
the protocol crate directly.
Layering gate: a row for dynamo-worker-selection (22 checks). CODEOWNERS
gains lib/router/worker-selection/ under the router area.
Validation: stage 41 tests; kv-router 554 + 2 + 7 (554 + 41
moved = the 595 before); builtin plugin and lib/llm kv_router 485 tests;
clippy clean on the stage crates and lib/llm, mocker, ext-proc, bench;
layering gate 22 checks; move-diff 13748 common / 102 new / 35 dropped (the new lines are #[inline] hints and visibility).
Interleaved A/B bench against the 5d-3b head at 10k workers: parity or
better once the stage's small accessors carry #[inline] (non-generic
functions the kernel calls per rank across the crate boundary; without
the hints the reference path measured +2%).
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
had a problem deploying
to
external_collaborator
November 2, 2026 13:53 — with
GitHub Actions
Failure
Owner
Author
|
Landed upstream as ai-dynamo#18055 (squash commit 79692cf on main, 2026-11-11). Closing this fork review unit. |
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Router split v2, unit 08 of 17. Stacked on #180 (unit 07).
Summary
dynamo-worker-selection(lib/router/worker-selection) picks one rank from a candidate table under a policy: the candidate rows and views over protocol's columns, the filter, scorer, picker API, the composed policy and runtime updates,CandidatePreparer, the KV-transfer source choice, the shipped default policy, the LoRA filter, conditional disaggregation,SelectionConfig,WorkerSelectionPolicyError; it depends on protocol only, withtestingandbenchfeatures, and its plugin API is tier 1 through kv-router's prelude. Five behavior-preserving boundary changes open the stage first (#123 the selection context holds fields, not the request; #124 the stage's own config slices; #125 KV transfer sources resolved before ranking; #126 selection over rows the kernel yields; #127 the surface a kernel in another crate needs), then #128 moves the files. Rule applied: the producer owns its output, and the stage never names the request or the flat router config. kv-router re-exports every name at the paths plugins and hosts already use. The rebase placed two upstream changes inside this range: ai-dynamo#15600'ssticky_min_load.rsand itsresolves_sticky_min_load_yamltest take&SelectionConfigat #124's commit, and ai-dynamo#17700'sSelectionLogFieldsimport,log_selection(request_id, &SchedulingResponse)head, andlog_fieldsconstruction are carried inside the stack'sfinish_selectioninscheduling/selector/mod.rsat #126's commit. Replaces fork PRs #123, #124, #125, #126, #127, #128.What moves
566285b68brefactor(router): make WorkerSelectionContext hold fields, not the request: thirteen value fields including the newrequest_idandshared_cache_hits; a behavior-preserving refactor, not a move; no move-diff.b90543338erefactor(router): give worker selection its own config slices:SelectionConfigandConditionalDisaggConfig; the factory, the policy, and the reference policy take the slice; behavior-preserving, not a move; no move-diff.290307639crefactor(router): resolve KV transfer sources before worker selection ranks them:KvTransferCandidateSourceto protocolkv_hints, the resolver in the router, the summary over resolved records. Move-diff (its body): 5,671 common / 88 new / 33 dropped (theResolvedTransferSourcerecord, the resolver, the summary's new signature and loop over records,from_partsand the three accessors, the extension trait and the cache's call, the call-site spelling).915e48ea8frefactor(router): select over rows the kernel yields, not the request:WorkerSelectionPolicy::select_rows(context, source, ...)replaces the request-reading path. Move-diff (its body): 10,748 common / 835 new / 314 dropped; the dropped lines areMaterializedSelectionInput,collect_policy_candidates,collect_prepared_candidates,push_transfer_target,WorkerSelectionPolicyStateRef, and the deferred fetch fill.a7525ecde9refactor(router): open the worker-selection stage to a kernel in another crate:WorkerSelectionContext::newandwith_*, public row fields, stage-ownedWorkerSelectionPolicyError,SelectionTrace*, andSharedCacheHits, re-exported where they were. Move-diff (its body): 16,744 common / 238 new / 125 dropped; the dropped lines are the picker internals (from_parts,softmax_sample_entries, the two inline pick loops), thefrom_requestfield copies, and thepub(crate)field declarations.7598a008derefactor(router): move the worker-selection stage to lib/router/worker-selection:plugins/worker_selection/{context, inputs, runtime, selection_config, error, trace, shared_cache, reference_picker}.rs, theWorkerFilter/WorkerScorer/WorkerPickertraits,scheduling/selector/policy.rs,kv_hints/sources.rs(askv_transfer.rs),conditional_disagg.rs,scheduling/lora_filter.rsmove as they are. Move-diff (its body): 13,748 common / 102 new / 35 dropped; the new lines are the#[inline]hints (75) and the visibility promotions, the dropped lines the six accessor calls inreference.rs, the protocol constant path,mod testsrenamedmod summary_tests, and rustfmt re-wrapping four lines of the dedented test module.Dependency edges
dynamo-worker-selection-> dynamo-router-protocol (router deps; dynamo-runtime, dynamo-kv-router, dynamo-kv-indexer, dynamo-slot-tracker, dynamo-session-affinity, dynamo-kv-events, dynamo-kv-hashing, dynamo-zmq, axum, hyper, reqwest, zmq, prometheus forbidden in its tree; direct column pinned in unit 14)Validation
Unit-level move-diff, run from the worktree at the unit head
7598a008de:python3 -I scripts/router_move_diff.py 182e257876 7598a008de --from <22 paths> --to <32 paths>, the union of the replaced fork PRs'--from/--tolists filtered to the paths present at each end (full command in the artifact/tmp/rr-split/unit-prs/08-move-diff.txt).1,541 new / 563 dropped. The range includes two boundary PRs with no move of their own (#123, #124) whose rewrites of
context.rs,inputs.rs,selection_config.rs,selector/policy.rs, andreference.rsappear here because those files are in the moved set; the four per-PR runs (#125 to #128) sum to 1,263 / 507. By category: the braces and punctuation of the new row API and its tests (206); visibility promotions for the kernel,pubconstructors, row fields,WorkerSelectionContext::newandwith_*(156 / 60); the 81#[inline]hints at the crate boundary (#128); theselect_rows(context, source, ...)API withCandidateRowSource,RankVisitor,CandidateRank,CandidatePreparer,PreparedRows,RowPick,SelectionFailure, and the trace builder, replacingMaterializedSelectionInput,collect_policy_candidates,collect_prepared_candidates,push_transfer_target,WorkerSelectionPolicyStateRef, and the deferred fetch fill (#126, #127);SelectionConfig/ConditionalDisaggConfigwith theirFromStrparsers and error strings replacingkv_router_configreads (#124); the thirteen context fields andtracks_prefill_tokens()-style accessors replacing the request reads (#123);ResolvedTransferSource, the resolver,from_parts, the three accessors (#125); the picker internals (from_parts,softmax_sample_entries, the inline pick loops) and thefrom_requestfield copies dropped (#127);mod testsrenamedmod summary_testsand the rustfmt re-wraps (#128). Each fork PR body explains its own lines.New and dropped normalized lines (1541 new, 563 dropped). The listing below is the 80 most repeated distinct lines of each kind (1038 distinct new, 446 distinct dropped in the artifact).
Chain v2, full mode, on the rebased stack head
ebd31cdf4d(every unit's commits are ancestors of it; the chain ran once on the head, not per unit), log/tmp/rr-split/srebase-validation.log: stage 1, the 13 router crates (clippy under default and all features,cargo test1,247 passed / 5 ignored, rustdoc with all features), 1 min 50 s; stage 2, kv-router (clippy, bench check, 229 tests including the prelude test), 1 min 20 s; stage 3, workspace clippy, 4 min 58 s; stage 4, hosts (dynamo-llmlib alone, its kv_router 492 / discovery 275 / pool_relay 261 tests, 13 host crates clippy, bindings clippy; the pre-existing kvbm-consolidator manifest failure is the one non-zero exit), 29 min 21 s; stage 5, gates (every kv-router feature alone and together, 6 public-API snapshots unchanged with kv-router at 918 items, layering gate 47 checks and--host66 checks with 0 failures, codeowners 7590/7590, fmt, 3 lockfiles--locked), 3 min 06 s; 40 min 35 s total, exit 0.Each replaced fork PR carried its own validation chain; see its body.
Tickets
Part of DYNO-177