Skip to content

refactor(router): point every host at the owning crates through stage host modules - #185

Closed
ishandhanani wants to merge 7 commits into
idhanani/rs2-11-kv-router-compositionfrom
idhanani/rs2-12-host-rewrite
Closed

ishandhanani wants to merge 7 commits into
idhanani/rs2-11-kv-router-compositionfrom
idhanani/rs2-12-host-rewrite

Conversation

@ishandhanani

Copy link
Copy Markdown
Owner

Router split v2, unit 12 of 17. Stacked on #184 (unit 11).

Summary

Step 8, the host rewrite: every in-tree host imports the owning crates instead of kv-router's shims. #148 points tier 1 (the builtin plugin crate, the policy catalogs, the six custom-policy examples) at dynamo_kv_router::plugins::prelude with no features; #149 gives each stage crate a host module naming exactly what in-tree hosts import from it (the explicit surface unit 14 gates); #150 ext-proc and the Python bindings, #151 lib/bench, the SGLang sidecar test, the C bindings, kvbm-consolidator, and the mocker servers' integration tests, #152 the mocker (27 files), #153 the frontend host lib/llm (108 files, 247 use lines) import dynamo_scheduler::host, the stage host modules, and the vocabulary crates; #154 makes the selection service's adapters optional inside the service so lib/llm drops the adapter features it never used. Rule applied: hosts name a stage through ::host:: only and the vocabulary crates directly; kv-router is for plugins and composition. The rebase placed three upstream changes inside this range: ai-dynamo#15600's sticky_min_load.rs takes plugins::prelude::* and the two session-affinity limits from dynamo_session_affinity (a new direct dependency of the builtin crate) at #148's commit; at #153's commit ai-dynamo#16983's deletion of lib/llm/src/http/service/generate.rs stands and its successor crate lib/generate-formats/vllm is re-pointed at dynamo_kv_hashing::routing and dynamo_kv_events::zmq_wire (its dynamo-kv-router dependency dropped), and ai-dynamo#16610's imports in lib/llm/src/kv_router/shared_cache.rs take their owner paths. Replaces fork PRs #148, #149, #150, #151, #152, #153, #154.

What moves

Dependency edges

No crate row changes. The --host check's HOST_FORBIDDEN drops prometheus (the kv-indexer export feature, kv-router metrics, reaches it by design until exposition moves to dynamo-metrics), and HOST_ADAPTER_EXEMPT="dynamo-zmq:zmq" names the one router crate allowed to reach zmq in a host's graph.

Validation

Unit-level move-diff, run from the worktree at the unit head 9243f1fc13: python3 -I scripts/router_move_diff.py 0c70aa229a 9243f1fc13 --from <1 paths> --to <1 paths>, the union of the replaced fork PRs' --from/--to lists filtered to the paths present at each end (full command in the artifact /tmp/rr-split/unit-prs/12-move-diff.txt).

summary from_normalized=55 to_normalized=51 common=51 new=0 dropped=4
result DIFFERS

0 new / 4 dropped: the #[cfg(not(feature = "metrics"))] fallback body of runtime_metrics.rs (Arc::new(KvIndexerMetrics::new_unregistered()), let _ = component;, its braces), which had no caller because lib/llm always enables metrics (#153). The other six fork PRs in the unit re-point imports and add host modules; they move no code.

New and dropped normalized lines (0 new, 4 dropped).
dropped 1x Arc::new(KvIndexerMetrics::new_unregistered())
dropped 1x let _ = component;
dropped 1x {
dropped 1x }

Chain v2, full mode, on the rebased stack head ebd31cdf4d (every unit's commits are ancestors of it; the chain ran once on the head, not per unit), log /tmp/rr-split/srebase-validation.log: stage 1, the 13 router crates (clippy under default and all features, cargo test 1,247 passed / 5 ignored, rustdoc with all features), 1 min 50 s; stage 2, kv-router (clippy, bench check, 229 tests including the prelude test), 1 min 20 s; stage 3, workspace clippy, 4 min 58 s; stage 4, hosts (dynamo-llm lib alone, its kv_router 492 / discovery 275 / pool_relay 261 tests, 13 host crates clippy, bindings clippy; the pre-existing kvbm-consolidator manifest failure is the one non-zero exit), 29 min 21 s; stage 5, gates (every kv-router feature alone and together, 6 public-API snapshots unchanged with kv-router at 918 items, layering gate 47 checks and --host 66 checks with 0 failures, codeowners 7590/7590, fmt, 3 lockfiles --locked), 3 min 06 s; 40 min 35 s total, exit 0.

Each replaced fork PR carried its own validation chain; see its body.

Tickets

Part of DYNO-181

Step 8a of the router split (DYNO-181). The builtin plugin crate, the
policy catalog, and the six custom-policy examples reached the plugin
contracts through five kv-router modules and the crate root, and
reached the registry types through services::selection, which exists
only with the selection service's adapter features. Every one of those
imports is now `use dynamo_kv_router::plugins::prelude::*;` (the kept
1.6 module paths plugins::{worker_selection, request_classifier} stay
where a file used them), and the eight crates depend on dynamo-kv-router
with no features: their normal dependency trees lose axum, hyper,
reqwest, zmq, and dynamo-zmq.

Two things sit outside the prelude on purpose. The builtin crate's
DefaultWorkerSelector is a synchronous adapter over the kernel's
selector API that the mocker's replay drives, so the crate depends on
dynamo-scheduler for selector::{WorkerSelector, WorkerSelectionInput}.
Tests and benches that drive a policy through the kernel's selector, or
compare against the kernel's reference selector, import it from
dynamo-scheduler as a dev-dependency (`bench` where the reference
selector is needed); the builtin lifecycle tests keep driving the
composed core through kv-router's local_selection_core with the adapter
features on the dev-dependency.

Validation: cargo clippy -D warnings --all-targets on the eight crates;
cargo test on the builtin plugin (56 + 5 + 1 + 3 + 20 + 3 + 4) and the
examples and catalogs (1 + 3 + 1 + 2); dynamo-scheduler 291 (testing),
clippy under testing, default, and all-features (the host list grew);
cargo check on dynamo-ext-proc, dynamo-llm, dynamo-mocker and the Python
bindings; kv-router public-api snapshot unchanged (1392 items); router
layering gate 28 checks; codeowners strict; fmt. cargo tree -e normal:
the seven policy crates go from five transport crates (axum, hyper,
reqwest, zmq, dynamo-zmq) to none; the EPP example keeps them through
dynamo-ext-proc, which is an HTTP server.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 8b of the router split (DYNO-181). In-tree hosts (the frontend in
lib/llm, the mocker, the bench harness, the Python bindings, ext-proc)
reach the stage crates through dynamo-kv-router's shim modules today,
and once those shims go they would reach into each crate's module tree
directly. Each stage crate now carries a `host` module: an explicit
`pub use` list of what an in-tree host names from it, derived from
every host's real imports (hosts inventory v2, section 3), nothing
else. kv-indexer, slot-tracker, session-affinity, worker-selection, and
admission gain one; the kernel's grows with the config, queue, overlap,
prefill-load, protocol, shared-cache, tracking-hash, and request types
hosts use, LocalScheduler for the mocker's replay, and the selection
service DTOs. Protocol, kv-events, and kv-hashing are vocabulary crates
and stay imported directly. No item moves; the host rewrites that follow
import through these lists and append what they find missing.

Two pre-existing lints surfaced because this is the first time the stage
crates ran clippy under default features and rustdoc with -D warnings:
worker-selection's select_rows matches a single arm once the reference
policy is compiled out (allowed there, with the reason), and a
slot-tracker doc link pointed at a private prelude path (now the
protocol crate's path), and three kernel doc links still used the
pre-move kv-router paths or linked a private constant (6-4b moved the
files; this is the kernel's first -D warnings rustdoc run). The
bench-only observation snapshot re-export is gated like its module.

Validation: on kv-indexer, slot-tracker, session-affinity,
worker-selection, admission, and the kernel: cargo check, cargo test
--features testing (first test binary: 583 / 130 / 10 / 41 / 22 / 291,
all green), clippy -D
warnings --all-targets under testing, default features, and
--all-features, and RUSTDOCFLAGS=-D warnings cargo doc --no-deps (the
kernel also under default features); kv-router check under the full
adapter set with all targets; cargo check on lib/llm, the mocker,
ext-proc, and the builtin plugin; kv-router public-api snapshot
unchanged; router layering gate 28 checks; codeowners strict; fmt.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…g crates

Step 8c of the router split (DYNO-181). The EPP ext-proc and the Python
bindings reached stage and kernel types through dynamo-kv-router's shim
modules (protocols, scheduling, zmq_wire, the crate root, and the
selection service's re-exports). They now import the kernel through
dynamo_scheduler::host, each stage through its host module
(kv-indexer, worker-selection, admission, session-affinity), and the
vocabulary crates (dynamo-router-protocol, dynamo-kv-events,
dynamo-kv-hashing) directly; the composition items stay on kv-router:
the flat config and its env readers, the plugin registry and
RouterPlugins, SelectionService and its builder, and the three service
entries the binaries run.

Features follow: ext-proc enables the three adapters the selection
service builder composes (ingest-zmq, replica-sync-zmq, peer-recovery)
plus metrics, and drops runtime-protocols, which it never used; the
bindings' kv-indexer, slot-tracker, and select-service features name
the adapters each binary runs, and custom-policy enables no kv-router
feature (the catalog needs none since 8a). The bindings manifest, which
lives outside the workspace, gains path dependencies on the owner
crates. The host lists grow by what these two hosts needed:
KvTransferEnforcement on the kernel, ApproximateCachePolicyKind on
kv-indexer, ConditionalDisaggPolicyKind on worker-selection.

The bindings' `protocols::*` glob had hidden three things: a name it
never used (WorkerWithDpRank), one used only in a comment
(PotentialLoad), and two the file shadows with its own pyclasses
(OverlapScores, RoutingConstraints), which the explicit imports now
leave out; the two Rust-side OverlapScores uses are fully qualified.

The layering gate's --host mode, which CI runs, failed on every host
before this change for two reasons that have nothing to do with hosts:
dynamo-zmq is the ZMQ socket helper and reaches zmq by definition, and
the kv-indexer `export` feature (kv-router `metrics`) reaches prometheus
by design until step 9c moves exposition to dynamo-metrics. The gate now
exempts the declared adapter crate for its own dependency
(HOST_ADAPTER_EXEMPT) and lists prometheus as a 9c return; with that,
--host passes on lib/llm and ext-proc.

Validation: ext-proc clippy -D warnings --all-targets and tests (167 +
1); the EPP example checks; the Python bindings check and clippy -D
warnings under default features, no default features, and
kv-indexer,slot-tracker,select-service; kv-router check under the full
adapter set; the three host crates check; kv-router public-api snapshot
unchanged; router layering gate --host 42 checks, 0 failures (it failed
on every host before: see above); codeowners strict; fmt on the
workspace and the bindings. cargo tree -p dynamo-ext-proc -e normal
names kv-router, protocol, scheduler, session-affinity, kv-events, and
the runtime; zmq and reqwest remain through the adapters it enables.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…tor at the owners

Step 8d of the router split (DYNO-181). lib/bench, the SGLang sidecar's
headless test, the C bindings, kvbm-consolidator, and the mocker
servers' integration tests reached indexer, slot-tracker, and wire types
through dynamo-kv-router's shim modules. They now import the KV indexer
and slot-tracker stages through their host modules, the DC relay's
cuckoo module directly (it stays a public module of kv-indexer), and the
vocabulary crates (dynamo-router-protocol, dynamo-kv-events,
dynamo-kv-hashing) directly. Only the bench keeps dynamo-kv-router, for
the flat config its offline replay bench parses; the sidecar, the C
bindings, the consolidator, and the mocker servers drop it.

The kv-indexer host list grows with the assembly the bench harness
builds by hand (KvIndexer, ThreadPoolIndexer, PositionalIndexer,
BranchShardedIndexer, the lower-tier and approximate-LRU types, the
metric names, the residency projections) and the slot-tracker list with
PrefillLoadHint, SequenceError, and SequenceRequest.

One correction: the bench's parity tests counted warn and error logs by
the targets dynamo_kv_router::indexer and dynamo_kv_router::sequences,
which no crate has emitted since the indexer and slot-tracker moved out
(steps 5a and 5b), so the counters could never fire. They now watch
dynamo_kv_indexer and dynamo_slot_tracker, and the active-sequences
bench's log filter names dynamo_slot_tracker.

The mocker's manifest gains kv-router's standalone-selection alias for
one step: the mocker still reads the session-affinity constants through
kv-router's selection service module, and since 8a (the builtin plugin
no longer enables that module) nothing in the mocker's own dependency
graph turned it on, so `cargo check -p dynamo-mocker` on its own failed
while workspace-wide builds, unified with lib/llm's features, hid it.
8e flips the mocker to dynamo_session_affinity and drops the alias.

Validation: cargo check -p dynamo-mocker alone passes again; kv-indexer
and slot-tracker (host lists grew) check and clippy -D warnings under
default and all features; lib/bench clippy -D warnings --all-targets
with router-bench,router-metrics and under default features; cargo test
-p dynamo-bench (37 + 9 + 18) and --features replay-bench for the
mooncake and active-sequences parity tests (9 + 18, with the corrected
log targets); the sidecar clippy and lib tests (123); the two mocker
server crates clippy; the C bindings check, clippy, and fmt; kv-router
public-api unchanged; router layering gate 28 checks; codeowners
strict; fmt. kvbm-consolidator is not a member of this workspace and
inherits `kvbm-logical`, which the root does not define, so it cannot
be built from this tree (before or after this change); its two import
flips and manifest were checked by inspection against the exporting
crates.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 8e of the router split (DYNO-181). The mocker's replay harness
(online and offline), its load generator, its engine observations, and
its ZMQ event service reached kernel, stage, and wire types through
dynamo-kv-router's shim modules. They now import the kernel through
dynamo_scheduler::host, each stage through its host module, and the
vocabulary crates directly; kv-router keeps the flat KvRouterConfig the
replay is configured with, its tracking-hash resolver, the plugin
surface, and the YAML-side BusyThresholdsExt.

LocalScheduler::new stays the replay harness's composition entry
(reached through the kernel's host module). The design's alternative,
composing the replay through SelectionCore behind a replay feature,
changes how replay drives the scheduler and needs a parity run; it is
deferred, not decided here.

Host lists grow with the replay's names: the kernel's with
SchedulingContext, ScheduleRequest, TrackingHashContext,
WorkerCapacityExt; session-affinity's with the table API the offline
extension drives (SessionAffinity, AffinityLease, Hold, AcquireStep,
AffinityError, subagent_group_affinity_id); slot-tracker's with
BusyRanksExt; admission's with WorkerPlacement; kv-indexer's with
SessionPrefixIndexer.

Validation: the five host crates clippy -D warnings --all-features
--all-targets (lists grew); mocker clippy -D warnings --all-targets
under default features and replay-bench; mocker lib tests 304; each
dependent checked alone (dynamo-llm, dynamo-bench, the SGLang and vLLM
mocker servers, ext-proc) plus the Python bindings; kv-router
public-api snapshot unchanged; router layering gate --host 42 checks, 0
failures (the mocker's own tree has no axum, hyper, reqwest, or zmq);
codeowners strict; fmt.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 8f of the router split (DYNO-181), the last host. lib/llm reached
every router type through dynamo-kv-router's shim modules: protocols,
identity, indexer, scheduling, selector, sequences, kv_hints, zmq_wire,
the crate root, and the selection service's re-exports. It now imports
the kernel through dynamo_scheduler::host, each stage through its host
module, and the vocabulary crates directly; kv-router keeps what it
owns: the flat KvRouterConfig and its env reader, the plugin registry
and RouterPlugins, and the kept plugins::{worker_selection,
request_classifier} paths for the plugin contract the frontend hosts.

The rewrite is table-driven (one target per kv-router path lib/llm
named, derived from the owner map and the host lists); the three
`protocols::*` and two `zmq_wire::*` globs become the explicit names
each file and its `use super::*` children use. KvIndexerMetricsExt, the
one runtime-backed constructor kv-router carried behind
runtime-protocols, moves into lib/llm's indexer module, so kv-router
drops its optional dynamo-runtime dependency and the feature stays as an
empty, deprecated name. The host lists grow with what the frontend
names (eligibility and booking types on the kernel, the indexer's
service DTOs and identity specs, the slot tracker's replication wire,
the admission queue's limit kind, the conditional-disaggregation
policy).

lib/llm depends on the nine owner crates directly and on kv-router with
`metrics` plus the three adapter features (`ingest-zmq`,
`replica-sync-zmq`, `peer-recovery`) that gate the whole
`services::selection` module; `http` and `runtime-protocols` are gone.
The frontend supplies its own index ingress and replica channels and
never sets indexer peers or a replica mesh, so the next step gates the
adapters inside the service and lib/llm drops them; until then its
dependency tree reaches zmq and reqwest through kv-router exactly as
before this change.

Validation: host crates clippy --all-features --all-targets -D warnings
and rustdoc -D warnings; kv-router clippy over the adapter features
with --all-targets, and its --all-features tree has no dynamo-runtime;
lib/llm clippy --all-targets -D warnings, `cargo check --lib` alone,
4000 lib tests (15 ignored) and every integration binary green; each
dependent checked alone (ext-proc, mocker, bench, custom-policy-builtin,
sglang-sidecar, sglang-mocker, vllm-mocker clippy --tests -D warnings;
the Python bindings with kv-indexer,slot-tracker,select-service; the
EPP example); lib/llm's depth-1 tree names kv-router, protocol,
kv-events, kv-hashing, and the six stage/kernel crates, and kv-router
resolves without http; public-api snapshot unchanged (1392 items);
layering gate --host 42 checks; codeowners strict; fmt; cargo hack
--each-feature on kv-router (15 configurations).

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 8g of the router split (DYNO-181). Step 7b gated
`services::selection` as one unit on `ingest-zmq`, `replica-sync-zmq`,
and `peer-recovery`, because the service builder composes the ZMQ
ingress, the replica mesh, and peer recovery. The frontend host uses
the same builder with its own index ingress and replica channels and
none of the adapters, so it had to enable all three to compile the
module (step 8f kept them for that reason alone).

The module now compiles with no adapter feature. The ZMQ-direct ingress
and the builder's ZMQ default index source need `ingest-zmq`; indexer
peers and the startup recovery need `ingest-zmq` + `peer-recovery`; the
replica mesh (`replica_sync`, the peer manager, the replica-peer
methods, the mesh shutdown) needs `replica-sync-zmq`; the standalone
CLI config and the HTTP server need all three plus `http`. Without the
ZMQ ingress the builder's default host is the kernel's passive ingress.
Behavior under any feature set that compiled before is unchanged: every
gated item keeps its body, and `IndexerPolicy` is imported from its
owner crate instead of through the gated indexer service module.

lib/llm depends on kv-router with `metrics` only; its dependency tree
no longer reaches zmq or reqwest through a router crate. ext-proc and
the Python bindings keep the features their services use.

Validation: cargo hack over every kv-router feature alone and together
(15 configurations, --all-targets); clippy -D warnings --all-targets
for ten feature combinations from none to the full set; kv-router
tests under none (119), ingest-zmq (139), replica-sync-zmq (123), and
the full set (225 + 7); rustdoc -D warnings under none and the full
set; lib/llm `cargo check --lib` alone, clippy --all-targets, 4000 lib
tests (15 ignored) and every integration binary; seven dependents
checked alone, the Python bindings, the EPP example, and ext-proc's
168 tests; lib/llm resolves kv-router with `default,metrics` and the
kv-router subtree under `metrics` has no zmq, dynamo-zmq, reqwest,
axum, or hyper; public-api snapshot unchanged (1392 items); layering
gate --host 42 checks; codeowners strict; fmt.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Owner Author

Landed upstream as ai-dynamo#18055 (squash commit 79692cf on main, 2026-11-11). Closing this fork review unit.

This branch had an error being deployed

1 failed deployment
external_collaborator — 9243f1fc Deployed Nov 2, 2026 by ishandhanani via ok-to-test #524
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation frontend refactor router

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant