Repository navigation
refactor(router): point every host at the owning crates through stage host modules - #185
Closed
ishandhanani wants to merge 7 commits into
Closed
ishandhanani wants to merge 7 commits into
ishandhanani wants to merge 7 commits into
Conversation
Step 8a of the router split (DYNO-181). The builtin plugin crate, the
policy catalog, and the six custom-policy examples reached the plugin
contracts through five kv-router modules and the crate root, and
reached the registry types through services::selection, which exists
only with the selection service's adapter features. Every one of those
imports is now `use dynamo_kv_router::plugins::prelude::*;` (the kept
1.6 module paths plugins::{worker_selection, request_classifier} stay
where a file used them), and the eight crates depend on dynamo-kv-router
with no features: their normal dependency trees lose axum, hyper,
reqwest, zmq, and dynamo-zmq.
Two things sit outside the prelude on purpose. The builtin crate's
DefaultWorkerSelector is a synchronous adapter over the kernel's
selector API that the mocker's replay drives, so the crate depends on
dynamo-scheduler for selector::{WorkerSelector, WorkerSelectionInput}.
Tests and benches that drive a policy through the kernel's selector, or
compare against the kernel's reference selector, import it from
dynamo-scheduler as a dev-dependency (`bench` where the reference
selector is needed); the builtin lifecycle tests keep driving the
composed core through kv-router's local_selection_core with the adapter
features on the dev-dependency.
Validation: cargo clippy -D warnings --all-targets on the eight crates;
cargo test on the builtin plugin (56 + 5 + 1 + 3 + 20 + 3 + 4) and the
examples and catalogs (1 + 3 + 1 + 2); dynamo-scheduler 291 (testing),
clippy under testing, default, and all-features (the host list grew);
cargo check on dynamo-ext-proc, dynamo-llm, dynamo-mocker and the Python
bindings; kv-router public-api snapshot unchanged (1392 items); router
layering gate 28 checks; codeowners strict; fmt. cargo tree -e normal:
the seven policy crates go from five transport crates (axum, hyper,
reqwest, zmq, dynamo-zmq) to none; the EPP example keeps them through
dynamo-ext-proc, which is an HTTP server.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 8b of the router split (DYNO-181). In-tree hosts (the frontend in lib/llm, the mocker, the bench harness, the Python bindings, ext-proc) reach the stage crates through dynamo-kv-router's shim modules today, and once those shims go they would reach into each crate's module tree directly. Each stage crate now carries a `host` module: an explicit `pub use` list of what an in-tree host names from it, derived from every host's real imports (hosts inventory v2, section 3), nothing else. kv-indexer, slot-tracker, session-affinity, worker-selection, and admission gain one; the kernel's grows with the config, queue, overlap, prefill-load, protocol, shared-cache, tracking-hash, and request types hosts use, LocalScheduler for the mocker's replay, and the selection service DTOs. Protocol, kv-events, and kv-hashing are vocabulary crates and stay imported directly. No item moves; the host rewrites that follow import through these lists and append what they find missing. Two pre-existing lints surfaced because this is the first time the stage crates ran clippy under default features and rustdoc with -D warnings: worker-selection's select_rows matches a single arm once the reference policy is compiled out (allowed there, with the reason), and a slot-tracker doc link pointed at a private prelude path (now the protocol crate's path), and three kernel doc links still used the pre-move kv-router paths or linked a private constant (6-4b moved the files; this is the kernel's first -D warnings rustdoc run). The bench-only observation snapshot re-export is gated like its module. Validation: on kv-indexer, slot-tracker, session-affinity, worker-selection, admission, and the kernel: cargo check, cargo test --features testing (first test binary: 583 / 130 / 10 / 41 / 22 / 291, all green), clippy -D warnings --all-targets under testing, default features, and --all-features, and RUSTDOCFLAGS=-D warnings cargo doc --no-deps (the kernel also under default features); kv-router check under the full adapter set with all targets; cargo check on lib/llm, the mocker, ext-proc, and the builtin plugin; kv-router public-api snapshot unchanged; router layering gate 28 checks; codeowners strict; fmt. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…g crates Step 8c of the router split (DYNO-181). The EPP ext-proc and the Python bindings reached stage and kernel types through dynamo-kv-router's shim modules (protocols, scheduling, zmq_wire, the crate root, and the selection service's re-exports). They now import the kernel through dynamo_scheduler::host, each stage through its host module (kv-indexer, worker-selection, admission, session-affinity), and the vocabulary crates (dynamo-router-protocol, dynamo-kv-events, dynamo-kv-hashing) directly; the composition items stay on kv-router: the flat config and its env readers, the plugin registry and RouterPlugins, SelectionService and its builder, and the three service entries the binaries run. Features follow: ext-proc enables the three adapters the selection service builder composes (ingest-zmq, replica-sync-zmq, peer-recovery) plus metrics, and drops runtime-protocols, which it never used; the bindings' kv-indexer, slot-tracker, and select-service features name the adapters each binary runs, and custom-policy enables no kv-router feature (the catalog needs none since 8a). The bindings manifest, which lives outside the workspace, gains path dependencies on the owner crates. The host lists grow by what these two hosts needed: KvTransferEnforcement on the kernel, ApproximateCachePolicyKind on kv-indexer, ConditionalDisaggPolicyKind on worker-selection. The bindings' `protocols::*` glob had hidden three things: a name it never used (WorkerWithDpRank), one used only in a comment (PotentialLoad), and two the file shadows with its own pyclasses (OverlapScores, RoutingConstraints), which the explicit imports now leave out; the two Rust-side OverlapScores uses are fully qualified. The layering gate's --host mode, which CI runs, failed on every host before this change for two reasons that have nothing to do with hosts: dynamo-zmq is the ZMQ socket helper and reaches zmq by definition, and the kv-indexer `export` feature (kv-router `metrics`) reaches prometheus by design until step 9c moves exposition to dynamo-metrics. The gate now exempts the declared adapter crate for its own dependency (HOST_ADAPTER_EXEMPT) and lists prometheus as a 9c return; with that, --host passes on lib/llm and ext-proc. Validation: ext-proc clippy -D warnings --all-targets and tests (167 + 1); the EPP example checks; the Python bindings check and clippy -D warnings under default features, no default features, and kv-indexer,slot-tracker,select-service; kv-router check under the full adapter set; the three host crates check; kv-router public-api snapshot unchanged; router layering gate --host 42 checks, 0 failures (it failed on every host before: see above); codeowners strict; fmt on the workspace and the bindings. cargo tree -p dynamo-ext-proc -e normal names kv-router, protocol, scheduler, session-affinity, kv-events, and the runtime; zmq and reqwest remain through the adapters it enables. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…tor at the owners Step 8d of the router split (DYNO-181). lib/bench, the SGLang sidecar's headless test, the C bindings, kvbm-consolidator, and the mocker servers' integration tests reached indexer, slot-tracker, and wire types through dynamo-kv-router's shim modules. They now import the KV indexer and slot-tracker stages through their host modules, the DC relay's cuckoo module directly (it stays a public module of kv-indexer), and the vocabulary crates (dynamo-router-protocol, dynamo-kv-events, dynamo-kv-hashing) directly. Only the bench keeps dynamo-kv-router, for the flat config its offline replay bench parses; the sidecar, the C bindings, the consolidator, and the mocker servers drop it. The kv-indexer host list grows with the assembly the bench harness builds by hand (KvIndexer, ThreadPoolIndexer, PositionalIndexer, BranchShardedIndexer, the lower-tier and approximate-LRU types, the metric names, the residency projections) and the slot-tracker list with PrefillLoadHint, SequenceError, and SequenceRequest. One correction: the bench's parity tests counted warn and error logs by the targets dynamo_kv_router::indexer and dynamo_kv_router::sequences, which no crate has emitted since the indexer and slot-tracker moved out (steps 5a and 5b), so the counters could never fire. They now watch dynamo_kv_indexer and dynamo_slot_tracker, and the active-sequences bench's log filter names dynamo_slot_tracker. The mocker's manifest gains kv-router's standalone-selection alias for one step: the mocker still reads the session-affinity constants through kv-router's selection service module, and since 8a (the builtin plugin no longer enables that module) nothing in the mocker's own dependency graph turned it on, so `cargo check -p dynamo-mocker` on its own failed while workspace-wide builds, unified with lib/llm's features, hid it. 8e flips the mocker to dynamo_session_affinity and drops the alias. Validation: cargo check -p dynamo-mocker alone passes again; kv-indexer and slot-tracker (host lists grew) check and clippy -D warnings under default and all features; lib/bench clippy -D warnings --all-targets with router-bench,router-metrics and under default features; cargo test -p dynamo-bench (37 + 9 + 18) and --features replay-bench for the mooncake and active-sequences parity tests (9 + 18, with the corrected log targets); the sidecar clippy and lib tests (123); the two mocker server crates clippy; the C bindings check, clippy, and fmt; kv-router public-api unchanged; router layering gate 28 checks; codeowners strict; fmt. kvbm-consolidator is not a member of this workspace and inherits `kvbm-logical`, which the root does not define, so it cannot be built from this tree (before or after this change); its two import flips and manifest were checked by inspection against the exporting crates. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 8e of the router split (DYNO-181). The mocker's replay harness (online and offline), its load generator, its engine observations, and its ZMQ event service reached kernel, stage, and wire types through dynamo-kv-router's shim modules. They now import the kernel through dynamo_scheduler::host, each stage through its host module, and the vocabulary crates directly; kv-router keeps the flat KvRouterConfig the replay is configured with, its tracking-hash resolver, the plugin surface, and the YAML-side BusyThresholdsExt. LocalScheduler::new stays the replay harness's composition entry (reached through the kernel's host module). The design's alternative, composing the replay through SelectionCore behind a replay feature, changes how replay drives the scheduler and needs a parity run; it is deferred, not decided here. Host lists grow with the replay's names: the kernel's with SchedulingContext, ScheduleRequest, TrackingHashContext, WorkerCapacityExt; session-affinity's with the table API the offline extension drives (SessionAffinity, AffinityLease, Hold, AcquireStep, AffinityError, subagent_group_affinity_id); slot-tracker's with BusyRanksExt; admission's with WorkerPlacement; kv-indexer's with SessionPrefixIndexer. Validation: the five host crates clippy -D warnings --all-features --all-targets (lists grew); mocker clippy -D warnings --all-targets under default features and replay-bench; mocker lib tests 304; each dependent checked alone (dynamo-llm, dynamo-bench, the SGLang and vLLM mocker servers, ext-proc) plus the Python bindings; kv-router public-api snapshot unchanged; router layering gate --host 42 checks, 0 failures (the mocker's own tree has no axum, hyper, reqwest, or zmq); codeowners strict; fmt. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 8f of the router split (DYNO-181), the last host. lib/llm reached
every router type through dynamo-kv-router's shim modules: protocols,
identity, indexer, scheduling, selector, sequences, kv_hints, zmq_wire,
the crate root, and the selection service's re-exports. It now imports
the kernel through dynamo_scheduler::host, each stage through its host
module, and the vocabulary crates directly; kv-router keeps what it
owns: the flat KvRouterConfig and its env reader, the plugin registry
and RouterPlugins, and the kept plugins::{worker_selection,
request_classifier} paths for the plugin contract the frontend hosts.
The rewrite is table-driven (one target per kv-router path lib/llm
named, derived from the owner map and the host lists); the three
`protocols::*` and two `zmq_wire::*` globs become the explicit names
each file and its `use super::*` children use. KvIndexerMetricsExt, the
one runtime-backed constructor kv-router carried behind
runtime-protocols, moves into lib/llm's indexer module, so kv-router
drops its optional dynamo-runtime dependency and the feature stays as an
empty, deprecated name. The host lists grow with what the frontend
names (eligibility and booking types on the kernel, the indexer's
service DTOs and identity specs, the slot tracker's replication wire,
the admission queue's limit kind, the conditional-disaggregation
policy).
lib/llm depends on the nine owner crates directly and on kv-router with
`metrics` plus the three adapter features (`ingest-zmq`,
`replica-sync-zmq`, `peer-recovery`) that gate the whole
`services::selection` module; `http` and `runtime-protocols` are gone.
The frontend supplies its own index ingress and replica channels and
never sets indexer peers or a replica mesh, so the next step gates the
adapters inside the service and lib/llm drops them; until then its
dependency tree reaches zmq and reqwest through kv-router exactly as
before this change.
Validation: host crates clippy --all-features --all-targets -D warnings
and rustdoc -D warnings; kv-router clippy over the adapter features
with --all-targets, and its --all-features tree has no dynamo-runtime;
lib/llm clippy --all-targets -D warnings, `cargo check --lib` alone,
4000 lib tests (15 ignored) and every integration binary green; each
dependent checked alone (ext-proc, mocker, bench, custom-policy-builtin,
sglang-sidecar, sglang-mocker, vllm-mocker clippy --tests -D warnings;
the Python bindings with kv-indexer,slot-tracker,select-service; the
EPP example); lib/llm's depth-1 tree names kv-router, protocol,
kv-events, kv-hashing, and the six stage/kernel crates, and kv-router
resolves without http; public-api snapshot unchanged (1392 items);
layering gate --host 42 checks; codeowners strict; fmt; cargo hack
--each-feature on kv-router (15 configurations).
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 8g of the router split (DYNO-181). Step 7b gated `services::selection` as one unit on `ingest-zmq`, `replica-sync-zmq`, and `peer-recovery`, because the service builder composes the ZMQ ingress, the replica mesh, and peer recovery. The frontend host uses the same builder with its own index ingress and replica channels and none of the adapters, so it had to enable all three to compile the module (step 8f kept them for that reason alone). The module now compiles with no adapter feature. The ZMQ-direct ingress and the builder's ZMQ default index source need `ingest-zmq`; indexer peers and the startup recovery need `ingest-zmq` + `peer-recovery`; the replica mesh (`replica_sync`, the peer manager, the replica-peer methods, the mesh shutdown) needs `replica-sync-zmq`; the standalone CLI config and the HTTP server need all three plus `http`. Without the ZMQ ingress the builder's default host is the kernel's passive ingress. Behavior under any feature set that compiled before is unchanged: every gated item keeps its body, and `IndexerPolicy` is imported from its owner crate instead of through the gated indexer service module. lib/llm depends on kv-router with `metrics` only; its dependency tree no longer reaches zmq or reqwest through a router crate. ext-proc and the Python bindings keep the features their services use. Validation: cargo hack over every kv-router feature alone and together (15 configurations, --all-targets); clippy -D warnings --all-targets for ten feature combinations from none to the full set; kv-router tests under none (119), ingest-zmq (139), replica-sync-zmq (123), and the full set (225 + 7); rustdoc -D warnings under none and the full set; lib/llm `cargo check --lib` alone, clippy --all-targets, 4000 lib tests (15 ignored) and every integration binary; seven dependents checked alone, the Python bindings, the EPP example, and ext-proc's 168 tests; lib/llm resolves kv-router with `default,metrics` and the kv-router subtree under `metrics` has no zmq, dynamo-zmq, reqwest, axum, or hyper; public-api snapshot unchanged (1392 items); layering gate --host 42 checks; codeowners strict; fmt. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
had a problem deploying
to
external_collaborator
November 2, 2026 13:53 — with
GitHub Actions
Failure
Owner
Author
|
Landed upstream as ai-dynamo#18055 (squash commit 79692cf on main, 2026-11-11). Closing this fork review unit. |
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Router split v2, unit 12 of 17. Stacked on #184 (unit 11).
Summary
Step 8, the host rewrite: every in-tree host imports the owning crates instead of kv-router's shims. #148 points tier 1 (the builtin plugin crate, the policy catalogs, the six custom-policy examples) at
dynamo_kv_router::plugins::preludewith no features; #149 gives each stage crate ahostmodule naming exactly what in-tree hosts import from it (the explicit surface unit 14 gates); #150 ext-proc and the Python bindings, #151lib/bench, the SGLang sidecar test, the C bindings, kvbm-consolidator, and the mocker servers' integration tests, #152 the mocker (27 files), #153 the frontend hostlib/llm(108 files, 247uselines) importdynamo_scheduler::host, the stage host modules, and the vocabulary crates; #154 makes the selection service's adapters optional inside the service so lib/llm drops the adapter features it never used. Rule applied: hosts name a stage through::host::only and the vocabulary crates directly; kv-router is for plugins and composition. The rebase placed three upstream changes inside this range: ai-dynamo#15600'ssticky_min_load.rstakesplugins::prelude::*and the two session-affinity limits fromdynamo_session_affinity(a new direct dependency of the builtin crate) at #148's commit; at #153's commit ai-dynamo#16983's deletion oflib/llm/src/http/service/generate.rsstands and its successor cratelib/generate-formats/vllmis re-pointed atdynamo_kv_hashing::routinganddynamo_kv_events::zmq_wire(itsdynamo-kv-routerdependency dropped), and ai-dynamo#16610's imports inlib/llm/src/kv_router/shared_cache.rstake their owner paths. Replaces fork PRs #148, #149, #150, #151, #152, #153, #154.What moves
670424df9brefactor(router): import the plugin surface through plugins::prelude: the builtin plugin crate, both catalogs, and the six policy examples; import re-points only, no move-diff.baaaa941c6feat(router): name each stage's host surface in a host module:host.rson kv-indexer, slot-tracker, session-affinity, worker-selection, admission, and the kernel; additive only, no move-diff.413162b3ebrefactor(router): point ext-proc and the Python bindings at the owning crates: import re-points only, no move-diff.b7ae6ecb4crefactor(router): point the bench, sidecar, C bindings, and consolidator at the owners:lib/bench(17 files), the SGLang sidecar test, the C bindings, kvbm-consolidator, the mocker servers' integration tests; import re-points only, no move-diff.a6fa878bcbrefactor(router): point the mocker at the owning crates: 27 files; import re-points only, no move-diff.c8d516f153refactor(router): point the frontend host at the owning crates: lib/llm (108 files, table-driven), andindexer/runtime_metrics.rsmoves from kv-router into lib/llm. Move-diff (its body,HEAD~1 HEAD --from lib/kv-router/src/indexer/runtime_metrics.rs --to lib/llm/src/kv_router/indexer/runtime_metrics.rs): 55 normalized lines in, 51 out, common 51, new 0, dropped 4 (the#[cfg(not(feature = "metrics"))]fallback body; lib/llm always enablesmetrics, so it had no caller).9243f1fc13refactor(router): make the selection service's adapters optional:services::selectionungated,ZmqDirectIngressand the builder's ZMQ default behindingest-zmq; lib/llm drops the adapter features; not a move; no move-diff.Dependency edges
No crate row changes. The
--hostcheck'sHOST_FORBIDDENdropsprometheus(the kv-indexerexportfeature, kv-routermetrics, reaches it by design until exposition moves todynamo-metrics), andHOST_ADAPTER_EXEMPT="dynamo-zmq:zmq"names the one router crate allowed to reachzmqin a host's graph.Validation
Unit-level move-diff, run from the worktree at the unit head
9243f1fc13:python3 -I scripts/router_move_diff.py 0c70aa229a 9243f1fc13 --from <1 paths> --to <1 paths>, the union of the replaced fork PRs'--from/--tolists filtered to the paths present at each end (full command in the artifact/tmp/rr-split/unit-prs/12-move-diff.txt).0 new / 4 dropped: the
#[cfg(not(feature = "metrics"))]fallback body ofruntime_metrics.rs(Arc::new(KvIndexerMetrics::new_unregistered()),let _ = component;, its braces), which had no caller because lib/llm always enablesmetrics(#153). The other six fork PRs in the unit re-point imports and add host modules; they move no code.New and dropped normalized lines (0 new, 4 dropped).
Chain v2, full mode, on the rebased stack head
ebd31cdf4d(every unit's commits are ancestors of it; the chain ran once on the head, not per unit), log/tmp/rr-split/srebase-validation.log: stage 1, the 13 router crates (clippy under default and all features,cargo test1,247 passed / 5 ignored, rustdoc with all features), 1 min 50 s; stage 2, kv-router (clippy, bench check, 229 tests including the prelude test), 1 min 20 s; stage 3, workspace clippy, 4 min 58 s; stage 4, hosts (dynamo-llmlib alone, its kv_router 492 / discovery 275 / pool_relay 261 tests, 13 host crates clippy, bindings clippy; the pre-existing kvbm-consolidator manifest failure is the one non-zero exit), 29 min 21 s; stage 5, gates (every kv-router feature alone and together, 6 public-API snapshots unchanged with kv-router at 918 items, layering gate 47 checks and--host66 checks with 0 failures, codeowners 7590/7590, fmt, 3 lockfiles--locked), 3 min 06 s; 40 min 35 s total, exit 0.Each replaced fork PR carried its own validation chain; see its body.
Tickets
Part of DYNO-181