Skip to content

refactor(router): protocol cleanup: load, hints, residency, candidates, multimodal, and ActiveLoad to their owners - #189

Closed
ishandhanani wants to merge 7 commits into
idhanani/rs2-15-dynamo-errorfrom
idhanani/rs2-16-protocol-cleanup
Closed

ishandhanani wants to merge 7 commits into
idhanani/rs2-15-dynamo-errorfrom
idhanani/rs2-16-protocol-cleanup

Conversation

@ishandhanani

Copy link
Copy Markdown
Owner

Router split v2, unit 16 of 17. Stacked on #188 (unit 15).

Summary

Protocol cleanup under the rule decided on 2026-11-01: a stage may depend on another stage for that stage's output types, and protocol keeps counterparty wire and identity only (stage edges over protocol hoisting). #163 dissolves protocol::load: the fullness rule (RankLoad, BusyThresholds) and the scheduler-load wire (SchedulerLoad, SchedulerGroup) go to the slot tracker, ActiveLoad, WorkerCapacity, and DEFAULT_MAX_BATCHED_TOKENS to protocol::worker, and worker-selection and admission gain a slot-tracker edge. #165 adds dynamo-kv-hints (lib/kv-hints, tier 1): the KV transfer hints the router hands engines, the KvHint wire and KV_HINT_TRANSFER_* keys, the transfer source choice, the worker transfer keys. #166 moves the residency vocabulary (tiers, domains, owners) to dynamo-kv-events. #169 makes kv-indexer own the transfer candidate vocabulary (KvTransferCandidateSource, label hashes) and the residency owner key, with kv-hints re-exporting from kv-indexer. #170 moves the block hash identities LocalBlockHash and ExternalSequenceBlockHash to kv-hashing and adds the leaf dynamo-multimodal (lib/multimodal, tier 1) for the multimodal identity both ends of the wire compute, so protocol has no relation to hashing or multimodal identity. #172 moves ActiveLoad to lib/llm, where its publisher and every reader live. After this unit protocol's modules are attempt columns identity metadata session worker worker_type. The rebase placed upstream ai-dynamo#16983 inside this range: at #170's second commit the generate.rs deletion stands and lib/generate-formats/vllm takes hash_mm_identifier, pad_value_for_mm_hash, and mark_mm_hash_for_extra_key from dynamo_multimodal; the ai-dynamo#17700 SelectionLogFields re-export and the ai-dynamo#16610 StorageTier/ExternalSequenceBlockHash imports in lib/llm follow their owners at #165's, #166's, and #170's commits. Replaces fork PRs #163, #165, #166, #169, #170 (two commits), #172.

What moves

  • refactor(router): move the fullness rule and scheduler-load wire to the slot tracker #163 0cb7114fe4 refactor(router): move the fullness rule and scheduler-load wire to the slot tracker: RankLoad and BusyThresholds to dynamo_slot_tracker::busy, SchedulerLoad and SchedulerGroup to dynamo_slot_tracker::wire, ActiveLoad/WorkerCapacity/DEFAULT_MAX_BATCHED_TOKENS to protocol::worker; protocol::load removed. Move-diff (its body): from_normalized=296 to_normalized=296 common=296 new=0 dropped=0, result MOVE (only comments changed in the moved text).
  • refactor(router): add dynamo-kv-hints, the hints the router hands engines #165 23e5b6d053 refactor(router): add dynamo-kv-hints, the hints the router hands engines: wire, keys, transfer from protocol's kv_hints, worker-selection's kv_transfer.rs, and the kernel's kv_hints/protocols. Move-diff (its body): from_normalized=1149 to_normalized=1229 common=1138 new=91 dropped=11, result DIFFERS by design, every differing line listed with its reason in the body: the config-reading WorkerKvTransferKeysExt::of body and the config-taking cache method dropped; WorkerKvTransferKeys::from_worker over fields, the kernel adapter's six-argument call, the closure-taking get_or_init, and the kernel test adapter_equals_the_direct_call new.
  • refactor(router): move the residency vocabulary to dynamo-kv-events #166 28000ec2e6 refactor(router): move the residency vocabulary to dynamo-kv-events: git mv lib/router/protocol/src/residency.rs lib/kv-events/src/residency.rs, body unchanged. Move-diff (its body): from_normalized=190 to_normalized=190 common=190 new=0 dropped=0, result MOVE.
  • refactor(router): kv-indexer owns the transfer candidate vocabulary #169 8122fcdee6 refactor(router): kv-indexer owns the transfer candidate vocabulary: KvTransferCandidateSource, the label hashes, and ResidencyOwnerKey into kv-indexer; kv-hints and kv-events re-point. Move-diff (its body, over the five files): from_normalized=573 to_normalized=573 common=573 new=0 dropped=0, result MOVE.
  • refactor(router): add dynamo-multimodal and move the block hash identities to dynamo-kv-hashing #170 77135c3ee8 refactor(router): move the block hash identities to dynamo-kv-hashing: block_hash.rs whole into kv-hashing, re-exported at its root. Move-diff (its body): from_normalized=67 to_normalized=67 common=67 new=0 dropped=0, result MOVE.
  • refactor(router): add dynamo-multimodal and move the block hash identities to dynamo-kv-hashing #170 9e05aaf294 feat(router): add dynamo-multimodal, the multimodal identity contract engines and the router share: mm.rs to multimodal/src/info.rs, the pad rules from kv-hashing's routing.rs to pad.rs, the extra-key marker from kv-events' extra_keys.rs to extra_key.rs. Move-diff (its body): from_normalized=1132 to_normalized=1137 common=1132 new=5 dropped=0, result DIFFERS by the tool's strict definition; the five new lines are #![deny(missing_docs)] and the mod tests { } wrappers of pad.rs and extra_key.rs, nothing dropped.
  • refactor(router): move ActiveLoad to lib/llm #172 3b6c929486 refactor(router): move ActiveLoad to lib/llm: the struct cut verbatim from protocol/src/worker.rs into lib/llm/src/kv_router/active_load.rs. Move-diff (its body): from_normalized=66 to_normalized=66 common=66 new=0 dropped=0, result MOVE.

Dependency edges

  • dynamo-multimodal -> no router crate; direct: serde, xxhash-rust
  • dynamo-kv-hints -> dynamo-router-protocol, dynamo-kv-events, dynamo-kv-hashing, dynamo-kv-indexer, dynamo-multimodal (router deps); direct: dynamo-kv-hashing, dynamo-kv-indexer, dynamo-router-protocol, rustc-hash, serde, serde_json
  • dynamo-kv-hashing -> dynamo-multimodal (was dynamo-router-protocol, now forbidden in its tree); direct: derive_builder, dynamo-multimodal, dynamo-tokens, serde, serde_json, thiserror, xxhash-rust
  • dynamo-kv-events -> dynamo-router-protocol, dynamo-kv-hashing, dynamo-multimodal; direct gains dynamo-multimodal
  • dynamo-kv-indexer -> dynamo-router-protocol, dynamo-kv-events, dynamo-kv-hashing, dynamo-multimodal (transitive); direct unchanged
  • dynamo-worker-selection -> dynamo-router-protocol, dynamo-slot-tracker; direct gains dynamo-slot-tracker
  • dynamo-admission -> dynamo-router-protocol, dynamo-slot-tracker; direct gains dynamo-slot-tracker
  • dynamo-scheduler -> adds dynamo-kv-hints, dynamo-multimodal; direct gains dynamo-kv-hints, dynamo-multimodal
  • dynamo-kv-router (no features) -> adds dynamo-kv-hints, dynamo-multimodal; COMPOSITION_DIRECT_DEPS gains both
  • ROUTER_FAMILY gains dynamo-multimodal and dynamo-kv-hints; ROUTER_CRATE_DIRS gains lib/multimodal and lib/kv-hints; the forbidden lists of protocol, zmq, slot-tracker, session-affinity, and error gain dynamo-multimodal

Validation

Unit-level move-diff, run from the worktree at the unit head 3b6c929486: python3 -I scripts/router_move_diff.py 168b7658a0 3b6c929486 --from <17 paths> --to <22 paths>, the union of the replaced fork PRs' --from/--to lists filtered to the paths present at each end (full command in the artifact /tmp/rr-split/unit-prs/16-move-diff.txt).

summary from_normalized=2989 to_normalized=3074 common=2978 new=96 dropped=11
result DIFFERS

96 new / 11 dropped: #165's documented delta (91 new / 11 dropped: WorkerKvTransferKeys::from_worker building the keys from fields, the kernel adapter's six-argument call, the closure-taking WorkerKvTransferKeysCache::get_or_init, and the kernel test adapter_equals_the_direct_call with its WorkerConfigLike fixture, replacing the config-reading of body and the config-taking cache method) plus #170's five structural lines (#![deny(missing_docs)] and the mod tests braces of pad.rs and extra_key.rs). #163, #166, #169, #170 commit 1, and #172 are exact moves (296, 190, 573, 67, 66 lines common, nothing new or dropped).

New and dropped normalized lines (96 new, 11 dropped).
new 1x #![deny(missing_docs)]
new 1x #[test]
new 2x &self,
new 1x ("rdma-island".to_string(), "a".to_string()),
new 1x ("zone".to_string(), "z1".to_string()),
new 1x )
new 1x ) -> &WorkerKvTransferKeys {
new 1x ) -> Option<KvHintTransferWorkerMetadata<'_>> {
new 1x ) -> Self {
new 1x );
new 1x 2
new 1x 2,
new 1x 4
new 1x 4,
new 2x None
new 1x Some("rdma-island")
new 1x Some("rdma-island"),
new 1x Some(&config.topology),
new 1x Some(&self.topology)
new 1x Some(KvHintTransferWorkerMetadata {
new 1x Some(KvTransferEnforcement::Preferred)
new 1x Some(KvTransferEnforcement::Preferred),
new 1x WorkerKvTransferKeys::from_worker(
new 1x ]),
new 1x assert!(keys.is_source(5) && !keys.is_source(4));
new 1x assert!(keys.worker_type().is_some());
new 1x assert_eq!(keys, direct);
new 1x assert_eq!(keys.target(), direct.target());
new 1x assert_eq!(keys.topology().len(), 2);
new 1x config.data_parallel_size(),
new 1x config.data_parallel_start_rank(),
new 1x config.kv_transfer_domain(),
new 1x config.kv_transfer_enforcement(),
new 1x config.topology_domains(),
new 1x dp_rank: u32,
new 1x dp_size: u32,
new 1x enforcement == Some(KvTransferEnforcement::Preferred),
new 1x enforcement: Option<KvTransferEnforcement>,
new 1x fn adapter_equals_the_direct_call() {
new 1x fn data_parallel_size(&self) -> u32 {
new 1x fn data_parallel_start_rank(&self) -> u32 {
new 1x fn kv_hint_transfer_metadata_for_dp_rank(
new 1x fn kv_transfer_domain(&self) -> Option<&str> {
new 1x fn kv_transfer_enforcement(&self) -> Option<KvTransferEnforcement> {
new 1x fn max_num_batched_tokens(&self) -> Option<u64> {
new 1x fn topology_domains(&self) -> Option<&HashMap<String, String>> {
new 1x fn total_kv_blocks(&self) -> Option<u64> {
new 1x for rank in start_rank..start_rank.saturating_add(dp_size) {
new 1x impl WorkerConfigLike for Worker {
new 1x impl WorkerKvTransferKeys {
new 1x init: impl FnOnce() -> WorkerKvTransferKeys,
new 1x let Some(metadata) = per_rank_metadata(rank) else {
new 1x let config = Worker {
new 1x let direct = WorkerKvTransferKeys::from_worker(
new 1x let keys = WorkerKvTransferKeys::of(&config);
new 1x let topology = topology.map(topology_keys).unwrap_or_default();
new 1x let transfer_domain = transfer_domain.map(label_key);
new 4x mod tests {
new 1x per_rank_metadata: impl Fn(u32) -> Option<KvHintTransferWorkerMetadata<'a>>,
new 1x pub fn from_worker<'a>(
new 1x pub fn get_or_init(
new 1x self.0.get_or_init(init)
new 1x source_control_endpoint: (dp_rank == 5).then_some("tcp://worker:23280"),
new 1x start_rank: u32,
new 1x struct Worker {
new 1x topology: HashMap::from([
new 1x topology: HashMap<String, String>,
new 1x topology: Option<&HashMap<String, String>>,
new 1x transfer_domain: Option<&str>,
new 1x worker_type: "prefill",
new 2x |rank| config.kv_hint_transfer_metadata_for_dp_rank(rank),
new 17x }
new 1x })
new 1x };
dropped 1x .map(topology_keys)
dropped 1x .topology_domains()
dropped 1x .unwrap_or_default();
dropped 1x config.kv_transfer_enforcement() == Some(KvTransferEnforcement::Preferred),
dropped 1x for rank in start..start.saturating_add(config.data_parallel_size()) {
dropped 1x let Some(metadata) = config.kv_hint_transfer_metadata_for_dp_rank(rank) else {
dropped 1x let start = config.data_parallel_start_rank();
dropped 1x let topology = config
dropped 1x let transfer_domain = config.kv_transfer_domain().map(label_key);
dropped 1x pub fn get_or_init<C: WorkerConfigLike + ?Sized>(&self, config: &C) -> &WorkerKvTransferKeys {
dropped 1x self.0.get_or_init(|| WorkerKvTransferKeys::of(config))

Chain v2, full mode, on the rebased stack head ebd31cdf4d (every unit's commits are ancestors of it; the chain ran once on the head, not per unit), log /tmp/rr-split/srebase-validation.log: stage 1, the 13 router crates (clippy under default and all features, cargo test 1,247 passed / 5 ignored, rustdoc with all features), 1 min 50 s; stage 2, kv-router (clippy, bench check, 229 tests including the prelude test), 1 min 20 s; stage 3, workspace clippy, 4 min 58 s; stage 4, hosts (dynamo-llm lib alone, its kv_router 492 / discovery 275 / pool_relay 261 tests, 13 host crates clippy, bindings clippy; the pre-existing kvbm-consolidator manifest failure is the one non-zero exit), 29 min 21 s; stage 5, gates (every kv-router feature alone and together, 6 public-API snapshots unchanged with kv-router at 918 items, layering gate 47 checks and --host 66 checks with 0 failures, codeowners 7590/7590, fmt, 3 lockfiles --locked), 3 min 06 s; 40 min 35 s total, exit 0.

Each replaced fork PR carried its own validation chain; see its body.

Tickets

Part of DYNO-184
Part of DYNO-185
Part of DYNO-186
Part of DYNO-191
Part of DYNO-190
Part of DYNO-192

…he slot tracker

Dissolve `dynamo_router_protocol::load`. The protocol crate keeps only
what a worker publishes about itself and what every crate names; a type
with one producing stage now lives in that stage, and a stage may depend
on another stage for that stage's output types instead of hoisting them
into protocol.

Moves:

- `RankLoad` and `BusyThresholds` (with `is_configured` and `has_room`)
  move to `dynamo_slot_tracker::busy`, replacing the re-export that was
  there. The fullness rule is the slot tracker's output: its projection
  implements `RankLoad`, and `FullRankCounts` applies the thresholds.
- `SchedulerLoad` and `SchedulerGroup` move to
  `dynamo_slot_tracker::wire` with their wire test. One tracker
  publishes the scheduler-load report about the ranks it books; it is
  the tracker's replica wire, not a worker-published fact.
- `ActiveLoad`, `WorkerCapacity`, and `DEFAULT_MAX_BATCHED_TOKENS` stay
  in protocol and move to `dynamo_router_protocol::worker`, beside the
  worker identities. The constant stays with
  `WorkerCapacity::prefill_budget`, which reads it; protocol cannot
  depend on the slot tracker, so the ticket's placement of the constant
  under slot-tracker does not hold.

New edges: `dynamo-admission` and `dynamo-worker-selection` now depend
on `dynamo-slot-tracker` for the fullness rule. The layering gate rows
for both crates name the edge (router deps become protocol plus
slot-tracker; slot-tracker leaves the forbidden list and joins the
direct deps). Hosts import `BusyThresholds`, `RankLoad`,
`SchedulerLoad`, and `SchedulerGroup` through
`dynamo_slot_tracker::host`; the kv-router and scheduler `protocols`
shims keep every existing public name and re-point. Protocol drops its
`rmp-serde` dev-dependency, which served only the moved test; the slot
tracker gains `rmp-serde` and `serde_json` as dev-dependencies. Crate
docs for protocol, slot-tracker, admission, and worker-selection name
the new owner and the new edges. No behavior change.

Validation:
Stage crates (dynamo-router-protocol, dynamo-slot-tracker,
dynamo-admission, dynamo-worker-selection, dynamo-scheduler): clippy
--all-targets -D warnings with default and --all-features, tests
(11, 131, 22, 44, 291 passed), rustdoc -D warnings. dynamo-kv-router:
clippy with http,ingest-zmq,replica-sync-zmq,peer-recovery,metrics,
testing --all-targets, bench check, 229 tests passed. Workspace clippy
--all-targets -D warnings. Hosts: dynamo-llm check, clippy, 485
kv_router and 261 pool_relay tests; clippy --tests for dynamo-mocker,
dynamo-bench, dynamo-ext-proc, dynamo-custom-policy-builtin,
dynamo-sglang-sidecar, dynamo-sglang-mocker, dynamo-vllm-mocker; the
Python bindings with kv-indexer,slot-tracker,select-service. Gates:
router_features_check (15/15), router_public_api_check (protocol
snapshot regenerated, kv-router snapshot changes only the path of
`BusyThresholds` in its `BusyThresholdsExt` impl, dynamo-error
unchanged), router_layering_check 41 checks and --host 60 checks with
0 failures, codeowners 7542/7542, cargo fmt --check, the three
lockfiles under --locked. router_move_diff from 11d495f over
load.rs, busy.rs, wire.rs, worker.rs: 296 normalized lines on each
side, new 0, dropped 0, result MOVE.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ines

KV transfer hints are what the router hands an engine so the selected
worker can fetch a KV prefix from another worker or cache owner. They
are the counterpart of `dynamo-kv-events` (what engines publish to the
router), and engines consume them: backend-common, the SGLang and vLLM
sidecars, and the Python engine components. Until now the feature was
spread over four crates (protocol, kv-indexer, worker-selection, the
scheduler kernel) and engines reached the wire through lib/llm. The new
tier-1 crate `lib/kv-hints` (`dynamo-kv-hints`) gives it one owner, one
AGENTS.md, one public-API snapshot, and a crate engine-side code can
depend on directly later. Among router crates it depends on
`dynamo-router-protocol` only; its direct dependencies are exactly
protocol, rustc-hash, serde, and serde_json.

Moves (git mv where a whole file moved):

- `lib/router/protocol/src/kv_hints.rs` -> `lib/kv-hints/src/wire.rs`
  whole: `KvHint`, `KvHintAction`, `KvSourceLocationsPayload`, the three
  `KV_HINT_TRANSFER_*` runtime keys, `KvTransferCandidateSource`. The
  two wire serde tests move with the types from the kernel. Protocol
  loses the module and with it its only reference to
  `ResidencyOwnerKey` outside `residency`.
- `lib/router/worker-selection/src/kv_transfer.rs` ->
  `lib/kv-hints/src/transfer.rs` whole: `WorkerKvTransferKeys`,
  `KvTransferTarget`, `KvTransferScope`, `ResolvedTransferSource`,
  `KvTransferSourceChoice`, `KvTransferSourceSummary`.
- From `dynamo_kv_indexer::kv_hints`: `label_key` and `topology_keys`
  (pure hashing) -> `lib/kv-hints/src/keys.rs`; the indexer re-exports
  them at their old path.
- From `dynamo_scheduler::kv_hints`: `WorkerKvTransferKeysCache` and the
  body of `WorkerKvTransferKeysExt::of`, now
  `WorkerKvTransferKeys::from_worker(start_rank, dp_size,
  per_rank_metadata, topology, transfer_domain, enforcement)` in `keys`.
- From `dynamo_scheduler::protocols`: `KvHintTransferWorkerMetadata` and
  `KvTransferEnforcement` with the enforcement serde test -> `keys`.

Stays, and why:

- `KvTransferCandidates` and `CacheOwnerKvTransferKeys` stay in
  `dynamo-kv-indexer`: they carry `Arc<ResidencyRoutingSnapshot>`, the
  indexer's projection. The producer owns its output; moving them would
  make the indexer and the new crate name each other.
- `resolve_transfer_sources` and its tests stay in the kernel: it joins
  the indexer's candidates with worker keys under a selection scope,
  which is the kernel's stage-to-stage conversion. The attach point in
  `selection/core/run.rs` is untouched.
- `WorkerKvTransferKeysExt` stays in the kernel as a thin adapter: `of`
  reads the fields off `WorkerConfigLike` and calls `from_worker`. One
  new kernel test asserts the adapter equals the direct call for a
  config with two ranks, one source endpoint, a topology map, a
  transfer domain, and `Preferred` enforcement.

Signature changes (no behavior change): the key builder takes extracted
fields instead of a `WorkerConfigLike`, and
`WorkerKvTransferKeysCache::get_or_init` takes the initializer closure
(`|| WorkerKvTransferKeys::of(self)`) instead of the config, because the
cache no longer lives beside the config trait. Its two callers
(`SelectionWorkerConfig` and the builtin plugin's selection bench) are
updated.

New edges: kv-indexer, scheduler, kv-router, and lib/llm depend on
`dynamo-kv-hints`. worker-selection does not: nothing in it consumed the
moved types, so the scheduler kernel and the prelude re-point to the new
crate instead. The layering gate gains a row for `dynamo-kv-hints`
(router deps exactly protocol; every stage, the kernel, kv-events,
kv-hashing, kv-router, runtime, tokio, and the transport crates
forbidden; direct deps exact) and `dynamo-kv-hints` joins
`ROUTER_FAMILY`, `ROUTER_CRATE_DIRS`, the composition crate's router
and direct dependency sets, and the kv-indexer and scheduler rows.
`scripts/router_public_api_check.sh` gains the fourth snapshot
`lib/kv-hints/public-api.txt`; the pre-commit trigger and the router
codeowners area claim `lib/kv-hints/`.

Public API: every existing path resolves. `dynamo_scheduler::kv_hints`,
`dynamo_scheduler::protocols`, and `dynamo_scheduler::host` re-export
the moved items (host also exposes `WorkerKvTransferKeysExt` beside the
cache, which the closure form of `get_or_init` needs); the kv-router
`kv_hints` and `protocols` shims and `plugins::prelude` re-point with
every name unchanged, so `lib/kv-router/public-api.txt` is byte-for-byte
identical. `lib/router/protocol/public-api.txt` loses its 28
`kv_hints::` items; the new snapshot has 142 items. Hosts in lib/llm
import the wire and keys from `dynamo_kv_hints` directly;
backend-common, the sidecars, the bindings, and the mocker keep their
current paths through lib/llm and the kernel.

`label_key` hashes with `DefaultHasher::new()`: deterministic within one
toolchain, not a cross-version contract. Both sides of every comparison
are computed in one process today; the crate doc records it as a
follow-up if keys ever cross a process boundary.

Validation:
dynamo-kv-hints: clippy --all-targets -D warnings (default and
--all-features), 4 tests, rustdoc -D warnings. Stage crates
(dynamo-router-protocol, dynamo-kv-indexer, dynamo-worker-selection,
dynamo-scheduler): clippy --all-targets -D warnings with default and
--all-features, tests with --all-features (11, 597, 43, 289 passed;
worker-selection 44 -> 43 and scheduler 291 -> 289 are the moved tests
plus the one new adapter test), rustdoc -D warnings (default and
--all-features). dynamo-kv-router: clippy with http,ingest-zmq,
replica-sync-zmq,peer-recovery,metrics,testing --all-targets, bench
check, 220 + 1 + 1 + 7 tests, the plugin_api_prelude and
plugin_api_compat tests. Workspace clippy --all-targets -D warnings.
Hosts: dynamo-llm check, clippy, 485 kv_router and 38 local_model
tests; clippy --tests for dynamo-mocker, dynamo-bench, dynamo-ext-proc,
dynamo-custom-policy-builtin, dynamo-worker-selection-policy-catalog,
dynamo-sglang-sidecar, dynamo-vllm-sidecar, dynamo-backend-common,
dynamo-sglang-mocker, dynamo-vllm-mocker,
dynamo-custom-policy-example-epp,
dynamo-custom-policy-example-soft-pin-repin; the Python bindings with
kv-indexer,slot-tracker,select-service. Gates: router_features_check,
router_public_api_check (kv-router snapshot identical, protocol
snapshot -28 kv_hints items, new kv-hints snapshot 142 items,
dynamo-error unchanged at 134), router_layering_check 44 checks and
--host 63 checks with 0 failures, codeowners 7541/7541, cargo fmt
--check, the three lockfiles under --locked. router_move_diff from
4ff5815 over protocol kv_hints.rs, worker-selection kv_transfer.rs,
kv-indexer kv_hints.rs, and the kernel's kv_hints.rs and protocols.rs
to the new crate and the two kernel files: common 1138, new 91,
dropped 11. The dropped lines are the old adapter body that read the
config and the config-taking get_or_init; the new lines are
from_worker, the six-argument adapter call, the closure-taking
get_or_init, and the new kernel test.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
The producer owns the type. `StorageTier` is a field of the KV event
payload engines publish (`RouterEvent::storage_tier`), the residency
domain rides the same wire (`residency_domain`), and the kvbm
consolidator publishes residency too. The owner is the KV-events crate,
not protocol and not a router stage: protocol keeps the counterparty
nouns every crate names (identity, worker, block_hash, mm, session,
attempt, columns, metadata, worker_type). After DYNO-185 removed
`kv_hints.rs`, no protocol module referenced `residency` any more, so
the module moves whole.

Move (git mv, body unchanged): `lib/router/protocol/src/residency.rs`
-> `lib/kv-events/src/residency.rs`: `StorageTier`, `ResidencyDomain`,
`ResidencyOwner`, `ResidencyOwnerKey`, `ResetScope`,
`WireResidencyDomain` with its manual serde impls, and
`UnsupportedResidencyDomain`. Its two `crate::` imports become
`dynamo_router_protocol::{identity, worker}`. The file carries no test
module of its own; the residency-domain compatibility tests already
live beside the event types in `lib/kv-events/src/event.rs`.
`dynamo-kv-events` gains `pub mod residency` and its crate doc, README,
and AGENTS name the vocabulary; protocol drops the module, and its
crate doc and README list the vocabulary under `dynamo-kv-events`.

Re-pointed, compiler-driven, 46 files. kv-events' own files
(`event.rs`, `placement.rs`, `zmq_wire/{mod,convert,tests}.rs`) use
`crate::residency`. Everyone else imports
`dynamo_kv_events::residency` directly (vocabulary crate, no host
module): lib/llm (16 files: the publisher, indexer, recovery, and
kv_consolidator modules, `mocker.rs`, the `kv_router.rs` test),
lib/mocker (10), lib/bench (4), kvbm-consolidator
(`ingress/zmq_subscriber.rs`), kv-router
(`services/indexer/registry.rs`, `services/selection/tests.rs`,
`benches/selection_core.rs`, `tests/standalone_indexer_http.rs`),
kv-indexer (`prelude.rs`, `residency.rs`, `cuckoo/dc.rs`), and
`lib/kv-hints/src/wire.rs`. The deprecated shims
`dynamo_kv_router::protocols` and `dynamo_scheduler::protocols`
re-point their `pub use` with every name unchanged.

One new edge the brief did not anticipate: `dynamo-kv-hints` names
`ResidencyOwnerKey` (`KvTransferCandidateSource::CacheOwner`), so it
now depends on `dynamo-kv-events`. The crate was documented as
depending on protocol only among router crates; its crate doc, README,
and AGENTS now say protocol and kv-events, and its layering gate row
changes with it: router deps exactly protocol, kv-events, kv-hashing;
direct deps gain kv-events; kv-events and kv-hashing leave its
forbidden list. No cycle: kv-events does not depend on kv-hints. The
root, Python-bindings, and kvbm-bindings lockfiles each gain the one
dependency line. No other gate row changes.

Snapshots: `scripts/router_public_api_check.sh` gains the fifth
snapshot `lib/kv-events/public-api.txt` (261 items: 55 `event::`, 127
`zmq_wire::`, 23 `placement::`, 18 `cursor::`, 53 `residency::`, plus
root impls). `lib/router/protocol/public-api.txt` loses its 42
`residency::` lines (36 items and 6 impls) and gains none, 293 -> 251.
`lib/kv-hints/public-api.txt` changes two defining-path lines, both
`CacheOwner(dynamo_kv_events::residency::ResidencyOwnerKey)`.
`lib/kv-router/public-api.txt` (1398) and `lib/error/public-api.txt`
(134) are byte-for-byte identical. `lib/router/README.md` and the
kv-router README deprecated-paths table name the new owner.

Validation:
dynamo-kv-events: clippy --all-targets -D warnings (default and
--all-features), 65 tests, rustdoc -D warnings (default and
--all-features). Stage crates dynamo-router-protocol, dynamo-kv-hints,
dynamo-kv-indexer, dynamo-scheduler: clippy --all-targets -D warnings
with default and --all-features, tests with --all-features (11, 4, 597 +
2 ignored, 289 + 1 ignored passed), rustdoc -D warnings (default and
--all-features). dynamo-kv-router: clippy with http,ingest-zmq,
replica-sync-zmq,peer-recovery,metrics,testing --all-targets, bench
check, 220 + 1 + 1 + 7 tests, the plugin_api_prelude and
plugin_api_compat tests. Workspace clippy --all-targets -D warnings.
Hosts: dynamo-llm check, clippy, 485 kv_router and 21
block_manager::kv_consolidator tests; clippy --tests for dynamo-mocker,
dynamo-bench, dynamo-ext-proc, dynamo-custom-policy-builtin,
dynamo-worker-selection-policy-catalog, dynamo-sglang-sidecar,
dynamo-vllm-sidecar, dynamo-backend-common, dynamo-sglang-mocker,
dynamo-vllm-mocker, dynamo-custom-policy-example-epp,
dynamo-custom-policy-example-soft-pin-repin; the Python bindings with
kv-indexer,slot-tracker,select-service. kvbm-consolidator does not build
in this workspace for a pre-existing reason unrelated to this change
(its manifest inherits `kvbm-logical` from a root workspace that does
not define it); its one import is re-pointed. Gates:
router_features_check, router_public_api_check (five snapshots match),
router_layering_check 44 checks and --host 63 checks with 0 failures,
codeowners 7548/7548, cargo fmt --check, the three lockfiles under
--locked. router_move_diff from e21776c over
`lib/router/protocol/src/residency.rs` to
`lib/kv-events/src/residency.rs`: 190 normalized lines on each side,
common 190, new 0, dropped 0, result MOVE.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
KvTransferCandidateSource (who holds a prefix a target could fetch),
label_key and topology_keys (the label hashing both sides of a source
match use), and ResidencyOwnerKey (the compact owner identity the
indexer computes in compact_key) are what the indexer emits with a
match. #165 placed the first three in dynamo-kv-hints and #166 the key
in dynamo-kv-events, which left the indexer depending on the feature
crate for its own output types. Rule: a feature crate depends on the
stage that produces its types, never the reverse; the producer owns
its outputs.

Moves, text verbatim (move-diff over the five files from e844c15:
573/573 common, new 0, dropped 0): KvTransferCandidateSource and its
From<WorkerWithDpRank> from lib/kv-hints/src/wire.rs and label_key and
topology_keys from lib/kv-hints/src/keys.rs into
lib/router/kv-indexer/src/kv_hints.rs beside KvTransferCandidates;
ResidencyOwnerKey from lib/kv-events/src/residency.rs into
lib/router/kv-indexer/src/residency.rs just before
ResidencyOwnerKeyExt. kv-hints gains dynamo-kv-indexer, drops
dynamo-kv-events (nothing else in it used that crate), and re-exports
the three names at every path it exposed (root, keys::, wire::), so
every existing consumer path resolves. kv-indexer drops
dynamo-kv-hints. kv-events drops the key it never used; ResidencyOwner
is untouched. Protocol gains nothing.

Edges: kv-indexer -> protocol, kv-events, kv-hashing (was also
kv-hints); kv-hints -> protocol, kv-indexer (was protocol, kv-events).
Gate rows for both crates updated, including kv-hints' forbidden list,
which can no longer name tokio, anyhow, or async-trait now that they
arrive through the indexer; scheduler and kv-events rows and
COMPOSITION_ROUTER_DEPS unchanged. Re-pointed: the kernel and
kv-router protocols shims take ResidencyOwnerKey from
dynamo_kv_indexer::residency (same public name); the plugin prelude
takes KvTransferCandidateSource from the indexer beside
KvTransferCandidates. Three lockfiles follow the manifests. Crate
docs, AGENTS, and READMEs for kv-indexer, kv-hints, and kv-events, the
lib/router and protocol README tables, name the new owner.

Snapshots: kv-events 261 -> 257 (the key, its impl, digest,
from_digest); kv-hints 142 -> 134 (the three re-exported names render
as pub use lines at root, keys::, and wire:: in place of their inlined
definitions, and the two source fields at both paths name the
indexer's defining path); kv-router, protocol, and dynamo-error
byte-identical. No behavior change. DYNO-191.

Validation: clippy (default and --all-features, --all-targets, -D
warnings), cargo test --all-features, and rustdoc -D warnings for
kv-events, kv-hints, kv-indexer, scheduler, and protocol; kv-router
clippy with all features, bench check, tests, and prelude tests;
workspace clippy; hosts alone (lib/llm check, clippy, kv_router tests;
twelve host crates clippy --tests; Python bindings clippy);
router_features_check, router_public_api_check, router_layering_check
plain and --host, codeowners --strict, cargo fmt --check, and the
three lockfiles --locked. The chain ran on top of fork PR #168
(e048753); after #168 was closed the commit was rebased onto #166's
head e844c15 and re-verified there: workspace clippy, the five
crates' tests --all-features, the gate --host, the five snapshots,
fmt --check, and the three lockfiles --locked.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
LocalBlockHash and ExternalSequenceBlockHash are what the hash contract
produces: compute_block_hash and compute_block_hash_for_seq return the
first, and the second is the engine-side identity the router recomputes
with this crate's rules so the two sides agree. Protocol is counterparty
wire and identity only, and the producer owns its outputs, so the two
newtypes move to dynamo-kv-hashing.

Move, text verbatim (git mv lib/router/protocol/src/block_hash.rs to
lib/kv-hashing/src/block_hash.rs; move-diff from 94513b3 over the
file and the kv-hashing root: 67/67 common, new 0, dropped 0; the mod
line and the root re-export are import lines the tool normalizes away):
both structs, From<u64> and From<i64>, the
four manual serde impls, and the two serialization tests. kv-hashing
re-exports both names at the root; routing.rs imports LocalBlockHash
from the crate and its doc link names crate::ExternalSequenceBlockHash.
Protocol drops the module; its crate doc and README table point at
kv-hashing. serde_json stays in protocol's dependencies: the
metadata.rs and worker_type.rs tests still use it, so the documented
exact set (serde, serde_json, thiserror) is unchanged.

Deprecated shims: lib/kv-router/src/protocols.rs and
lib/router/scheduler/src/protocols.rs drop ExternalSequenceBlockHash
and LocalBlockHash (owner decision: no shim for the block hash names).
dynamo_kv_router::LocalBlockHash stays at the crate root, deprecated
as before, now sourced from dynamo_kv_hashing. The plugins prelude
(lib/kv-router/src/plugins/prelude.rs) is the tier-1 plugin API, not a
shim: it keeps ExternalSequenceBlockHash, re-pointed; the kv-indexer
prelude likewise, so the builtin plugin tests and bench compile
unchanged through the prelude.

Re-pointed, compiler-driven: 57 files from
dynamo_router_protocol::block_hash:: to dynamo_kv_hashing:: (lib/llm
20, kv-indexer 10 incl. prelude and bench, bench 9, kv-router 7 incl.
the prelude and the standalone indexer test, mocker 5, kv-events 3,
kv-hints wire.rs, the C and Python bindings), plus the kernel's own
twelve files that took the names through crate::protocols (kv_hints,
local, overlap_refresh, queue, selection/core/{mod,operation,tests},
selection/{input,pending}, selector/policy_tests, tracking_hash,
types) and kv-router's core_host_tests, which import from
dynamo_kv_hashing directly.

Edges: kv-hints gains dynamo-kv-hashing (its wire names
ExternalSequenceBlockHash; the crate was already in its closure through
the indexer); every other importer already depended on kv-hashing.
Gate: the kv-hints row's direct set gains dynamo-kv-hashing; no other
row changes (kv-hashing keeps protocol for the mm types until the next
commit). Three lockfiles: the kv-hints block gains dynamo-kv-hashing.
Docs: kv-hashing crate doc, AGENTS, and README (the LocalBlockHash home
path), protocol crate doc and README table, kv-events AGENTS and
README, kv-hints crate doc, AGENTS, and README, and the lib/router
README rows name the new owner.

Snapshots: protocol 251 -> 236 (the block_hash module, both structs,
From<u64>, From<i64>, the four serde impls and their fns); kv-router
1398 -> 1396 (protocols::ExternalSequenceBlockHash and
protocols::LocalBlockHash gone; the eight compute_seq_hashes_for_
tracking lines now name dynamo_kv_hashing::block_hash::LocalBlockHash);
kv-hints 134 and kv-events 257 change defining paths only (2 and 4
lines); dynamo-error byte-identical. No behavior change. DYNO-190.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
… engines and the router share

A multimodal block's KV identity is its tokens plus the mm_hash of every
object it touches, and the frontend and the engine must compute it the
same way. The pieces that make them agree were split across three
crates: the per-block and per-request metadata in protocol (mm.rs),
the pad and identifier rules in kv-hashing (routing.rs), and the vLLM
extra-key marker in kv-events (extra_keys.rs). None of them is a
counterparty wire type or an identity every crate names, so protocol
should not hold them, and the pad rules are not hashing. They become
one leaf crate, dynamo-multimodal (lib/multimodal, tier 1), that
depends on serde and xxhash-rust and on no Dynamo crate.

Moves, text verbatim (move-diff from the previous commit over the four
source files and the seven destinations: 1132/1137 common 1132, new 5, dropped 0; the five
new normalized lines are the crate's #![deny(missing_docs)] and the
mod tests { } wrappers of pad.rs and extra_key.rs, nothing else):
  1. lib/router/protocol/src/mm.rs -> lib/multimodal/src/info.rs whole
     (git mv): BlockMmObjectInfo, BlockExtraInfo, RequestMmObjectInfo,
     RequestExtraInfo and to_block_level.
  2. lib/kv-hashing/src/routing.rs -> lib/multimodal/src/pad.rs:
     MM_PAD_SHIFT_VALUE and MM_PAD_HASH_MASK with the SGLang drift
     comment, pad_value_for_mm_hash, hash_mm_identifier, and their
     three tests. routing.rs keeps BlockHashOptions and every
     compute_* and consumes dynamo_multimodal::BlockExtraInfo.
  3. lib/kv-events/src/zmq_wire/extra_keys.rs ->
     lib/multimodal/src/extra_key.rs: mark_mm_hash_for_extra_key,
     parse_mm_hash_from_extra_key, and only_frontend_padded_mm_hashes_
     are_parsed. kv-events keeps DYNAMO_CACHE_SALT_PREFIX and the two
     extra_keys_to_* codec functions (they take the ZMQ wire item) and
     imports the parser from dynamo-multimodal; zmq_wire::mod.rs drops
     the two moved names from its re-export list (no shim).
The crate root is the data path (frontend hash_mm_identifier ->
pad_value_for_mm_hash placeholder tokens for SGLang or the
mark_mm_hash_for_extra_key marker for vLLM; RequestExtraInfo ->
BlockExtraInfo per block -> kv-hashing folds mm_hash into the block
hash; engine events -> kv-events parses the marker back) plus private
modules and root re-exports by name; README, AGENTS, and its
`@AGENTS.md` twin state what it owns, the exact dependency set, and what
stays elsewhere (the kv-events codec and placeholder-run
normalization, kv-hashing's compute_block_hash*, lib/llm's
model-specific mm routing, and kv-hashing's PLH RequestMmObjectInfo, a
different shape with the same name, untouched).

Re-pointed, compiler-driven: kv-hashing routing.rs (import and tests),
kv-events event.rs and zmq_wire/{convert,deserialize,extra_keys,
filter,tests,types}.rs, the kernel's selection/input.rs, kv-router's
scheduling/config.rs and services/selection/tests.rs, lib/llm's
kv_router.rs, prefill_router/query.rs, routing_host/{kv_selection,
request_guard}.rs, publisher/{mod,tests}.rs, preprocessor.rs,
protocols/common.rs, protocols/common/preprocessor.rs, and
http/service/generate.rs, and the Python bindings' llm/kv.rs: every
dynamo_router_protocol::mm::, dynamo_kv_hashing::routing::{pad_value_
for_mm_hash, hash_mm_identifier}, and dynamo_kv_events::zmq_wire::
mark_mm_hash_for_extra_key path now reads dynamo_multimodal::. The
deprecated protocols shims in kv-router and the kernel keep every mm
name (the four structs, the two constants, the two functions),
sourced from dynamo-multimodal; the plugins prelude names none of
them.

Edges: kv-hashing -> multimodal (and drops protocol: nothing left in
the crate names it, verified by grep over src, tests, and benches);
kv-events, scheduler, kv-router, lib/llm, and the Python bindings gain
multimodal; kv-indexer and kv-hints reach it only through kv-hashing;
protocol now has no relation to hashing or to multimodal identity.
Workspace member and workspace dependency added; three lockfiles
follow. Gate: new dynamo-multimodal row (no router deps, every router
crate and transport forbidden, direct deps exactly serde xxhash-rust),
ROUTER_FAMILY, ROUTER_CRATE_DIRS, COMPOSITION_ROUTER_DEPS, and
COMPOSITION_DIRECT_DEPS gain it; kv-hashing row router deps become
dynamo-multimodal with protocol forbidden and the direct set swapping
protocol for multimodal; kv-events and scheduler rows gain it in the
closure and the direct set; kv-indexer and kv-hints rows gain it in
the closure only; protocol, zmq, error, slot-tracker,
session-affinity, worker-selection, and admission rows forbid it.
Codeowners: lib/multimodal/ under the router area plus a router +
multimodal co-ownership glob mirroring extra_keys.rs; CODEOWNERS
regenerated (+2 rules). Docs: protocol crate doc and README table,
kv-hashing AGENTS, kv-events README and AGENTS, lib/router README
(fifteen crates). Sixth public-api snapshot added for the new crate.

Snapshots: multimodal new (19 items: the four structs and their
fields, to_block_level, the two constants, the four functions);
protocol 236 -> 223 (the mm module, the four structs, their fields,
and to_block_level); kv-events 257 -> 255 (mark_mm_hash_for_extra_key
and parse_mm_hash_from_extra_key gone; five lines name
dynamo_multimodal::info::BlockExtraInfo instead of protocol's:
KvCacheStoredBlockData::mm_extra_info, RawKvEvent::BlockStored::
block_mm_infos, StoredBlockOptions::mm_extra_info,
create_stored_blocks, extra_keys_to_block_mm_infos); kv-router
1396, kv-hints 134, and dynamo-error 134 byte-identical (the shims'
mm names render as pub use lines without a defining path). No behavior change. DYNO-190.

Validation: chain /tmp/rr-split/chain-190.sh, log s190-validation.log on
the build host; every stage exit 0 except the pre-existing
kvbm-consolidator manifest failure. clippy default and --all-features
(--all-targets, -D warnings), cargo test --all-features, and rustdoc
-D warnings for multimodal (4), kv-hashing (19 + 17 + 3), protocol (9),
kv-events (64), kv-hints (4), kv-indexer (597 + 2 ignored), scheduler
(289 + 1 ignored); kv-router clippy with all features, bench check,
tests (220 + 1 + 1 + 7), prelude and compat tests (1 + 1); workspace
clippy; hosts alone: lib/llm check, clippy, tests kv_router 485,
preprocessor 383 + 2 ignored, protocols::common 132,
http::service::generate 36; thirteen host packages clippy --tests
including libdynamo_llm; Python bindings clippy with
kv-indexer,slot-tracker,select-service; router_features_check;
router_public_api_check (six snapshots match); router_layering_check
47 checks, 0 failures and --host 66 checks, 0 failures; codeowners
--strict 7549/7549; cargo fmt --check; three lockfiles --locked.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ActiveLoad is the load message a worker publishes on the event plane.
Its publisher (kv_router::publisher::worker_metrics) and every reader
(kv_router::metrics_subscriber, discovery::worker_monitor, pool_relay)
live in lib/llm, and no router crate names it: the slot tracker has its
own in-process SchedulerLoadSnapshot and only linked to ActiveLoad from
a doc comment. Protocol is counterparty wire and identity only, and the
producer owns its outputs, so the struct moves to
lib/llm/src/kv_router/active_load.rs (new file; pub mod active_load and
pub use active_load::ActiveLoad at the kv_router root). The
dynamo-router-worker crate idea is dropped: the worker vocabulary
(WorkerId, DpRank, WorkerWithDpRank, WorkerAffinityTarget,
WorkerCapacity, DEFAULT_MAX_BATCHED_TOKENS) stays in protocol::worker.

Move, text verbatim: the struct with its derives, serde attribute, and
docs. Move-diff from ecc0595 over protocol worker.rs to worker.rs
plus the new file: 66/66 common, new 0, dropped 0, MOVE (the new file's two use lines, for
WorkerId/DpRank and serde, are import lines the tool normalizes away).
Re-pointed to crate::kv_router::ActiveLoad: kv_router/metrics_subscriber,
kv_router/publisher/worker_metrics, kv_router/publisher/tests (2),
discovery/worker_monitor (2), pool_relay/{pool_registry,load},
pool_relay/wan/grpc/conformance. pool_relay/host.rs and
kv_router/metrics.rs name ActiveLoad in log strings and prose only.
slot-tracker multi_worker.rs: the doc link becomes plain text naming
dynamo_llm::kv_router::ActiveLoad (slot-tracker cannot link into
lib/llm). No Cargo.toml edge changes (lib/llm already depends on
protocol and serde), so the three lockfiles are unchanged.

Deprecated shims: lib/kv-router/src/protocols.rs and
lib/router/scheduler/src/protocols.rs drop ActiveLoad now rather than
deprecating it, because neither crate can see lib/llm; the kv-router
README "Deprecated paths" table gains a protocols::ActiveLoad row that
points at dynamo_llm::kv_router::ActiveLoad. Nothing imported the name
through either shim. The Python and C bindings never named it.

Docs: protocol worker.rs module doc drops "its active load"; the
protocol crate doc and README table name lib/llm as the owner; the
kv_router AGENTS module map gains active_load.rs.

Gate: no row changes. Snapshots: protocol 223 -> 217 (the struct and
its five fields under worker::); kv-router 1396 -> 1395
(protocols::ActiveLoad, the early shim removal); kv-events 255,
kv-hints 134, error 134, multimodal 19 byte-identical. No behavior
change. DYNO-192.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Owner Author

Landed upstream as ai-dynamo#18055 (squash commit 79692cf on main, 2026-11-11). Closing this fork review unit.

This branch had an error being deployed

1 failed deployment
external_collaborator — 3b6c9294 Deployed Nov 2, 2026 by ishandhanani via ok-to-test #528
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation frontend refactor router

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant