Skip to content

refactor(router): move RouterHintSourceMetadata to kv-indexer and WorkerAffinityTarget to session-affinity - #195

Closed
ishandhanani wants to merge 2 commits into
idhanani/dyn-194-trim-guardrailsfrom
idhanani/dyn-195-metadata-and-affinity-target
Closed

ishandhanani wants to merge 2 commits into
idhanani/dyn-194-trim-guardrailsfrom
idhanani/dyn-195-metadata-and-affinity-target

Conversation

@ishandhanani

Copy link
Copy Markdown
Owner

Router split v2 follow-up (DYNO-195). Stacked on #193.

Summary

Two move-only commits, each with its own move-diff; no behavior change. Owner decisions (2026-11-03): protocol should read as a composition of types each with one or a few consumers; attempt stays whole (its minter is the only writer of the private id field, and the id is named by two stages that must not depend on each other); the new stage edges admission -> session-affinity and worker-selection -> session-affinity are accepted, including tokio, tokio-util, dashmap, and blake3 entering worker-selection's tree.

  • Commit 1, refactor(router): move RouterHintSourceMetadata to kv-indexer (2d8fa36). The struct, its derives and docs, and its serde test move whole from lib/router/protocol/src/metadata.rs (deleted, with its pub mod metadata; line) to lib/router/kv-indexer/src/residency.rs, just before ResolvedRouterHintSource, which wraps it. Who writes it: hosts, when they publish a cache owner's hint source in runtime configuration (lib/llm discovery/kv_state_agent.rs, the state-agent publishers and the attachment owner, the Python bindings' state-agent config). Who reads it: the indexer's residency snapshot, which resolves it into ResolvedRouterHintSource and the transfer keys the kernel hands worker selection; no other stage names it. Why kv-indexer and not kv-hints: kv-hints depends on kv-indexer for the transfer candidate vocabulary (DYNO-191) and residency.rs embeds the metadata, so a kv-hints home would make the indexer name a kv-hints type and close a cycle. kv-indexer exports it at the crate root and in host (the host convention); kv_hints.rs imports crate::residency::RouterHintSourceMetadata; the crate's AGENTS.md "owns" sentence names it. Re-pointed, compiler-driven: lib/llm discovery/kv_state_agent.rs, kv_router/publisher/{attachment_owner,state_agent,state_agent_host}.rs, kv_router/indexer/recovery/state_agent.rs, and lib/bindings/python/rust/llm/kv_state_agent.rs through dynamo_kv_indexer::host; the kernel's kv_hints.rs and selection/core/tests.rs (tests) through the residency path; the bench's mooncake_lower_tier.rs through the crate root; the indexer's own lower_tier.rs tests through crate::residency. No Cargo.toml change: every importer already depends on dynamo-kv-indexer. Protocol's crate doc lists the type under "Deliberately elsewhere"; the README's "deliberately not here" table names it in the kv-indexer row (the README has no module table, so no row is removed); protocol's AGENTS.md module list drops metadata. Gate rows unchanged. 19 files, +67 -71 (16 .rs files, +64 -68).
  • Commit 2, refactor(router): move WorkerAffinityTarget to session-affinity (55f3568). pub struct WorkerAffinityTarget with its derives and doc, impl WorkerAffinityTarget { new }, and impl From<WorkerWithDpRank> for WorkerAffinityTarget move whole from lib/router/protocol/src/worker.rs to lib/router/session-affinity/src/lib.rs, where the alias line pub use dynamo_router_protocol::worker::WorkerAffinityTarget as AffinityTarget; becomes the definition plus pub type AffinityTarget = WorkerAffinityTarget;, so both names keep resolving; the crate's host module exports it (hosts set it on requests). WorkerWithDpRank and its constructors stay in protocol. Who writes it: session affinity (the binding table, AffinityLease::target, wire::AffinityBindingEvent::target(), rebound_target) and hosts (lib/llm sets the request's affinity_target from its session coordinator). Who reads it: admission's classifier (the classify request's affinity target) and worker selection's context (WorkerSelectionContext, the soft-pin filters), with the kernel carrying it between them (filter.rs, selection/*, types.rs). Edge cost, stated plainly: lib/router/admission/Cargo.toml and lib/router/worker-selection/Cargo.toml gain dynamo-session-affinity = { workspace = true } (the root, Python bindings, and KVBM bindings lockfiles each gain the two entries), so session-affinity's tree enters worker-selection's: cargo tree -p dynamo-worker-selection -e normal | grep -c tokio reads 4 before and 6 after (tokio and tokio-util; dashmap and blake3 arrive with them). The direction stays one way: session-affinity names neither stage, and a reverse edge is now a cargo cycle. Re-pointed, compiler-driven: worker-selection context.rs; admission classifier/mod.rs; the kernel's filter.rs, types.rs, selection/{types,catalog}.rs, selection/core/{mod,operation}.rs (run.rs and the core tests reach it through super::*, so they need no edit), selector/reference.rs (test); lib/llm kv_router.rs and routing_host/kv_selection.rs through dynamo_session_affinity::host; plugins::prelude re-exports it from dynamo_session_affinity under the same name (plugin_api_prelude passes unchanged; the soft-pin-repin example compiles through the prelude). lib/router/README.md and the three AGENTS.md and README files state the new edges. 28 files, +71 -52 (15 .rs files, +47 -36).

Move-diffs

Commit 1, python3 -I scripts/router_move_diff.py 0ebe1e5139 WORKTREE --from lib/router/protocol/src/metadata.rs lib/router/kv-indexer/src/residency.rs --to lib/router/kv-indexer/src/residency.rs:

from lib/router/protocol/src/metadata.rs raw=39 normalized=20 found=20 fraction=1.000
from lib/router/kv-indexer/src/residency.rs raw=204 normalized=155 found=155 fraction=1.000
to lib/router/kv-indexer/src/residency.rs raw=237 normalized=175
summary from_normalized=175 to_normalized=175 common=175 new=0 dropped=0
result MOVE

Commit 2, python3 -I scripts/router_move_diff.py 2d8fa3618e WORKTREE --from lib/router/protocol/src/worker.rs lib/router/session-affinity/src/lib.rs --to lib/router/protocol/src/worker.rs lib/router/session-affinity/src/lib.rs:

from lib/router/protocol/src/worker.rs raw=97 normalized=57 found=57 fraction=1.000
from lib/router/session-affinity/src/lib.rs raw=1426 normalized=1210 found=1210 fraction=1.000
to lib/router/protocol/src/worker.rs raw=78 normalized=42
to lib/router/session-affinity/src/lib.rs raw=1447 normalized=1226
summary from_normalized=1267 to_normalized=1268 common=1267 new=1 dropped=0
new 1x pub type AffinityTarget = WorkerAffinityTarget;
result DIFFERS

The one new line is the alias the brief asked for; the old pub use ... as AffinityTarget; is a use line and normalization drops it on both sides. Over the union from 0ebe1e5139 (both --from sets into both --to sets): common=1442 new=1 dropped=0, the same line.

Gate rows and the planted violation

Rows changed in scripts/router_layering_check.sh (exact direct Dynamo set | forbidden anywhere in the --all-features closure):

  • dynamo-session-affinity: direct dynamo-router-protocol (unchanged); forbidden gains dynamo-worker-selection dynamo-admission (the direction).
  • dynamo-worker-selection: direct dynamo-router-protocol dynamo-session-affinity dynamo-slot-tracker (was without session-affinity); forbidden drops dynamo-session-affinity.
  • dynamo-admission: direct dynamo-router-protocol dynamo-session-affinity dynamo-slot-tracker dynamo-tokens; forbidden drops dynamo-session-affinity.

40 checks plain, 59 with --host, 0 failures (the counts #193 left).

Planted dynamo-worker-selection = { workspace = true } in lib/router/session-affinity/Cargo.toml, alone, then reverted (git status clean and cargo metadata --locked passing afterwards):

$ cargo metadata --format-version 1 >/dev/null
error: cyclic package dependency: package `dynamo-session-affinity v1.7.0 (.../lib/router/session-affinity)` depends on itself. Cycle:
package `dynamo-session-affinity v1.7.0 (.../lib/router/session-affinity)`
    ... which satisfies path dependency `dynamo-session-affinity` (locked to 1.7.0) of package `dynamo-worker-selection v1.7.0 (.../lib/router/worker-selection)`
    ... which satisfies path dependency `dynamo-worker-selection` (locked to 1.7.0) of package `dynamo-session-affinity v1.7.0 (.../lib/router/session-affinity)`
    ... which satisfies path dependency `dynamo-session-affinity` (locked to 1.7.0) of package `dynamo-admission v1.7.0 (.../lib/router/admission)`
cargo metadata exit 101
$ bash scripts/router_layering_check.sh
ok    dynamo-error: Dynamo deps == {} (direct)
ERROR cargo tree -p dynamo-error --all-features: error: cyclic package dependency: package `dynamo-session-affinity v1.7.0 (...)` depends on itself. Cycle: ...
gate exit 2

With the forward edges in place, session-affinity -> worker-selection (or -> admission) is a cycle by construction: cargo refuses the graph before the gate's table check can run, the gate's first cargo tree fails, and the gate exits 2 (ERROR, fail closed) rather than printing a FAIL row. The forbidden entries document the direction for the day a forward edge goes away.

Validation

Validation chain v2 in MODE=full (mandatory: two stage Cargo.toml edges change), TAG=195 LLM_EXTRA_TESTS="discovery pool_relay" with both move specs in MOVE_SPECS_FILE, run once on the final head 55f356872d after commit 2 (log /tmp/rr-split/s195-validation.log on the build host). Every stage exit 0 except the pre-existing kvbm-consolidator manifest failure (unrelated, untouched); chain exit 0.

  • Per-stage wall time: stage 1 router crates 1 min 52 s; stage 2 kv-router 1 min 04 s; stage 3 workspace clippy 3 min 56 s; stage 4 hosts (full, per-host clippy) 22 min 32 s; stage 5 gates, codeowners, fmt, lockfiles 2 min 15 s; stage 6 move-diffs under a second. Total 31 min 39 s (04:20:40 to 04:52:19). target 23 GB after the run.
  • Stage 1, thirteen router crates: cargo clippy --all-targets -- -D warnings (default and --all-features); cargo test --all-features: admission 22, error 29, kv-events 64, kv-hashing 19 + 17 + 3, kv-hints 4, kv-indexer 600 + 2 ignored (599 before: the moved serde test), multimodal 4, protocol 8 (9 before), scheduler 289 + 1 ignored, session-affinity 10, slot-tracker 131, worker-selection 43, zmq 2, doc-tests 2 + 2 ignored (1,247 passed, 5 ignored, the same total as refactor(router): plugins use the prelude; drop the API snapshots; slim the gate and AGENTS files #193); RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features.
  • Stage 2: cargo clippy -p dynamo-kv-router --features http,ingest-zmq,replica-sync-zmq,peer-recovery,metrics,testing --all-targets -- -D warnings; cargo check --features bench --benches; tests 220 + 1 + 7; plugin_api_prelude 1 passed, the test unchanged.
  • Stage 3: cargo clippy --workspace --all-targets -- -D warnings clean.
  • Stage 4 (full): cargo check -p dynamo-llm --lib alone; cargo test -p dynamo-llm --lib for kv_router (492), discovery (275), pool_relay (261); per-host cargo clippy -p <host> --tests -- -D warnings exit 0 for dynamo-mocker, dynamo-bench, dynamo-ext-proc, dynamo-custom-policy-builtin, dynamo-worker-selection-policy-catalog, dynamo-sglang-sidecar, dynamo-vllm-sidecar, dynamo-backend-common, dynamo-sglang-mocker, dynamo-vllm-mocker, dynamo-custom-policy-example-epp, dynamo-custom-policy-example-soft-pin-repin, libdynamo_llm; kvbm-consolidator exit 101 (manifest, pre-existing); lib/bindings/python cargo clippy --features kv-indexer,slot-tracker,select-service -- -D warnings exit 0.
  • Stage 5: scripts/router_features_check.sh (every kv-router feature alone and together); scripts/router_layering_check.sh 40 checks, 0 failures and --host 59 checks, 0 failures; python3 .github/codeowners/build_codeowners.py --strict 7586/7586 explicitly owned (7587 before: metadata.rs is gone); cargo fmt --all -- --check; cargo metadata --locked for the root, lib/bindings/python, and lib/bindings/kvbm lockfiles.
  • Stage 6: move-diff 1 common=175 new=0 dropped=0 MOVE; move-diff 2 common=1267 new=1 dropped=0 (the alias line), as above.
  • Before the chain, on each commit's tree. Commit 1 (log s195-c1-check.log): cargo check --tests for protocol, kv-indexer, scheduler, bench; cargo check -p dynamo-llm --tests; bindings check; the moved serde test green in kv-indexer and protocol down to 8 tests; gate 40/0; move-diff MOVE. Commit 2 (log s195-c2-check.log): cargo check --all-features --tests for the four stage crates, protocol, and the kernel; kv-router with the chain's feature set, the mocker, builtin, the soft-pin-repin and EPP examples, the bench; plugin_api_prelude 1; session-affinity 10 and protocol 8 tests; cargo check -p dynamo-llm --tests; bindings check; gate 40/0 and --host 59/0; fmt; three lockfiles --locked; cargo tree -p dynamo-worker-selection -e normal | grep -c tokio 6 (4 at the base).
  • Disk: target 23 GB; 293 GB free.

Owner decision 2026-11-03: protocol reads as a composition of types
each with one or a few consumers. RouterHintSourceMetadata is the
cache-owner hint-source record hosts publish in runtime configuration
and the KV indexer's residency snapshot resolves
(ResolvedRouterHintSource wraps it); no other stage names it. It moves
whole, with its serde test, from lib/router/protocol/src/metadata.rs
(deleted, with its module line) to
lib/router/kv-indexer/src/residency.rs, just before the struct that
wraps it. It belongs to the indexer and not to dynamo-kv-hints because
kv-hints depends on the indexer for the transfer candidate vocabulary;
the indexer naming a kv-hints type would close a cycle.

kv-indexer exports it at the crate root and in host (hosts publish
it). Re-pointed: lib/llm discovery/kv_state_agent.rs,
kv_router/publisher/{attachment_owner,state_agent,state_agent_host}.rs,
kv_router/indexer/recovery/state_agent.rs, and the Python bindings'
kv_state_agent.rs import dynamo_kv_indexer::host; the kernel's
kv_hints and selection core tests, the bench's mooncake_lower_tier,
and the indexer's own kv_hints and lower_tier tests import the
residency path or the crate root. No Cargo.toml change: every importer
already depends on dynamo-kv-indexer. Protocol's crate doc and README
list the type under dynamo-kv-indexer; its AGENTS.md module list drops
metadata. Gate rows unchanged (40 checks, 0 failures).

Move-diff from 0ebe1e5 over metadata.rs and residency.rs into
residency.rs: common 175, new 0, dropped 0, MOVE.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Owner decision 2026-11-03: protocol reads as a composition of types
each with one or a few consumers. WorkerAffinityTarget is the target a
session binds to: session affinity writes it (the table, the lease,
the replication wire's target()), admission's classifier and worker
selection's context read it as the request's requested target, the
kernel carries it between them, and hosts set it on requests. It moves
whole (struct, derives, doc, new, From<WorkerWithDpRank>) from
lib/router/protocol/src/worker.rs to
lib/router/session-affinity/src/lib.rs, where the alias line
`pub use ...::WorkerAffinityTarget as AffinityTarget` becomes the
definition plus `pub type AffinityTarget = WorkerAffinityTarget;`, so
both names keep resolving; the host module exports it too.
WorkerWithDpRank and its constructors stay in protocol.

New stage edges, accepted by the owner: dynamo-admission and
dynamo-worker-selection depend on dynamo-session-affinity (three
lockfiles updated). The cost is session-affinity's tree entering
worker-selection's: tokio, tokio-util, dashmap, and blake3
(`cargo tree -p dynamo-worker-selection -e normal | grep -c tokio`:
4 before, 6 after). The direction stays one way: session-affinity
names neither stage.

Re-pointed: worker-selection context.rs, admission classifier/mod.rs,
the kernel's filter.rs, types.rs, selection/{types,catalog}.rs,
selection/core/{mod,operation}.rs (run.rs and the core tests reach it
through super::*), selector/reference.rs; lib/llm kv_router.rs and
routing_host/kv_selection.rs import dynamo_session_affinity::host; the
plugins prelude re-exports it from dynamo_session_affinity under the
same name (plugin_api_prelude passes unchanged; the soft-pin-repin
example compiles through it).

Gate: dynamo-worker-selection's direct set becomes {protocol,
session-affinity, slot-tracker}, dynamo-admission's gains
session-affinity, each drops session-affinity from its forbidden list;
dynamo-session-affinity's forbidden list gains dynamo-worker-selection
and dynamo-admission. 40 checks plain, 59 with --host, 0 failures.
Planted dynamo-worker-selection in session-affinity's [dependencies]:
cargo rejects the cycle (session-affinity -> worker-selection ->
session-affinity) and the gate exits 2 with ERROR on its first cargo
tree, fail closed; reverted. lib/router/README.md and the three
AGENTS.md and README files state the new edges.

Move-diff from 2d8fa36 over worker.rs and session-affinity lib.rs:
common 1267, new 1 (the alias line), dropped 0.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-195-metadata-and-affinity-target branch from 55f3568 to dc1537a Compare November 4, 2026 09:30
@ishandhanani

Copy link
Copy Markdown
Owner Author

Landed upstream as ai-dynamo#18055 (squash commit 79692cf on main, 2026-11-11). Closing this fork review unit.

This branch had an error being deployed

1 failed deployment
external_collaborator — dc1537af Deployed Nov 4, 2026 by ishandhanani via ok-to-test #537
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation refactor router

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant