Repository navigation
refactor(router): split dynamo-kv-router into vocabulary, stage, and kernel crates - #18055
Merged
ishandhanani merged 129 commits intoNov 11, 2026
Merged
Conversation
Extract the identities and worker-facing vocabulary every router crate names into lib/router/protocol: worker and rank ids, partition, pool, owner, and domain identities, WorkerType, the residency vocabulary, the block-hash newtypes, worker load and metadata wire types, KV hint wire types, SessionContext, and AttemptId. lib/kv-router keeps pub use shims at every old path, so no other crate's imports change. AttemptId has no public constructor any more. AttemptIdMinter::mint is the only way to obtain one and yields fresh ids only; the slot tracker's RequestIndex holds the one minter. Tests use for_testing behind the testing feature, benches for_benchmark behind bench. ResidencyOwner's BLAKE3 compact_key is hashing, not wire, so it becomes the ResidencyOwnerKeyExt extension trait in dynamo_kv_router::protocols. The crate depends on exactly serde, serde_json, and thiserror. scripts/router_layering_check.sh enforces that by set equality, forbids behavior in the crate, and fails closed; it runs in pre-commit and, with --host, in the Rust CI job. scripts/router_move_diff.py shows the move is a move: 3,858 lines found, 45 new (minter, extension trait, one test), 9 dropped (the replaced constructor paths). Step 1 of the router split v2 plan. DYNO-170. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Move the router's block and sequence hash functions out of lib/kv-router/src/protocols.rs into dynamo_kv_hashing::routing: the XXH3 seed, compute_block_hash, BlockHashOptions, the multimodal pad constants and pad_value_for_mm_hash, hash_mm_identifier, compute_block_hash_for_seq and its seeded variants, compute_next_seq_hash, compute_seq_hash_for_block, compute_seq_hash_for_tokens_with_seeds, and TokensWithHashes. The eighteen tests that exercise them move with them. The three seeded helpers that were pub(crate) become pub with their existing docs, because the tracking hash in kv-router calls them across the crate boundary. Move the multimodal block and request metadata (BlockMmObjectInfo, BlockExtraInfo, RequestMmObjectInfo, RequestExtraInfo) into dynamo_router_protocol::mm: KV events, the hash contract, and the frontend preprocessor all name it, so it is shared vocabulary. dynamo-kv-hashing gains dependencies on dynamo-router-protocol and xxhash-rust; protocol does not depend on it. lib/kv-router keeps pub use shims at the old paths, so no other crate changes. The layering gate gains a dynamo-kv-hashing row and forbids kv-hashing in protocol's tree. Step 3a of the router split v2 plan, taken before step 2 because the ZMQ event codec calls these helpers. DYNO-171. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ine codec Move everything about KV cache events out of lib/kv-router into lib/kv-events: the event types (KvCacheEvents, KvCacheEvent, KvCacheEventData, store, remove, and stored-block data, KvCacheEventError), the RouterEvent envelope and its builders, KV_EVENT_SUBJECT, the engine ZMQ msgpack codec (zmq_wire), the replay cursor (recovery/cursor.rs), and the publisher placement helpers (Placement, PlacementOwner, PlacementEvent). The RouterEvent legacy-wire tests and the router event constructor test move with the types; the codec's tests move with the codec. Both ends of the KV-event wire depend on this crate: workers, sidecars, and the C bindings publish with it, the router's indexer ingests with it. It depends on dynamo-router-protocol and dynamo-kv-hashing plus serde, rmp-serde, rustc-hash, thiserror, and tracing; no sockets, tasks, or servers. lib/kv-router keeps pub use shims at every old path, including the zmq_wire and recovery module paths, so no other crate's imports change. KvCacheEventError is non-exhaustive, which binds only across crates, so kv-router's two exhaustive matches on it (behind the metrics feature) gain a wildcard arm and an "unknown" status counter. The layering gate gains a dynamo-kv-events row; CODEOWNERS claims lib/kv-events and follows the moved extra_keys override. Step 2 of the router split v2 plan. DYNO-172. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Move lib/kv-router/src/services/common/zmq.rs to lib/zmq as the dynamo-zmq crate: the async ZmqSocket over tokio's AsyncFd, multipart send and receive, the process-wide context pool sized by DYN_ZMQ_IO_THREADS, the SUB, DEALER, and bound PUB constructors, and validate_endpoint. The file moves verbatim; the feature gates that selected functions per standalone service are dropped because the crate is unconditional, and pub(crate) becomes pub. The error type stays anyhow for now so callers do not change; it becomes a thiserror enum when those callers move into the router adapter layer. lib/kv-router re-exports the crate at services::common::zmq behind the same standalone features, so no caller changes. The layering gate gains a dynamo-zmq row; CODEOWNERS claims lib/zmq. Step 4 of the router split v2 plan. DYNO-173. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…gate, and counters Create lib/router/kv-indexer, the KV indexer stage crate, and move its first slice out of lib/kv-router: the cuckoo-filter pool index (cuckoo and its eleven modules), the hash lifecycle delegate, and the indexer counters. The files move verbatim with their tests; imports that pointed at kv-router's protocols shims now name dynamo-router-protocol, dynamo-kv-events, and dynamo-kv-hashing directly, and pub(crate) items that sibling files still in kv-router use become documented pub. Two things could not move. KvIndexerMetrics::from_component needs a runtime Component, so it becomes the KvIndexerMetricsExt extension trait in dynamo_kv_router::indexer behind runtime-protocols, built on the new KvIndexerMetrics::from_counters; the one lib/llm caller gains the trait import. The counters are Prometheus IntCounterVec fields, so the stage carries a transitional `export` feature that kv-router's `metrics` forwards to; the metrics behavior change replaces it with atomics. lib/kv-router re-exports cuckoo, the delegate, and the counters at their old paths. The layering gate gains a dynamo-kv-indexer row that allows prometheus only under export; CODEOWNERS claims the crate. Step 5a, slice 1 of the router split v2 plan. DYNO-174. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…o dynamo-kv-indexer Slice 2 of step 5a. Move out of lib/kv-router into lib/router/kv-indexer: the radix trees (radix_tree, positional, compressed_radix), the concurrent compressed tree and its sixteen modules, the ThreadPoolIndexer and BranchShardedIndexer wrappers with the shard handle, KvIndexer, approximate LRU retention and pruning, the bench-only observation plumbing, the indexer traits (KvIndexerInterface, SyncIndexer), the task and request types, the active-set helper, the residency projection and routing snapshot, overlap scores, KV transfer candidates with the cache owner's hashed hint keys, the shared test helpers (now dynamo_kv_indexer::testing), the indexer_delegate bench, and the index README. The files move verbatim with their tests; shared sibling-crate names come through a crate-private explicit prelude. Three items stay in lib/kv-router by design and are cut, not moved: SharedKvCache is a kernel hook and lands in scheduling/shared_cache.rs; TieredMatchProvider and the Wire<->LowerTierMatchDetails conversions join the lower-tier files they return; the request-plane DTOs (KvState*, WorkerKvQuery*, IndexerQuery*, IndexerRecordRoutingDecision*) become indexer/api.rs because lib/llm implements the runtime MaybeError on them. The LocalKvIndexer tests and the delegate tests stay with local.rs and the lower tier (indexer/local_tests.rs, indexer/delegate_tests.rs) until those slices move; one radix-tree test compares the dump payloads directly instead of wrapping them in WorkerKvQueryResponse. Items that files still in kv-router call become documented pub: mix_worker, ApproximateLruTask::complete and ApproximateLruReply, CacheOwnerKvTransferKeys fields, label_key and topology_keys, and the test-only KvIndexer::snapshot_event_sender (gated on `testing`). lib/kv-router re-exports the crate at indexer::*, protocols::, kv_hints::, and test_utils::. The gate row allows tokio for the indexer: the stage rule forbids sockets and HTTP, not channels and actors. Step 5a, slice 2 of the router split v2 plan. DYNO-174. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…mo-kv-indexer Slice 3 of step 5a. Move lower_tier.rs (LowerTierIndexer, the continuation index over (parent, local) -> child edges with residency-owner tables, LowerTierContinuation, LowerTierMatchDetails, KvTransferExtensions, WorkerBlockIndex) and lower_tier_indexers.rs (LowerTierIndexers, TieredMatchDetails, LowerTierQueryOptions, TieredMatchProvider, the tier-walk helpers) verbatim from lib/kv-router into lib/router/kv-indexer, with their tests. Imports that named kv-router's shims now go through the crate prelude, which gains ResidencyOwnerKey, ResidencyProjection, and ResidencyRoutingSnapshot. lib/kv-router keeps every old path through its indexer::* re-export; nothing outside kv-router changes. Step 5a, slice 3 of the router split v2 plan. DYNO-174. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
The indexer's request-plane responses (WorkerKvQueryResponse, IndexerQueryResponse, IndexerRecordRoutingDecisionResponse) implemented dynamo_runtime's MaybeError inside lib/kv-router behind the runtime-protocols feature. The DTOs are about to move into dynamo-kv-indexer, which must not depend on dynamo-runtime, and neither kv-router nor lib/llm may implement a foreign trait on a foreign type. Add lib/llm/src/kv_router/indexer/reply.rs with three #[serde(transparent)] newtypes (WorkerKvQueryReply, IndexerQueryReply, IndexerRecordRoutingDecisionReply) that implement the runtime MaybeError exactly as the removed impls did: Error(message) is the error state, and from_err produces it. Use the newtypes at the typed boundaries only: the AddressedPushRouter stream in the worker query transport, the PushRouter and Ingress generics and the two served engines in remote.rs, and the WorkerKvQueryEngine stream builders. Every other caller keeps the bare enums through From in both directions. The wire encoding is unchanged (transparent newtype); tests pin that and the error mapping. Remove the three runtime impls from lib/kv-router's indexer/api.rs; the runtime-protocols feature keeps KvIndexerMetricsExt. Prerequisite for step 5a slice 4 of the router split v2 plan. DYNO-174. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ocal indexer, and DTOs
Slice 4 of step 5a, the last move into lib/router/kv-indexer:
- api: the request-plane DTOs (KvState*, WorkerKvQuery*, IndexerQuery*,
IndexerRecordRoutingDecision*, the endpoint names) and the indexer's own
MaybeError trait, verbatim.
- local: LocalKvIndexer, with its tests and the delegate tests that build
lower-tier and local variants.
- backend, lookup, recording, session_updates: the ingest backend
(Indexer, IndexerPolicy, RemotePrimary, SideIndexer, create_indexer*),
the lookup pipeline, approximate request leases, and session updates.
- session_prefix_index: SessionPrefixIndexer; the cleanup cadence becomes a
per-stage constant instead of reaching into kv-router's cleanup module.
- identity: IndexerIdentitySpec, ExplicitIdentityMap, IdentitySpecError,
CanonicalIdentityMaterial, the explicit-identity limits and tag constants,
and their tests. kv-router keeps the protocol re-export and
default_routing_group.
- config: ApproximateCachePolicyKind moves here, and the new IndexerConfig
slice (use_kv_events, approximate_cache_policy, predicted_ttl_secs,
ttl_secs) replaces KvRouterConfig in IndexerPolicy::from_router_config,
now from_config with the same checks and messages. kv-router converts
once with From<&KvRouterConfig> for IndexerConfig at its three call
sites; the policy tests build IndexerConfig literals.
backend::test_util is pub under `testing` for kv-router's service tests;
the test-only Indexer::flush drops its standalone-selection gate.
lib/kv-router re-exports everything at the old paths: indexer::*,
services::indexer::{backend, lookup, recording, session_updates},
session_prefix_index, identity::*, config::ApproximateCachePolicyKind.
Nothing outside kv-router changes.
Step 5a, slice 4 of the router split v2 plan. DYNO-174.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
WorkerCapacity (a rank's advertised max_num_batched_tokens and total_kv_blocks, zero meaning unreported) and DEFAULT_MAX_BATCHED_TOKENS lived in worker selection's inputs module, but the slot tracker stores a capacity per rank and BusyThresholds judges fullness against it. Stage crates do not depend on each other, so the type moves to dynamo_router_protocol::load beside SchedulerLoad and ActiveLoad: the values cross the process boundary in the worker's runtime config, and every stage names them. prefill_budget becomes pub for the same reason. WorkerCapacity::of(&impl WorkerConfigLike) cannot follow (the config trait is router-side), so it becomes the WorkerCapacityExt extension trait next to the trait, re-exported from plugins::worker_selection; the five kv-router call sites and the one mocker call site import it. inputs.rs re-exports both names at their old paths. Step 5b-0 of the router split v2 plan (prerequisite for dynamo-slot-tracker). DYNO-175. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 5b of the router split v2 plan. Create lib/router/slot-tracker and move the slot tracker out of lib/kv-router verbatim with its tests: the active-sequence trackers (single, multi_worker), the prompt registry with its membership trie and load projection, the block and prefill trackers, the request index that holds the one AttemptIdMinter, worker topology, the replication effects, the sharded lock, and cleanup.rs (used only by the membership trie). Three items join it from other kv-router files because the slot tracker owns them: the replication wire (ActiveSequenceEvent*, ActiveBlockEvent*, PrefillEvent*, PrefillLoadHint, MAX_REPLICA_BATCH_*) from protocols.rs as `wire`, BusyThresholds and FullRankCounts from scheduling/policy_config.rs as `busy`, and SchedulerBookingDescriptor from scheduling/queue.rs as `booking` (documented, no longer doc-hidden). Two seams needed shaping so the stage never sees router-side types. BusyThresholds::has_room is generic over a RankLoad reading trait with two accessors; the slot tracker's WorkerLoadProjection implements it, and kv-router implements it for worker selection's WorkerLoadInput, so every existing call keeps its text. worker_is_full(&impl WorkerConfigLike) becomes worker_is_full_in_ranks(worker_id, dp_ranks, capacity, loads) in the stage plus kv-router's BusyThresholdsExt::worker_is_full with the old signature; the mocker imports the trait. rank_is_full reads the projection map directly instead of building a LOAD row (same result). Stage operations the router drives (the booking lifecycle methods on ActiveSequencesMultiWorker, SequenceTrackerOptions and its fields, RateLimitedFailureLog::record, PrefillCompletion, WorkerLoadReader and its get, the load projection pass) become documented pub; request_booking drops its standalone-slot-tracker gate. The crate depends on dynamo-router-protocol and dynamo-tokens only (kv-hashing is a dev-dependency for one test); tokio stays for channels and timers. lib/kv-router aliases the crate as `sequences` and re-exports the wire, busy, and booking names at their old paths; it drops crossbeam-utils, indexmap, seqlock, and slotmap, which only the tracker used. The layering gate gains the slot-tracker row; CODEOWNERS claims the crate. DYNO-175. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…tage crate Step 5c of the router split v2 plan. Create lib/router/session-affinity and move services/selection/affinity.rs out of lib/kv-router verbatim as the crate root: the SessionAffinity table, WeakSessionAffinity, the acquire protocol (Hold, AffinityInitialization, AffinityLease), AffinityVersion, the AffinityReplicaSink port, AffinityError, SessionAffinityConfig and SessionAffinityMode, the limits, subagent_group_affinity_id, and the tests. AffinityBindingEvent moves from the replica mesh (services/common/replica_sync.rs) into the crate's `wire` module with its target() and version() accessors made pub and ungated: both ends of that wire are affinity tables, and the mesh that carries it stays a kv-router adapter. SessionAffinity::validate_ttl becomes documented pub for the router's config validation. The crate depends on dynamo-router-protocol only among router crates (WorkerAffinityTarget is the protocol's); feature `testing` gates the test hooks the file already had. lib/kv-router aliases the crate at services::selection::affinity and re-exports the wire type at its old path, so lib/llm's session_affinity module keeps every import. The layering gate gains the session-affinity row; CODEOWNERS claims the crate. DYNO-176. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…quest WorkerSelectionContext, the view filters, scorers, and pickers read, held a reference to the kernel's SchedulingRequest and answered its accessors by reaching into it. The worker-selection stage must not depend on the kernel's request type, so the context now carries the values a policy may read: request id, ISL tokens, request blocks, block size, prefill-token tracking, pinned worker, temperature override, session context, affinity target, expected output tokens, the two priorities, and the shared-cache hits. The kernel builds it once per selection with from_request. Two accessors are new (request_id, shared_cache_hits); the existing accessors return the same values from the copied fields. The reference selector was the only reader of the request through the context: it now uses the accessors, and the per-worker local transfer prefix it needs for the fetchable-blocks term is passed in by the caller, which already computes it from the request. Plugin crates read the context through accessors only and do not change. Step 5d-0 of the router split v2 plan (worker-selection boundary). DYNO-177. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Worker-selection code read the flat KvRouterConfig: the policy factory type took it, WorkerSelectionPolicy::new carried it, the reference selector scored with its weights, and ConditionalDisaggPolicy read six of its fields. The stage must not depend on the router's flat configuration, so two slices now own those fields. SelectionConfig (plugins::worker_selection::selection_config) holds the scoring weights, the sampling temperature, the shared-cache multiplier and type, and conditional_disagg_enabled, with Default equal to the flat defaults, From<&KvRouterConfig> next to the flat type, and validate() using the validators that moved with it. SharedCacheType moves into the slice module. WorkerSelectionPolicyFactory takes &SelectionConfig; WorkerSelectionPolicy::new, new_with_filters, and reference, and the reference DefaultWorkerSelector, take SelectionConfig. The registry, the selection core, lib/llm's prepared-policy wrapper, the mocker, and the in-tree plugins convert once at the boundary. ConditionalDisaggConfig (conditional_disagg) holds the six conditional_disagg_* fields with Default and From<&KvRouterConfig>; ConditionalDisaggPolicyKind moves beside it; make_conditional_disagg_policy and the policies' from_config take the slice, and lib/llm's prefill leg converts once. config:: re-exports every moved name and the root exports the two slices, so existing paths resolve. The builtin plugin's default policy, its parameters, and its selector take SelectionConfig; the four example policies and the builtin, kv-router, and lib/llm tests convert where they built a flat config for selection. Step 5d-1 of the router split v2 plan (worker-selection boundary). DYNO-177. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
… ranks them The transfer-source summary that credits a candidate worker for the KV prefix it could fetch read two things the worker-selection stage must not see: each worker's config (through WorkerConfigLike::kv_transfer_keys) and the indexer's residency routing snapshot (through KvTransferCandidates.routing_snapshot). Split the step in two. The router resolves first: kv_hints::resolve_transfer_sources walks the request's candidates with today's skips (sources no longer than every target's local prefix or longer than the retained chain, worker ranks that publish no source endpoint, cache owners missing from the snapshot, the attached worker's topology for an owner that publishes none) and yields ResolvedTransferSource records: who holds the prefix, how long it is, the hashed hint worker type, the attached worker, and the topology labels. KvTransferSourceSummary::new takes those records and the scope and ranks them exactly as before; source_for is unchanged. Worker keys move the same way. WorkerKvTransferKeys keeps its data, target(), and a from_parts constructor over already-hashed labels, with worker_type() and topology() accessors; deriving the keys from a config becomes the WorkerKvTransferKeysExt extension trait beside WorkerConfigLike, and WorkerKvTransferKeysCache moves next to it. KvTransferCandidateSource moves to dynamo_router_protocol::kv_hints as shared vocabulary; the indexer and kv-router re-export it at the old paths. The label hashing (label_key, topology_keys) stays with the indexer, which builds the snapshot keys; the router calls it for worker keys. Step 5d-2 of the router split v2 plan (worker-selection boundary). DYNO-177. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 5d-3 of the router split (DYNO-177). The composed worker-selection policy no longer reads SchedulingRequest, RoutingEligibility, or worker configs. The kernel implements a CandidateRowSource (walk the eligible ranks, each a CandidateRank that yields its row at the requested columns and its capacity; a rank's capacity and booked load by worker) and drives the policy through WorkerSelectionPolicy::select_rows( context, source, prepared, trace), which returns a RowPick or a SelectionFailure. The walk takes a RankVisitor whose visit method is generic over the rank, so the walk, the rank, and the row build are monomorphized together; a dyn visitor cost 11 ns per rank. The kernel maps the pick to WorkerSelectionResult, logs the selection, and turns NoCandidates into overloaded-vs-no-endpoints from eligibility; the stage reports AllFiltered only when the source yielded a row. Rows still materialize lazily: a policy with filters asks a rank for the filters' columns and for the remaining columns only once it is kept, and the no-filter path stays one pass. CandidateData gains a u32 raw_cached_tokens column (packed beside inputs; the row stays 128 bytes) that the kernel fills for tracked requests, so the stage folds the best eligible cache reuse over kept rows without the request. The fetch column is filled as each row is built. Its summary's scope is every eligible rank's target and local prefix (KvTransferScope::empty + add_target), computed once; this admits only sources no kept rank can use, which source_for already rejects, so kept rows see the same sources as the kept-rows scope did. The selected worker's fetch source keeps its single-target scope when no scorer declared KV_TRANSFER. CandidatePreparer::prepare_with owns the pooled buffer and the kernel's CandidatePreparerExt::prepare builds rows into it; PreparedCandidates is now the kernel wrapper carrying the fetch summary beside the stage's PreparedRows. Selection-trace assembly and the per-candidate TRACE log move into the stage with the scratch rows they read. The reference selector keeps reading the request through the kernel's row source. Validation: kv-router 594 + 2 + 7 (four new stage-only tests drive select_rows over a table with no request in sight), builtin plugin tests, lib/llm kv_router tests, clippy clean, layering gate 20 checks. criterion worker_selection against the parent commit at 10k workers: custom_no_filter +1.6%, custom_keep_all_filter +1.2%, custom_capacity_filter +2.4%, unused_preferred_taint_metadata no change, reference direct -1.6%; default_policy_wrapper -2.5% to -8% at 32 and 1024 workers, +9-11 ns per selection at 2 workers. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…her crate Step 5d-3b of the router split (DYNO-177): the last boundary changes before the stage becomes its own crate, each one something the kernel could do only because it shared a crate with the stage. - WorkerSelectionContext gains a public constructor: new(isl_tokens, block_size) with every other field at its default plus with_* setters; the kernel's from_request becomes selection_context in the selector. - The row types the kernel fills have public fields: CandidateData, WorkerCacheData, WorkerLoadInput, WorkerTransferInput, CacheSnapshot, and WorkerInputRows with new/empty/read. - What the stage derived from kernel-only types leaves it: the config- and projection-derived constructors are now the WorkerCapacityExt, WorkerInfoExt, and WorkerLoadInputExt traits in selector::derive, and the facade re-exports them at their old paths. WorkerInfo::of becomes WorkerInfoExt::of. - RankLoad moves to the protocol crate's load module so both the slot tracker's projection and the stage's load column implement one trait without either stage naming the other; the slot tracker re-exports it. - WorkerSelectionPolicyError, the selection trace types, and SharedCacheHits move into the stage's modules; scheduling::types and protocols re-export them, so every existing path still resolves. - The reference picker moves into the stage behind the bench and test features as a push-based pick: begin(temperature), offer(worker, cost), finish(). The kernel's pick_default_worker offers scores in eligibility order, or sorted order when the picker is seeded, and never touches the RNG or the softmax scratch. Same draws, same ties. Validation: protocol 10, slot-tracker 130, kv-router 595 + 2 + 7, builtin plugin, lib/llm kv_router 485 tests; clippy clean on stage crates and lib/llm, mocker, ext-proc, bench; layering gate 20 checks; move-diff 16,744 common / 238 new / 125 dropped. Interleaved A/B bench against the 5d-3 head at 10k workers: composed paths at parity, the test-only reference path within about 1 ns per rank once the pick state mirrors the old loop's locals (its first cut measured +3.8%). Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…r-selection
Step 5d-4 of the router split (DYNO-177): dynamo-worker-selection, a
stage crate that depends on dynamo-router-protocol and nothing router-
side beyond it. A move: the files that made up the stage inside
lib/kv-router move as they are, and lib/kv-router re-exports every name
at the path plugins and hosts already use.
Moved: plugins/worker_selection/{context, inputs, runtime,
selection_config, error, trace, shared_cache, reference_picker}.rs, the
plugin traits from the plugins/worker_selection.rs facade,
scheduling/selector/policy.rs (the composed policy, CandidateRowSource,
RankVisitor, CandidateRank, CandidatePreparer, PreparedRows, the trace
builder), kv_hints/sources.rs as kv_transfer.rs, conditional_disagg.rs,
and scheduling/lora_filter.rs.
Stays in lib/kv-router: the WorkerSelector trait and its adapter, the
request-to-row source, the derived columns (selector/derive.rs), the
reference selector, the policy factory type, the registry, the host
instance config (plugins/worker_selection/config.rs), and the resolver
for kv-transfer sources. The policy tests that drive WorkerSelector with
a SchedulingRequest stay as selector/policy_tests.rs; the kv-transfer
summary tests that resolve against a routing snapshot stay in
kv_hints.rs; the stage keeps its own row-source, trace, picker, scope,
and conditional-disagg tests.
Visibility the crate boundary required: the selection config slice's
defaults and validators, which the flat router config shares, are pub,
as are the conditional-disagg defaults, WorkerInputs::ALL, and
WorkerSelectionPolicy::reference_parts; the reference selector reads the
context through its accessors. DEFAULT_MAX_BATCHED_TOKENS is taken from
the protocol crate directly.
Layering gate: a row for dynamo-worker-selection (22 checks). CODEOWNERS
gains lib/router/worker-selection/ under the router area.
Validation: stage 41 tests; kv-router 554 + 2 + 7 (554 + 41
moved = the 595 before); builtin plugin and lib/llm kv_router 485 tests;
clippy clean on the stage crates and lib/llm, mocker, ext-proc, bench;
layering gate 22 checks; move-diff 13748 common / 102 new / 35 dropped (the new lines are #[inline] hints and visibility).
Interleaved A/B bench against the 5d-3b head at 10k workers: parity or
better once the stage's small accessors carry #[inline] (non-generic
functions the kernel calls per rank across the crate boundary; without
the hints the reference path measured +2%).
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 5e-0 of the router split (DYNO-178). The column-group bitset a stage component declares (WorkerInputs) and the static facts about a rank every column carries (WorkerInfo: identity plus advertised capacity) move from dynamo-worker-selection to a new columns module in dynamo-router-protocol. Worker selection builds its rows from them and admission's classifier will read workers through them, so neither stage should have to name the other. Worker selection re-exports both, so every existing path in kv-router, lib/llm, and the plugins holds. WorkerInputs::without, which only the composed policy used, is public in its new home. Validation: protocol 10 and worker-selection 41 tests; kv-router 554 + 2 + 7; builtin plugin and lib/llm kv_router 485 tests; clippy clean on the stage crates and lib/llm, mocker, ext-proc, bench; layering gate 22 checks; move-diff 458 common / 1 new / 1 dropped. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
… view Step 5e-1 of the router split (DYNO-178). RequestClassifier::classify takes `&dyn WorkerView` instead of worker selection's `WorkerCandidates<'_>`. The view is the admission stage's own vocabulary: one row per eligible rank, `info(row)` for identity and capacity, and `load(row)` / `cache(row)` as plain column structs (WorkerLoadView, WorkerCacheView) that read as None unless the classifier declared the group. The kernel implements it over worker selection's rows with the ClassifierRows newtype (scheduling/classifier_rows.rs), so each column means what it means in a filter or scorer, and the admission crate will depend on the protocol crate alone. This is the one plugin-API signature change of the admission step. The in-tree implementations (the ThunderAgent classifier, the lib/llm watcher and routing-host test classifiers, the kv-router and builtin test classifiers) change their parameter type and nothing else; none of them read the table beyond `is_empty`. Groups other than CACHE and LOAD are selection's and read as None through the view; no in-tree classifier declared them. NoWorkers is the empty view. Validation: kv-router 554 + 2 + 7; builtin plugin all targets; lib/llm kv_router 485 and discovery watcher 36 tests; clippy clean on the stage crates and lib/llm, mocker, ext-proc, bench; downstream check clean; layering gate 22 checks; move-diff over the touched kv-router files 9,243 common / 124 new / 39 dropped (the view module, the newtype, and the signature and test-site changes). Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 5e-2 of the router split (DYNO-178). The resolved half of the
policy config leaves the YAML behind.
- BusyThresholds (the per-rank limits, is_configured, has_room) moves
to dynamo-router-protocol's load module beside WorkerCapacity and
RankLoad: admission configures it per class, the kernel's gate
evaluates it, classifiers read it. The slot tracker re-exports it and
keeps the rank checks over its own projection as BusyRanksExt
(rank_is_full, worker_is_full_in_ranks).
- RouterQueuePolicy, PolicyClassConfig, PolicyProfile, and the family-
bucket classifier move to scheduling/admission_config.rs, the stage's
resolved config, with a public validated constructor:
PolicyProfile::family_buckets(FamilyBucketTable) refuses an index out
of range, a bucket list that does not start at zero or is not
strictly increasing, or a table whose size is not families times
buckets, returning AdmissionConfigError::Validation. The raw YAML
forms, the file reader, RouterSettings, the identifier rules, and
every message the resolver emits today stay in policy_config.rs; the
resolver builds the table and calls the constructor, and
RouterPolicyConfigError wraps the stage's error.
- config.rs and policy_config.rs re-export every moved name, so
dynamo_kv_router::RouterQueuePolicy, scheduling::{BusyThresholds,
PolicyClassConfig, PolicyProfile, RouterPolicyConfigError}, and the
slot tracker's busy::BusyThresholds all still resolve. The mocker's
offline replay, the one caller of rank_is_full outside the slot
tracker, imports BusyRanksExt beside BusyThresholdsExt.
Validation: protocol 10, slot-tracker 130, worker-selection 41,
kv-router 554 + 2 + 7 (resolver messages unchanged), builtin plugin,
lib/llm kv_router 485, mocker 304; clippy clean on the stage crates and
kv-router, lib/llm, mocker, ext-proc, bench; layering gate 22 checks;
move-diff 3878 common / 109 new / 10 dropped.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…the view Step 5e-3 of the router split (DYNO-178). RequestClassifierRuntime no longer names KvSchedulerError: classify_with, begin_request, and the terminal-delivery wait return ClassificationError, with one variant per failure the runtime can detect (Panicked, Failed, DuplicateRequestId, InvalidMetadata, LifecycleEnded, Shutdown) and the same messages the kernel's variants carried. The kernel converts with From<ClassificationError> for KvSchedulerError, so its error surface, the selection service's status mapping, and every host path are unchanged. classify_with takes `&dyn WorkerView`; the scheduler wraps the rows it built in ClassifierRows at its two call sites, and the runtime's tests classify over the empty view. WorkerView gains a Sync supertrait: views are read-only snapshots, and a classification future that borrows one must stay Send for spawned callers. Validation: kv-router 554 + 2 + 7; builtin plugin all targets; lib/llm kv_router 485 and discovery watcher 36; clippy clean on kv-router, builtin, lib/llm, mocker, ext-proc, bench; downstream check clean; layering gate 22 checks; move-diff 3,788 common / 91 new / 64 dropped. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 5e-4 of the router split (DYNO-178): dynamo-admission, a stage
crate that depends on dynamo-router-protocol and nothing router-side
beyond it. A move: the files that made up the stage inside
lib/kv-router move as they are, and lib/kv-router re-exports every name
at the path plugins and hosts already use.
Moved: scheduling/policy_queue.rs as queue, scheduling/queue_admission
as placement, scheduling/admission_config.rs as profile, and the
classifier side as classifier/{mod (the plugin API), inputs, progress,
view, runtime (scheduling/request_classifier.rs: the runtime and the
request lifecycle)}.
Stays in lib/kv-router: the YAML policy file and its resolver
(policy_config.rs), the classifier instance config and the registries,
the gate over the slot projection, ClassifierRows (the WorkerView over
worker selection's rows), and the ClassificationError conversion. The
24 policy-queue tests build their profiles through the YAML resolver,
so they stay as scheduling/policy_queue_tests.rs and read the queue's
internals through three testing-gated accessors (due_entry_count,
class_deficit, class_blocks_worker); the entry-size test moves with the
queue. The classifier runtime's tests move with it.
Visibility the crate boundary required: the kernel-facing constructors
and scheduling methods are pub (ClassifyRequest::with_*, into_queue_
inputs; ClassificationOverrides and QueueMetadata with their fields;
PolicyQueue::{has_ready, any_ready_head, next_due_at, recheck_worker,
enqueue_with_due_at, take_if_in_class, take_expired};
RequestClassifierRuntime and its methods), documented as kernel-facing
in the crate docs instead of #[doc(hidden)]. ClassifyRequest's fields
stay private: the runtime that reads them is a child module.
Layering gate: a row for dynamo-admission (24 checks). CODEOWNERS gains
lib/router/admission/ under the router area.
Validation: admission 22 tests; kv-router 532 + 2 + 7 (532 + 22 moved = the
554 before); builtin plugin, lib/llm kv_router 485 and watcher 36,
mocker 304; clippy clean on admission, kv-router, builtin, lib/llm,
mocker, ext-proc, bench; layering gate 24 checks; move-diff 3,941
common / 55 new / 49 dropped (visibility and the test hooks).
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Step 6-0 of the router split (DYNO-179). The scheduler kernel stops reading the flat KvRouterConfig. RouterConfig (scheduling/ router_config.rs) holds exactly what the kernel reads: the worker- selection and conditional-disagg slices, use_kv_events, track_prefill_tokens and assume_kv_reuse with their override-aware helpers, the queue recheck interval, and a PolicyProfileSource that resolves a partition's profile by model name. RouterConfig::validate runs the selection slice's checks plus a non-zero recheck interval and yields Validated<RouterConfig>, the only config SelectionCore::new_inner accepts (invariant G4); the flat config converts with From<&KvRouterConfig>, resolving profiles lazily through the policy file it loaded so a bad file still fails where policy_profile failed before. OverlapAnalysis::new, cache_hit_estimates_from_tiered_matches, and TieredOverlapRefresher::new take CacheHitWeights, a two-field view of the selection slice's host and disk cache-hit weights, instead of the whole flat config; the kernel derives it from config.selection, and lib/llm and the mocker pass CacheHitWeights::from(&config) at their five call sites. SelectionCore holds the validated slice; its try_new_local constructors still take KvRouterConfig and convert, so tests and benches are unchanged. Validation: cargo test -p dynamo-kv-router --features standalone-selection,standalone-slot-tracker,metrics,testing (532 lib + 2 plugin-API + 7 standalone HTTP); clippy -D warnings on kv-router (all targets, same features), dynamo-llm, dynamo-mocker, dynamo-ext-proc, dynamo-custom-policy-builtin, dynamo-bench (--tests), dynamo-sglang-sidecar; builtin tests 56+5+1+3+20+3+4; cargo test -p dynamo-llm kv_router (485) and --lib watcher (36); dynamo-mocker --lib 304; router layering gate 24/24; CODEOWNERS strict clean; cargo fmt --check clean; move-diff vs 37e939a lists only the RouterConfig/Validated/CacheHitWeights additions and the removed KvRouterConfig reads (summary common=16814 new=198 dropped=58). Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
… selection core Step 6-1a of the router split (DYNO-179). Three adapters the scheduler kernel still reached are cut at the boundary the kernel crate will have, without moving any file across crates yet. SelectionError no longer implements axum's IntoResponse. It keeps status_code() -> u16 and kind() as data (the numbers come from a private status module instead of http::StatusCode); the response body, including the classifier-failure sanitizer and the QueueRejected details, moves to server.rs as selection_error_response(), which the eleven handler sites call. A free function rather than an impl: once the error type lives in the kernel crate, kv-router cannot implement a foreign trait for it. The kernel's default KV index source is LocalIndexIngress, which opens a partition's index and nothing else. SelectionServiceBuilder::new installs ZmqDirectIngress explicitly (the standalone service's behavior is unchanged; kv_index() still overrides it), and the ZMQ adapter moves to zmq_ingress.rs with its endpoint validation and listener reconciliation. try_new_local, whose callers are all tests and benches, now gets the passive ingress, which with use_kv_events = false was never consulted. services/common/replica_sync.rs splits in two. The channel plumbing the selection core and the slot-tracker service drive (scoped sequence events, ReplicaSyncConfig, the per-partition publisher and subscriber, ReplicaInbox, the session-affinity sink, the host channels and setup_scoped_replica_sync) is services/common/replication.rs, with no ZMQ import; the ZMQ mesh (publisher task, PeerManager, peer commands, topics, ReplicaSyncRuntime, setup_replica_sync) stays in replica_sync.rs and re-exports the moved names at their previous visibility, so every existing path still resolves and the tests stay where they were. Validation: cargo test -p dynamo-kv-router --features standalone-selection,standalone-slot-tracker,metrics,testing (532 lib + 2 plugin-API + 7 standalone HTTP), and --no-default-features --features standalone-slot-tracker --lib (332); clippy -D warnings on kv-router all targets under the full feature set, under standalone-slot-tracker alone, and under standalone-selection alone (the split replica-sync module compiles in all three), and on dynamo-llm, dynamo-mocker, dynamo-ext-proc, dynamo-sglang-sidecar, dynamo-custom-policy-builtin, dynamo-bench (--tests); builtin tests 56+5+1+3+20+3+4; cargo test -p dynamo-llm kv_router (485) and --lib watcher (36); dynamo-mocker --lib 304; router layering gate 24 checks, 0 failures; CODEOWNERS strict clean; cargo fmt --check clean; move-diff vs e9256a3 common=7281 new=123 dropped=62, every line one of the boundary edits named above. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ners wrap it Step 6-1b of the router split (DYNO-179). The selection core's last transport-bound dependency was WorkerRegistry: one struct holding the per-partition indexers, their policy and readiness signal, and the ZMQ listener records, watermarks, and peers, with the ZMQ listener spawned from its register path. The index half becomes dynamo_kv_indexer::registry::IndexRegistry, a stage type: partition -> indexer created on demand with the registry's policy and metrics, the ready watch, and the claim-vs-remove lifecycle lock. claim() runs under a LifecycleGuard the caller holds and rejects a block-size mismatch with the same message register() produced; release_if_unused() takes the lock, asks the caller whether anything still names the partition, and removes otherwise (never when the registry retains empty indexers); get_or_create_indexer() stays the lock-free path recovery dumps and embedding hosts use. kv-router's WorkerRegistry keeps the listeners, watermarks, and peers and holds an Arc<IndexRegistry> it delegates to, so the standalone indexer service, its HTTP tests, and the slot tracker keep their calls. KvEventIngress speaks IndexRegistry: open() receives it, reconcile() and detach() no longer take a registry, and a new attach(index, cancel_token) hook lets the core hand its index registry and cancellation scope to whichever ingress the builder installed. ZmqDirectIngress builds its listener registry there, over the core's index and inside the core's cancellation scope, so shutdown() cancels listeners exactly as before. SelectionCore holds Arc<IndexRegistry> and exposes it; peer recovery (reqwest) and the /dump snapshot move from the core to the service over that accessor. lib/llm's RuntimeIngress only changes its open() signature. Validation: cargo test -p dynamo-kv-indexer --features testing --lib (583); cargo test -p dynamo-kv-router under the full service feature set (532 lib + 2 plugin-API + 7 standalone HTTP), under standalone-slot-tracker alone (332), and under standalone-indexer alone (335 + 1 + 6); clippy -D warnings all targets on kv-indexer (testing) and on kv-router under the full, slot-tracker-only, indexer-only, and selection-only feature sets, and on dynamo-llm, dynamo-mocker, dynamo-ext-proc, dynamo-sglang-sidecar, dynamo-custom-policy-builtin, dynamo-bench (--tests); builtin tests 56+5+1+3+20+3+4; cargo test -p dynamo-llm kv_router (485) and --lib watcher (36); dynamo-mocker --lib 304; router layering gate 24 checks, 0 failures; CODEOWNERS strict clean; cargo fmt --check clean; move-diff vs 768e5ea common=7551 new=272 dropped=129, every line one of the boundary edits named above (the indexer creation path and the lock-free accessors moved verbatim). Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…l module Step 6-2a of the router split (DYNO-179). lib/kv-router/src/protocols.rs mixed two things: re-exports of the stage crates' wire types, and the router-local request vocabulary the scheduler kernel owns per the split design (section 4, rows 40-41): KvHintTransferWorkerMetadata, WorkerConfigLike, KvTransferEnforcement, RoutingConstraints, RouterEventSink, RouterRequest, PotentialLoad, RouterResponse, WorkerSelectionResult, and the two private helpers they use. Those items move verbatim to scheduling/protocols.rs, the kernel module that leaves with the rest of scheduling/ in 6-3, together with the tests that exercise them. protocols.rs keeps the stage re-exports and re-exports the moved names at their current paths, so no importer inside or outside the crate changes. Validation: move-diff vs 70059d9 common=557 new=2 dropped=0 (the second test module's braces); cargo test -p dynamo-kv-router under the full service feature set (532 lib + 2 plugin-API + 7 standalone HTTP) and with default features (304 lib); clippy -D warnings all targets on kv-router under both, and on dynamo-llm, dynamo-mocker, dynamo-ext-proc, dynamo-sglang-sidecar, dynamo-custom-policy-builtin, dynamo-bench (--tests); builtin tests 56+5+1+3+20+3+4; cargo test -p dynamo-llm kv_router (485) and --lib watcher (36); dynamo-mocker --lib 304; router layering gate 24 checks, 0 failures; CODEOWNERS strict clean; cargo fmt --check clean. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…fig slice
Step 6-2b of the router split (DYNO-179). TrackingHashContext::from_config
read the flat KvRouterConfig, validated the option combination, and
opened the key file. The kernel must not read files or the flat config,
so the context now takes a resolved slice.
scheduling/tracking_hash.rs (kernel module, moves with the kernel in
6-3) holds the algorithm, the scope, the context with its hashing and
seed derivation, and the new TrackingHashConfig { algorithm, key_id,
provider_key }; TrackingHashContext::new(config) is the only
constructor there. lib/kv-router/src/tracking_hash.rs keeps the
flat-config side: validate_tracking_hash_options (the flat config's
validator still calls it), tracking_hash_config(&KvRouterConfig) with
the former from_config body (validation, key-file read, size check,
same messages), and a from_config shim over the two, so the twelve
callers do not change. The vector, framing, and zeroize tests move
with the context and build their keyed configs from bytes; the
key-file and option-validation tests stay with the resolver.
Validation: move-diff vs 5b21071 common=503 new=52 dropped=11 (the
config struct, new(), the resolver wrapper, and the tests' constructor
lines; hashing and key loading moved verbatim); cargo test -p dynamo-kv-router under the full
service feature set (532 lib + 2 plugin-API + 7 standalone HTTP) and
with default features (304 lib); clippy -D warnings all targets on
kv-router under both, and on dynamo-llm, dynamo-mocker, dynamo-ext-proc,
dynamo-sglang-sidecar, dynamo-custom-policy-builtin, dynamo-bench
(--tests); builtin tests 56+5+1+3+20+3+4; cargo test -p dynamo-llm
kv_router (485) and --lib watcher (36); dynamo-mocker --lib 304; router
layering gate 24 checks, 0 failures; CODEOWNERS strict clean; cargo fmt
--check clean.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…are numbers Follow-up to 6-1a (DYNO-179). Taking axum out of the kernel's error type left six hand-written status numbers behind a private module: HTTP vocabulary without the HTTP type. The owner's call is to keep the types-only http crate for the codes, so SelectionError::status() now returns http::StatusCode, status_code() is its u16 view, and the HTTP adapter uses the StatusCode directly instead of re-parsing a number. The classification itself is unchanged; moving it onto dynamo-error's error class is the E2 follow-up once #110 merges. http 1.4 joins the workspace dependencies (already in the lockfile through axum). Validation: cargo test -p dynamo-kv-router under the full service feature set (532 lib + 2 plugin-API + 7 standalone HTTP) and with default features (304 lib); clippy -D warnings all targets on kv-router under both, and on dynamo-llm, dynamo-mocker, dynamo-ext-proc, dynamo-sglang-sidecar, dynamo-custom-policy-builtin, dynamo-bench (--tests); builtin tests 56+5+1+3+20+3+4; cargo test -p dynamo-llm kv_router (485) and --lib watcher (36); dynamo-mocker --lib 304; router layering gate 24 checks, 0 failures; CODEOWNERS strict clean; cargo fmt --check clean; Cargo.lock only adds http 1.4.1 to kv-router's list. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…to the kernel config Step 6-2c of the router split (DYNO-179). RouterConfigOverride is the per-request slice of the router configuration the kernel honors (it already shaped RouterConfig::assume_kv_reuse in 6-0), and RouterPrefillLoadModel is a kernel setting the flat config carries. Both lived in scheduling/config.rs beside the flat KvRouterConfig and its parsers. They move verbatim to scheduling/router_config.rs, the kernel config module that leaves with the kernel in 6-3: RouterPrefillLoadModel with its Display, FromStr, and is_enabled; RouterConfigOverride with its builder derive, validate, the RouterConfigOverrideSerde compatibility shape and TryFrom, and the deprecated overlap_score_weight helper only that TryFrom uses. config.rs re-exports both names, so crate::config, scheduling::config, and the lib/llm imports keep resolving. Validation: move-diff vs 0d9fa86 common=2483 new=0 dropped=0 (an exact move); cargo test -p dynamo-kv-router under the full service feature set (532 lib + 2 plugin-API + 7 standalone HTTP) and with default features (304 lib); clippy -D warnings all targets on kv-router under both, and on dynamo-llm, dynamo-mocker, dynamo-ext-proc, dynamo-sglang-sidecar, dynamo-custom-policy-builtin, dynamo-bench (--tests); builtin tests 56+5+1+3+20+3+4; cargo test -p dynamo-llm kv_router (485) and --lib watcher (36); dynamo-mocker --lib 304; router layering gate 24 checks, 0 failures; CODEOWNERS strict clean; cargo fmt --check clean. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
This was referenced Nov 11, 2026
1 task done
This was referenced Nov 11, 2026
This was referenced Nov 11, 2026
This was referenced Nov 11, 2026
Contributor
|
damn dude |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The composition crate
dynamo-kv-routerbecomes 12 crates underlib/: five vocabulary crates (dynamo-router-protocol,dynamo-kv-events,dynamo-kv-hashing,dynamo-kv-hints,dynamo-error), five stage crates underlib/router/(dynamo-kv-indexer,dynamo-slot-tracker,dynamo-session-affinity,dynamo-worker-selection,dynamo-admission), the kernel (dynamo-scheduler), anddynamo-kv-routeras the composition: the selection service, the plugin registry andplugins::prelude, the flatKvRouterConfigand its parsers, and every adapter (HTTP servers, ZMQ ingest, replica mesh, peer recovery, metrics) behind a feature withdefault = [].Why: one producer per type and no
commoncrate, so a change to a wire shape, a stage, or a host has one home; in-tree hosts import a stage through itshostmodule and the vocabulary crates directly, plugins importdynamo_kv_router::plugins::preludeand depend ondynamo-kv-routeralone, andscripts/router_layering_check.shpins every Dynamo-family edge, the forbidden closures, the composition crate's no-features tree, and the hosts' graphs in CI (--host).Deliberately not in this PR: behavior changes (the step 9 items stay parked: one
InputTrigger, worker-only pins withOption<DpRank>, theaffinity_targetversuspinned_workerpair, folding session affinity into the kernel, the ZMQ helper dedupe) anddynamo-metrics.Rules
dynamo-router-protocolcarries no executable behavior and no topic modules; what computes stays with its producer.dynamo-kv-hintsnamesdynamo-kv-indexer; the indexer never names the hints.lib/router/kv-router/README.md.dynamo-kv-routeralone and importsplugins::prelude; the gate's plugin table enforces it for the builtin, catalog, and example crates.Crates
dynamo-errorlib/errorDynamoError,ErrorClass::http_status, the reason catalog,PublicDetailsdynamo-router-protocollib/router/protocolWorkerType;WorkerCapacity; the candidate columns (WorkerInputs,WorkerInfo);AttemptIdand its minter;SessionContextdynamo-kv-hashinglib/kv-hashingLocalBlockHash,ExternalSequenceBlockHash, block and sequence hashes, the tracking-hash context; itsmultimodalmodule is the multimodal identity both ends of the wire compute (per-block and per-request metadata, the pad and identifier rules, the extra-key marker)dynamo-tokensdynamo-kv-eventslib/kv-eventsdynamo-router-protocol,dynamo-kv-hashingdynamo-kv-hintslib/router/kv-hintsKvHintwire, the transfer source choice (with #17657's fetch continuations), the worker transfer keysdynamo-router-protocol,dynamo-kv-hashing,dynamo-kv-indexerdynamo-kv-indexerlib/router/kv-indexerIndexerConfigdynamo-router-protocol,dynamo-kv-events,dynamo-kv-hashing,dynamo-tokensdynamo-slot-trackerlib/router/slot-trackerdynamo-router-protocol,dynamo-tokensdynamo-session-affinitylib/router/session-affinityWorkerAffinityTarget, its replication wiredynamo-router-protocoldynamo-worker-selectionlib/router/worker-selectionSelectionConfigdynamo-router-protocol,dynamo-slot-tracker,dynamo-session-affinitydynamo-admissionlib/router/admissionAdmissionConfigdynamo-router-protocol,dynamo-slot-tracker,dynamo-session-affinity,dynamo-tokensdynamo-schedulerlib/router/schedulerSchedulingRequest, eligibility, the hook traits hosts implement,SelectionCore, the queue actor, the stage conversions,Validated<RouterConfig>,KvSchedulerErrorandSelectionErrordynamo-kv-events,dynamo-kv-hashing,dynamo-kv-hints,dynamo-router-protocol,dynamo-tokensdynamo-kv-routerlib/router/kv-routerSelectionServiceBuilder, the plugin registry,plugins::prelude, the flatKvRouterConfigand its CLI, env, and policy-file parsers, every adapter behind a feature (http,ingest-zmq,replica-sync-zmq,peer-recovery,metrics), the private ZMQ socket helperdynamo-scheduler, the five stages, the vocabulary cratesdynamo-llm(the frontend host) anddynamo-runtimeare on the request path but are not router crates; the runtime depends on no router crate.Compatibility paths
The split lands in 1.8.0 while 1.7.0 ships
main's unsplitdynamo-kv-router, so the 1.7 import paths ofdynamo_kv_routerkeep compiling in 1.8 and 1.9 and are removed in 1.10 (owner decision, 2026-11-10; paths only: no alias features, no feature on the compat module). One file,lib/router/kv-router/src/compat.rs, holds the inventory as named re-exports from the owner crates: 660 pre-split paths (617 distinct lines: 273 plain structs as#[deprecated]type aliases that warn with the new path, 344pub uselines for traits, functions, constants, enums, generic and tuple types) across 32 pre-split modules, the crate root, and the 9 root module aliases; 14 items and thetest_utilsblock carry the owner's feature cfg.lib.rsmounts the root modules and items by name;scheduling,scheduling::config,services,services::selection, andtest_utilsstill exist and mount their blocks by name from their own files. 207 pre-split paths were already unchanged. Not re-exported: the 9 names of the dummy shared-cache server (services::shared_cache), removed by the stack, andprotocols::ActiveLoad, which lives indynamo_llm::kv_router(this crate cannot depend ondynamo-llm). The schedule, the module-to-owner table, and the removed names are on the deprecations page (docs/fern/pages/reference/general/releases/deprecations.mdx, Upcoming section). Inventory: rustdoc JSON oforigin/main'slib/kv-router(hidden items included) mapped to rustdoc JSON of the owner crates at this head, each mapping checked by normalized signature. Proof: an out-of-workspace probe crate with oneuseper inventory path compiles with 797 paths on default features, 858 withhttp,ingest-zmq,replica-sync-zmq,peer-recovery, andmetrics, and 903 withbenchandtestingadded.Dependency graph
Dynamo-family edges, from the gate rows at HEAD (
dynamo-tokens, a pre-existing leaf, omitted).graph BT error[dynamo-error] router_protocol[dynamo-router-protocol] kv_hashing[dynamo-kv-hashing] kv_events[dynamo-kv-events] kv_hints[dynamo-kv-hints] kv_indexer[dynamo-kv-indexer] slot_tracker[dynamo-slot-tracker] session_affinity[dynamo-session-affinity] worker_selection[dynamo-worker-selection] admission[dynamo-admission] scheduler[dynamo-scheduler] kv_router[dynamo-kv-router] kv_events --> kv_hashing kv_events --> router_protocol kv_hints --> kv_hashing kv_hints --> kv_indexer kv_hints --> router_protocol kv_indexer --> kv_events kv_indexer --> kv_hashing kv_indexer --> router_protocol slot_tracker --> router_protocol session_affinity --> router_protocol worker_selection --> router_protocol worker_selection --> session_affinity worker_selection --> slot_tracker admission --> router_protocol admission --> session_affinity admission --> slot_tracker scheduler --> admission scheduler --> kv_events scheduler --> kv_hashing scheduler --> kv_hints scheduler --> kv_indexer scheduler --> router_protocol scheduler --> session_affinity scheduler --> slot_tracker scheduler --> worker_selection kv_router --> admission kv_router --> kv_events kv_router --> kv_hashing kv_router --> kv_hints kv_router --> kv_indexer kv_router --> router_protocol kv_router --> scheduler kv_router --> session_affinity kv_router --> slot_tracker kv_router --> worker_selectionHow to review
The 20 review units on the fork, in order; each unit's body carries its move-diff artifact (what moved verbatim versus what is new). The stack commits are in this PR unsquashed in the same order, followed by one merge commit absorbing upstream
mainas of182039dbb5, then the three review follow-up commits listed under Review follow-ups. A second merge commita30e067f9athen absorbs upstreammainas of53015f9b05(36 commits): kthui's policy-class batch #17723 flatten policy classes, #17724 EDF ordering, #17725 controlled delay (the policy queue, EDF order, class budgets and head expiry inlib/router/admission; the queue actor's class timers, due-time checks, DRR-on-admission and retained overlap refresh inlib/router/scheduler, which also hosts the restoredpolicy.rsscore keys; the flat YAMLpolicy_classes/default_policy_classresolver inlib/router/kv-router;UnknownPolicyClassmapped to 400 in lib/llm and ext-proc), #17437 image-aware KV routing for TRT-LLM sidecars (thetrtllmwire module inlib/kv-events, the sidecar importing kv-events, kv-hashing, multimodal and protocol directly), #14775 prefill cancellation, #17716/#18077 pool relay, #17707 NVTX range, #18102 LiveEngine buffering and #17887 the shared replay clock (lib/llm, bindings, mocker, bench; auto-merged). A third merge commite626076861absorbs upstreammainas of02331f0d6a(5 commits): #18008 Fx-set worker availability (WorkerIdSetandAvailableWorkerIdsin the kernel'stypes.rs/filter.rs/queue.rs, exported to hosts viadynamo_scheduler::hostand to plugins via the prelude), #18084 CRTC sweep trigger (lib/router/kv-indexer), #17900 shared bench lane primitive (lib/bench), #17947 and #17625 outside the router. A fourth merge commite7b2b4fa04absorbs upstreammainas of20c5f11f3d(5 commits): #18009 prepared rows kept past the reject thresholds (the kernel'sfilter.rs/queue.rs/selector/mod.rsdrop full workers' prepared rows before the policy adopts them;PreparedRows::retainand thedp_sizecolumn inlib/router/worker-selection), #18108 CRTC decode-store append (lib/router/kv-indexer), #16258 vLLM token-limit defaults (lib/llm preprocessor, vLLM sidecar), #18152 and #16426 outside the router. A fifth merge commit4feb38413dabsorbs upstreammainas off580aacddc(4 commits, no conflicts): #18116 plugin selectors for the transcription router (lib/llm discovery watcher), #17691 DynamoDB registration store (global discovery service), #16288 and #16723 outside the router. A sixth merge commitdfd1bba7b1absorbs upstreammainas ofdc1fef6951(1 commit, no conflicts): #15413 operator RollingUpdate, outside the router. A seventh merge commitee45cf2d1cabsorbs upstreammainas of295854c125(1 commit, no conflicts): #16737 SGLang EAGLE3 request-migration e2e test undertests/plus three docs pages, outside the router. An eighth merge commitb6bfa517eeabsorbs upstreammainas of2fb3b132fc(1 commit, no conflicts): the automated community-events docs refresh (README.md,docs/fern/components/events.generated.ts), outside the router. A ninth merge commitd33927a031absorbs upstreammainas ofed5b75d51e(26 commits, 8 conflicts resolved into the owner crates): #17910 strict boolean worker flags (the kernel'sselection/types.rsWorkerRequest/WorkerPatchRequest, carried by rename detection), #18240 configurable session-affinity entry limit (lib/llm readsSessionAffinityConfigandMAX_SESSION_AFFINITY_ENTRIESfromdynamo_session_affinity::host, which now exports the limit), #18145 EPP co-ownership (theareas.yamlglobs re-pointed tolib/router/kv-router/src/plugins/registry.rs, the kv-router and kernel halves ofconfig.rs, the kernel'sselection/*, kv-router'sservices/selection/*andsession-affinity/src/lib.rs, owners kept), #17708 nvtx CI steps next to the stack's feature-build steps, #17623 dependency bumps (mocker keepsfastrandnext tozip; lockfiles regenerated), #17675 DynamoDB global session affinity (newlib/global-router-store-dynamodbmember), #18278 replay synthetic sessions (mocker, bindings), #16607 use-case docs refactor (the stack'sDYN_LOGdirective fix carried to the moved multimodal KV routing page), and the lib/llm, runtime, planner, operator, recipes and docs commits outside the router. A tenth merge commit19e2aa67a9absorbs upstreammainas ofe7cb9a9d2b(1 commit, 1 conflict): #17792 (the 1.8 two-tier default notice and thehost_default_factoryseam) in lib/llm'skv_router/plugins.rsandentrypoint/input/http.rs; the one conflict was the import line where the stack'sWorkerInputsand upstream'sWorkerSelectionPolicyFactoryshare ausepath, both kept. An eleventh merge commit86b7bf415babsorbs upstreammainas of4c5706219a(1 commit, no conflicts): the automated community-events docs refresh (README.md,docs/fern/components/events.generated.ts), outside the router. A twelfth merge commitf7a0ec44d9absorbs upstreammainas ofc43e48e9c8(1 commit, 6 conflicts resolved into the owner crates): #17896 standalonedynamo-loracrate (lib/lora). Upstream wires it todynamo-kv-routerforWorkerWithDpRank,ActiveSequenceEvent/ActiveSequenceEventDataandLoraWorkerFilter; the stack serves those fromdynamo-router-protocol,dynamo-slot-trackeranddynamo-worker-selection, solib/lora/Cargo.tomlnames those three in place ofdynamo-kv-router(the re-pointingdynamo-vllm-generatereceived in the first merge; the trait is defined in worker-selection and implemented in lora, so no router crate reachesdynamo-lora). The five conflictedlib/lorafiles were import blocks where rename detection carried the stack's re-points onto the moved files, resolved as upstream's flattenedcrate::paths plus the stack's imports;lib/llm/Cargo.tomlkeeps the stack's router crate list plusdynamo-lora; three lockfiles regenerated; CODEOWNERS regenerated with upstream's owners on everylib/lorapath. A thirteenth merge commit5dc7edfa94absorbs upstreammainas offa98520149(1 commit, no conflicts): #18192 operator LPX agent-runtime sidecar target, outside the router. A fourteenth merge commit68005ade79absorbs upstreammainas ofd9e5845dd7(1 commit, no conflicts): the automated community-events docs refresh (docs/fern/components/events.generated.ts), outside the router. A fifteenth merge commitdbbe5e42feabsorbs upstreammainas of57b5a57b24(2 commits, 3 modify/delete conflicts on shims the stack deleted, resolved into the owner crates): #15773 session-affinity hits logged and measured like KV selections (WorkerAffinityTarget::matchesinlib/router/session-affinity;SelectionLogFields.session_affinity, theWorkerSelector::applies_worker_filtersdefault, theselectionlog field that replaces theSelected pinned workerline, and the session-affinity fact in the kernel'sprotocols.rsandselector/mod.rs;WorkerSelectionPolicy::applies_worker_filtersas an inherent method inlib/router/worker-selectionwith the kernel's trait impl delegating, theuses_exclusive_affinity_targetshape; the best-eligible raw-cache widening with three queue tests, the wire-pin affinity filter with two core tests, and the reference-selector override carried into the kernel by rename detection; the replay adapter override inlib/mocker; lib/llm host logging, affinity-target forwarding, metrics docs and two tests), plus #15671 the Qwen3-ASR vLLM example outside the router. A sixteenth merge commit7e00036163absorbs upstreammainas ofd2ceb6a9b8(2 commits, no conflicts): #17978 release-note contributor credits and #18236 DeepSeek V4 recipes, outside the router. A seventeenth merge commit61da52673cabsorbs upstreammainas ofaa620b7181(2 commits, no conflicts): #18334 Nemotron 3.5 Super VL recipe defaults and #17937 multimodal audio-loader client errors, outside the router. An eighteenth merge commit7ab262c4a0absorbs upstreammainas of3d477a4129(1 commit, no conflicts): #16735 planner startup no longer blocked on replicas pending startup capacity (Python undercomponents/src/dynamo/planner), outside the router. A nineteenth merge commitb14856a71eabsorbs upstreammainas of197c7e56dd(1 commit, no conflicts): #18255 Kimi-K3.1-Pro recipes aligned with recipe best practices (recipes/kimi-k3.1-proplus its catalog entry and model-recipes page underdocs/fern/pages/recipes), outside the router. A twentieth merge commitbb61c38da3absorbs upstreammainas of7d8eab2b94(1 commit, no conflicts): #17575 Kubernetes discovery persistence bounded (lib/runtime/src/discovery/kube.rsandkube/crd.rs, one new runtime test), outside the router. A twenty-first merge commit0ac05d1d90absorbs upstreammainas of2680b5064d(1 commit, no conflicts): #18252 themuseparser alias deprecated in favor ofmuse_glimmer(lib/llmpreprocessor.rsandunified_parser.rsalias warning plus one unit test,components/src/dynamo/commonruntime args and tests, three tool-calling docs pages), outside the router. A twenty-second merge commit9bcad0cde3absorbs upstreammainas ofad8ecd2e5d(1 commit, no conflicts): #18293 SGLangMultimodalProcessorOutputimported fromreq_types(components/src/dynamo/sglang/_compat.py), outside the router. A twenty-third merge commit791179e68fabsorbs upstreammainas of5c17fb8ab6(1 commit, no conflicts): #15856 lifecycle badges required on recipe targets (the recipe catalog schema, validator and backfill list underdocs/fern/pages/recipes/_catalog,docs/tests/test_recipe_badges.py, and oneexamplesglob plus two cases in.github/filters.yamlandtest-filters.jsnext to the stack's router filter lines; the filters self-test passes 92 / 92), outside the router. A twenty-fourth merge commit6fd5106465absorbs upstreammainas ofef8d3b1a0b(3 commits, two import-only conflicts): #18072DYN_LORA_S3_TLS_INSECUREfor S3 LoRA downloads (lib/lora/src/source.rsplus three unit tests,lib/runtime/src/config/environment_names.rs, two LoRA docs pages), #17250 EPP tokenization moved off the async workers with stage and stall metrics (deploy/inference-gateway/ext-proc: newtokio_runtime.rsandlog_budget.rs,metrics.rs,envoy_helpers.rs,server.rs,epp.rs, the tokioschedule-latencyfeature; gateway and observability docs) taken whole, the two conflicts being adjacent import lines where the stack's owner-crate imports (dynamo_scheduler::host::{RouterConfigOverride, RoutingConstraints}inepp.rs,dynamo_kv_events::zmq_wire::DYNAMO_CACHE_SALT_PREFIXinserver.rs) are kept next to upstream's newtokio::sync::Semaphoreandcrate::tokio_runtimeimports, so the merged files differ from upstream only by those re-pointed imports, and #18003 EFA installer, Mooncake 0.3.14 and NIXL bumps (container/only); validated with the slim chain,cargo test -p dynamo-ext-proc(192 + 1 passed, all 23 new and 2 renamed or rewritten tests by name) andcargo test -p dynamo-lora(157 + 19 passed, the three new S3 TLS tests by name). A twenty-fifth merge commit58d4775769absorbs upstreammainas ofdec6c2200e(1 commit, no conflicts): #17751 plannerload_scaling_down_sensitivity0 semantics corrected (six planner docs pages), outside the router. A twenty-sixth merge commite361e9607babsorbs upstreammainas of1d6e9e0846(1 commit, no conflicts): #18338 dynamo-runtime parallel CPU-only test timeout raised to 45m (.github/workflows/dynamo-pipeline.yml), outside the router. A twenty-seventh merge commit0a5745b148absorbs upstreammainas ofbaba4997ea(1 commit, no conflicts): #17517 metrics review prompt (.ai/metrics-review.md, applied in.coderabbit.yaml, thepr-checklistskill and the review-prompts README;lib/runtime/src/metrics/prometheus_names.rsmodule doc comments; one newareas.yamlglob with CODEOWNERS regenerated identical), outside the router. A twenty-eighth merge commitdfe78dab37absorbs upstreammainas of16c7f5d176(2 commits, no conflicts): #18198 router replicas registered through global discovery (lib/global-common,lib/global-discovery-client,lib/global-discovery-servicewith the newbackends/routers.rsandtests/router_registration.rs,lib/global-frontend, whose README auto-merged with upstream's new section next to the stack'sdynamo_router_protocol::identity::PoolIdre-point, and the global-discovery-service configuration page) and #17227 TRT-LLM container images on the upstream NIXL stack (container/,tests/dependencies/test_trtllm_nixl.py, TRT-LLM and release docs), outside the router. A twenty-ninth merge commite69788f9d2absorbs upstreammainas of58be22021c(1 commit, no conflicts): #18295 Worker registration fetches HF models without weight files (lib/backend-common/src/worker.rs:LocalModel::fetch(&source, true)inbuild_local_modelplus one new test), outside the router. A thirtieth merge commit079928f7edabsorbs upstreammainas of403ead2f65(2 commits, no conflicts): #18311 TRT-LLM guided-decoding grammar rejections returned as 400 (newcomponents/src/dynamo/trtllm/errors.py,handler_base.pyraisingInvalidArgumentbefore any output token, two new tests) and #18324 Responses API echoesdescription: nullon function tools (echoed_tools_jsoninlib/llm/src/protocols/openai/responses/template_tools.rs, used byNvResponseserialization and the stream converter's lifecycle snapshots, two new tests), outside the router. A thirty-first merge commit5e36cd5bd6absorbs upstreammainas of51a6b3f2f3(1 commit, no conflicts): #18353 deploy tests ride out 60 s vCluster tunnel restarts (tests/deploy/vcluster_utils.pyraisesVCLUSTER_CONNECTION_RETRY_LIMITfrom 3 to 12,dgdr_utils.pybounds the connection retries insidewait_for_phaseby the phase deadline withasyncio.wait_for,test_dgdr_utils.pyuses the constant and gains two tests), outside the router. A thirty-second merge commit8536781e4aabsorbs upstreammainas of4948d19a4a(1 commit, no conflicts): the automateddocs: refresh community eventscalendar refresh (docs/fern/components/events.generated.ts,GENERATED_ON2026-11-06 -> 2026-11-07), outside the router. A thirty-third merge commiteccdcd1ffbabsorbs upstreammainas of335d2f6a18(1 commit, no conflicts): #17730 Harbor and Substrate agents tutorial (newexamples/agents/harbor-substrate/tree, new and edited docs pages underdocs/fern/pages/{use-cases/agents,kubernetes/agent-sandboxes}with assets anddocs/fern/index.yml;.github/codeowners/areas.yamlgives theagentsarea the globexamples/agents/and the generatedCODEOWNERSfollows, regenerated byte-identical withbuild_codeowners.py --strictpassing), outside the router. A thirty-fourth merge commitbdc8812aababsorbs upstreammainas of1dd1eefd9d(1 commit, no conflicts): #18053 SGLang reasoning gate advertised for multimodal LLM workers (components/src/dynamo/sglang/register.pykeys the gate ondynamo_args.multimodal_worker/multimodal_encode_workerinstead of the handler type, so frontend-decoded media workers advertise it;tests/test_runtime_metadata.pyreparametrized overworker_flagswith areasoning_gate_expectedcolumn), Python only, outside the router. A thirty-fifth merge commit84de392d42absorbs upstreammainas of05ee045712(1 commit, no textual conflicts, one semantic conflict resolved into the owner crates): #17893 runtime LoRA adapter identity (lib/lora/src/runtime.rstaken whole; it bounds worker configs by the router'sWorkerConfigLikeand readsRoutingConstraints, which upstream imports fromdynamo_kv_router::protocolsand the stack serves fromdynamo_scheduler::host, solib/loragains thedynamo-schedulerhost edge that lib/llm, ext-proc and mocker already have, withWorkerIdfromdynamo_router_protocol::worker; the file differs from upstream only in those four path lines;lib/llm/src/lora.rsRuntimeLoraWorkerFactsimpl and theDYN_LORA_*environment names auto-merged; three lockfiles regenerated, one line each), validated withcargo test -p dynamo-lora(171 + 19 passed, all 14 newruntime::testsby name),cargo test -p dynamo-llm kv_router(505 passed),cargo test -p dynamo-runtime environment_namesand lora clippy-D warnings. A thirty-sixth merge commiteb3e41ca84absorbs upstreammainas of27f6052cd5(1 commit, no conflicts): #18302 split tool-call openers preserved after reasoning (components/src/dynamo/frontend/prepost.pyholds back the trailing text after a reasoning end that is a proper prefix of a tool-start marker in a new_pending_tool_openerand prepends it to the next delta, so an opener split across chunks starts tool buffering instead of leaking as content;tests/test_vllm_processor_unit.pygains the parametrizedtest_hermes_split_tool_opener_at_reasoning_end), Python only, outside the router. A thirty-seventh merge commit8d6d7dffe6absorbs upstreammainas ofb977beb708(1 commit, no conflicts): the automateddocs: refresh community eventsdata refresh (no PR number; one line ofdocs/fern/components/events.generated.ts), generated docs data only, outside the router. A thirty-eighth merge commit281cf248b5absorbs upstreammainas of02a6ccbb0e(1 commit, four conflicts): #18281 worker-selection policies written inline per role (eachworker_selectionrole selects a bare policy type name or a{type, parameters}mapping; theinstances:list is deprecated for removal in v1.8 and still accepted with a warning), with the newWorkerSelectionPolicySelectionenum and per-role parser kept besideWorkerSelectionConfiginlib/router/kv-router/src/plugins/worker_selection/config.rs,WorkerSelectionPolicySelectionsand the CLI/env/YAML precedence in kv-routerscheduling/config.rs, and the registry's bare-type and inline resolution in kv-routerplugins/registry.rs; the kernel,dynamo-worker-selectionanddynamo-admissionare untouched. The four conflicts are import and test-helper hunks (lib/router/worker-selection/src/lib.rskeeps the stack's trait module, upstream's one-line re-export landing in the kv-routerplugins/worker_selection.rsshim instead;registry.rs,scheduling/config.rsandpolicy_config.rskeep the stack's owner-crate imports beside upstream's new ones) plus one semantic conflict (upstream's new registry test passes&KvRouterConfigto the factory, the stack's takes&SelectionConfig::from(&config)like its existing tests);plugins::preludeexportsWorkerSelectionPolicySelection. Validated with the slim chain (28 min, every stage 0 except the expected kvbm-consolidator manifest failure),cargo test -p dynamo-kv-router(251 + 2 + 7 passed, the registry, policy_config, config-precedence and services::selection tests by name),cargo test -p dynamo-router-policies(110 passed, all nine fixture tests by name), the mocker, ext-proc and lib/llm watcher fixture tests by name, andtest_kv_router_args.py(91 passed). A thirty-ninth merge commit19d3753420absorbs upstreammainas of3a8226a768(19 commits, one conflict): #17923 no block-lookup growth for existing-key updates, landing inlib/router/kv-indexer/src/concurrent_radix_tree_compressed/block_lookup.rs(rename-detected; the file differs from upstream only by itscrate::preludeimport); #17741 dynamo-tokenizers 4.1.2, dynamo-renderer 9.0.0 andtokenizers0.23.2 together with #17520's dynamo-protocols 8.5.0 pin, whose rootCargo.tomlhunk was the one conflict (upstream's pins kept beside the stack's workspace members and path crates; the four lockfiles regenerated with no drift); #18380 and #18448 bindingspy_convert.rs, the directtoken_idsandextra_argsreads inRoutedEngine.generate(llm/kv.rsextract_token_idsnow delegates to it, the stack's owner-crate imports kept); the other fourteen commits are lib/llm frontend code (chat-completions passthrough, media decoders and EXIF, speculative prefill, images aggregator), lib/runtime TCP_NODELAY, Python frontend, vllm, trtllm and common, recipes and docs, all auto-merged. Validated with the slim chain (40 min, every stage 0 except the expected kvbm-consolidator manifest failure), the CRTC tests (70 passed including the newexisting_keys_do_not_grow_a_full_lookup), upstream's lib/llm tests by name (514 passed across the media, speculative-prefill, chat-completions and images modules;tokenizer_cache2 passed), the bindings clippy with--all-targets, andtest_preprocessed_request_conversion.py(14 passed on a wheel built from the merged tree). A fortieth merge commit7ba3ab340cabsorbs upstreammainas of24c7f0c38e(2 commits, two conflicts): #18247 Kimi K3 prompt controls in the SGLang processor (Python frontend only) and #18345 the Anthropic streaming and Qwen grammar fixes (dynamo-renderer 10.1.0, dynamo-protocols 10.0.1, dynamo-parsers 11.0.11, dynamo-parsers-v2 0.10.1; lib/llm Anthropic stream converter and types, OpenAI validation with two new tests, the chat-completions parsers and aggregators,perf/logprobs.rs,preprocessor.rs,discovery/watcher.rs, and the generate-format tests), all outside the router. The conflicts are the rootCargo.tomlpin hunk again (upstream'sdynamo-protocolspin kept beside the stack's path crates) and two hunks inlib/bindings/kvbm/Cargo.lockwhose upstream side was empty beside the stack's kv-crate entries (kept, then the four lockfiles regenerated with no drift). Validated with the slim chain (27 min, every stage 0 except the expected kvbm-consolidator manifest failure) and upstream's lib/llm tests by name (validateand the Anthropic and chat-completions modules 478 passed including both new validation tests,aggregators36 including the new v2-parser test,logprob_analysis_integration6,postprocessor_parsing_stream117). A forty-first merge commitb4053467aaabsorbs upstreammainas of72ca6717d9(2 commits, one conflict): #17889 generic candidate policy hooks (lib/llm only:routing_policy.rsgains the crate-privateHostedCandidatePolicyandKvAdmissionPolicytraits, the newlora_routing.rsholdsLegacyLoraPolicyandLegacyLoraAdmissionwith the load lease, the routing host'slorafield becomescandidate_policyand the request guard's load guard an_attempt_lease; both new files importSchedulingRequest,ScheduleModeandKvSchedulerErrorfromdynamo_scheduler::host,HostedWorkerPickerfromdynamo_worker_selection::hostandWorkerWithDpRankfromdynamo_router_protocol::workerinstead of the pre-splitdynamo_kv_routerpaths; both traits take lib/llm'sPreprocessedRequestandRuntimeConfigWatch, so nothing moves into a router crate and the plugin prelude is unchanged) and #18490 one docs page. The conflict isrouting_host/hosted.rs, where upstream deletedselect_lora_targetand the stack had added.map_err(KvSchedulerError::from)to its picker call (the owner crate'sHostedWorkerPicker::selectreturnsSelectionFailure); the deletion is taken and the mapping travels with the code intolora_routing.rs. Validated with the slim chain (22 min, every stage 0 except the expected kvbm-consolidator manifest failure, 2,670 tests), the routing-host test module by name (79 passed including the extendedbuiltin_lora_keeps_separate_selection_and_cleanup), upstream's two newlora_routingtests, andcargo check -p dynamo-kv-router --all-featuresfor the compatibility paths. A forty-second merge commit984b3646b3absorbs upstreammainas of02235ef5cb(1 commit, no conflicts): #17891 runtime LoRA placement policy (lib/lora only: the newpolicy.rsvalidates the router mode for LoRA serving and places a versioned request-time adapter overRuntimeLoraWorkerFacts + WorkerConfigLikeunderRoutingConstraints,filter.rsgainsfilter_worker_ids_for_runtime_lora; lib/lora is a kernel host, so the newpolicy.rsimport ofdynamo_kv_router::{WorkerConfigLike, protocols::RoutingConstraints}is re-pointed todynamo_scheduler::host, which already re-exports both, and nothing moves into a router crate). Validated at Tier B:cargo check --workspace --all-targets,cargo test -p dynamo-lora --all-targets(178 + 19 passed, 1 ignored) with upstream's seven new tests by name,cargo clippy -p dynamo-lora --all-targets -- -D warnings,cargo check -p dynamo-kv-router --all-featuresfor the compatibility paths, the layering gates (36 / 0,--host55 / 0), pre-commit over the round's files andorigin/main..HEAD, and the four lockfiles--locked. A forty-third merge commita9b3636e5cabsorbs upstreammainas of3a524d0d70(1 commit, 9 conflicts): #17895 safe runtime LoRA routing (the kernel's newAccountingAttributionenum and its threading throughScheduleRequest,SchedulingRequest,SelectionOperation,PendingSelectionand the reservation booking land whole inlib/router/scheduler, re-exported throughdynamo_scheduler::hostand the plugin prelude, with the 1.7 pathdynamo_kv_router::scheduling::AccountingAttributionkept incompat.rs; lib/llm'srouting_policy.rs,lora_routing.rs,kv_router.rsandrouting_host/{hosted,kv_selection}.rsand lib/lora's newtests/accounting.rskeep upstream's content with theirdynamo_kv_routerimports re-pointed todynamo_scheduler::host,dynamo_worker_selection::host,dynamo_slot_tracker::hostanddynamo_router_protocol::worker, plus the stack'sKvSchedulerErrormapping on the hosted picker call in the newLoraHostedPolicy; the stack-onlyqueue_yaml_tests.rsliteral and upstream's new kernel test import were the two fixes upstream could not see; the example policy crates, builtin policies, mocker and the kv-router bench take the new field viaDefault::default()). Validated at Tier C: slim chain 24 min 43 s (09:28:48 to 09:53:31) with every stage 0 except the expected kvbm-consolidator manifest failure, 2,689 tests passed / 0 failed over 22 suites (router crates 1,351 incl. the scheduler's 356, kv-router 251 + 2 + 7 and the prelude test 2, llmkv_router517,discovery283,pool_relay269,lora_routing7), then by name: selection core 99 (incl. upstream'sexplicit_reservation_preserves_legacy_adapter_attributionandaccounting_attribution_preserves_routing_identity_and_reservation_replay), kernel selector/types/pending 60,cargo test -p dynamo-lora --all-targets181 + 1 + 19 (1 ignored) with upstream's threeruntime::testsandtests/accounting.rsby name, llmkv_router::routing_host123 and upstream's fourteen new lib/llm tests by name (routing host, hosted, kv_selection, builder, activation, entrypoint, migration) plus the sixlora_routingtests, the two example policy crates 3 + 2,dynamo-router-policies71 + 5 + 1 + 3 + 20 + 3, mockerreplay184,cargo check -p dynamo-kv-router --all-features0 for the compatibility paths; the layering gates (36 / 0,--host55 / 0), codeowners strict, fmt in all four workspaces, machete, pre-commit 10 / 0 / 30 on the round's 46 files and 22 / 0 / 18 overorigin/main..HEAD, and the four lockfiles--lockedwith no drift. A forty-fourth merge commitc4c5e8cebcabsorbs upstreammainas of1a08f91b25(1 commit, no conflicts): the automateddocs: refresh community eventsdata refresh of one line indocs/fern/components/events.generated.ts, no PR number, no router-owned path; Tier A:cargo check --workspace --all-targets0, the layering gates (36 / 0,--host55 / 0), pre-commit 12 / 0 / 28 on the round's file and 22 / 0 / 18 overorigin/main..HEAD, and the four lockfiles--lockedwith no drift. A forty-fifth merge commit455fd49ecaabsorbs upstreammainas of6b2b25cddc(1 commit, no conflicts): #17190 expands the operator's kube-api-linter to the full API conventions, 34 files all underdeploy/operator/(.golangci-kal.yml, Dockerfile, Makefile, theapi/v1alpha1andapi/v1beta1Go types and conversions, the roundtrip fuzz tests, the CRD bases YAML,api/scripts/generate_pydantic_from_go.py) plus the two generated API reference docs pages, taken whole; no router-owned path, no Rust; Tier A:cargo check --workspace --all-targets0, the layering gates (36 / 0,--host55 / 0), pre-commit 17 / 0 / 23 on the round's 34 files and 22 / 0 / 18 overorigin/main..HEAD, and the four lockfiles--lockedwith no drift. A forty-sixth merge commitf24e0fa464absorbs upstreammainas of4b1a320423(1 commit, no conflicts): #17943 drops thecache_transceiver_configkey from the LLaVA TRT-LLM aggregated engine config, one fileexamples/backends/trtllm/engine_configs/llava-v1.6-mistral-7b-hf/agg.yaml(+1 / -2), taken whole; no router-owned path, no Rust, no Python; Tier A:cargo check --workspace --all-targets0, the layering gates (36 / 0,--host55 / 0), pre-commit 11 / 0 / 29 on the round's file and 22 / 0 / 18 overorigin/main..HEAD, and the four lockfiles--lockedwith no drift. A forty-seventh merge commiteb8e9cde4cabsorbs upstreammainas off42e5b1fdf(4 commits, one conflict): #18384 admits sequential disagg decode with transferred prompt KV in the lib/llm mocker frontend (lib/llm/src/mocker.rs, auto-merged; theLiveEngine::submit_decodehandoff it calls stays inlib/mocker); #18199 keeps mm-routing opt-in for the default wheel (the sglang and trtllm sidecars gain a[features]table withdefault = ["mm-routing"], the bindings passdefault-features = falseto both and the wheel'smm-routingfeature enables theirs; the one conflict,lib/sidecar/trtllm/Cargo.toml, resolved as upstream's features block and baredynamo-llmline with the split'sdynamo-kv-eventsdependency kept in place of the pre-splitdynamo-kv-routeredge); #18351 moves Planner tests to their own CI lane (two workflows,container/Dockerfile.test,pytest.mark.plannerorcoreon seven test files); #18429 keeps declared Responses tools in the prompt underallowed_tools(three lib/llm files). No router-owned path, no lockfile change; Tier C: chain v2 full on the final tree (every stage 0, 2,682 tests passed, gate 36 / 0 and--host55 / 0), the four upstream tests by name, both sidecars with and without default features, the default wheel resolving nodynamo-llm/mm-routingedge, pre-commit 17 / 0 / 23 on the round's 17 files and 22 / 0 / 18 overorigin/main..HEAD, and the four lockfiles--lockedwith no drift. A forty-eighth merge commit4df5cd20dcabsorbs upstreammainas ofc2a6b68bc7(2 commits, no conflicts): #18425 forwardslogit_biasfrom the SGLang chat processor (components/src/dynamo/frontend/sglang_processor.pyconverts the JSON's str keys to the int-keyedPreprocessedRequestmap;components/src/dynamo/sglang/request_handlers/llm/decode_handler.pystringifies them again for SGLang's msgpack IPC; one new frontend unit test, one decode-handler assertion update); #18316 makes the GMS V1 SGLang plugin honor disabled memory-saver requests (lib/gpu_memory_service/v1/integrations/sglang/plugin.pydefers to the originalTorchMemorySaverAdapter.createwhenenableis false, plus its test). Python only, taken whole; no router-owned path, no Rust, no lockfile change; Tier A plus the touched Python tests:cargo check --workspace --all-targets0, the layering gates (36 / 0,--host55 / 0), pre-commit 14 / 0 / 26 on the round's six files and 22 / 0 / 18 overorigin/main..HEAD, the four lockfiles--lockedwith no drift, and the three touched pytest files (783 passed, including the round's six tests by name) in the worktree venv with sglang 0.5.23. A forty-ninth merge commit37231331e6absorbs upstreammainas of8510b90261(1 commit, no conflicts): #18465 returns 400 for media modalities the frontend cannot decode (lib/llm/src/preprocessor/media/loader.rs: a missing image, video, or audio decoder, themedia-ffmpegfeature being off, or an unknown content part type now maps toInvalidArgumentwith a fixed public message instead of an unclassified 500; new testtest_unsupported_media_is_invalid_argument). Rust in dynamo-llm's media preprocessor only, taken whole; no router-owned path, no lockfile change; Tier A plus the round's test:cargo check --workspace --all-targets0,cargo test -p dynamo-llm --lib preprocessor::media::loader29 / 0 including the new test by name, the layering gates (36 / 0,--host55 / 0), pre-commit 10 / 0 / 30 on the round's file and 22 / 0 / 18 overorigin/main..HEAD, and the four lockfiles--lockedwith no drift. A fiftieth merge commit046e51ad24absorbs upstreammainas of227196a725(2 commits, no conflicts): #18367 drops the trailing dot before the PodCliqueSet name hash (deploy/operator/internal/dynamo/graph.go, with the case added tograph_test.go), and #18188 makes component, disaggregated-set and Grove restart progress wait for the requested restart revision (deploy/operator/internal/controller: thedgd_*_restart_progress_resolver.goresolvers anddgd_grove_program.go, plus controller and envtest coverage; one line in thedynamo-graph-deployment.mdxKubernetes API page). Go operator code only, taken whole; no router-owned path, no Rust, no Python, no lockfile change; Tier A:cargo check --workspace --all-targets0, the layering gates (36 / 0,--host55 / 0), pre-commit 11 / 0 / 29 on the round's 14 files and 22 / 0 / 18 overorigin/main..HEAD, and the four lockfiles--lockedwith no drift. A fifty-first merge commit21cab08d6cabsorbs upstreammainas of104e488026(6 commits, no conflicts): #18464 records public Responses tool names and namespaces in the request trace (lib/llm/src/request_trace,http/service/openai.rs,protocols/openai/responses/tool_names.rs), #18321 reports queued requests in vLLM FPM heartbeats during long passes (components/src/dynamo/common/forward_pass_metrics.py,vllm/instrumented_scheduler.py, the newtest_vllm_instrumented_scheduler.py), #18404, #18405 and #18406 read, detokenize and edit the response items ofMigration,Backendand the preprocessor in place (lib/llm/src/migration.rs,backend.rs,preprocessor.rs, the newpreprocessor/edit.rsandtests/test_streaming_usage.rs), and #18416 rejects unsupported modalities before decoding in the vLLM multimodal request processor (vllm/multimodal_utils/request_processor.pyand its test). All taken whole; the split's re-pointed imports inopenai.rs,migration.rs,preprocessor.rsandrequest_trace/mod.rssurvive the auto-merge; no router-owned path, noCargo.tomlor lockfile change. Tier C slim: chain v2 (router crates, kv-router, workspace clippy, hosts, bindings,cargo hack, gates 36 / 0 and--host55 / 0, codeowners, fmt, machete, four lockfiles) 2,682 passed / 0 failed,cargo check --workspace --all-targets0, by nametest_streaming_usage15,migration67,backend117,preprocessor417,request_trace88,http::service::openai171,cargo check -p dynamo-llm --features mm-routing --all-targets0, pre-commit 15 / 0 / 25 on the round's 18 files and 22 / 0 / 18 overorigin/main..HEAD; in the worktree venvtest_recv_forward_pass_metrics.py14 / 0 (the two newtest_vllm_*files need vLLM, which that venv does not carry). A fifty-second merge commitbe6d22917ethen absorbs upstreammainas of33cfa1271c(3 commits: #18520 the runtimeController's pending stop or kill wait made cheap to re-poll (lib/runtime/src/pipeline/context.rs: anAtomicU8state plus aNotifyreplace thewatchpair, two new tests), #18426logit_biasforwarded in SGLang multimodal E/PD and E/P/D requests (components/src/dynamo/sglang/protocol.py,request_handlers/multimodal/worker_handler.py, one new parametrized test intest_sglang_decode_handler.py), #17924 the TRT-LLM runtime image's system-site datasets, fsspec, filelock and pyasn1 refresh (container/templates/trtllm_runtime.Dockerfile)). All taken whole and byte-identical tomain(the split touches none of the five files); no conflicts, no router-owned path, noCargo.tomlor lockfile change. Tier C slim: chain v2 2,682 passed / 0 failed with gates 36 / 0 and--host55 / 0,cargo check --workspace --all-targets0 before and after folding the two later commits, by namecargo test -p dynamo-runtime --lib pipeline::context11 andpipeline335 with both new tests ok under--exact, pre-commit 14 / 0 / 26 on the round's 5 files and 22 / 0 / 18 overorigin/main..HEAD, four lockfiles--locked0 with no drift; in the worktree venv (sglang 0.5.23)test_sglang_decode_handler.py280 / 0 with the newlogit_biastest 6 / 0 by name.dynamo-router-protocol, the wire vocabulary crate, and the fail-closed layering gatedynamo-kv-hashingdynamo-kv-events: event types, the engine ZMQ codec, the cursordynamo-zmq, the socket helper (folded back in unit 20)dynamo-kv-indexerin four slicesWorkerCapacityto protocol anddynamo-slot-trackerdynamo-session-affinitylib/router/worker-selectionlib/router/admissionlib/router/schedulerplugins::preludehostmodulesdynamo-errorActiveLoadto their owners (unit 16 introduceddynamo-multimodal; a later commit,1e45e0395f, folded it intodynamo-kv-hashingas themultimodalmodule after the crates.io name collision was found)RouterHintSourceMetadatato kv-indexer,WorkerAffinityTargetto session-affinitydynamo-zmqback into kv-router behind its ZMQ featuresThe merge commit absorbs these router PRs, each placed in the crate that owns the code at the stack head:
&selfplugin traits andWorkerInputs::ACTIVE_REQUESTSindynamo-worker-selectionanddynamo-router-protocol;HostedWorkerPickerindynamo-worker-selection; theInstanceSelectoradapter inlib/llm.router_modesindynamo-router-policiesthrough the prelude;resolve_typein kv-router's registry; the frontend's hosted path throughdynamo_worker_selection::host.dynamo-kv-indexer, the bucketed summary indynamo-kv-hints, the interval filter in the kernel's resolver.lib/llm;supports_kv_transfer_chain_retentionon the flat config; its indexer-versus-config test in kv-router.dynamo-kv-events's codec; the codec-plus-indexer tests indynamo-kv-indexer; theengine_owned_mm_identityflag through the kernel'sWorkerRequest, kv-router's registry and run config, and the bindings.lib/llm, the multimodal pad docs, since folded intodynamo-kv-hashing'smultimodalmodule).464ff290fe: perf(kv-router): log the selected worker from the routing task #17700 (selected-worker log fields in the kernel'sprotocols), feat(router): attribute Mooncake session residency from SGLang's direct linker #16610 (session residency in kv-indexer), feat(router): add sticky min-load worker-selection policy #15600 (sticky min-load policy through the prelude), refactor(frontend): unify native generate handlers behind format plugins #16983 (the vLLM generate-formats crate re-pointed to kv-hashing, kv-events, and multimodal).Validation
Chain v2 in full mode on the merged head (
/tmp/rr-split/chain-v2.sh,MODE=full,LLM_EXTRA_TESTS="discovery pool_relay"): 40 min 27 s wall, every stage exit 0 except the pre-existingkvbm-consolidatormanifest failure the chain expects.-D warningsunder default and all features,cargo test --all-features1,277 passed / 5 ignored (kv-indexer 609, scheduler 296, slot-tracker 131, kv-events 69, worker-selection 54, kv-hashing 19 + 17 + 3, error 29, admission 22, session-affinity 10, protocol 8, kv-hints 4, multimodal 4), rustdoc-D warningswith all features.http,ingest-zmq,replica-sync-zmq,peer-recovery,metrics,testing, thebench-feature bench check, 235 tests (226 + 2 + 7) plus the prelude integration test (2).cargo clippy --workspace --all-targets -D warnings(4 min 41 s): clean.dynamo-llmlib alone; itskv_router493,discovery281,pool_relay264 tests; clippy--testsper host for the 13 host crates (mocker, bench, ext-proc, router-policies, policy catalog, SGLang and vLLM sidecars, backend-common, SGLang and vLLM mockers, the EPP and soft-pin-repin examples,libdynamo_llm); the Python bindings crate withkv-indexer,slot-tracker,select-service.cargo hack check --each-featureover kv-router's features (then wrapped inrouter_features_check.sh, now the inline line inrust-feature-builds),router_layering_check.sh38 checks / 0 failures and--host57 checks / 0 failures, codeowners 7711/7711 explicitly owned,cargo fmt --check, the three lockfiles--locked.Upstream tests re-run by name in their new crates:
dynamo-kv-indexermm_exclusion_tests, 7 passed.dynamo-kv-eventspartial_tail*,undersized_block_before_any_full_block_still_reaches_conversion_and_warns,all_blocks_cleared_resets_partial_tail_filter_history, 5 passed.dynamo-kv-indexerfetch_continuation*androuter_hint_source_rejoins_only_at_independent_frontier, 4 passed;dynamo-kv-hintstransfertests, 2 passed;dynamo-schedulercontinuation_choices_match_single_source_inventory,fetch_continuation_books_one_source_from_each_actual_local_frontier,continuation_island_preference_uses_only_sources_reachable_from_local_depth,cache_owners_attached_to_the_target_are_skipped,longest_prefix_beyond_the_local_one_wins, 5 passed.dynamo-worker-selectionhosted::tests, 11 passed;dynamo-schedulerhosted_rejected_kv_policies_select_through_the_kerneland the threeACTIVE_REQUESTStests, 4 passed;dynamo-kv-routerplugin_api_prelude(carries the former compat test's hosted-picker coverage), 2 passed;dynamo-router-policiesdefault_picker_sampling_buffers_are_safe_across_threads_and_rolesandseeded_selection_matches_reference_across_cache_and_load_shapes, 2 passed.dynamo-router-policiesrouter_modesunit tests 5 passed,tests/router_modes.rs3 passed;dynamo-llmkv_router::routing_host110 passed (including the five hosted-mode tests).dynamo-kv-routerupsert_carries_engine_owned_mm_identity_to_the_listener,equivalent_mm_identity_defaults_preserve_cached_overlap,config_predicts_kv_transfer_chain_retention,shutdown_cancels_listeners_but_keeps_parent_alive, 4 passed.dynamo-mockerreplaytests 182 passed (the 12kv_fetch*,kv_transfer_keys, androuter_hint_metadata_covers_each_rank_and_keys_are_built_oncetests among them).pre-commit run --from-ref origin/main --to-ref HEADon the current head: 20 hooks passed, 20 skipped for having no applicable files, 0 failed, no hook-made edits (the router-layering hook is gone, see Review follow-ups).Size against
mainat182039dbb5: 563 files changed, 26,158 insertions, 20,211 deletions (git diff --shortstat origin/main...HEAD); 77.rsfiles added, 12 deleted, 254 modified. Moved-versus-new text over every.rsfile changed (scripts/router_move_diff.py origin/main HEAD, 266 source files deleted or modified against 331 added or modified): 199,955 normalized lines before, 205,423 after, 198,436 common (96.6% of the resulting text is moved verbatim), 6,987 new (3.4%: the hosted picker port, the bucketed transfer summary, the relocated and new tests, the prelude andhostlists, the gate script), 1,519 dropped (0.76% of the base: the deleted shim modules and the deadSchedulingPolicyand dummy shared-cache server).CI status
Pre-existing on upstream
main, not caused by this PR:planner / CPU Test(amd64 and arm64) anddynamo-runtime / test / sequential(amd64 and arm64) fail onmain's own commits182039dbb5and73014901d4with the same three tests,test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two]and::test_worker_router_mode_matches_the_real_worker_parser[args3-env3](both becausecomponents/src/dynamo/common/configuration/groups/router_args.pyno longer listspower-of-twowhile the planner test still passes it) andtest_frontend_parser_builder.py::test_no_registered_env_var_is_read_from_os_environ. Undercomponents/this PR changes only path strings that follow the moves (one test fixture path, three docstring or comment references), none of the planner or parser code those tests exercise. Also pre-existing onmain: #18405's newbackend::tests::backend_output_is_the_same_when_the_engine_is_pending_between_chunksis nondeterministic (14 of 20 solo runs fail on this head;lib/llm/src/backend.rsis byte-identical tomain): the bare-EOF flush iteratesdecoders: HashMap<u32, Decoder>, so the two synthetic per-choiceSTOPchunks come out in hash order and the test compares two independent runs.Head
be6d22917eis the fifty-second upstream merge (upstreammain33cfa1271c, 3 commits: #18520 the runtimeController's pending stop or kill wait made cheap to re-poll (lib/runtime/src/pipeline/context.rs,AtomicU8state plusNotify, two new tests), #18426logit_biasforwarded in SGLang multimodal E/PD and E/P/D requests (sglang/protocol.py,multimodal/worker_handler.py, one new parametrized test), #17924 TRT-LLM runtime image system-site dependency refresh (trtllm_runtime.Dockerfile); no conflicts, no router-owned path, noCargo.tomlor lockfile change, all five files byte-identical tomain; Tier C slim: chain v2 2,682 passed / 0 failed with gates 36 / 0 and--host55 / 0,cargo check --workspace --all-targets0 before and after folding the two later commits, by namecargo test -p dynamo-runtime --lib pipeline::context11 andpipeline335 with both new tests--exact, pre-commit 14 / 0 / 26 on the round's 5 files and 22 / 0 / 18 overorigin/main..HEAD, four lockfiles--locked0 with no drift,test_sglang_decode_handler.py280 / 0 in the worktree venv with the newlogit_biastest 6 / 0 by name), on top of21cab08d6c, the fifty-first upstream merge (upstreammain104e488026, 6 commits: #18464 request-trace public Responses tool names and namespaces (lib/llm/src/request_trace,http/service/openai.rs,tool_names.rs), #18321 queued requests in vLLM FPM heartbeats (forward_pass_metrics.py,instrumented_scheduler.py, new test), #18404 / #18405 / #18406 in-place response items inMigration,Backendand the preprocessor (migration.rs,backend.rs,preprocessor.rs, newpreprocessor/edit.rsandtests/test_streaming_usage.rs), #18416 unsupported modalities rejected before decoding (vllm/multimodal_utils/request_processor.pyand its test); no conflicts, no router-owned path, noCargo.tomlor lockfile change, the split's re-pointed imports intact; Tier C slim: chain v2 2,682 passed / 0 failed with gates 36 / 0 and--host55 / 0,cargo check --workspace --all-targets0, by nametest_streaming_usage15,migration67,backend117,preprocessor417,request_trace88,http::service::openai171,cargo check -p dynamo-llm --features mm-routing --all-targets0, pre-commit 15 / 0 / 25 on the round's 18 files and 22 / 0 / 18 overorigin/main..HEAD, four lockfiles--locked0 with no drift,test_recv_forward_pass_metrics.py14 / 0 in the worktree venv), on top of046e51ad24, the fiftieth upstream merge (upstreammain227196a725, 2 commits: #18367 the PodCliqueSet name hash no longer follows a trailing dot (deploy/operator/internal/dynamo/graph.go, one test case), #18188 component, disaggregated-set and Grove restart progress waits for the requested restart revision (deploy/operator/internal/controllerrestart progress resolvers and Grove program, controller and envtest tests, one line in thedynamo-graph-deployment.mdxKubernetes API page); no conflicts, no router-owned path, no Rust, no Python, no lockfile change; Tier A:cargo check --workspace --all-targets0, gate 36 / 0 and--host55 / 0, pre-commit 11 / 0 / 29 on the round's 14 files and 22 / 0 / 18 overorigin/main..HEAD, four lockfiles--locked0 with no drift), on top of37231331e6, the forty-ninth upstream merge (upstreammain8510b90261, 1 commit: #18465 media modalities the frontend cannot decode answered with 400 (lib/llm/src/preprocessor/media/loader.rs, one new test); no conflicts, no router-owned path, no lockfile change; Tier A plus the round's test:cargo check --workspace --all-targets0,cargo test -p dynamo-llm --lib preprocessor::media::loader29 / 0 withtest_unsupported_media_is_invalid_argumentok, gate 36 / 0 and--host55 / 0, pre-commit 10 / 0 / 30 on the round's file and 22 / 0 / 18 overorigin/main..HEAD, four lockfiles--locked0 with no drift), on top of4df5cd20dc, the forty-eighth upstream merge (upstreammainc2a6b68bc7, 2 commits: #18425logit_biasforwarded from the SGLang chat processor to the decode handler (frontendsglang_processor.py, SGLangdecode_handler.py, two tests), #18316 the GMS V1 SGLang plugin honors disabled memory-saver requests (lib/gpu_memory_serviceplugin and test); no conflicts, no router-owned path, no Rust, no lockfile change; Tier A plus the touched Python tests:cargo check --workspace --all-targets0, gate 36 / 0 and--host55 / 0, pre-commit 14 / 0 / 26 on the round's 6 files and 22 / 0 / 18 overorigin/main..HEAD, four lockfiles--locked0 with no drift, the three touched pytest files 783 / 0 in the worktree venv with sglang 0.5.23 and the round's six tests by name), on top ofeb8e9cde4c, the forty-seventh upstream merge (upstreammainf42e5b1fdf, 4 commits: #18384 sequential disagg decode admission in the lib/llm mocker frontend, #18199 mm-routing opt-in for the default wheel (sidecar[features]tables,default-features = falsein the bindings), #18351 Planner tests moved to their own CI lane, #18429 declared Responses tools kept in the prompt underallowed_tools; one conflict,lib/sidecar/trtllm/Cargo.toml, resolved keeping the split'sdynamo-kv-eventsedge under upstream's new features block; no router-owned path, no lockfile change; Tier C: chain v2 full on the final tree every stage 0 with 2,682 tests passed, gate 36 / 0 and--host55 / 0, the four upstream tests by name, sidecars with and without default features,cargo check --workspace --all-targets0, pre-commit 17 / 0 / 23 on the round's 17 files and 22 / 0 / 18 overorigin/main..HEAD, four lockfiles--locked0 with no drift), on top off24e0fa464, the forty-sixth upstream merge (upstreammain4b1a320423, 1 commit: #17943 drops thecache_transceiver_configkey from the LLaVA TRT-LLM aggregated engine config, one fileexamples/backends/trtllm/engine_configs/llava-v1.6-mistral-7b-hf/agg.yaml(+1 / -2), taken whole; no conflicts, no router-owned path, no Rust, no Python; Tier A:cargo check --workspace --all-targets0, gate 36 / 0 and--host55 / 0, pre-commit 11 / 0 / 29 on the round's file and 22 / 0 / 18 overorigin/main..HEAD, four lockfiles--locked0 with no drift), on top of455fd49eca, the forty-fifth upstream merge (upstreammain6b2b25cddc, 1 commit: #17190 kube-api-linter expanded to the full API conventions, 34 files all underdeploy/operator/(.golangci-kal.yml, Dockerfile, Makefile, theapi/v1alpha1andapi/v1beta1Go types and conversions, the roundtrip fuzz tests, the CRD bases YAML,api/scripts/generate_pydantic_from_go.py) plus the two generated API reference docs pages, taken whole; no conflicts, no router-owned path, no Rust; Tier A:cargo check --workspace --all-targets0, gate 36 / 0 and--host55 / 0, pre-commit 17 / 0 / 23 on the round's 34 files and 22 / 0 / 18 overorigin/main..HEAD, four lockfiles--locked0 with no drift), on top ofc4c5e8cebc, the forty-fourth upstream merge (upstreammain1a08f91b25, 1 commit: the automateddocs: refresh community eventsdata refresh of one line indocs/fern/components/events.generated.ts, no PR number; no conflicts, no router-owned path; Tier A:cargo check --workspace --all-targets0, gate 36 / 0 and--host55 / 0, pre-commit 12 / 0 / 28 on the round's file and 22 / 0 / 18 overorigin/main..HEAD, four lockfiles--locked0 with no drift), on top ofa9b3636e5c, the forty-third upstream merge (upstreammain3a524d0d70, 1 commit: #17895 safe runtime LoRA routing; 9 conflicts resolved into the kernel (AccountingAttributionand its threading, whole, inlib/router/scheduler, re-exported viadynamo_scheduler::host, the prelude and acompat.rsline for the 1.7 path) and into lib/llm and lib/lora with owner-crate re-points only, plus the stack'sKvSchedulerErrormapping on the newLoraHostedPolicy's picker call; Tier C: slim chain 24 min 43 s with every stage 0 except the expected kvbm-consolidator, 2,689 tests passed / 0 failed, selection core 99 and kernel selector/types/pending 60 by name, lora 181 + 1 + 19 with upstream's four new tests by name, llm routing host 123, upstream's fourteen lib/llm tests and sixlora_routingtests by name, example crates 3 + 2, router-policies 103, mocker replay 184,cargo check -p dynamo-kv-router --all-features0, gate 36 / 0 and--host55 / 0, pre-commit 10 / 0 / 30 and 22 / 0 / 18, four lockfiles--locked0 with no drift), on top of984b3646b3, the forty-second upstream merge (upstreammain02235ef5cb, 1 commit: #17891 runtime LoRA placement policy, lib/lora only, no conflicts; the newpolicy.rsimport ofdynamo_kv_router::{WorkerConfigLike, protocols::RoutingConstraints}is re-pointed todynamo_scheduler::host, which already re-exports both, because lib/lora has nodynamo-kv-routeredge after the split; Tier B:cargo check --workspace --all-targets0,cargo test -p dynamo-lora --all-targets178 + 19 passed / 1 ignored / 0 failed with upstream's seven newpolicy::testsandfilter::testsby name (17 passed),cargo clippy -p dynamo-lora --all-targets -- -D warnings0,cargo check -p dynamo-kv-router --all-features0, gate 36 / 0 and--host55 / 0, pre-commit 10 / 0 / 30 on the round's files and 22 / 0 / 18 overorigin/main..HEAD, four lockfiles--locked0 with no drift), on top ofb4053467aa, the forty-first upstream merge (upstreammain72ca6717d9, 2 commits: #17889 generic candidate policy hooks, lib/llm only, with therouting_host/hosted.rsconflict resolved by taking upstream's removal ofselect_lora_targetand carrying the stack'sKvSchedulerErrormapping into the newlora_routing.rs, whose imports androuting_policy.rs's point atdynamo_scheduler::host,dynamo_worker_selection::hostanddynamo_router_protocol::worker, and #18490 one docs page; Tier C: slim chain 22 min 14 s with every stage 0 except the expected kvbm-consolidator, 2,670 tests passed / 0 failed, routing-host tests 79 by name including the extendedbuiltin_lora_keeps_separate_selection_and_cleanup,lora_routing2,cargo check -p dynamo-kv-router --all-features0, gate 36 / 0 and--host55 / 0, pre-commit 11 / 0 / 29 on the round's files and 22 / 0 / 18 overorigin/main..HEAD, four lockfiles--locked0), on top of5d7ada5f65, the compatibility-paths commit (one commit, 9 files; see Compatibility paths above; slim chain 30 min 19 s with every stage 0 except the expected kvbm-consolidator, 2,668 tests passed / 0 failed, gate 36 / 0 and--host55 / 0,cargo hack --each-feature0, out-of-workspace probes 797 / 858 / 903 paths, pre-commit on the 9 files) on top of7ba3ab340c, the fortieth upstream merge (upstreammain24c7f0c38e, 2 commits: #18247 Kimi K3 prompt controls, Python only, and #18345 dependency bumps to dynamo-renderer 10.1.0 / dynamo-protocols 10.0.1 / dynamo-parsers 11.0.11 / dynamo-parsers-v2 0.10.1 with lib/llm protocol code; rootCargo.tomlpin hunk and two empty-upstream hunks in the kvbm lockfile resolved, four lockfiles regenerated with no drift; Tier C: slim chain 27 min with every stage 0 except the expected kvbm-consolidator, kv-router 251 + 2 + 7, llmkv_router505, lib/llm by name 478 + 36 + 6 + 117, gate 36 / 0 and--host55 / 0, pre-commit 22 / 0 / 18, four lockfiles--locked0), on top of the thirty-ninth merge19d3753420(upstream3a8226a768, 19 commits: #17923 block-lookup growth fix inlib/router/kv-indexer, #17741 dynamo-tokenizers 4.1.2 / dynamo-renderer 9.0.0 / tokenizers 0.23.2, #17520 dynamo-protocols 8.5.0, #18380 and #18448 bindingspy_convert.rs, and lib/llm, lib/runtime, Python, recipes and docs changes; one rootCargo.tomlconflict resolved as upstream's pins beside the stack's path crates, four lockfiles regenerated with no drift; Tier C: slim chain 40 min with every stage 0 except the expected kvbm-consolidator, kv-indexer 613 and CRTC 70 by name, kv-router 251 + 2 + 7, llmkv_router505 plus 514 named lib/llm tests andtokenizer_cache2, gate 36 / 0 and--host55 / 0, pre-commit 22 / 0 / 18, four lockfiles--locked0,test_preprocessed_request_conversion.py14 passed on a Modal-built wheel), on top of the thirty-eighth merge281cf248b5(upstream02a6ccbb0e, #18281 worker-selection policies written inline per role: the newWorkerSelectionPolicySelectionenum, per-role parser and bare-type or inline registry resolution all kept inlib/router/kv-routerbesideWorkerSelectionConfig; four import or test-helper conflicts resolved to the stack's owner-crate imports plus one semantic conflict (&SelectionConfig::from(&config)in upstream's new registry test); Tier C: slim chain 28 min with every stage 0 except the expected kvbm-consolidator, kv-router 251 + 2 + 7, router-policies 110, llmkv_router505, gate 36 / 0 and--host55 / 0, pre-commit 22 / 0 / 18, four lockfiles--locked0), on top of the thirty-seventh merge8d6d7dffe6(upstreamb977beb708, the automateddocs: refresh community eventsdata refresh of one line indocs/fern/components/events.generated.ts, no PR number; no conflicts), on top of the thirty-sixth mergeeb3e41ca84(upstream27f6052cd5, #18302 split tool-call openers preserved after reasoning:components/src/dynamo/frontend/prepost.pyholds a partial tool-start marker after a reasoning end in_pending_tool_openerand prepends it to the next delta, with the parametrizedtest_hermes_split_tool_opener_at_reasoning_endintests/test_vllm_processor_unit.py; Python only, no router-owned path, no conflicts; Tier A:cargo check --workspace --all-targets0, gate 36 / 0 and--host55 / 0, pre-commit 22 / 0 / 18, four lockfiles--locked0), on top of the thirty-fifth merge84de392d42(upstream05ee045712, #17893 runtime LoRA adapter identity:lib/lora/src/runtime.rstaken whole with itsWorkerConfigLikeandRoutingConstraintsimports re-pointed todynamo_scheduler::hostandWorkerIdtodynamo_router_protocol::worker,lib/lora/Cargo.tomlgaining thedynamo-schedulerhost edge, three lockfiles regenerated; no textual conflicts, one semantic conflict;cargo test -p dynamo-lora171 + 19 passed with all 14 new tests by name,cargo test -p dynamo-llm kv_router505 passed), on top of the thirty-fourth mergebdc8812aab(upstream1dd1eefd9d, #18053 SGLang reasoning gate advertised for multimodal LLM workers incomponents/src/dynamo/sglang, Python only; no conflicts), on top of the thirty-third mergeeccdcd1ffb(upstream335d2f6a18, #17730 Harbor and Substrate agents tutorial: newexamples/agents/harbor-substrate/tree, docs pages and assets,areas.yamlagents glob with the generatedCODEOWNERSregenerated byte-identical; no conflicts), on top of the thirty-second merge8536781e4a(upstream4948d19a4a, the automateddocs: refresh community eventscalendar refresh indocs/fern/components/events.generated.ts, one line; no conflicts), on top of the thirty-first merge5e36cd5bd6(upstream51a6b3f2f3, #18353 deploy tests ride out 60 s vCluster tunnel restarts intests/deploy/{dgdr_utils,test_dgdr_utils,vcluster_utils}.pywith two new tests; no conflicts), on top of the thirtieth merge079928f7ed(upstream403ead2f65, #18311 TRT-LLM guided-decoding grammar rejections returned as 400 incomponents/src/dynamo/trtllmwith two new tests, and #18324 Responses API echoingdescription: nullon function tools inlib/llm/src/protocols/openai/responseswith two new tests; no conflicts), on top of the twenty-ninth mergee69788f9d2(upstream58be22021c, #18295 Worker registration fetches HF models without weight files inlib/backend-common/src/worker.rswith one new test; no conflicts), on top of the twenty-eighth mergedfe78dab37(upstream16c7f5d176, #18198 router replicas registered through global discovery in thelib/global-*crates with two new integration test files, and #17227 TRT-LLM container images on the upstream NIXL stack; no conflicts), on top of the twenty-seventh merge0a5745b148(upstreambaba4997ea, #17517 metrics review prompt, review config and alib/runtimemodule doc comment, no Rust code, no conflicts), the twenty-sixth mergee361e9607b(upstream1d6e9e0846, #18338 dynamo-runtime parallel CPU-only test timeout, one workflow file, no conflicts), the twenty-fifth merge58d4775769(upstreamdec6c2200e, #17751 planner sensitivity docs, six docs pages, no conflicts), the twenty-fourth merge6fd5106465(upstreamef8d3b1a0b, #18072 LoRA S3 TLS knob, #17250 EPP tokenization off the async workers with stage metrics, #18003 container bumps; two import-only conflicts in ext-procepp.rsandserver.rsresolved by keeping the stack's owner-crate imports beside upstream's new ones; slim chain green,cargo test -p dynamo-ext-proc192 + 1 passed including all 23 new and 2 renamed or rewritten tests,cargo test -p dynamo-lora157 + 19 passed including the three new tests), the twenty-third merge791179e68f(upstream5c17fb8ab6, #15856 recipe lifecycle badges, docs/recipes catalog and CI filters only, no conflicts), the twenty-second merge9bcad0cde3(upstreamad8ecd2e5d, #18293 SGLangMultimodalProcessorOutputimport fix, one Python file, no conflicts), the twenty-first merge0ac05d1d90(upstream2680b5064d, #18252museparser alias deprecation, lib/llm parser code plus frontend Python and docs, no conflicts), the twentieth mergebb61c38da3(upstream7d8eab2b94, #17575 Kubernetes discovery persistence bounded,lib/runtimeonly, no conflicts), the nineteenth mergeb14856a71e(upstream197c7e56dd, #18255 Kimi-K3.1-Pro recipe alignment, recipes and docs only, no conflicts), the eighteenth merge7ab262c4a0(upstream3d477a4129, #16735 planner startup capacity, Python only, no conflicts), the seventeenth merge61da52673c(upstreamaa620b7181, #18334 recipes and #17937 multimodal audio loader, no conflicts), the sixteenth merge7e00036163(upstreamd2ceb6a9b8, #17978 release-note credits and #18236 DeepSeek V4 recipes, no conflicts) and the fifteenth mergedbbe5e42fe(upstream57b5a57b24, #15773 session-affinity hits logged and measured like KV selections, placed into session-affinity, worker-selection and the kernel, plus #15671; chain v2 slim 26:18 green with 2,664 tests, the round's seven kernel tests, two lib/llm tests,dynamo-router-policies110 and mocker 322 by name, gate 36 / 55); CI on it is running.Fixed on the previous heads: the TRT-LLM
test_deployment[aggregated_router-2]failure was the test's ownDYN_LOGnaming the pre-splitdynamo_kv_router::scheduling::selectortarget, andCheck for broken markdown linksflagged two README links thelib/router/move lengthened (both in the fourth follow-up commit); before that, thecargo machetefindings and thelib/bindings/pythoncargo fmt --checkfailure, the SPDX headers, the CI-filter coverage of the feature script, the stale docs link, and thelib/runtime/exampleslockfile.Regression check against the base wheel
Same scenario on two wheels, the PR head
6c50b1d0f2and its merge parent182039dbb5(both built the same way), compared output for output. Every item had identical outcomes on both wheels:tests/router -m "router and gpu_0", 69 ids) and the bindings and config unit tests (178 ids, plus the planner file with the two pre-existing failures).round-robin,random,kv,direct,least-loaded,device-aware-weighted) and every builtin policy (default cost function, two-tier, thunderagent, sticky min-load): 200 requests each, same per-request selection fields and logits; worker labels differ only where equal-cost ties break through an unseeded RNG.dynamo-benchreplay-bench): canonical reports byte-identical on four golden rows and four forced-pressure rows (vLLM and SGLang, aggregated and disaggregated, preemption and retraction)./metricsnames, labels, and help strings;prometheus_names.pycodegen;libdynamo_llm_capiexported symbols (31);cargo test -p dynamo-ext-proc(168).gpu_1router cases on one L4, and on L40S the fulltests/router/test_router_e2e_with_sglang.py(one and two GPUs, including disaggregated and EAGLE) plus the KV-routertests/serve/test_sglang.pyconfigs: same pass set, andSelected worker ... logit=logged once per routed request on both wheels.The one difference found is the log-target rename that the fourth follow-up commit addresses:
DYN_LOG=...,dynamo_kv_router=debugon the old wheel also enabled the selection-core debug lines that now live underdynamo_scheduler; addingdynamo_scheduler=debugrestores the identical line set.Review follow-ups
Four commits on top of the merge commit, from the review and the first CI run on 2026-11-04:
dynamo-kv-routeranddynamo-kv-hintsmoved underlib/router/(lib/router/kv-router,lib/router/kv-hints): crates only the router produces live there; the shared contract crates (lib/kv-events,lib/kv-hashing,lib/multimodalat the time,lib/error) andlib/router-plugins/stay. 65 files renamed, 59 byte-identical and 6 changed only in a path string (theAGENTS.mdheaders and oneinclude_str!); the orphansrc/sequences/docs directory is deleted; every path reference in the manifests, codeowners, the gate script, docs, skills, and comments follows, and the lockfiles are untouched by the move.rust-clippyjob'sCheck router crate layeringstep, with--host) and runs locally on demand; its pre-commit hook is dropped. The per-feature build check is onecargo hack check --locked -p dynamo-kv-router --each-feature --all-targetsline insiderust-feature-builds, andscripts/router_features_check.shis gone.scripts/router_move_diff.pystays as a dev tool.dynamo-kv-router:arc-swap,crossbeam-queue,derive_builder,flume,futures-util,xxhash-rust;dynamo-scheduler:dashmap,ordered-float;dynamo-admission:serde_json;dynamo-kv-indexer:seqlock;libdynamo_llm:dynamo-router-protocol), the three affected lockfiles are regenerated, andcargo fmt --allis applied in every workspace (two import blocks inlib/bindings/python).dynamo-multimodalis folded intodynamo-kv-hashingas itsmultimodalmodule (1e45e0395f): the namedynamo-multimodalis already published on crates.io byai-dynamo/frontend-crates(0.1.0 to 0.2.4), socargo package -p dynamo-kv-hashingcould not resolvedynamo-multimodal = "^1.7.0". Every one of its ten dependents already depended ondynamo-kv-hashing, which already depended on it, so no dependency edge changes and the node disappears from the graph (12 crates). The three source files moved verbatim (git mv, byte-identical);lib.rsbecamemultimodal/mod.rswith the samemodlines andpub uselist; the 60dynamo_multimodal::paths in 30 files aredynamo_kv_hashing::multimodal::, the root re-exports none of it. Workspace member and dependency, nine manifest lines, the gate row and lists, and thelib/multimodal/codeowners glob (nowlib/kv-hashing/src/multimodal/, router, multimodal and kv-memory teams) follow; the four lockfiles are regenerated.cargo package --locked --no-verify -p dynamo-tokens -p dynamo-kv-hashingsucceeds on the folded tree. Validation of the fold: chain v2 in full mode on its tree, 38 min 28 s, every stage exit 0 except the pre-existing kvbm-consolidator manifest failure (2,656 tests passed, 0 failed, 5 ignored; the 11 router crates' clippy, tests, and rustdoc under default and all features; kv-router 237 plus the prelude test; workspace clippy;dynamo-llmlib alone and itskv_router503,discovery,pool_relaytests; per-host clippy--testsfor the 13 hosts and the bindings; the gate 36 plain / 55 host checks, 0 failures; codeowners strict; fmt in all four workspaces; machete; four lockfiles--locked); by name,dynamo-kv-hashing23 + 17 + 3 (the fourmultimodal::tests among them),dynamo-kv-events74 (extra_key5, pad normalization 5),dynamo-kv-indexermm_exclusion7,dynamo-vllm-generate37,dynamo-trtllm-sidecar93 + 3,dynamo-llmpreprocessor397;scripts/router_move_diff.pyover the four files: 147 normalized lines before, 146 after, 146 common, 0 new, 1 dropped (the#![deny(missing_docs)]the crate root already sets); pre-commit 22 passed.dynamo_kv_router::scheduling::selector=becomesdynamo_scheduler::selector=,dynamo_kv_router::scheduling=debugbecomesdynamo_scheduler=debug,dynamo_admission=debug,dynamo_worker_selection=debug, and the nemotron recipes that setdynamo_kv_router=debugnow also name the eight crates that code moved into (the TRT-LLM serve test'sDYN_LOGwas whytest_deployment[aggregated_router-2]missed itsSelected worker ... logit=line). The deprecations page carries the same mapping. Two README links thelib/router/move lengthened by one directory are corrected.Validation of the three commits: chain v2 in full mode (
MODE=full,LLM_EXTRA_TESTS="discovery pool_relay") on the final tree, 34 min 18 s wall for the rebuild the manifest changes force, every stage exit 0 except the pre-existingkvbm-consolidatormanifest failure and one hit of a pre-existing race-test flake,dynamo-kv-indexerconcurrent_radix_tree_compressed::tests::race_tests::cleanup::race_cleanup_with_dead_child_reuse_keeps_restored_child(unexpected edge lengths: [2, 2, 3]; the module and the test are identical tomainmodulo import paths, and 1 of 30 isolated runs fails), after which the router test stage was re-run with--no-fail-fast(1,277 passed, 0 failed, 5 ignored) and the whole chain once more on the same tree (2 min 43 s warm; 2,552 tests passed, 0 failed, 5 ignored). Covered: the 12 router crates' clippy, tests, and rustdoc under default and all features; kv-router 237 tests plus the prelude test;cargo clippy --workspace --all-targets -D warnings;dynamo-llmlib alone and itskv_router493,discovery281,pool_relay264 tests; per-host clippy--testsfor the 13 hosts and the Python bindings crate; the layering gate 38 checks and--host57 checks, 0 failures; codeowners 7707/7707 explicitly owned;cargo fmt --checkin all four workspaces;cargo macheteclean apart from the three pre-existingkvbm-*manifest load errors; the four lockfiles--locked;git diff -Mover the move commit shows 65 renames (59 byte-identical, 6 changed only in a path string) and no lockfile change. The first chain pass, on the move and CI commits alone, took 39 min 0 s and was green throughout.Release note
crates.io carries
dynamo-runtime,dynamo-llm,dynamo-kv-router, anddynamo-kv-hashingat 1.6.2 and none of the ten crates this PR adds. This lands in 1.8.0, and that release must publish the new crates first:dynamo-errorbeforedynamo-runtime, and the router crates beforedynamo-kv-routeranddynamo-llm..github/scripts/stage_crates.pyalready discovers and orders them (45 crates at this head instead of 35 onmain, withdynamo-errorfirst), so the release script needs no change, only the awareness that the first publish of each new name happens in that release.Tickets
Part of DYNO-170
Part of DYNO-171
Part of DYNO-172
Part of DYNO-173
Part of DYNO-174
Part of DYNO-175
Part of DYNO-176
Part of DYNO-177
Part of DYNO-178
Part of DYNO-179
Part of DYNO-180
Part of DYNO-181
Part of DYNO-182
Part of DYNO-183
Part of DYNO-184
Part of DYNO-185
Part of DYNO-186
Part of DYNO-190
Part of DYNO-191
Part of DYNO-192
Part of DYNO-193
Part of DYNO-194
Part of DYNO-195
Part of DYNO-196
DEP to follow; this draft is for early review of the shape.