Skip to content

refactor(discovery): move the model catalog and the routed-pipeline builder into dynamo-discovery - #227

Draft
ishandhanani wants to merge 7 commits into
idhanani/dyno-228-worker-sidefrom
idhanani/dyno-230-discovery
Draft

ishandhanani wants to merge 7 commits into
idhanani/dyno-228-worker-sidefrom
idhanani/dyno-230-discovery

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

Layer 14 of the lib/llm split stack (DYNO-230). Base: #226 (worker-side code into dynamo-backend-common), so this PR shows only this layer. It moves the frontend's model catalog and the routed-pipeline builder out of dynamo-llm into the new crate dynamo-discovery (lib/discovery). The crate sits between the frontend (dynamo-llm: HTTP, gRPC, entrypoint), which calls into it, and dynamo-router-host, whose routers it builds. It does not depend on dynamo-llm.

What moved

  • dynamo_llm::discovery::X is dynamo_discovery::X: ModelManager, ModelWatcher, ModelDiscoveryController, Model, WorkerSet, the readiness evaluator, the image-edit and router-policy registries and the worker inventory (lib/llm/src/discovery.rs is the crate root, discovery/*.rs its modules under the same names).
  • The routed-pipeline builder: PreprocessedRouting, build_preprocessed_routing and the types ChatEngineFactoryCallback and PrefillRoutedEngine are dynamo_discovery::{...} (src/pipeline.rs). They were dynamo_llm::entrypoint::{...}.
  • dynamo_llm::kv_router::shared_cache (the HiCache shared KV cache client, built by ModelManager) is dynamo_discovery::shared_cache. dynamo-llm's kv_router module is gone.
  • lib/llm/src/scheduler_load.rs (the scheduler-load publisher ModelManager hands to every KV router) is the private module dynamo_discovery::scheduler_load.
  • dynamo_llm::engines (echo engines, StreamingEngine, EmbeddingEngine, the adapters) is dynamo_discovery::engines.
  • The frontend's eight LoRA allocation gauges, register_lora_allocation_metrics, controller_metrics and LoadEstimatorSubscription are dynamo_discovery::lora::{...}.
  • 21 files move with git mv in the move commit: 19 Rust files (19,970 lines with tests) and the two agent files of the WorkerSet admission contract, now lib/discovery/src/AGENTS.md. scripts/router_move_diff.py over the Rust files of the move commit: 16,534 normalized lines before, 16,535 after, 16,523 common, 12 new, 11 dropped. The 11 dropped lines are the six sample-model paths and five inline crate::discovery:: paths, each of which has its rewritten counterpart among the new ones; the twelfth new line is the test_utils include.
  • lib/llm/src goes from 138,195 to 118,184 lines of Rust (240 to 221 files). What dynamo-llm still declares: block_manager 40.5k (behind its feature), http 35.3k, pool_relay 27.8k, mocker 6.4k, grpc 3.1k, entrypoint 2.0k, tokens.rs 1.7k, generate_format 490, cuda.rs 207, frontend_load.rs 162, common 55, utils 46 (lora_name_to_id), and the re-export modules lora.rs, namespace.rs, worker_type.rs and tokenizers.
  • No compatibility re-exports: dynamo-llm has no pub use dynamo_discovery, and the old paths are deleted.

Commits

Five in-place cuts, one widening commit and one move commit. Each builds and passes on its own.

Commit What
a07644101e utils/retry.rs (Backoff, FailureStreak) moves to dynamo_runtime::utils::retry
294da232b6 SCHEDULER_LOAD_SUBJECT, SCHEDULER_LOAD_PUBLISH_INTERVAL and serving_load_publishing_enabled are defined in dynamo_sync::load
18c9fdf2ee 48 paths in discovery name the owning crate instead of a dynamo-llm re-export module
f8a2c802aa The LoRA allocation gauges, controller_metrics and the load-estimator subscription move next to ModelManager as discovery::lora
3af919b930 The routed-pipeline builder and the chat engine factory types move from entrypoint to discovery::pipeline
207aac425c 61 crate-private catalog items that the frontend uses become pub
89edb624d6 The move: git mv, path rewrites, manifests, docs, codeowners

Edges

Edge Resolution Alternative rejected
discovery::watcher calls entrypoint::input::build_preprocessed_routing_with_session_affinity_mode and stores an entrypoint::ChatEngineFactoryCallback; entrypoint/input/common.rs names ModelManager and ModelWatcher. The one real loop 3af919b930: what names a router or Migration moves to discovery/pipeline.rs (PreprocessedRouting with build_pipeline and build_preprocessed_pipeline, build_preprocessed_routing*, wait_for_min_initial_workers, router_client, two tests, the two callback types). prepare_engine, PreparedEngine, the local chain build_pipeline, build_completions_pipeline, link_pipeline, EngineConfig and the input runners stay in entrypoint, which now imports discovery one way A trait for the builder that entrypoint implements and the watcher calls: a new trait and an injection point for code that has one caller
ModelManager::ensure_lora_controller calls lora::controller_metrics(), which clones eight gauges defined in http::service::metrics; the HTTP service imports ModelManager f8a2c802aa: the gauges, their registration function and controller_metrics move next to their only writer as discovery::lora. Names, labels, help texts and the registration point and order are unchanged Passing the gauges into ModelManager (every constructor call would have to supply them or lose the series); defining them in dynamo-lora (it takes its handles from the frontend on purpose)
ModelManager uses lora::LoadEstimatorSubscription (lora/subscription.rs) f8a2c802aa: discovery/lora/subscription.rs, with its only caller dynamo-lora or dynamo-sync: a new dependency between the two for one caller
ModelManager passes scheduler_load::start_scheduler_load_publisher to KvRouter; the publisher reads pool_relay::serving_load_publishing_enabled(); the Pool Relay reads the subject and cadence from scheduler_load 294da232b6: the subject, the cadence and the flag are dynamo_sync::load::{...}, next to KV_METRICS_SUBJECT. The publisher moves with the catalog in the move commit The publisher in dynamo-router-host, which owns the SelectionService it reads: that undoes the injection layer 11 made and is a refactor
scheduler_load.rs, frontend_load.rs and three Pool Relay loops use utils::retry a07644101e: dynamo_runtime::utils::retry. It is generic (names only Duration) A copy per crate
ModelManager builds kv_router::shared_cache::HicacheSharedKvCache It moves with the catalog; it names no dynamo-llm module Leaving it in dynamo-llm behind the SharedKvCache trait: ModelManager constructs it
Nine tests of the catalog register engines::EchoBidirectionalEngine or StreamingEngineAdapter::new(make_echo_engine()). No production code of the catalog uses engines engines.rs moves with the catalog. Production users, all in dynamo-llm: EngineConfig::InProcessText, the adapter in entrypoint/input/{common, http, grpc, endpoint}.rs, and make_echo_engine in the Python bindings A dev-dependency on dynamo-llm (a cycle, and cargo tree would show it); dynamo-backend-common (a new dev edge to a worker-side crate); dynamo-frontend-protocols (hoists stage code into the protocol crate); stub engines in the tests (a copy)
48 paths reach WorkerType, NamespaceFilter, NamespacePrefixMode, Tokenizer and eleven dynamo-lora items through dynamo-llm re-export modules 18c9fdf2ee: they name the owner (dynamo-router-protocol 29, dynamo-lora 15, dynamo-runtime 3, dynamo-tokenizers 1). namespace.rs, worker_type.rs, lora.rs and tokenizers stay in dynamo-llm for its other callers Moving the re-export modules
ModelWatcher carries Arc<dynamo_http_server::metrics::Metrics> into PreprocessedRouting::build_pipeline for Migration::from_mdc; impl From<CommittedModelView> for dynamo_http_server::load::ModelLoadView dynamo-http-server is a direct dependency. No moved code serves HTTP; dynamo-router-host and dynamo-sync have the same edge for the same handle. Decision to confirm (below) Making ModelWatcher and the builder generic over the metrics handle: a refactor that hides the edge without removing it from the tree
Ten tests call crate::test_utils::{run_isolated, isolated_command, assert_isolated_success, tracing_interest_guard} The crate root includes lib/runtime/src/test_utils.rs under cfg(test), as the other split crates do n/a
Six tests load tests/data/sample-models/mock-llama-3.1-8b-instruct relative to the crate directory They name ../llm/tests/data/sample-models/...; the fixture stays with its other users, as for dynamo-preprocessor Moving or copying the fixture

What was widened

61 definitions, compiler-driven (E0603, E0624, E0616, E0451, E0365, private_interfaces): four modules (image_edits, readiness, router_policy, worker_inventory) and one re-export in discovery.rs; nine methods, router_policies, CommittedModelView and its name in model_manager.rs; GenerateEngineSelection, SystemOneExecutionSelection, SystemOneSelectionError, ten of their fields, validate_dispatch and Model::decision_capabilities in model.rs; three ModelWatcher methods; WorkerSet::{chat_engine, load_thresholds}; SNAPSHOT_KEY, ImageEngineSelection with two fields and ImageEditSnapshot; RouterPolicyTarget and RouterPolicyRegistry with three methods; WorkerGroupState, WorkerGroupObservation with eight fields and one method, WorkerInventory with two methods. 85 pub(crate) and pub(super) declarations keep their visibility. No test-only item crosses the boundary, so the crate has no testing feature. No trait was added.

Callers

  • In lib/llm, 32 Rust files name dynamo_discovery: the entrypoints, the HTTP and gRPC services, frontend_load.rs, two Pool Relay files and five integration tests. lib.rs loses four module declarations.
  • Outside lib/llm: the Python bindings (rust/llm/{entrypoint, kv, routed_engine}.rs), the Runtime EPP (deploy/inference-gateway/ext-proc/src/epp.rs), the sidecar testkit (two test files), and the tests of dynamo-generate-format-catalog and dynamo-global-frontend. Their manifests gain the dependency (dev-dependency for the last three).
  • dynamo-llm drops dashmap, dynamo-kv-hints, dynamo-worker-selection and libc, and the testing feature of dynamo-router-host in its dev-dependencies.

Log targets

  • 74 production log statements change prefix: dynamo_llm::discovery::{allocator 2, controller 16, endpoint_card 2, model 3, model_manager 11, watcher 15} are dynamo_discovery::{...}; dynamo_llm::kv_router::shared_cache (18) is dynamo_discovery::shared_cache; dynamo_llm::scheduler_load (3) is dynamo_discovery::scheduler_load; dynamo_llm::lora::subscription (4) is dynamo_discovery::lora::subscription. Targets are not pinned.
  • dynamo_llm::kv_router=<level> is dynamo_discovery::shared_cache=<level> in 17 places: examples/backends/{sglang,vllm}/launch/agg_multimodal_router.sh, the six recipes/nemotron-3-ultra/vllm/disagg-*/deploy.yaml, tests/mm_router/test_router_rust_mm_{frontend_decode,glm5_next,router}_e2e.py, tests/router/test_router_e2e_with_mockers.py, the docs pages recipes/cli-templates/{sglang,vllm}.mdx, multimodal-kv-routing.md and exact-video-kv-routing.md, and the EPP note in the 1.7.0 release notes.
  • tests/router/test_router_e2e_with_mockers.py set DYN_LOG=info,dynamo_llm::discovery=debug and now sets info,dynamo_discovery=debug: the same lines, plus the debug lines of shared_cache, scheduler_load and lora::subscription, which are in the crate now.

Plumbing

  • lib/discovery/AGENTS.md states what the crate owns, depends on and must not depend on. The guides of dynamo-llm, dynamo-router-host, dynamo-sync and the KV publisher, and the comments and docs that cite the moved files, point to the new place.
  • Codeowners: every moved file keeps its owners. lib/discovery/ is runtime's; src/pipeline.rs, src/lora.rs (shared with observability, which co-owned the gauges' old file) and src/lora/ are frontend's; src/scheduler_load.rs is router's; src/shared_cache.rs is router's and kv-memory's; lib/sync/src/load/serving_load.rs is router's and global-router's. The router PR label covers lib/discovery/src/shared_cache.rs. The CI path filters match lib/** and need no change.
  • Lock files: the root, lib/bindings/python and lib/bindings/kvbm ones change (no new third-party package). lib/runtime/examples does not change.
  • The move is reproducible: /tmp/llm-split/moves/dyno-230-widen.py reproduces the widening commit from its parent, and /tmp/llm-split/moves/dyno-230.py reproduces the move commit from the widening commit (same tree hash).

Behaviour

No discovery watch or registration order, WorkerSet admission, readiness evaluation, router construction or configuration, pipeline stage order, metric name or label, log message, env var, subject or wire format changes.

Decisions to confirm

  • engines.rs lives in dynamo-discovery. It is there because the catalog's tests need it below them; its production users are in the entrypoint.
  • dynamo-discovery depends on dynamo-http-server directly, for metrics::Metrics and load::ModelLoadView. The cut that removes the edge from dynamo-sync, dynamo-router-host and this crate at once is to move lib/http-server/src/{metrics, load}.rs into a crate that is not the HTTP server.
  • The scheduler-load subject, cadence and enabling flag live in dynamo_sync::load.
  • The LoRA allocation gauges are defined in dynamo_discovery::lora and registered by the HTTP service.
  • readiness.rs moves with the catalog here; the Pool Relay layer moves it below both.

Validation

Baseline is the parent 2b7750bb15, head is 89edb624d6; the same commands ran at both on the Modal cargo runner.

  • cargo test -p dynamo-llm -p dynamo-backend-common -p dynamo-router-host -p dynamo-sync -p dynamo-global-frontend --no-fail-fast (plus -p dynamo-discovery at the head): baseline 49 binaries, 2,618 passed, 0 failed, 22 ignored; head 51 binaries, 2,621 passed, 0 failed, 22 ignored.
  • Test names (-- --list): 2,640 at baseline, 2,643 at head. 2,402 keep their binary and path. 237 are in dynamo_discovery with the same leaf name: discovery::{model_manager 58, model 57, watcher 37, controller 26, worker_set 11, readiness 6, allocator 2, image_edits 2} lose the discovery:: prefix, kv_router::shared_cache (30) is shared_cache, entrypoint::input::common (2) is pipeline, and engines (2), scheduler_load (2), lora::subscription (1) and lora::metrics_tests (1) keep their path. One, utils::retry::tests::failure_streak_starts_once_and_reports_its_length_on_recovery, is in dynamo-runtime and passes there. 0 missing. 4 added: the self-tests of lib/runtime/src/test_utils.rs, which the new crate root includes.
  • cargo test -p dynamo-sidecar-testkit with the sidecars built: 113 passed at both. The tests of the four sidecars, dynamo-sidecar-common, the three mocker servers and dynamo-mocker-backend pass at both.
  • Crates whose sources changed, at the head: cargo test -p dynamo-discovery alone 241 passed; cargo test -p dynamo-runtime --lib 914 passed, 2 ignored; dynamo-ext-proc 193 passed; dynamo-generate-format-catalog 12 passed; dynamo-lora and dynamo-model-card pass.
  • Every commit on its own: at each of the six commits before the move, cargo check --workspace --all-targets, cargo clippy --no-deps -p dynamo-llm -p dynamo-sync -p dynamo-runtime --all-targets -- -D warnings, cargo check --locked in lib/bindings/python, cargo fmt --check, the four lock files, the regenerated CODEOWNERS, and cargo test -p dynamo-llm -p dynamo-sync --lib (1,271 and 191 passed, 0 failed) pass.
  • cargo check --workspace --all-targets, cargo clippy --no-deps --workspace --all-targets -- -D warnings, cargo clippy --no-deps -p dynamo-discovery --all-targets -- -D warnings, cargo fmt --all -- --check: pass.
  • cargo check --locked in lib/bindings/python (also clippy -D warnings and fmt), lib/bindings/c, lib/bindings/kvbm, and lib/runtime/examples (--all-targets): pass. cargo metadata --locked passes for the four lock files.
  • Features: no feature gates moved code. cargo check -p dynamo-llm --all-targets with ckf-diagnostics, mm-routing and testing-etcd, and --features kv-router-stress --bench kv_router_bench: pass. The call in lib/llm/tests/http_metrics.rs that stopped --features integration is fixed in refactor(llm): serve HTTP through dynamo-http-server #209; that check passes there, at refactor(llm): move the request path into dynamo-router-host #225 and at the top of the stack (refactor(pool-relay): move the Pool Relay into dynamo-pool-relay #228).
  • cargo machete at the root, in lib/bindings/python and in lib/runtime/examples: clean at the parent and at the head (the root after dynamo-llm dropped the four dependencies).
  • uvx pre-commit run --from-ref 2b7750bb15 --to-ref HEAD: pass. build_codeowners.py --strict: pass; CODEOWNERS is regenerated.
  • OpenAPI: cargo run -p dynamo-llm --bin generate-frontend-openapi writes the same file, SHA-256 706f15d951891703fa911b9c659ac54992fa7144365ce4cd6b2a179a7630ede5.
  • Graph: cargo tree -p dynamo-discovery | grep -c dynamo-llm prints 0. Direct Dynamo dependencies: dynamo-router-host, dynamo-sync, dynamo-preprocessor, dynamo-frontend-protocols, dynamo-model-card, dynamo-lora, dynamo-runtime, dynamo-http-server, dynamo-kv-router, dynamo-scheduler, dynamo-worker-selection, dynamo-slot-tracker, dynamo-kv-indexer, dynamo-kv-hashing, dynamo-kv-hints, dynamo-kv-events, dynamo-router-protocol, and the published dynamo-protocols, dynamo-decisions, dynamo-renderer, dynamo-tokenizers; for tests also dynamo-router-policies. Its normal dependency tree has 681 crates; dynamo-llm's has 729.
  • Not run: the Python e2e tests whose DYN_LOG filters changed (they need GPUs or a cluster); media-ffmpeg builds (no FFmpeg on the runner).

Closes DYNO-230

… dynamo-runtime

DYNO-230, layer 14 of splitting lib/llm into smaller crates. The model catalog (`discovery`) and the Pool Relay are about to become crates of their own, and neither may depend on dynamo-llm. Both pace their reconnect loops with `dynamo_llm::utils::retry`: the scheduler-load publisher that discovery hands to the KV router, the frontend-load publisher, and three loops of the Pool Relay. The module is generic (a capped exponential `Backoff` and a `FailureStreak` that says when to warn and when to report recovery; it names only `std::time::Duration`), so it moves below all of them.

- `lib/llm/src/utils/retry.rs` is `lib/runtime/src/utils/retry.rs` (`git mv`), declared as `dynamo_runtime::utils::retry`. Its eight `pub(crate)` items become `pub`; nothing else in the file changes. Its unit test moves with it.
- The five users name the new path: `scheduler_load.rs`, `frontend_load.rs`, `pool_relay/host.rs`, `pool_relay/serving_load/frontend.rs` and `pool_relay/serving_load/roster.rs`.
- No compatibility re-export: `dynamo_llm::utils::retry` is gone. It was crate-private, so no other crate named it.

No delay, cap, log message or log level changes.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…e serving-load flag in dynamo-sync

DYNO-230, layer 14 of splitting lib/llm into smaller crates. The scheduler-load stream has one publisher and one subscriber on opposite sides of the split: discovery hands `start_scheduler_load_publisher` to every KV router it builds, and the Pool Relay subscribes. The publisher read its enabling flag from `pool_relay`, and the Relay read the subject and the cadence from `scheduler_load`, so the two modules imported each other. The vocabulary both sides share moves below both, next to `KV_METRICS_SUBJECT` and the other worker-load messages.

- New `dynamo_sync::load::{SCHEDULER_LOAD_SUBJECT, SCHEDULER_LOAD_PUBLISH_INTERVAL, serving_load_publishing_enabled}` in `lib/sync/src/load/serving_load.rs`. The three definitions and their doc comments are the ones from `lib/llm/src/scheduler_load.rs` and `lib/llm/src/pool_relay.rs`; the subject is still `scheduler_load`, the interval one second, and the flag still reads `DYN_SERVING_LOAD_PUBLISH`.
- `scheduler_load.rs` keeps the publisher and imports the three names. The HTTP service, which checks the same flag before it starts the frontend-load publisher, and `pool_relay/{host, load, serving_load}.rs` name `dynamo_sync::load`.
- No compatibility re-export: `dynamo_llm::scheduler_load::SCHEDULER_LOAD_SUBJECT` and the crate-private `pool_relay::serving_load_publishing_enabled` are gone; no other crate named them.
- `lib/sync/AGENTS.md` lists the vocabulary under `load`.
- Codeowners: `lib/sync/src/load/serving_load.rs` is router's and global-router's, the owners of the two files its definitions came from.

No subject, cadence, env var or wire format changes. `scheduler_load` no longer imports `pool_relay`.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
DYNO-230, layer 14 of splitting lib/llm into smaller crates. The model catalog (`discovery`) is about to become the crate `dynamo-discovery`, which may not depend on dynamo-llm. Forty-eight of its paths reached another crate's item through a dynamo-llm module that only re-exports it. They now name the owning crate, so they keep resolving after the move.

- `crate::worker_type::WorkerType` is `dynamo_router_protocol::worker_type::WorkerType` (29 paths in `controller.rs`, `model.rs`, `model_manager.rs`, `readiness.rs`, `router_policy.rs`, `watcher.rs`, `worker_inventory.rs` and `worker_set.rs`).
- `crate::namespace::{NamespaceFilter, NamespacePrefixMode}` is `dynamo_runtime::namespace::{NamespaceFilter, NamespacePrefixMode}` (3 paths in `controller.rs` and `watcher.rs`).
- `crate::lora::{LoraFilter, LoraRoutingTable, LoraStateTracker, LoraAllocationConfig, LoadEstimatorConfig, LoraController, LORA_REQUIRES_REGISTRATION, lora_serving_enabled, load_estimator::LoadEstimator, state_tracker::{LoraWorkerObservation, LoraWorkerProjection}}` is `dynamo_lora::...` (15 paths in `model_manager.rs` and `watcher.rs`). `LoadEstimatorSubscription` and `controller_metrics`, which dynamo-llm defines, keep their paths.
- `crate::tokenizers::Tokenizer` is `dynamo_tokenizers::Tokenizer` (1 path in `watcher.rs`).

The re-exporting modules (`worker_type`, `namespace`, `lora`, `tokenizers`) stay for the rest of dynamo-llm and its callers. No item moves and no behaviour changes; the longer paths re-wrap a few lines.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
… subscription with the model manager that uses them

DYNO-230, layer 14 of splitting lib/llm into smaller crates. The model catalog (`discovery`) is about to become the crate `dynamo-discovery`, which may not depend on dynamo-llm. `ModelManager` starts the LoRA allocation controller and the load-estimator event subscription of every endpoint, and for both it reached into modules that stay: `lora::controller_metrics()`, which cloned eight gauges defined in `http::service::metrics`, and `lora::LoadEstimatorSubscription`. The HTTP service imports `ModelManager`, so the gauges were one half of a loop. `ModelManager` is their only writer and the only caller of the subscription, so both now live next to it as `discovery::lora`.

- New `lib/llm/src/discovery/lora.rs` holds, unchanged: the eight `LORA_*_GAUGE` statics and `register_lora_allocation_metrics` from `http/service/metrics.rs`, and `controller_metrics` from `lora.rs`. The gauges keep their names (`dynamo_frontend_lora_*`), help texts and labels, and the HTTP service registers them at the same point of its start-up, in the same order, now through `crate::discovery::lora::register_lora_allocation_metrics`.
- `lib/llm/src/lora/subscription.rs` and `lib/llm/src/lora/metrics_tests.rs` are `lib/llm/src/discovery/lora/subscription.rs` and `.../metrics_tests.rs` (`git mv`). The subscription is unchanged. The test imports the `dynamo-lora` names it used through the old module's glob.
- `lib/llm/src/lora.rs` keeps only `pub use dynamo_lora::*`, which the Python bindings and the vLLM sidecar use. No compatibility re-export of the moved items: `dynamo_llm::lora::{LoadEstimatorSubscription, controller_metrics}` and `dynamo_llm::http::service::metrics::{LORA_*_GAUGE, register_lora_allocation_metrics}` are gone; nothing outside dynamo-llm named them.
- Alternatives rejected: passing the gauges into `ModelManager` (every constructor call, in the bindings and in tests, would have to supply them or lose the series); defining them in `dynamo-lora` (that crate takes its metric handles from the serving frontend on purpose); defining the subscription in `dynamo-lora` or `dynamo-sync` (a new dependency between the two for one caller).
- Codeowners: `discovery/lora.rs` and `discovery/lora/` stay with frontend, as `lora.rs` and `lora/` were, and `discovery/lora.rs` is shared with observability, which co-owns the file the gauges came from. The two LoRA operations pages cited `lib/llm/src/lora/` for adapter download and caching, which is `lib/lora/`.

No metric name, label, help text, registration order, subject or log message changes.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…actory types from the entrypoint into discovery

DYNO-230, layer 14 of splitting lib/llm into smaller crates. The model catalog (`discovery`) is about to become the crate `dynamo-discovery`, which may not depend on dynamo-llm. `discovery::watcher` and `entrypoint` imported each other: the watcher called `entrypoint::input::build_preprocessed_routing_with_session_affinity_mode` and stored an `entrypoint::ChatEngineFactoryCallback`, while `entrypoint/input/common.rs` named `ModelManager` and `ModelWatcher`. This was the one real loop between discovery and the rest of dynamo-llm. The builder of the routed pipeline exists for the watcher, which builds one per WorkerSet; it moves next to it.

- New `lib/llm/src/discovery/pipeline.rs` holds, unchanged, what turns a model card and a client into the routed chain (OpenAIPreprocessor, Migration, Backend, EncoderRouter, PrefillRouter, RoutingHost over the push router): `PreprocessedRouting` with `build_pipeline` and `build_preprocessed_pipeline`, `build_preprocessed_routing`, `build_preprocessed_routing_with_session_affinity_mode`, the private `wait_for_min_initial_workers` and `router_client`, and the two tests of that code. From `entrypoint.rs` it takes the types the watcher stores and calls: `PrefillRoutedEngine` and `ChatEngineFactoryCallback`.
- Where the line falls: a function moves when it names a router (`RoutingHost`, `KvRouter`, `PrefillRouter`, `EncoderRouter`) or `Migration`. `entrypoint/input/common.rs` keeps the code that turns an `EngineConfig` into an engine and names no router: `prepare_engine`, `PreparedEngine`, and the local chain `build_pipeline`, `build_completions_pipeline` and `link_pipeline` (preprocessor, backend, in-process engine) that the HTTP and gRPC inputs use for an in-process token engine. `EngineConfig` and the input runners (`http`, `grpc`, `text`, `endpoint`) stay in `entrypoint`, which now imports discovery and is not imported by it.
- Inside the moved code two paths name the owning crate: `crate::lora::runtime::RuntimeLoraConfig` is `dynamo_lora::runtime::RuntimeLoraConfig`, and `crate::tokenizers::Tokenizer` is `dynamo_tokenizers::Tokenizer`. The moved test module imports `ModelManager` and `DistributedRuntime`, which it used through the old file's imports.
- No compatibility re-export: `dynamo_llm::entrypoint::{PreprocessedRouting, build_preprocessed_routing, ChatEngineFactoryCallback, PrefillRoutedEngine}` and `dynamo_llm::entrypoint::input::{PreprocessedRouting, build_preprocessed_routing}` are gone. The four names are `dynamo_llm::discovery::{...}`. Callers updated: `entrypoint.rs`, `entrypoint/input/http.rs`, and the Python bindings (`rust/llm/entrypoint.rs`, `rust/llm/routed_engine.rs`).
- Codeowners: `discovery/pipeline.rs` stays with frontend, which owns the file it came from.

No stage order, router construction, session-affinity, LoRA validation, worker wait, error text or log message changes.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
DYNO-230, layer 14 of splitting lib/llm into smaller crates. The model catalog (`discovery`) is about to move into `dynamo-discovery`, below dynamo-llm. Sixty-one of its definitions are crate-private today and are used by callers that stay in dynamo-llm: the HTTP service, the entrypoint, the frontend-load publisher and the Pool Relay. That stops working across a crate boundary, so they become `pub`.

- `discovery.rs`: the modules `image_edits`, `readiness`, `router_policy` and `worker_inventory`, and the re-export of `GenerateEngineSelection`, `SystemOneExecutionSelection` and `SystemOneSelectionError`.
- `model_manager.rs`: the methods `worker_inventory`, `worker_group_available_ids`, `get_committed_model`, `committed_model_views`, `registered_model_readiness`, `select_image_edit_engine`, `get_generate_engine_for_capability_with_routing`, `get_systemone_execution_selection` and `accepts_load_thresholds`; the field `ModelManager::router_policies`; `CommittedModelView` and its field `name`.
- `model.rs`: `GenerateEngineSelection` with its fields `engine`, `kv_cache_block_size`, `lora_name` and `tower_connector_lora_enabled`; `SystemOneExecutionSelection` with its fields `canonical_model`, `engine`, `card`, `preprocessor`, `admitted_ids` and `lifecycle_cancellation` and its method `validate_dispatch`; `SystemOneSelectionError`; `Model::decision_capabilities`.
- `watcher.rs`: `ModelWatcher::new_with_plugins`, `set_generate_engine_capabilities` and `set_systemone_enabled`, which the HTTP input calls.
- `worker_set.rs`: the fields `WorkerSet::chat_engine` and `WorkerSet::load_thresholds`. Only tests of the HTTP service set them.
- `image_edits.rs`: `SNAPSHOT_KEY`, `ImageEngineSelection` with its fields `engine` and `snapshot`, and `ImageEditSnapshot`.
- `router_policy.rs`: `RouterPolicyTarget`, and `RouterPolicyRegistry` with `register`, `get` and `list`.
- `worker_inventory.rs`: `WorkerGroupState`; `WorkerGroupObservation` with its fields `endpoint`, `model`, `model_type`, `worker_type`, `workers`, `committed`, `checksum_mismatches` and `state` and its method `metric_worker_type`; `WorkerInventory` with `publish` and `snapshot`.

The list is compiler-driven: the catalog was compiled as its own crate, and an item was widened only when a caller outside it failed to reach it (E0603, E0624, E0616, E0451, E0365) or when a widened item takes or returns it (`private_interfaces`). The remaining 85 `pub(crate)` and `pub(super)` declarations of the catalog keep their visibility. No test-only item needed an export: the tests that stay in dynamo-llm use none of the catalog's `cfg(test)` helpers. Inside dynamo-llm the change has no effect.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…uilder into dynamo-discovery

DYNO-230, layer 14 of splitting lib/llm into smaller crates. The frontend's model catalog becomes the workspace crate `dynamo-discovery` at `lib/discovery`: `dynamo_llm::discovery::X` is `dynamo_discovery::X`. The crate sits between the frontend (`dynamo-llm`: HTTP, gRPC, entrypoint), which calls into it, and `dynamo-router-host`, whose routers it builds. It does not depend on dynamo-llm.

- `lib/llm/src/discovery.rs` is the crate root. `discovery/{allocator, controller, endpoint_card, image_edits, lora, model, model_manager, pipeline, readiness, router_policy, watcher, worker_inventory, worker_set}.rs` and `discovery/lora/` are its modules under the same names; `pipeline.rs` is the routed-pipeline builder that the commits before this one moved out of the entrypoint. Three modules that the catalog builds, starts or tests with join it: `kv_router/shared_cache.rs` (the HiCache shared KV cache client, built by `ModelManager`) is `dynamo_discovery::shared_cache`, `scheduler_load.rs` (the publisher `ModelManager` hands to every KV router) is the private module `scheduler_load`, and `engines.rs` is `dynamo_discovery::engines`. dynamo-llm's `kv_router` module, which held nothing else, is gone.
- `engines.rs` (the echo engines, `StreamingEngine`, `EmbeddingEngine`, `ValidateEngine`, `EngineDispatcher` and the two adapters) moves because nine tests of the catalog register its echo engines, and a crate below dynamo-llm cannot borrow them from it. Its production users all stay in dynamo-llm and now name `dynamo_discovery::engines`: `EngineConfig::InProcessText` and the adapter in `entrypoint` (`common`, `http`, `grpc`, `endpoint`), and `make_echo_engine` in the Python bindings. It needs nothing the catalog does not already depend on.
- 21 files move with `git mv`: 19 Rust files (19,970 lines with tests), and the two agent files of the WorkerSet admission contract, now `lib/discovery/src/AGENTS.md`. Inside the moved Rust files `crate::discovery::` becomes `crate::` (9 lines), `crate::kv_router::shared_cache` becomes `crate::shared_cache`, six tests name the sample model as `../llm/tests/data/sample-models/...` (the fixture stays in `lib/llm/tests/data` with its other users, as for `dynamo-preprocessor`), and one doc link in `scheduler_load.rs` names `frontend_load` in words. The crate root gains its doc comment, the three module declarations and the `test_utils` include.
- No compatibility re-export: dynamo-llm has no `pub use dynamo_discovery`, and `dynamo_llm::{discovery, engines, kv_router, scheduler_load}` are gone. The callers name the crate: 32 files in `lib/llm` (the entrypoints, the HTTP and gRPC services, the frontend-load publisher, two Pool Relay files, five integration tests), the Python bindings (`rust/llm/{entrypoint, kv, routed_engine}.rs`), the Runtime EPP, the sidecar testkit, and the tests of `dynamo-generate-format-catalog` and `dynamo-global-frontend`.
- `dynamo-http-server` is a direct dependency of the new crate, for two types that crate defines: the request metrics handle `metrics::Metrics`, which `ModelWatcher` carries into `PreprocessedRouting::build_pipeline` for `Migration::from_mdc` (`dynamo-router-host` takes it), and `load::ModelLoadView`, for `impl From<CommittedModelView>`. No moved code serves HTTP. `dynamo-router-host` and `dynamo-sync` depend on `dynamo-http-server` for the same handle already. The smallest cut that removes the edge from all four crates is to move `metrics.rs` and `load.rs` out of `dynamo-http-server` into a crate that is not the HTTP server; that is a change to an earlier layer and is not made here.
- No cargo feature gates moved code, and no test-only item is used across the new boundary, so the crate has no `testing` feature. No integration test or bench of `lib/llm` moves: every one that names the catalog also serves it through the HTTP service or the entrypoint.
- The crate root includes `lib/runtime/src/test_utils.rs` under `cfg(test)`, as dynamo-llm and the other split crates do; the watcher and model manager tests use its isolated-process helpers.
- dynamo-llm drops `dashmap`, `dynamo-kv-hints`, `dynamo-worker-selection` and `libc`, which only the moved code used.
- Log statements without an explicit target in the moved code (74 in production code) report `dynamo_discovery::{allocator, controller, endpoint_card, model, model_manager, watcher}` where they reported `dynamo_llm::discovery::...`, `dynamo_discovery::shared_cache` for `dynamo_llm::kv_router::shared_cache`, `dynamo_discovery::scheduler_load` for `dynamo_llm::scheduler_load`, and `dynamo_discovery::lora::subscription` for `dynamo_llm::lora::subscription`. Every in-repo log filter that named `dynamo_llm::kv_router=<level>`, which since the publisher left covered only the shared cache client, now reads `dynamo_discovery::shared_cache=<level>`: two launch scripts, six recipe manifests, four e2e tests, four docs pages and the EPP note in the 1.7.0 release notes. `tests/router/test_router_e2e_with_mockers.py` set `dynamo_llm::discovery=debug` and now sets `dynamo_discovery=debug`, which shows the same lines and, at debug, those of the three modules that joined the crate.
- `lib/discovery/AGENTS.md` states what the crate owns and must not depend on; the guides of `dynamo-llm`, `dynamo-router-host`, `dynamo-sync` and the KV publisher point to the new place, as do the comments and docs that cite the moved files.
- Codeowners: every moved file keeps its owners. `lib/discovery/` is runtime's, as `lib/llm/src/discovery/` was; `src/pipeline.rs`, `src/lora.rs` and `src/lora/` stay with frontend (`lora.rs` shared with observability), `src/scheduler_load.rs` with router, `src/shared_cache.rs` with router and kv-memory. The `router` PR label covers `lib/discovery/src/shared_cache.rs`.

No discovery watch or registration order, WorkerSet admission, readiness evaluation, router construction or configuration, pipeline stage order, metric, log message, env var, subject or wire format changes.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/dyno-230-discovery branch from e070133 to d412774 Compare November 16, 2026 20:34
@ishandhanani
ishandhanani force-pushed the idhanani/dyno-228-worker-side branch from 1fe1191 to cb71696 Compare November 16, 2026 20:34

This branch had an error being deployed

1 failed deployment
external_collaborator — d412774e Deployed Nov 16, 2026 by ishandhanani via ok-to-test #739
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant