Repository navigation
refactor(discovery): move the model catalog and the routed-pipeline builder into dynamo-discovery - #227
Draft
ishandhanani wants to merge 7 commits into
Draft
refactor(discovery): move the model catalog and the routed-pipeline builder into dynamo-discovery#227ishandhanani wants to merge 7 commits into
ishandhanani wants to merge 7 commits into
Conversation
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 03:05 — with
GitHub Actions
Failure
ishandhanani
force-pushed
the
idhanani/dyno-228-worker-side
branch
from
November 16, 2026 07:07
2b7750b to
a8de954
Compare
ishandhanani
force-pushed
the
idhanani/dyno-230-discovery
branch
from
November 16, 2026 07:07
89edb62 to
476fcdf
Compare
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 07:07 — with
GitHub Actions
Failure
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 07:07 — with
GitHub Actions
Failure
This was referenced Nov 16, 2026
ishandhanani
force-pushed
the
idhanani/dyno-228-worker-side
branch
from
November 16, 2026 16:25
a8de954 to
1fe1191
Compare
ishandhanani
force-pushed
the
idhanani/dyno-230-discovery
branch
from
November 16, 2026 16:25
476fcdf to
e070133
Compare
… dynamo-runtime DYNO-230, layer 14 of splitting lib/llm into smaller crates. The model catalog (`discovery`) and the Pool Relay are about to become crates of their own, and neither may depend on dynamo-llm. Both pace their reconnect loops with `dynamo_llm::utils::retry`: the scheduler-load publisher that discovery hands to the KV router, the frontend-load publisher, and three loops of the Pool Relay. The module is generic (a capped exponential `Backoff` and a `FailureStreak` that says when to warn and when to report recovery; it names only `std::time::Duration`), so it moves below all of them. - `lib/llm/src/utils/retry.rs` is `lib/runtime/src/utils/retry.rs` (`git mv`), declared as `dynamo_runtime::utils::retry`. Its eight `pub(crate)` items become `pub`; nothing else in the file changes. Its unit test moves with it. - The five users name the new path: `scheduler_load.rs`, `frontend_load.rs`, `pool_relay/host.rs`, `pool_relay/serving_load/frontend.rs` and `pool_relay/serving_load/roster.rs`. - No compatibility re-export: `dynamo_llm::utils::retry` is gone. It was crate-private, so no other crate named it. No delay, cap, log message or log level changes. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…e serving-load flag in dynamo-sync
DYNO-230, layer 14 of splitting lib/llm into smaller crates. The scheduler-load stream has one publisher and one subscriber on opposite sides of the split: discovery hands `start_scheduler_load_publisher` to every KV router it builds, and the Pool Relay subscribes. The publisher read its enabling flag from `pool_relay`, and the Relay read the subject and the cadence from `scheduler_load`, so the two modules imported each other. The vocabulary both sides share moves below both, next to `KV_METRICS_SUBJECT` and the other worker-load messages.
- New `dynamo_sync::load::{SCHEDULER_LOAD_SUBJECT, SCHEDULER_LOAD_PUBLISH_INTERVAL, serving_load_publishing_enabled}` in `lib/sync/src/load/serving_load.rs`. The three definitions and their doc comments are the ones from `lib/llm/src/scheduler_load.rs` and `lib/llm/src/pool_relay.rs`; the subject is still `scheduler_load`, the interval one second, and the flag still reads `DYN_SERVING_LOAD_PUBLISH`.
- `scheduler_load.rs` keeps the publisher and imports the three names. The HTTP service, which checks the same flag before it starts the frontend-load publisher, and `pool_relay/{host, load, serving_load}.rs` name `dynamo_sync::load`.
- No compatibility re-export: `dynamo_llm::scheduler_load::SCHEDULER_LOAD_SUBJECT` and the crate-private `pool_relay::serving_load_publishing_enabled` are gone; no other crate named them.
- `lib/sync/AGENTS.md` lists the vocabulary under `load`.
- Codeowners: `lib/sync/src/load/serving_load.rs` is router's and global-router's, the owners of the two files its definitions came from.
No subject, cadence, env var or wire format changes. `scheduler_load` no longer imports `pool_relay`.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
DYNO-230, layer 14 of splitting lib/llm into smaller crates. The model catalog (`discovery`) is about to become the crate `dynamo-discovery`, which may not depend on dynamo-llm. Forty-eight of its paths reached another crate's item through a dynamo-llm module that only re-exports it. They now name the owning crate, so they keep resolving after the move.
- `crate::worker_type::WorkerType` is `dynamo_router_protocol::worker_type::WorkerType` (29 paths in `controller.rs`, `model.rs`, `model_manager.rs`, `readiness.rs`, `router_policy.rs`, `watcher.rs`, `worker_inventory.rs` and `worker_set.rs`).
- `crate::namespace::{NamespaceFilter, NamespacePrefixMode}` is `dynamo_runtime::namespace::{NamespaceFilter, NamespacePrefixMode}` (3 paths in `controller.rs` and `watcher.rs`).
- `crate::lora::{LoraFilter, LoraRoutingTable, LoraStateTracker, LoraAllocationConfig, LoadEstimatorConfig, LoraController, LORA_REQUIRES_REGISTRATION, lora_serving_enabled, load_estimator::LoadEstimator, state_tracker::{LoraWorkerObservation, LoraWorkerProjection}}` is `dynamo_lora::...` (15 paths in `model_manager.rs` and `watcher.rs`). `LoadEstimatorSubscription` and `controller_metrics`, which dynamo-llm defines, keep their paths.
- `crate::tokenizers::Tokenizer` is `dynamo_tokenizers::Tokenizer` (1 path in `watcher.rs`).
The re-exporting modules (`worker_type`, `namespace`, `lora`, `tokenizers`) stay for the rest of dynamo-llm and its callers. No item moves and no behaviour changes; the longer paths re-wrap a few lines.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
… subscription with the model manager that uses them
DYNO-230, layer 14 of splitting lib/llm into smaller crates. The model catalog (`discovery`) is about to become the crate `dynamo-discovery`, which may not depend on dynamo-llm. `ModelManager` starts the LoRA allocation controller and the load-estimator event subscription of every endpoint, and for both it reached into modules that stay: `lora::controller_metrics()`, which cloned eight gauges defined in `http::service::metrics`, and `lora::LoadEstimatorSubscription`. The HTTP service imports `ModelManager`, so the gauges were one half of a loop. `ModelManager` is their only writer and the only caller of the subscription, so both now live next to it as `discovery::lora`.
- New `lib/llm/src/discovery/lora.rs` holds, unchanged: the eight `LORA_*_GAUGE` statics and `register_lora_allocation_metrics` from `http/service/metrics.rs`, and `controller_metrics` from `lora.rs`. The gauges keep their names (`dynamo_frontend_lora_*`), help texts and labels, and the HTTP service registers them at the same point of its start-up, in the same order, now through `crate::discovery::lora::register_lora_allocation_metrics`.
- `lib/llm/src/lora/subscription.rs` and `lib/llm/src/lora/metrics_tests.rs` are `lib/llm/src/discovery/lora/subscription.rs` and `.../metrics_tests.rs` (`git mv`). The subscription is unchanged. The test imports the `dynamo-lora` names it used through the old module's glob.
- `lib/llm/src/lora.rs` keeps only `pub use dynamo_lora::*`, which the Python bindings and the vLLM sidecar use. No compatibility re-export of the moved items: `dynamo_llm::lora::{LoadEstimatorSubscription, controller_metrics}` and `dynamo_llm::http::service::metrics::{LORA_*_GAUGE, register_lora_allocation_metrics}` are gone; nothing outside dynamo-llm named them.
- Alternatives rejected: passing the gauges into `ModelManager` (every constructor call, in the bindings and in tests, would have to supply them or lose the series); defining them in `dynamo-lora` (that crate takes its metric handles from the serving frontend on purpose); defining the subscription in `dynamo-lora` or `dynamo-sync` (a new dependency between the two for one caller).
- Codeowners: `discovery/lora.rs` and `discovery/lora/` stay with frontend, as `lora.rs` and `lora/` were, and `discovery/lora.rs` is shared with observability, which co-owns the file the gauges came from. The two LoRA operations pages cited `lib/llm/src/lora/` for adapter download and caching, which is `lib/lora/`.
No metric name, label, help text, registration order, subject or log message changes.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…actory types from the entrypoint into discovery
DYNO-230, layer 14 of splitting lib/llm into smaller crates. The model catalog (`discovery`) is about to become the crate `dynamo-discovery`, which may not depend on dynamo-llm. `discovery::watcher` and `entrypoint` imported each other: the watcher called `entrypoint::input::build_preprocessed_routing_with_session_affinity_mode` and stored an `entrypoint::ChatEngineFactoryCallback`, while `entrypoint/input/common.rs` named `ModelManager` and `ModelWatcher`. This was the one real loop between discovery and the rest of dynamo-llm. The builder of the routed pipeline exists for the watcher, which builds one per WorkerSet; it moves next to it.
- New `lib/llm/src/discovery/pipeline.rs` holds, unchanged, what turns a model card and a client into the routed chain (OpenAIPreprocessor, Migration, Backend, EncoderRouter, PrefillRouter, RoutingHost over the push router): `PreprocessedRouting` with `build_pipeline` and `build_preprocessed_pipeline`, `build_preprocessed_routing`, `build_preprocessed_routing_with_session_affinity_mode`, the private `wait_for_min_initial_workers` and `router_client`, and the two tests of that code. From `entrypoint.rs` it takes the types the watcher stores and calls: `PrefillRoutedEngine` and `ChatEngineFactoryCallback`.
- Where the line falls: a function moves when it names a router (`RoutingHost`, `KvRouter`, `PrefillRouter`, `EncoderRouter`) or `Migration`. `entrypoint/input/common.rs` keeps the code that turns an `EngineConfig` into an engine and names no router: `prepare_engine`, `PreparedEngine`, and the local chain `build_pipeline`, `build_completions_pipeline` and `link_pipeline` (preprocessor, backend, in-process engine) that the HTTP and gRPC inputs use for an in-process token engine. `EngineConfig` and the input runners (`http`, `grpc`, `text`, `endpoint`) stay in `entrypoint`, which now imports discovery and is not imported by it.
- Inside the moved code two paths name the owning crate: `crate::lora::runtime::RuntimeLoraConfig` is `dynamo_lora::runtime::RuntimeLoraConfig`, and `crate::tokenizers::Tokenizer` is `dynamo_tokenizers::Tokenizer`. The moved test module imports `ModelManager` and `DistributedRuntime`, which it used through the old file's imports.
- No compatibility re-export: `dynamo_llm::entrypoint::{PreprocessedRouting, build_preprocessed_routing, ChatEngineFactoryCallback, PrefillRoutedEngine}` and `dynamo_llm::entrypoint::input::{PreprocessedRouting, build_preprocessed_routing}` are gone. The four names are `dynamo_llm::discovery::{...}`. Callers updated: `entrypoint.rs`, `entrypoint/input/http.rs`, and the Python bindings (`rust/llm/entrypoint.rs`, `rust/llm/routed_engine.rs`).
- Codeowners: `discovery/pipeline.rs` stays with frontend, which owns the file it came from.
No stage order, router construction, session-affinity, LoRA validation, worker wait, error text or log message changes.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
DYNO-230, layer 14 of splitting lib/llm into smaller crates. The model catalog (`discovery`) is about to move into `dynamo-discovery`, below dynamo-llm. Sixty-one of its definitions are crate-private today and are used by callers that stay in dynamo-llm: the HTTP service, the entrypoint, the frontend-load publisher and the Pool Relay. That stops working across a crate boundary, so they become `pub`. - `discovery.rs`: the modules `image_edits`, `readiness`, `router_policy` and `worker_inventory`, and the re-export of `GenerateEngineSelection`, `SystemOneExecutionSelection` and `SystemOneSelectionError`. - `model_manager.rs`: the methods `worker_inventory`, `worker_group_available_ids`, `get_committed_model`, `committed_model_views`, `registered_model_readiness`, `select_image_edit_engine`, `get_generate_engine_for_capability_with_routing`, `get_systemone_execution_selection` and `accepts_load_thresholds`; the field `ModelManager::router_policies`; `CommittedModelView` and its field `name`. - `model.rs`: `GenerateEngineSelection` with its fields `engine`, `kv_cache_block_size`, `lora_name` and `tower_connector_lora_enabled`; `SystemOneExecutionSelection` with its fields `canonical_model`, `engine`, `card`, `preprocessor`, `admitted_ids` and `lifecycle_cancellation` and its method `validate_dispatch`; `SystemOneSelectionError`; `Model::decision_capabilities`. - `watcher.rs`: `ModelWatcher::new_with_plugins`, `set_generate_engine_capabilities` and `set_systemone_enabled`, which the HTTP input calls. - `worker_set.rs`: the fields `WorkerSet::chat_engine` and `WorkerSet::load_thresholds`. Only tests of the HTTP service set them. - `image_edits.rs`: `SNAPSHOT_KEY`, `ImageEngineSelection` with its fields `engine` and `snapshot`, and `ImageEditSnapshot`. - `router_policy.rs`: `RouterPolicyTarget`, and `RouterPolicyRegistry` with `register`, `get` and `list`. - `worker_inventory.rs`: `WorkerGroupState`; `WorkerGroupObservation` with its fields `endpoint`, `model`, `model_type`, `worker_type`, `workers`, `committed`, `checksum_mismatches` and `state` and its method `metric_worker_type`; `WorkerInventory` with `publish` and `snapshot`. The list is compiler-driven: the catalog was compiled as its own crate, and an item was widened only when a caller outside it failed to reach it (E0603, E0624, E0616, E0451, E0365) or when a widened item takes or returns it (`private_interfaces`). The remaining 85 `pub(crate)` and `pub(super)` declarations of the catalog keep their visibility. No test-only item needed an export: the tests that stay in dynamo-llm use none of the catalog's `cfg(test)` helpers. Inside dynamo-llm the change has no effect. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…uilder into dynamo-discovery
DYNO-230, layer 14 of splitting lib/llm into smaller crates. The frontend's model catalog becomes the workspace crate `dynamo-discovery` at `lib/discovery`: `dynamo_llm::discovery::X` is `dynamo_discovery::X`. The crate sits between the frontend (`dynamo-llm`: HTTP, gRPC, entrypoint), which calls into it, and `dynamo-router-host`, whose routers it builds. It does not depend on dynamo-llm.
- `lib/llm/src/discovery.rs` is the crate root. `discovery/{allocator, controller, endpoint_card, image_edits, lora, model, model_manager, pipeline, readiness, router_policy, watcher, worker_inventory, worker_set}.rs` and `discovery/lora/` are its modules under the same names; `pipeline.rs` is the routed-pipeline builder that the commits before this one moved out of the entrypoint. Three modules that the catalog builds, starts or tests with join it: `kv_router/shared_cache.rs` (the HiCache shared KV cache client, built by `ModelManager`) is `dynamo_discovery::shared_cache`, `scheduler_load.rs` (the publisher `ModelManager` hands to every KV router) is the private module `scheduler_load`, and `engines.rs` is `dynamo_discovery::engines`. dynamo-llm's `kv_router` module, which held nothing else, is gone.
- `engines.rs` (the echo engines, `StreamingEngine`, `EmbeddingEngine`, `ValidateEngine`, `EngineDispatcher` and the two adapters) moves because nine tests of the catalog register its echo engines, and a crate below dynamo-llm cannot borrow them from it. Its production users all stay in dynamo-llm and now name `dynamo_discovery::engines`: `EngineConfig::InProcessText` and the adapter in `entrypoint` (`common`, `http`, `grpc`, `endpoint`), and `make_echo_engine` in the Python bindings. It needs nothing the catalog does not already depend on.
- 21 files move with `git mv`: 19 Rust files (19,970 lines with tests), and the two agent files of the WorkerSet admission contract, now `lib/discovery/src/AGENTS.md`. Inside the moved Rust files `crate::discovery::` becomes `crate::` (9 lines), `crate::kv_router::shared_cache` becomes `crate::shared_cache`, six tests name the sample model as `../llm/tests/data/sample-models/...` (the fixture stays in `lib/llm/tests/data` with its other users, as for `dynamo-preprocessor`), and one doc link in `scheduler_load.rs` names `frontend_load` in words. The crate root gains its doc comment, the three module declarations and the `test_utils` include.
- No compatibility re-export: dynamo-llm has no `pub use dynamo_discovery`, and `dynamo_llm::{discovery, engines, kv_router, scheduler_load}` are gone. The callers name the crate: 32 files in `lib/llm` (the entrypoints, the HTTP and gRPC services, the frontend-load publisher, two Pool Relay files, five integration tests), the Python bindings (`rust/llm/{entrypoint, kv, routed_engine}.rs`), the Runtime EPP, the sidecar testkit, and the tests of `dynamo-generate-format-catalog` and `dynamo-global-frontend`.
- `dynamo-http-server` is a direct dependency of the new crate, for two types that crate defines: the request metrics handle `metrics::Metrics`, which `ModelWatcher` carries into `PreprocessedRouting::build_pipeline` for `Migration::from_mdc` (`dynamo-router-host` takes it), and `load::ModelLoadView`, for `impl From<CommittedModelView>`. No moved code serves HTTP. `dynamo-router-host` and `dynamo-sync` depend on `dynamo-http-server` for the same handle already. The smallest cut that removes the edge from all four crates is to move `metrics.rs` and `load.rs` out of `dynamo-http-server` into a crate that is not the HTTP server; that is a change to an earlier layer and is not made here.
- No cargo feature gates moved code, and no test-only item is used across the new boundary, so the crate has no `testing` feature. No integration test or bench of `lib/llm` moves: every one that names the catalog also serves it through the HTTP service or the entrypoint.
- The crate root includes `lib/runtime/src/test_utils.rs` under `cfg(test)`, as dynamo-llm and the other split crates do; the watcher and model manager tests use its isolated-process helpers.
- dynamo-llm drops `dashmap`, `dynamo-kv-hints`, `dynamo-worker-selection` and `libc`, which only the moved code used.
- Log statements without an explicit target in the moved code (74 in production code) report `dynamo_discovery::{allocator, controller, endpoint_card, model, model_manager, watcher}` where they reported `dynamo_llm::discovery::...`, `dynamo_discovery::shared_cache` for `dynamo_llm::kv_router::shared_cache`, `dynamo_discovery::scheduler_load` for `dynamo_llm::scheduler_load`, and `dynamo_discovery::lora::subscription` for `dynamo_llm::lora::subscription`. Every in-repo log filter that named `dynamo_llm::kv_router=<level>`, which since the publisher left covered only the shared cache client, now reads `dynamo_discovery::shared_cache=<level>`: two launch scripts, six recipe manifests, four e2e tests, four docs pages and the EPP note in the 1.7.0 release notes. `tests/router/test_router_e2e_with_mockers.py` set `dynamo_llm::discovery=debug` and now sets `dynamo_discovery=debug`, which shows the same lines and, at debug, those of the three modules that joined the crate.
- `lib/discovery/AGENTS.md` states what the crate owns and must not depend on; the guides of `dynamo-llm`, `dynamo-router-host`, `dynamo-sync` and the KV publisher point to the new place, as do the comments and docs that cite the moved files.
- Codeowners: every moved file keeps its owners. `lib/discovery/` is runtime's, as `lib/llm/src/discovery/` was; `src/pipeline.rs`, `src/lora.rs` and `src/lora/` stay with frontend (`lora.rs` shared with observability), `src/scheduler_load.rs` with router, `src/shared_cache.rs` with router and kv-memory. The `router` PR label covers `lib/discovery/src/shared_cache.rs`.
No discovery watch or registration order, WorkerSet admission, readiness evaluation, router construction or configuration, pipeline stage order, metric, log message, env var, subject or wire format changes.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
force-pushed
the
idhanani/dyno-230-discovery
branch
from
November 16, 2026 20:34
e070133 to
d412774
Compare
ishandhanani
force-pushed
the
idhanani/dyno-228-worker-side
branch
from
November 16, 2026 20:34
1fe1191 to
cb71696
Compare
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 20:34 — with
GitHub Actions
Failure
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 20:34 — with
GitHub Actions
Failure
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Layer 14 of the lib/llm split stack (DYNO-230). Base: #226 (worker-side code into
dynamo-backend-common), so this PR shows only this layer. It moves the frontend's model catalog and the routed-pipeline builder out ofdynamo-llminto the new cratedynamo-discovery(lib/discovery). The crate sits between the frontend (dynamo-llm: HTTP, gRPC, entrypoint), which calls into it, anddynamo-router-host, whose routers it builds. It does not depend ondynamo-llm.What moved
dynamo_llm::discovery::Xisdynamo_discovery::X:ModelManager,ModelWatcher,ModelDiscoveryController,Model,WorkerSet, the readiness evaluator, the image-edit and router-policy registries and the worker inventory (lib/llm/src/discovery.rsis the crate root,discovery/*.rsits modules under the same names).PreprocessedRouting,build_preprocessed_routingand the typesChatEngineFactoryCallbackandPrefillRoutedEnginearedynamo_discovery::{...}(src/pipeline.rs). They weredynamo_llm::entrypoint::{...}.dynamo_llm::kv_router::shared_cache(the HiCache shared KV cache client, built byModelManager) isdynamo_discovery::shared_cache.dynamo-llm'skv_routermodule is gone.lib/llm/src/scheduler_load.rs(the scheduler-load publisherModelManagerhands to every KV router) is the private moduledynamo_discovery::scheduler_load.dynamo_llm::engines(echo engines,StreamingEngine,EmbeddingEngine, the adapters) isdynamo_discovery::engines.register_lora_allocation_metrics,controller_metricsandLoadEstimatorSubscriptionaredynamo_discovery::lora::{...}.git mvin the move commit: 19 Rust files (19,970 lines with tests) and the two agent files of the WorkerSet admission contract, nowlib/discovery/src/AGENTS.md.scripts/router_move_diff.pyover the Rust files of the move commit: 16,534 normalized lines before, 16,535 after, 16,523 common, 12 new, 11 dropped. The 11 dropped lines are the six sample-model paths and five inlinecrate::discovery::paths, each of which has its rewritten counterpart among the new ones; the twelfth new line is thetest_utilsinclude.lib/llm/srcgoes from 138,195 to 118,184 lines of Rust (240 to 221 files). Whatdynamo-llmstill declares:block_manager40.5k (behind its feature),http35.3k,pool_relay27.8k,mocker6.4k,grpc3.1k,entrypoint2.0k,tokens.rs1.7k,generate_format490,cuda.rs207,frontend_load.rs162,common55,utils46 (lora_name_to_id), and the re-export moduleslora.rs,namespace.rs,worker_type.rsandtokenizers.dynamo-llmhas nopub use dynamo_discovery, and the old paths are deleted.Commits
Five in-place cuts, one widening commit and one move commit. Each builds and passes on its own.
a07644101eutils/retry.rs(Backoff,FailureStreak) moves todynamo_runtime::utils::retry294da232b6SCHEDULER_LOAD_SUBJECT,SCHEDULER_LOAD_PUBLISH_INTERVALandserving_load_publishing_enabledare defined indynamo_sync::load18c9fdf2eedynamo-llmre-export modulef8a2c802aacontroller_metricsand the load-estimator subscription move next toModelManagerasdiscovery::lora3af919b930entrypointtodiscovery::pipeline207aac425cpub89edb624d6git mv, path rewrites, manifests, docs, codeownersEdges
discovery::watchercallsentrypoint::input::build_preprocessed_routing_with_session_affinity_modeand stores anentrypoint::ChatEngineFactoryCallback;entrypoint/input/common.rsnamesModelManagerandModelWatcher. The one real loop3af919b930: what names a router orMigrationmoves todiscovery/pipeline.rs(PreprocessedRoutingwithbuild_pipelineandbuild_preprocessed_pipeline,build_preprocessed_routing*,wait_for_min_initial_workers,router_client, two tests, the two callback types).prepare_engine,PreparedEngine, the local chainbuild_pipeline,build_completions_pipeline,link_pipeline,EngineConfigand the input runners stay inentrypoint, which now imports discovery one wayentrypointimplements and the watcher calls: a new trait and an injection point for code that has one callerModelManager::ensure_lora_controllercallslora::controller_metrics(), which clones eight gauges defined inhttp::service::metrics; the HTTP service importsModelManagerf8a2c802aa: the gauges, their registration function andcontroller_metricsmove next to their only writer asdiscovery::lora. Names, labels, help texts and the registration point and order are unchangedModelManager(every constructor call would have to supply them or lose the series); defining them indynamo-lora(it takes its handles from the frontend on purpose)ModelManageruseslora::LoadEstimatorSubscription(lora/subscription.rs)f8a2c802aa:discovery/lora/subscription.rs, with its only callerdynamo-loraordynamo-sync: a new dependency between the two for one callerModelManagerpassesscheduler_load::start_scheduler_load_publishertoKvRouter; the publisher readspool_relay::serving_load_publishing_enabled(); the Pool Relay reads the subject and cadence fromscheduler_load294da232b6: the subject, the cadence and the flag aredynamo_sync::load::{...}, next toKV_METRICS_SUBJECT. The publisher moves with the catalog in the move commitdynamo-router-host, which owns theSelectionServiceit reads: that undoes the injection layer 11 made and is a refactorscheduler_load.rs,frontend_load.rsand three Pool Relay loops useutils::retrya07644101e:dynamo_runtime::utils::retry. It is generic (names onlyDuration)ModelManagerbuildskv_router::shared_cache::HicacheSharedKvCachedynamo-llmmoduledynamo-llmbehind theSharedKvCachetrait:ModelManagerconstructs itengines::EchoBidirectionalEngineorStreamingEngineAdapter::new(make_echo_engine()). No production code of the catalog usesenginesengines.rsmoves with the catalog. Production users, all indynamo-llm:EngineConfig::InProcessText, the adapter inentrypoint/input/{common, http, grpc, endpoint}.rs, andmake_echo_enginein the Python bindingsdynamo-llm(a cycle, andcargo treewould show it);dynamo-backend-common(a new dev edge to a worker-side crate);dynamo-frontend-protocols(hoists stage code into the protocol crate); stub engines in the tests (a copy)WorkerType,NamespaceFilter,NamespacePrefixMode,Tokenizerand elevendynamo-loraitems throughdynamo-llmre-export modules18c9fdf2ee: they name the owner (dynamo-router-protocol29,dynamo-lora15,dynamo-runtime3,dynamo-tokenizers1).namespace.rs,worker_type.rs,lora.rsandtokenizersstay indynamo-llmfor its other callersModelWatchercarriesArc<dynamo_http_server::metrics::Metrics>intoPreprocessedRouting::build_pipelineforMigration::from_mdc;impl From<CommittedModelView> for dynamo_http_server::load::ModelLoadViewdynamo-http-serveris a direct dependency. No moved code serves HTTP;dynamo-router-hostanddynamo-synchave the same edge for the same handle. Decision to confirm (below)ModelWatcherand the builder generic over the metrics handle: a refactor that hides the edge without removing it from the treecrate::test_utils::{run_isolated, isolated_command, assert_isolated_success, tracing_interest_guard}lib/runtime/src/test_utils.rsundercfg(test), as the other split crates dotests/data/sample-models/mock-llama-3.1-8b-instructrelative to the crate directory../llm/tests/data/sample-models/...; the fixture stays with its other users, as fordynamo-preprocessorWhat was widened
61 definitions, compiler-driven (E0603, E0624, E0616, E0451, E0365,
private_interfaces): four modules (image_edits,readiness,router_policy,worker_inventory) and one re-export indiscovery.rs; nine methods,router_policies,CommittedModelViewand itsnameinmodel_manager.rs;GenerateEngineSelection,SystemOneExecutionSelection,SystemOneSelectionError, ten of their fields,validate_dispatchandModel::decision_capabilitiesinmodel.rs; threeModelWatchermethods;WorkerSet::{chat_engine, load_thresholds};SNAPSHOT_KEY,ImageEngineSelectionwith two fields andImageEditSnapshot;RouterPolicyTargetandRouterPolicyRegistrywith three methods;WorkerGroupState,WorkerGroupObservationwith eight fields and one method,WorkerInventorywith two methods. 85pub(crate)andpub(super)declarations keep their visibility. No test-only item crosses the boundary, so the crate has notestingfeature. No trait was added.Callers
lib/llm, 32 Rust files namedynamo_discovery: the entrypoints, the HTTP and gRPC services,frontend_load.rs, two Pool Relay files and five integration tests.lib.rsloses four module declarations.lib/llm: the Python bindings (rust/llm/{entrypoint, kv, routed_engine}.rs), the Runtime EPP (deploy/inference-gateway/ext-proc/src/epp.rs), the sidecar testkit (two test files), and the tests ofdynamo-generate-format-cataloganddynamo-global-frontend. Their manifests gain the dependency (dev-dependency for the last three).dynamo-llmdropsdashmap,dynamo-kv-hints,dynamo-worker-selectionandlibc, and thetestingfeature ofdynamo-router-hostin its dev-dependencies.Log targets
dynamo_llm::discovery::{allocator 2, controller 16, endpoint_card 2, model 3, model_manager 11, watcher 15}aredynamo_discovery::{...};dynamo_llm::kv_router::shared_cache(18) isdynamo_discovery::shared_cache;dynamo_llm::scheduler_load(3) isdynamo_discovery::scheduler_load;dynamo_llm::lora::subscription(4) isdynamo_discovery::lora::subscription. Targets are not pinned.dynamo_llm::kv_router=<level>isdynamo_discovery::shared_cache=<level>in 17 places:examples/backends/{sglang,vllm}/launch/agg_multimodal_router.sh, the sixrecipes/nemotron-3-ultra/vllm/disagg-*/deploy.yaml,tests/mm_router/test_router_rust_mm_{frontend_decode,glm5_next,router}_e2e.py,tests/router/test_router_e2e_with_mockers.py, the docs pagesrecipes/cli-templates/{sglang,vllm}.mdx,multimodal-kv-routing.mdandexact-video-kv-routing.md, and the EPP note in the 1.7.0 release notes.tests/router/test_router_e2e_with_mockers.pysetDYN_LOG=info,dynamo_llm::discovery=debugand now setsinfo,dynamo_discovery=debug: the same lines, plus the debug lines ofshared_cache,scheduler_loadandlora::subscription, which are in the crate now.Plumbing
lib/discovery/AGENTS.mdstates what the crate owns, depends on and must not depend on. The guides ofdynamo-llm,dynamo-router-host,dynamo-syncand the KV publisher, and the comments and docs that cite the moved files, point to the new place.lib/discovery/is runtime's;src/pipeline.rs,src/lora.rs(shared with observability, which co-owned the gauges' old file) andsrc/lora/are frontend's;src/scheduler_load.rsis router's;src/shared_cache.rsis router's and kv-memory's;lib/sync/src/load/serving_load.rsis router's and global-router's. TherouterPR label coverslib/discovery/src/shared_cache.rs. The CI path filters matchlib/**and need no change.lib/bindings/pythonandlib/bindings/kvbmones change (no new third-party package).lib/runtime/examplesdoes not change./tmp/llm-split/moves/dyno-230-widen.pyreproduces the widening commit from its parent, and/tmp/llm-split/moves/dyno-230.pyreproduces the move commit from the widening commit (same tree hash).Behaviour
No discovery watch or registration order, WorkerSet admission, readiness evaluation, router construction or configuration, pipeline stage order, metric name or label, log message, env var, subject or wire format changes.
Decisions to confirm
engines.rslives indynamo-discovery. It is there because the catalog's tests need it below them; its production users are in the entrypoint.dynamo-discoverydepends ondynamo-http-serverdirectly, formetrics::Metricsandload::ModelLoadView. The cut that removes the edge fromdynamo-sync,dynamo-router-hostand this crate at once is to movelib/http-server/src/{metrics, load}.rsinto a crate that is not the HTTP server.dynamo_sync::load.dynamo_discovery::loraand registered by the HTTP service.readiness.rsmoves with the catalog here; the Pool Relay layer moves it below both.Validation
Baseline is the parent
2b7750bb15, head is89edb624d6; the same commands ran at both on the Modal cargo runner.cargo test -p dynamo-llm -p dynamo-backend-common -p dynamo-router-host -p dynamo-sync -p dynamo-global-frontend --no-fail-fast(plus-p dynamo-discoveryat the head): baseline 49 binaries, 2,618 passed, 0 failed, 22 ignored; head 51 binaries, 2,621 passed, 0 failed, 22 ignored.-- --list): 2,640 at baseline, 2,643 at head. 2,402 keep their binary and path. 237 are indynamo_discoverywith the same leaf name:discovery::{model_manager 58, model 57, watcher 37, controller 26, worker_set 11, readiness 6, allocator 2, image_edits 2}lose thediscovery::prefix,kv_router::shared_cache(30) isshared_cache,entrypoint::input::common(2) ispipeline, andengines(2),scheduler_load(2),lora::subscription(1) andlora::metrics_tests(1) keep their path. One,utils::retry::tests::failure_streak_starts_once_and_reports_its_length_on_recovery, is indynamo-runtimeand passes there. 0 missing. 4 added: the self-tests oflib/runtime/src/test_utils.rs, which the new crate root includes.cargo test -p dynamo-sidecar-testkitwith the sidecars built: 113 passed at both. The tests of the four sidecars,dynamo-sidecar-common, the three mocker servers anddynamo-mocker-backendpass at both.cargo test -p dynamo-discoveryalone 241 passed;cargo test -p dynamo-runtime --lib914 passed, 2 ignored;dynamo-ext-proc193 passed;dynamo-generate-format-catalog12 passed;dynamo-loraanddynamo-model-cardpass.cargo check --workspace --all-targets,cargo clippy --no-deps -p dynamo-llm -p dynamo-sync -p dynamo-runtime --all-targets -- -D warnings,cargo check --lockedinlib/bindings/python,cargo fmt --check, the four lock files, the regeneratedCODEOWNERS, andcargo test -p dynamo-llm -p dynamo-sync --lib(1,271 and 191 passed, 0 failed) pass.cargo check --workspace --all-targets,cargo clippy --no-deps --workspace --all-targets -- -D warnings,cargo clippy --no-deps -p dynamo-discovery --all-targets -- -D warnings,cargo fmt --all -- --check: pass.cargo check --lockedinlib/bindings/python(also clippy-D warningsand fmt),lib/bindings/c,lib/bindings/kvbm, andlib/runtime/examples(--all-targets): pass.cargo metadata --lockedpasses for the four lock files.cargo check -p dynamo-llm --all-targetswithckf-diagnostics,mm-routingandtesting-etcd, and--features kv-router-stress --bench kv_router_bench: pass. The call inlib/llm/tests/http_metrics.rsthat stopped--features integrationis fixed in refactor(llm): serve HTTP through dynamo-http-server #209; that check passes there, at refactor(llm): move the request path into dynamo-router-host #225 and at the top of the stack (refactor(pool-relay): move the Pool Relay into dynamo-pool-relay #228).cargo macheteat the root, inlib/bindings/pythonand inlib/runtime/examples: clean at the parent and at the head (the root afterdynamo-llmdropped the four dependencies).uvx pre-commit run --from-ref 2b7750bb15 --to-ref HEAD: pass.build_codeowners.py --strict: pass;CODEOWNERSis regenerated.cargo run -p dynamo-llm --bin generate-frontend-openapiwrites the same file, SHA-256706f15d951891703fa911b9c659ac54992fa7144365ce4cd6b2a179a7630ede5.cargo tree -p dynamo-discovery | grep -c dynamo-llmprints 0. Direct Dynamo dependencies:dynamo-router-host,dynamo-sync,dynamo-preprocessor,dynamo-frontend-protocols,dynamo-model-card,dynamo-lora,dynamo-runtime,dynamo-http-server,dynamo-kv-router,dynamo-scheduler,dynamo-worker-selection,dynamo-slot-tracker,dynamo-kv-indexer,dynamo-kv-hashing,dynamo-kv-hints,dynamo-kv-events,dynamo-router-protocol, and the publisheddynamo-protocols,dynamo-decisions,dynamo-renderer,dynamo-tokenizers; for tests alsodynamo-router-policies. Its normal dependency tree has 681 crates;dynamo-llm's has 729.DYN_LOGfilters changed (they need GPUs or a cluster);media-ffmpegbuilds (no FFmpeg on the runner).Closes DYNO-230