Skip to content

refactor(llm): move router state sync into dynamo-sync - #223

Draft
ishandhanani wants to merge 2 commits into
idhanani/dyno-222-discovery-splitfrom
idhanani/dyno-226-sync
Draft

ishandhanani wants to merge 2 commits into
idhanani/dyno-222-discovery-splitfrom
idhanani/dyno-226-sync

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Summary

Layer 10 of the lib/llm split stack (DYNO-226). Base: #222 (the in-place regroup under dynamo_llm::sync), so this PR shows only this layer. It moves that module out of dynamo-llm into a new workspace crate, dynamo-sync, at lib/sync: the code that keeps a router's view of its workers up to date. Target layering, top to bottom: discovery (model catalog), the router host (request path), dynamo-sync. The first two stay in dynamo-llm.

What moved

  • lib/llm/src/sync.rs is the crate root and lib/llm/src/sync/ its modules: dynamo_llm::sync::X is now dynamo_sync::X.
  • indexer (KV index ingest, recovery, remote indexer), sequence (active-sequence replica sync), load (ActiveLoad and its subscription, KvWorkerMonitor, LoadThresholdHandle, RoutingLoadContext), membership (KV event-source membership and the state-agent vocabulary), runtime_configs (RuntimeConfigWatch), metrics (worker load, worker status, KV and active-sequence ingress, remote indexer), direct_zmq_fan_in and direct_zmq_sub_pool.
  • 36 files move with git mv (24,578 lines of Rust with tests); 13 are byte-identical. Inside them only crate::sync:: becomes crate::, plus the testing gates below. lib/llm/src goes from 216,237 to 191,664 lines of Rust (324 to 288 files).
  • No compatibility re-exports: dynamo-llm has no pub use dynamo_sync.
  • No integration test or bench of dynamo-llm moves: none of them names a sync item. kv_router_bench drives the router and stays.

Commits

One in-place commit inside dynamo-llm, which builds and passes on its own, then one move commit.

Edge Resolution Alternative rejected
229 pub(crate) and pub(super) declarations in sync (re-exports included), some used by the router host, discovery, the worker-side publisher, the Pool Relay, session affinity and the LoRA load estimator 67682046b7: 76 definitions and 7 struct fields become pub, with the re-exports that name them. Exactly what a caller outside the crate failed to reach when sync was compiled as its own crate, with and without ckf-diagnostics, plus the types in their signatures. 130 declarations stay crate-private. The list is in the commit message and below Re-exporting through dynamo-llm; widening whole modules
Seven test-only items that dynamo-llm's tests use (cfg(test) does not cross a crate boundary): indexer::test_util, indexer::{WorkerQueryClient, WorkerQueryTransport}, WorkerQueryClient::new_target_for_test, RuntimeIngress::has_subscription, KvStateAgentStatusCell::snapshot, KvSourceMembershipWatch::shares_coordinator_with, metrics::test_fixtures Exported behind a testing feature of the new crate, which dynamo-llm enables as a dev-dependency (the pattern of dynamo-preprocessor and dynamo-frontend-protocols) Copying the fixtures; making them unconditional
The Cargo feature ckf-diagnostics gates moved code (5 cfg sites: the worker query client's health snapshot) The new crate has the feature and dynamo-llm forwards its own, so dynamo-llm/ckf-diagnostics keeps working unchanged Leaving the feature only in dynamo-llm, which would leave the snapshot uncompiled
The index metrics use items of dynamo-kv-indexer behind its export feature dynamo-sync enables dynamo-kv-indexer/export, as dynamo-llm does Relying on feature unification through dynamo-llm
Test-only: one test calls crate::test_utils::tracing_interest_guard The crate root includes lib/runtime/src/test_utils.rs under cfg(test), as dynamo-llm and the other split crates do, with allow(dead_code) because only the tracing guard is used here Exporting the helper from dynamo-runtime, which changes a lower layer
Orphan rule Nothing to do: no impl for a sync type lives outside the moved module, and the moved code implements nothing for a type that stays n/a

What became public (67682046b7)

  • Modules: direct_zmq_fan_in, direct_zmq_sub_pool, load::metrics_subscriber, load::routing_load, and the test modules indexer::test_util and metrics::test_fixtures.
  • direct_zmq_fan_in: ContinuityMode, FanInEvent, FanInObservation, start_direct_zmq_fan_in.
  • direct_zmq_sub_pool: KV_ZMQ_RCVHWM, RawZmqFrames, DirectZmqSubPool with new_raw, register_raw, shutdown, DirectZmqSubRegistration with its receiver and disconnected fields, and DirectZmqSubItem, its default type parameter.
  • load: KvMetricsSubscriber with for_endpoint_id and next; RouterLoadSource::{metric_label, from_worker_role_or_metric, monitors_sequence_load}; SchedulerLoadSender::{disabled, route_busy_thresholds}; LoadThresholdHandle::{disabled, is_enabled}.
  • membership: KvSourceMembershipCoordinator with start and subscribe; KvSourceMembershipView::matches_binding_inputs; KvSourceMembershipWatch::shares_coordinator_with.
  • sequence: host_replica_channels, ReplicaIngress with start, ActiveSequenceEventPublisher::channel, RuntimeSequenceSubscriber::for_endpoint.
  • metrics: RouterWorkerStatusMetrics, RouterWorkerRegistration, RegistrationAvailabilityProvider, test_fixtures::gather_pef.
  • indexer: RuntimeIngress with start, indexer, set_task_guard, has_subscription; RuntimeIngressArgs; KvEventSourceRequirement with derive and should_subscribe; the re-exports of ApproximateRequestLease and TieredMatchDetails; test_util::store_event.
  • indexer recovery: DEFAULT_RECOVERY_ATTEMPT_TIMEOUT, RecoveryResetReason, RecoveryTarget, RecoverySupervisor with client and shutdown, start_target_subscriber, TargetFaultDisposition, WorkerQueryHealthSnapshot with its five fields, KvStateAgentStatusCell with new, load, store, store_outbound_cursor, snapshot, start_worker_kv_query_endpoint, start_worker_kv_query_endpoint_with_status, WorkerQueryClient with sync_membership, handle_target_fault, reject_source, handle_live_batch, health_snapshot, new_target_for_test, the WorkerQueryTransport trait, and IndexerRecoveryTarget, the default type parameter of WorkerQueryClient.

Callers

  • 42 source files of lib/llm/src name dynamo_sync: the router host (kv_router.rs, kv_router/{embedded,managed,metrics,request_lease,routing_host,shared_cache}.rs, routing_host/, prefill_router/, publisher/), discovery, the entrypoint, the HTTP service, the Pool Relay, session affinity, LoRA and the mocker.
  • Outside lib/llm, six Rust files do: lib/bindings/python/rust/{lib.rs,llm/kv.rs,llm/kv_state_agent.rs}, lib/sidecar/sglang/src/headless.rs, lib/sidecar/trtllm/src/headless.rs and the test lib/sidecar/testkit/tests/router_sidecar_mocker_integration.rs. Their manifests gain the dependency (a dev-dependency for the testkit). Two doc comments in lib/router and the lib/router/kv-router README name the new path. lib/backend-common, lib/mocker/servers/* and the EPP name none of the moved items.
  • dynamo-llm drops one dependency, dynamo-tokens, which only the remote indexer used.

Crate plumbing

  • Workspace member and [workspace.dependencies] entry; SPDX headers; lib/sync/AGENTS.md (what the crate owns, what it must not depend on, how to check it) with the sibling CLAUDE.md.
  • Dependency graph: cargo tree -p dynamo-sync | grep -c dynamo-llm prints 0. Direct Dynamo dependencies: dynamo-runtime, dynamo-kv-indexer, dynamo-kv-events, dynamo-kv-hashing, dynamo-slot-tracker, dynamo-scheduler, dynamo-router-protocol, dynamo-kv-router, dynamo-model-card, dynamo-frontend-protocols, dynamo-http-server, dynamo-tokens, and dynamo-kv-hints for tests. None is above the crate; dynamo-preprocessor and dynamo-telemetry are not in its graph.
  • Codeowners: all 36 moved files keep their owners (router for the index, sequence, load subscription, routing load and metrics files; runtime for the rest; the two existing epp co-ownerships). The new manifest and AGENTS.md go to runtime, the owner of lib/llm/. The router PR label covers lib/sync/**. The CI path filters match lib/** and need no change.
  • kv_router/AGENTS.md, kv_router/README.md, the router testing guide and the KV event replay comparison cite the new paths.

Behaviour

  • No metric name or label, log message text, env var, event-plane subject, wire format, timing or ordering changes. Metrics registered by the moved code keep their names and registration points.
  • One difference that follows from any move between crates: the 166 log statements in the moved code without an explicit target now report dynamo_sync::indexer::* (105), dynamo_sync::load::* (23), dynamo_sync::direct_zmq_fan_in (11), dynamo_sync::direct_zmq_sub_pool (11), dynamo_sync::membership::kv_source_watch (6), dynamo_sync::sequence and dynamo_sync::sequence::direct_zmq (6) and dynamo_sync::runtime_configs (4), where they reported dynamo_llm::sync::.... No target is pinned.
  • DYN_LOG examples keep selecting the same lines: every filter that refactor(llm): group index, slot, load and membership sync under one module #222 pointed at dynamo_llm::sync[::...] now names dynamo_sync[::...] (the two CLI template pages, the two multimodal KV routing pages, the vLLM and SGLang agg_multimodal_router.sh launchers, the three tests/mm_router end-to-end tests, tests/router/test_router_e2e_with_mockers.py, and the EPP note on the deprecations page). Checked live on the top of the stack against main: the disaggregated startup and overload tests capture 50 worker-monitor debug lines (51 on main) and 4 recovery query-endpoint debug lines (4 on main).

Validation

Cargo commands ran on a remote builder with the repository toolchain (1.96.1).

  • Baseline at 46d311b, cargo test -p dynamo-llm --no-fail-fast: 36 binaries, 2396 passed, 0 failed, 17 ignored.
  • Head, same command plus -p dynamo-sync: 38 binaries, 2400 passed, 0 failed, 17 ignored. The two extra binaries are the new crate's unit and doc tests. dynamo-llm unit tests go from 2041 passed and 4 ignored to 1854 and 4; the new crate has 191 (the 187 that moved and the 4 tests of the included test_utils.rs, which also run in dynamo-llm).
  • Test accounting from cargo test -- --list: all 2413 baseline names exist at head, none is missing. 2226 keep binary and path, 187 moved from dynamo-llm's unit tests to the same leaf name in dynamo-sync's (sync::X is X). The 4 added names are the test_utils tests in the new crate.
  • After the in-place commit: cargo check -p dynamo-llm --all-targets, the same with --features ckf-diagnostics, and cargo clippy --no-deps -p dynamo-llm --all-targets -- -D warnings pass; cargo test -p dynamo-llm --lib -- sync:: passes 191.
  • Head: cargo check --workspace --all-targets, cargo clippy --no-deps --workspace --all-targets -- -D warnings, cargo fmt --all -- --check (root and lib/bindings/python), and cargo check --locked in lib/bindings/python, lib/bindings/kvbm and lib/runtime/examples pass.
  • The new crate alone: cargo check -p dynamo-sync passes without features and with testing; cargo check -p dynamo-sync --all-targets --features ckf-diagnostics and clippy with ckf-diagnostics,testing and -D warnings pass; cargo test -p dynamo-sync passes 191, also with --features ckf-diagnostics.
  • ckf-diagnostics: cargo check -p dynamo-llm --features ckf-diagnostics --all-targets and clippy with -D warnings pass.
  • Router and mocker path without etcd or NATS: cargo test -p dynamo-sidecar-testkit --no-fail-fast with the four sidecar binaries built first: 113 passed, 0 failed at the baseline and at the head, of which 52 in router_sidecar_mocker_integration. Other crates whose sources changed: dynamo-sglang-sidecar (135 passed) and dynamo-trtllm-sidecar (96 passed, 7 ignored).
  • cargo machete at the root, lib/bindings/python and lib/runtime/examples reports no unused dependency. cargo metadata --locked passes for the four lock files; root, lib/bindings/python and lib/bindings/kvbm gain dynamo-sync, lib/runtime/examples is unchanged.
  • pre-commit run --from-ref idhanani/dyno-222-discovery-split --to-ref HEAD passes. build_codeowners.py --strict passes with CODEOWNERS matching emit_codeowners.py, and test_codeowners.py passes (157 tests).
  • Not run: the Python router e2e tests (tests/router/), which need etcd and NATS; the 17 ignored tests (the same as at baseline); the benches.
  • Both commits are scripted, and re-running each script on its commit's parent reproduces the commit's tree exactly (b562e4d471 for the move).

Closes DYNO-226

DYNO-226, layer 10 of splitting lib/llm into smaller crates. `dynamo_llm::sync` is about to move into its own crate below dynamo-llm. The router host, discovery, the worker-side publisher, the Pool Relay, session affinity and the LoRA load estimator use items of it that are `pub(crate)` today, which stops working across a crate boundary.

76 definitions and 7 struct fields become `pub`, together with the re-exports that name them, and nothing else: the list comes from compiling `sync` as its own crate, with and without the `ckf-diagnostics` feature, and widening only what a caller outside it failed to reach, plus the types that appear in the signatures of those items.

- modules: `direct_zmq_fan_in`, `direct_zmq_sub_pool`, `load::metrics_subscriber`, `load::routing_load`, and the test modules `indexer::test_util` and `metrics::test_fixtures`
- `direct_zmq_fan_in`: `ContinuityMode`, `FanInEvent`, `FanInObservation`, `start_direct_zmq_fan_in`
- `direct_zmq_sub_pool`: `KV_ZMQ_RCVHWM`, `RawZmqFrames`, `DirectZmqSubPool` with `new_raw`, `register_raw`, `shutdown`, `DirectZmqSubRegistration` with its `receiver` and `disconnected` fields, and `DirectZmqSubItem`, its default type parameter
- `load`: `KvMetricsSubscriber` with `for_endpoint_id` and `next`; `RouterLoadSource::{metric_label, from_worker_role_or_metric, monitors_sequence_load}`; `SchedulerLoadSender::{disabled, route_busy_thresholds}`; `LoadThresholdHandle::{disabled, is_enabled}`
- `membership`: `KvSourceMembershipCoordinator` with `start` and `subscribe`, `KvSourceMembershipView::matches_binding_inputs`, and the test-only `KvSourceMembershipWatch::shares_coordinator_with`
- `sequence`: `host_replica_channels`, `ReplicaIngress` with `start`, `ActiveSequenceEventPublisher::channel`, `RuntimeSequenceSubscriber::for_endpoint`
- `metrics`: `RouterWorkerStatusMetrics`, `RouterWorkerRegistration`, `RegistrationAvailabilityProvider`, and the test helper `test_fixtures::gather_pef`
- `indexer`: `RuntimeIngress` with `start`, `indexer`, `set_task_guard` and the test-only `has_subscription`; `RuntimeIngressArgs`; `KvEventSourceRequirement` with `derive` and `should_subscribe`; the re-exports of `ApproximateRequestLease` and `TieredMatchDetails`; the test fixture `test_util::store_event`
- `indexer` recovery: `DEFAULT_RECOVERY_ATTEMPT_TIMEOUT`, `RecoveryResetReason`, `RecoveryTarget`, `RecoverySupervisor` with `client` and `shutdown`, `start_target_subscriber`, `TargetFaultDisposition`, `WorkerQueryHealthSnapshot` with its five fields, `KvStateAgentStatusCell` with `new`, `load`, `store`, `store_outbound_cursor` and the test-only `snapshot`, `start_worker_kv_query_endpoint`, `start_worker_kv_query_endpoint_with_status`; `WorkerQueryClient` with `sync_membership`, `handle_target_fault`, `reject_source`, `handle_live_batch`, `health_snapshot` and the test-only `new_target_for_test`; the `WorkerQueryTransport` trait; and `IndexerRecoveryTarget`, the default type parameter of `WorkerQueryClient`

Inside dynamo-llm the change has no effect. The remaining 130 `pub(crate)` and `pub(super)` declarations in `sync` keep their visibility.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
DYNO-226, layer 10 of splitting lib/llm into smaller crates. `dynamo_llm::sync`, the code that keeps a router's view of its workers up to date, becomes the workspace crate `dynamo-sync` at `lib/sync`: `dynamo_llm::sync::X` is `dynamo_sync::X`. The crate sits below the router host and discovery, which stay in dynamo-llm, and does not depend on dynamo-llm or dynamo-preprocessor.

- `lib/llm/src/sync.rs` is the crate root and `lib/llm/src/sync/` its modules: `indexer`, `sequence`, `load`, `membership`, `runtime_configs`, `metrics`, `direct_zmq_fan_in`, `direct_zmq_sub_pool`. 36 files move with `git mv`. Inside them only `crate::sync::` becomes `crate::`.
- No compatibility re-export: dynamo-llm has no `pub use dynamo_sync`. The callers name the crate: 42 files in `lib/llm/src`, the Python bindings, the SGLang and TensorRT-LLM sidecars and one sidecar testkit test.
- Test-only items that dynamo-llm's tests use are exported behind the crate's `testing` feature, because `cfg(test)` does not cross a crate boundary: `indexer::test_util`, `indexer::{WorkerQueryClient, WorkerQueryTransport}`, `WorkerQueryClient::new_target_for_test`, `RuntimeIngress::has_subscription`, `KvStateAgentStatusCell::snapshot`, `KvSourceMembershipWatch::shares_coordinator_with` and `metrics::test_fixtures`. dynamo-llm enables the feature as a dev-dependency.
- The `ckf-diagnostics` feature gates moved code (the worker query client's health snapshot). The new crate has the feature and dynamo-llm forwards its own to it.
- The crate root includes `lib/runtime/src/test_utils.rs` under `cfg(test)`, as dynamo-llm and the other split crates do; one moved test uses its tracing guard.
- dynamo-llm drops `dynamo-tokens`, which only the remote indexer used.
- Log statements without an explicit target in the moved code report `dynamo_sync::...` where they reported `dynamo_llm::sync::...`. The router e2e test that set `dynamo_llm::sync::load::worker_monitor=debug` names the new target.
- Codeowners: every moved file keeps its owners. The `router` PR label covers `lib/sync/**`.

No metric, message, env var, subject or wire format changes.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

This branch had an error being deployed

1 failed deployment
external_collaborator — 91246096 Deployed Nov 16, 2026 by ishandhanani via ok-to-test #735
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant