Repository navigation
refactor(llm): move router state sync into dynamo-sync - #223
Draft
ishandhanani wants to merge 2 commits into
Draft
ishandhanani wants to merge 2 commits into
ishandhanani wants to merge 2 commits into
Conversation
ishandhanani
had a problem deploying
to
external_collaborator
November 15, 2026 19:41 — with
GitHub Actions
Failure
ishandhanani
force-pushed
the
idhanani/dyno-222-discovery-split
branch
from
November 16, 2026 07:07
46d311b to
e2179e6
Compare
ishandhanani
force-pushed
the
idhanani/dyno-226-sync
branch
from
November 16, 2026 07:07
8df840b to
3a824e7
Compare
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 07:07 — with
GitHub Actions
Failure
This was referenced Nov 16, 2026
ishandhanani
force-pushed
the
idhanani/dyno-222-discovery-split
branch
from
November 16, 2026 16:25
e2179e6 to
f45d06d
Compare
ishandhanani
force-pushed
the
idhanani/dyno-226-sync
branch
from
November 16, 2026 16:25
3a824e7 to
60f46b8
Compare
DYNO-226, layer 10 of splitting lib/llm into smaller crates. `dynamo_llm::sync` is about to move into its own crate below dynamo-llm. The router host, discovery, the worker-side publisher, the Pool Relay, session affinity and the LoRA load estimator use items of it that are `pub(crate)` today, which stops working across a crate boundary.
76 definitions and 7 struct fields become `pub`, together with the re-exports that name them, and nothing else: the list comes from compiling `sync` as its own crate, with and without the `ckf-diagnostics` feature, and widening only what a caller outside it failed to reach, plus the types that appear in the signatures of those items.
- modules: `direct_zmq_fan_in`, `direct_zmq_sub_pool`, `load::metrics_subscriber`, `load::routing_load`, and the test modules `indexer::test_util` and `metrics::test_fixtures`
- `direct_zmq_fan_in`: `ContinuityMode`, `FanInEvent`, `FanInObservation`, `start_direct_zmq_fan_in`
- `direct_zmq_sub_pool`: `KV_ZMQ_RCVHWM`, `RawZmqFrames`, `DirectZmqSubPool` with `new_raw`, `register_raw`, `shutdown`, `DirectZmqSubRegistration` with its `receiver` and `disconnected` fields, and `DirectZmqSubItem`, its default type parameter
- `load`: `KvMetricsSubscriber` with `for_endpoint_id` and `next`; `RouterLoadSource::{metric_label, from_worker_role_or_metric, monitors_sequence_load}`; `SchedulerLoadSender::{disabled, route_busy_thresholds}`; `LoadThresholdHandle::{disabled, is_enabled}`
- `membership`: `KvSourceMembershipCoordinator` with `start` and `subscribe`, `KvSourceMembershipView::matches_binding_inputs`, and the test-only `KvSourceMembershipWatch::shares_coordinator_with`
- `sequence`: `host_replica_channels`, `ReplicaIngress` with `start`, `ActiveSequenceEventPublisher::channel`, `RuntimeSequenceSubscriber::for_endpoint`
- `metrics`: `RouterWorkerStatusMetrics`, `RouterWorkerRegistration`, `RegistrationAvailabilityProvider`, and the test helper `test_fixtures::gather_pef`
- `indexer`: `RuntimeIngress` with `start`, `indexer`, `set_task_guard` and the test-only `has_subscription`; `RuntimeIngressArgs`; `KvEventSourceRequirement` with `derive` and `should_subscribe`; the re-exports of `ApproximateRequestLease` and `TieredMatchDetails`; the test fixture `test_util::store_event`
- `indexer` recovery: `DEFAULT_RECOVERY_ATTEMPT_TIMEOUT`, `RecoveryResetReason`, `RecoveryTarget`, `RecoverySupervisor` with `client` and `shutdown`, `start_target_subscriber`, `TargetFaultDisposition`, `WorkerQueryHealthSnapshot` with its five fields, `KvStateAgentStatusCell` with `new`, `load`, `store`, `store_outbound_cursor` and the test-only `snapshot`, `start_worker_kv_query_endpoint`, `start_worker_kv_query_endpoint_with_status`; `WorkerQueryClient` with `sync_membership`, `handle_target_fault`, `reject_source`, `handle_live_batch`, `health_snapshot` and the test-only `new_target_for_test`; the `WorkerQueryTransport` trait; and `IndexerRecoveryTarget`, the default type parameter of `WorkerQueryClient`
Inside dynamo-llm the change has no effect. The remaining 130 `pub(crate)` and `pub(super)` declarations in `sync` keep their visibility.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
DYNO-226, layer 10 of splitting lib/llm into smaller crates. `dynamo_llm::sync`, the code that keeps a router's view of its workers up to date, becomes the workspace crate `dynamo-sync` at `lib/sync`: `dynamo_llm::sync::X` is `dynamo_sync::X`. The crate sits below the router host and discovery, which stay in dynamo-llm, and does not depend on dynamo-llm or dynamo-preprocessor.
- `lib/llm/src/sync.rs` is the crate root and `lib/llm/src/sync/` its modules: `indexer`, `sequence`, `load`, `membership`, `runtime_configs`, `metrics`, `direct_zmq_fan_in`, `direct_zmq_sub_pool`. 36 files move with `git mv`. Inside them only `crate::sync::` becomes `crate::`.
- No compatibility re-export: dynamo-llm has no `pub use dynamo_sync`. The callers name the crate: 42 files in `lib/llm/src`, the Python bindings, the SGLang and TensorRT-LLM sidecars and one sidecar testkit test.
- Test-only items that dynamo-llm's tests use are exported behind the crate's `testing` feature, because `cfg(test)` does not cross a crate boundary: `indexer::test_util`, `indexer::{WorkerQueryClient, WorkerQueryTransport}`, `WorkerQueryClient::new_target_for_test`, `RuntimeIngress::has_subscription`, `KvStateAgentStatusCell::snapshot`, `KvSourceMembershipWatch::shares_coordinator_with` and `metrics::test_fixtures`. dynamo-llm enables the feature as a dev-dependency.
- The `ckf-diagnostics` feature gates moved code (the worker query client's health snapshot). The new crate has the feature and dynamo-llm forwards its own to it.
- The crate root includes `lib/runtime/src/test_utils.rs` under `cfg(test)`, as dynamo-llm and the other split crates do; one moved test uses its tracing guard.
- dynamo-llm drops `dynamo-tokens`, which only the remote indexer used.
- Log statements without an explicit target in the moved code report `dynamo_sync::...` where they reported `dynamo_llm::sync::...`. The router e2e test that set `dynamo_llm::sync::load::worker_monitor=debug` names the new target.
- Codeowners: every moved file keeps its owners. The `router` PR label covers `lib/sync/**`.
No metric, message, env var, subject or wire format changes.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
force-pushed
the
idhanani/dyno-226-sync
branch
from
November 16, 2026 20:34
60f46b8 to
9124609
Compare
ishandhanani
had a problem deploying
to
external_collaborator
November 16, 2026 20:34 — with
GitHub Actions
Failure
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Layer 10 of the lib/llm split stack (DYNO-226). Base: #222 (the in-place regroup under
dynamo_llm::sync), so this PR shows only this layer. It moves that module out ofdynamo-llminto a new workspace crate,dynamo-sync, atlib/sync: the code that keeps a router's view of its workers up to date. Target layering, top to bottom: discovery (model catalog), the router host (request path),dynamo-sync. The first two stay indynamo-llm.What moved
lib/llm/src/sync.rsis the crate root andlib/llm/src/sync/its modules:dynamo_llm::sync::Xis nowdynamo_sync::X.indexer(KV index ingest, recovery, remote indexer),sequence(active-sequence replica sync),load(ActiveLoadand its subscription,KvWorkerMonitor,LoadThresholdHandle,RoutingLoadContext),membership(KV event-source membership and the state-agent vocabulary),runtime_configs(RuntimeConfigWatch),metrics(worker load, worker status, KV and active-sequence ingress, remote indexer),direct_zmq_fan_inanddirect_zmq_sub_pool.git mv(24,578 lines of Rust with tests); 13 are byte-identical. Inside them onlycrate::sync::becomescrate::, plus thetestinggates below.lib/llm/srcgoes from 216,237 to 191,664 lines of Rust (324 to 288 files).dynamo-llmhas nopub use dynamo_sync.dynamo-llmmoves: none of them names a sync item.kv_router_benchdrives the router and stays.Commits
One in-place commit inside
dynamo-llm, which builds and passes on its own, then one move commit.pub(crate)andpub(super)declarations insync(re-exports included), some used by the router host, discovery, the worker-side publisher, the Pool Relay, session affinity and the LoRA load estimator67682046b7: 76 definitions and 7 struct fields becomepub, with the re-exports that name them. Exactly what a caller outside the crate failed to reach whensyncwas compiled as its own crate, with and withoutckf-diagnostics, plus the types in their signatures. 130 declarations stay crate-private. The list is in the commit message and belowdynamo-llm; widening whole modulesdynamo-llm's tests use (cfg(test)does not cross a crate boundary):indexer::test_util,indexer::{WorkerQueryClient, WorkerQueryTransport},WorkerQueryClient::new_target_for_test,RuntimeIngress::has_subscription,KvStateAgentStatusCell::snapshot,KvSourceMembershipWatch::shares_coordinator_with,metrics::test_fixturestestingfeature of the new crate, whichdynamo-llmenables as a dev-dependency (the pattern ofdynamo-preprocessoranddynamo-frontend-protocols)ckf-diagnosticsgates moved code (5cfgsites: the worker query client's health snapshot)dynamo-llmforwards its own, sodynamo-llm/ckf-diagnosticskeeps working unchangeddynamo-llm, which would leave the snapshot uncompileddynamo-kv-indexerbehind itsexportfeaturedynamo-syncenablesdynamo-kv-indexer/export, asdynamo-llmdoesdynamo-llmcrate::test_utils::tracing_interest_guardlib/runtime/src/test_utils.rsundercfg(test), asdynamo-llmand the other split crates do, withallow(dead_code)because only the tracing guard is used heredynamo-runtime, which changes a lower layerWhat became public (
67682046b7)direct_zmq_fan_in,direct_zmq_sub_pool,load::metrics_subscriber,load::routing_load, and the test modulesindexer::test_utilandmetrics::test_fixtures.direct_zmq_fan_in:ContinuityMode,FanInEvent,FanInObservation,start_direct_zmq_fan_in.direct_zmq_sub_pool:KV_ZMQ_RCVHWM,RawZmqFrames,DirectZmqSubPoolwithnew_raw,register_raw,shutdown,DirectZmqSubRegistrationwith itsreceiveranddisconnectedfields, andDirectZmqSubItem, its default type parameter.load:KvMetricsSubscriberwithfor_endpoint_idandnext;RouterLoadSource::{metric_label, from_worker_role_or_metric, monitors_sequence_load};SchedulerLoadSender::{disabled, route_busy_thresholds};LoadThresholdHandle::{disabled, is_enabled}.membership:KvSourceMembershipCoordinatorwithstartandsubscribe;KvSourceMembershipView::matches_binding_inputs;KvSourceMembershipWatch::shares_coordinator_with.sequence:host_replica_channels,ReplicaIngresswithstart,ActiveSequenceEventPublisher::channel,RuntimeSequenceSubscriber::for_endpoint.metrics:RouterWorkerStatusMetrics,RouterWorkerRegistration,RegistrationAvailabilityProvider,test_fixtures::gather_pef.indexer:RuntimeIngresswithstart,indexer,set_task_guard,has_subscription;RuntimeIngressArgs;KvEventSourceRequirementwithderiveandshould_subscribe; the re-exports ofApproximateRequestLeaseandTieredMatchDetails;test_util::store_event.indexerrecovery:DEFAULT_RECOVERY_ATTEMPT_TIMEOUT,RecoveryResetReason,RecoveryTarget,RecoverySupervisorwithclientandshutdown,start_target_subscriber,TargetFaultDisposition,WorkerQueryHealthSnapshotwith its five fields,KvStateAgentStatusCellwithnew,load,store,store_outbound_cursor,snapshot,start_worker_kv_query_endpoint,start_worker_kv_query_endpoint_with_status,WorkerQueryClientwithsync_membership,handle_target_fault,reject_source,handle_live_batch,health_snapshot,new_target_for_test, theWorkerQueryTransporttrait, andIndexerRecoveryTarget, the default type parameter ofWorkerQueryClient.Callers
lib/llm/srcnamedynamo_sync: the router host (kv_router.rs,kv_router/{embedded,managed,metrics,request_lease,routing_host,shared_cache}.rs,routing_host/,prefill_router/,publisher/), discovery, the entrypoint, the HTTP service, the Pool Relay, session affinity, LoRA and the mocker.lib/llm, six Rust files do:lib/bindings/python/rust/{lib.rs,llm/kv.rs,llm/kv_state_agent.rs},lib/sidecar/sglang/src/headless.rs,lib/sidecar/trtllm/src/headless.rsand the testlib/sidecar/testkit/tests/router_sidecar_mocker_integration.rs. Their manifests gain the dependency (a dev-dependency for the testkit). Two doc comments inlib/routerand thelib/router/kv-routerREADME name the new path.lib/backend-common,lib/mocker/servers/*and the EPP name none of the moved items.dynamo-llmdrops one dependency,dynamo-tokens, which only the remote indexer used.Crate plumbing
[workspace.dependencies]entry; SPDX headers;lib/sync/AGENTS.md(what the crate owns, what it must not depend on, how to check it) with the siblingCLAUDE.md.cargo tree -p dynamo-sync | grep -c dynamo-llmprints 0. Direct Dynamo dependencies:dynamo-runtime,dynamo-kv-indexer,dynamo-kv-events,dynamo-kv-hashing,dynamo-slot-tracker,dynamo-scheduler,dynamo-router-protocol,dynamo-kv-router,dynamo-model-card,dynamo-frontend-protocols,dynamo-http-server,dynamo-tokens, anddynamo-kv-hintsfor tests. None is above the crate;dynamo-preprocessoranddynamo-telemetryare not in its graph.eppco-ownerships). The new manifest andAGENTS.mdgo to runtime, the owner oflib/llm/. TherouterPR label coverslib/sync/**. The CI path filters matchlib/**and need no change.kv_router/AGENTS.md,kv_router/README.md, the router testing guide and the KV event replay comparison cite the new paths.Behaviour
dynamo_sync::indexer::*(105),dynamo_sync::load::*(23),dynamo_sync::direct_zmq_fan_in(11),dynamo_sync::direct_zmq_sub_pool(11),dynamo_sync::membership::kv_source_watch(6),dynamo_sync::sequenceanddynamo_sync::sequence::direct_zmq(6) anddynamo_sync::runtime_configs(4), where they reporteddynamo_llm::sync::.... No target is pinned.DYN_LOGexamples keep selecting the same lines: every filter that refactor(llm): group index, slot, load and membership sync under one module #222 pointed atdynamo_llm::sync[::...]now namesdynamo_sync[::...](the two CLI template pages, the two multimodal KV routing pages, the vLLM and SGLangagg_multimodal_router.shlaunchers, the threetests/mm_routerend-to-end tests,tests/router/test_router_e2e_with_mockers.py, and the EPP note on the deprecations page). Checked live on the top of the stack againstmain: the disaggregated startup and overload tests capture 50 worker-monitor debug lines (51 onmain) and 4 recovery query-endpoint debug lines (4 onmain).Validation
Cargo commands ran on a remote builder with the repository toolchain (1.96.1).
cargo test -p dynamo-llm --no-fail-fast: 36 binaries, 2396 passed, 0 failed, 17 ignored.-p dynamo-sync: 38 binaries, 2400 passed, 0 failed, 17 ignored. The two extra binaries are the new crate's unit and doc tests.dynamo-llmunit tests go from 2041 passed and 4 ignored to 1854 and 4; the new crate has 191 (the 187 that moved and the 4 tests of the includedtest_utils.rs, which also run indynamo-llm).cargo test -- --list: all 2413 baseline names exist at head, none is missing. 2226 keep binary and path, 187 moved fromdynamo-llm's unit tests to the same leaf name indynamo-sync's (sync::XisX). The 4 added names are thetest_utilstests in the new crate.cargo check -p dynamo-llm --all-targets, the same with--features ckf-diagnostics, andcargo clippy --no-deps -p dynamo-llm --all-targets -- -D warningspass;cargo test -p dynamo-llm --lib -- sync::passes 191.cargo check --workspace --all-targets,cargo clippy --no-deps --workspace --all-targets -- -D warnings,cargo fmt --all -- --check(root andlib/bindings/python), andcargo check --lockedinlib/bindings/python,lib/bindings/kvbmandlib/runtime/examplespass.cargo check -p dynamo-syncpasses without features and withtesting;cargo check -p dynamo-sync --all-targets --features ckf-diagnosticsand clippy withckf-diagnostics,testingand-D warningspass;cargo test -p dynamo-syncpasses 191, also with--features ckf-diagnostics.ckf-diagnostics:cargo check -p dynamo-llm --features ckf-diagnostics --all-targetsand clippy with-D warningspass.cargo test -p dynamo-sidecar-testkit --no-fail-fastwith the four sidecar binaries built first: 113 passed, 0 failed at the baseline and at the head, of which 52 inrouter_sidecar_mocker_integration. Other crates whose sources changed:dynamo-sglang-sidecar(135 passed) anddynamo-trtllm-sidecar(96 passed, 7 ignored).cargo macheteat the root,lib/bindings/pythonandlib/runtime/examplesreports no unused dependency.cargo metadata --lockedpasses for the four lock files; root,lib/bindings/pythonandlib/bindings/kvbmgaindynamo-sync,lib/runtime/examplesis unchanged.pre-commit run --from-ref idhanani/dyno-222-discovery-split --to-ref HEADpasses.build_codeowners.py --strictpasses withCODEOWNERSmatchingemit_codeowners.py, andtest_codeowners.pypasses (157 tests).tests/router/), which need etcd and NATS; the 17 ignored tests (the same as at baseline); the benches.b562e4d471for the move).Closes DYNO-226