Repository navigation
perf(discovery): reconcile off the runtime, clone only matching cards - #16470
peter-xxxxx wants to merge 2 commits into
Conversation
Every RECONCILIATION_INTERVAL the frontend's model discovery controller lists the model cards it can see and re-normalizes each of them. On the Kubernetes backend that list cloned every model card in the cluster (card JSON included) under the list_state read lock before the query predicate ran, and the controller then deserialized and checksummed every own card inline on its tokio task. Both halves ran on runtime worker threads, so every request in flight on that frontend waited for them: a 200 Hz /live probe measures a stall of 3-6 ms per tick on a 342-object cluster, and the per-object cost extrapolates to 40-55 ms per tick on a 2,400-object cluster, every 30 s, on every frontend. A client that fans out across many calls per request sees that stall as its p99. Three changes, each small on its own: - DiscoveryMetadata::filter iterates the matching map by reference and clones only the instances the query selects. The result set is unchanged for every query kind; the per-cluster-object clone is gone for every caller of list(), list_and_watch()'s initial list and its lag resync included. - KubeDiscoveryClient::list takes a shallow snapshot (one Arc per worker) under the read lock, releases it, and runs the filter on spawn_blocking. - The controller's periodic reconcile normalizes the listed cards on spawn_blocking inside the reconciliation task it already spawns, and the controller task only applies the verdicts. The resync_revision and per-instance revision checks, the removal logic and the log lines are unchanged; a card superseded by a newer event is now normalized and then dropped instead of skipped, which does not change what is admitted or logged. ControllerHost::list_instances receives the NamespaceFilter so that an exact namespace lists NamespacedModels instead of AllModels; Global and Prefix filters still list AllModels, because no query expresses a prefix. The Resync event path keeps normalizing inline on the controller task, as before. One side effect of the offload: a panic inside normalize or filter now surfaces as a failed reconciliation (a warning) or a list error instead of unwinding the controller task. Closes ai-dynamo#16444 Signed-off-by: Suyang Xu <suyax@amazon.com>
|
👋 Hi peter-xxxxx! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
| card_json: serde_json::json!({}), | ||
| model_suffix: None, | ||
| }; | ||
| let filters = [ |
There was a problem hiding this comment.
This 3×2×5 matrix re-tests unchanged NamespaceFilter and DiscoveryInstance::matches behavior even though reconciliation_query depends only on the filter enum variant, not prefix mode or candidate namespace. The retained exact-query assertion plus one direct assertion each for Global and Prefix protects every changed mapping without these duplicate combinations.
🤖 AI Fix
Replace the nested filter/mode/namespace matrix with direct assertions that Global and Prefix return AllModels, retaining the existing Exact-to-NamespacedModels assertion.
| } | ||
|
|
||
| /// Filter this metadata by query | ||
| /// Filter this metadata by query, cloning only the instances the query selects. |
There was a problem hiding this comment.
The added “cloning only…” clause narrates this implementation’s iterator order rather than documenting a caller-visible contract or why that ordering must be preserved. The method body already makes the clone-after-filter behavior explicit, so retain only the concise description of what the method returns.
🤖 AI Fix
Remove the implementation-detail clause and leave the doc comment as /// Filter this metadata by query.
|
/ok to test 4850957 |
|
CC @tmonty12 @mohammedabdulwahhab on discovery changes |
❌ Dynamo PR CI failed — run 40037879582 (attempt 1) on
|
| Framework | 1-GPU arm64 |
|---|---|
| vLLM | ❌ 1 |
Failure details
1 job failed: the vLLM arm64 CPU-only pytest stage died with a self-hosted runner container-hook error before pytest reported any results; no test failures were recorded.
❌ vllm-runtime / Test cuda13.0, arm64: runner "custom container implementation failed" during pre_merge_cpu pytest
Job: vllm-runtime / Test cuda13.0, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 120708683031 -R ai-dynamo/dynamo --log-failed
23:34:16 Run bash .github/scripts/pytest-local.sh (TEST_TYPE=pre_merge_cpu, PYTEST_MARKS="pre_merge and vllm and gpu_0 and not e2e")
▶️ Executing (cwd=/workspace): python -m pytest -n auto --dist=loadscope ...
23:34:49 [Datadog Test Optimization] WARNING ... rerunfailures is installed alongside Datadog Test Optimization ... (xdist workers starting)
... no further output for ~9.5 minutes ...
23:44:16 ##[error]Executing the custom container implementation failed. Please contact your self hosted runner administrator.
...
❌ pre_merge_cpu: step outcome 'failure'
✅ pre_merge_cpu_e2e: pytest exited 0
##[error]A pytest stage did not record a passing result. See this step's log for the stage.
After the xdist workers started, the pre_merge_cpu pytest stage produced no output for about 9.5 minutes. It then ended with the runner's k8s container-hook error, well before the step's 30-minute timeout. No pytest summary, FAILED lines or allure results were written. The later pre_merge_cpu_e2e stage in the same job passed (tests/serve/test_sample.py::test_sample_deployment[aggregated]). This looks like an infrastructure failure: the runner lost its exec into the test container, which can also happen when the pod is killed for running out of resources. Rerunning the failed job is likely enough. If it fails the same way again, look for a hang in that test set.
For agents
{"pr": 16470, "run_id": 40037879582, "run_attempt": 1, "head_sha": "4850957e28931fc1dd9aa1fe88f78b6d7410fd47",
"failures": [{"job": "vllm-runtime / Test cuda13.0, arm64", "job_id": 120708683031, "failed_step": "Run CPU-only tests (parallelized)", "signature": "Executing the custom container implementation failed (runner container hook) during pre_merge_cpu pytest; no test results recorded", "tests": [], "log_cmd": "gh run view --job 120708683031 -R ai-dynamo/dynamo --log-failed"}]}Posted automatically by Devin for run 40037879582. Updated on every full-CI run of this PR.
Overview:
The frontend's 30 s model reconcile stalls the tokio runtime on every tick:
KubeDiscoveryClient::list(AllModels)clones every model card in the cluster under thelist_stateread lock before the query predicate runs, andapply_reconciliationthen deserializes and checksums every own card inline on the controller task. On a 342-object cluster that is a 3-6 ms stall every 30 s per frontend; the per-object cost extrapolates to 40-55 ms per tick on a 2,400-object cluster, and a client that fans out across many calls per request sees the stall as its p99. This PR keeps that work off the runtime workers and stops cloning cards the query does not select.Details:
DiscoveryMetadata::filter(lib/runtime) iterates the matching map by reference and clones only the instances the query selects. Same result set for every query kind;filter_instancesis folded in. Everylist()caller,list_and_watch()'s initial list and its lag resync benefit.KubeDiscoveryClient::listtakes a shallow snapshot oflist_state(oneArcper worker) under the read lock, releases it, and runs the filter ontokio::task::spawn_blocking. TheKubeDiscoveryClient::list returning … for query=…info line is unchanged.ModelDiscoveryController: the periodic reconcile normalizes the listed cards (ControllerHost::normalize: deserialize, legacy-topology and tokenizer normalization, shape validation,mdcsum, LoRA fingerprint, video contract) onspawn_blockinginside the reconciliation task it already spawns;apply_normalized_reconciliationon the controller task only applies the verdicts. Theresync_revisiondiscard, the per-instanceinstance_revisionscheck, the removal set,retainand the log lines are unchanged. The one behavioural difference is that a card superseded by a newer event is normalized (off-thread) and then dropped, instead of being skipped before normalization; it is neither admitted nor logged, as before. TheDiscoveryEvent::Resyncpath still normalizes inline on the controller task through the keptapply_reconciliationwrapper.ControllerHost::list_instancesreceives theNamespaceFilter.ModelWatcherlistsNamespacedModels { namespace }forNamespaceFilter::Exact(exactly the setnormalizecan admit under bothNamespacePrefixModes) and keepsAllModelsforGlobalandPrefix, since noDiscoveryQueryvariant expresses a prefix. Foreign-namespace keys were never admitted; with the narrower list their staleinstance_revisionsentries are pruned at the next tick instead of lingering.normalizeorfilternow surfaces as a failed reconciliation (warning) or alist()error instead of unwinding the controller task.metadata.rschecksfilteragainst thematchespredicate for every endpoint and model query kind;controller.rsdrivesstart_reconciliationthrough the spawned task andapply_normalized_reconciliationwith an admitted card, a foreign-namespace card, a malformed card, a non-model instance, a card superseded by an in-flight event, and a vanished card, plus the list-error path;watcher.rschecks thatreconciliation_querylists every card the filter admits under both prefix modes and exactly those for an exact namespace.Where should the reviewer start?
lib/llm/src/discovery/controller.rs:start_reconciliation,normalize_reconciliation,apply_reconciliation/apply_normalized_reconciliation(the split), then the testa_periodic_list_is_normalized_off_the_controller_and_applied_as_before.lib/runtime/src/discovery/kube.rs:list(lock scope).lib/runtime/src/discovery/metadata.rs:filter.lib/llm/src/discovery/watcher.rs:reconciliation_queryandlist_instances.Summary
Two complementary fixes for the 30 s reconcile stall: run the list filter and the per-card normalization on blocking threads (the latency fix), and clone only matching cards while listing only the namespace the frontend serves when the filter is exact (the CPU fix).
Validation
Commands run on a build host with Rust 1.96.1 (the repo's toolchain), on this branch at 938f4e4 + this diff:
Results:
cargo fmt -- --checkclean; clippy clean with-D warningsfor both crates;dynamo-runtimediscovery::100 passed, 0 failed;dynamo-llmdiscovery::241 passed, 0 failed (in one of five runsdiscovery::model_manager::tests::disaggregated_pair_sharing_an_alias_reports_one_claim_per_role, which this PR does not touch, failed on its captured tracing events and passed on every rerun, patched and unpatched); pre-commit: every applicable hook passed (codespell, case conflicts, merge conflicts, shebangs, line endings, trailing whitespace, AGENTS/CLAUDE pairs).Measured on a Kubernetes cluster with 341 discovery objects: a frontend-only image built from this branch (
--framework triton --target runtime, no engine) swapped onto one frontend of a deployment whose worker stayed on 1.4.1, then a 200 HzGET /liveprobe inside the frontend pod for 200 s plus 130 s with a 2 ms threshold, aligned with the frontend log.main7d82c10 (same cluster, same day)The branch frontend ran 13 reconcile ticks at a 30.000 s period during the two probes (
KubeDiscoveryClient::list returning 341-342 instances for query=AllModels, so this deployment uses a namespace prefix and exercised the offload and the clone-only-matches path, not theNamespacedModelsnarrowing). Probe 1: 48,222 calls, p50 0.08 / p99 0.19 / p99.9 0.25 / max 1.48 ms, no sample over 2 ms (the only 1-2 ms sample is the first request on the new connection). Probe 2: 31,339 calls, p99.9 0.26 / max 1.38 ms, again none over 2 ms. The same probe onmain7d82c10 minutes earlier on the same cluster showed every tick (2.6-6.1 ms). Compatibility with the 1.4.1 worker held:/v1/modelslists the model, a classify request returned the same probabilities as on 1.4.1 andmain, no WARN or ERROR lines, 0 restarts. (/healthon this branch still returns every instance, 271 KB here; that is #16443.)Related Issues
Summary by CodeRabbit