Skip to content

perf(discovery): reconcile off the runtime, clone only matching cards - #16470

Open
peter-xxxxx wants to merge 2 commits into
ai-dynamo:mainfrom
peter-xxxxx:suyax/discovery-reconcile-offload
Open

peter-xxxxx wants to merge 2 commits into
ai-dynamo:mainfrom
peter-xxxxx:suyax/discovery-reconcile-offload

Conversation

@peter-xxxxx

@peter-xxxxx peter-xxxxx commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Overview:

The frontend's 30 s model reconcile stalls the tokio runtime on every tick: KubeDiscoveryClient::list(AllModels) clones every model card in the cluster under the list_state read lock before the query predicate runs, and apply_reconciliation then deserializes and checksums every own card inline on the controller task. On a 342-object cluster that is a 3-6 ms stall every 30 s per frontend; the per-object cost extrapolates to 40-55 ms per tick on a 2,400-object cluster, and a client that fans out across many calls per request sees the stall as its p99. This PR keeps that work off the runtime workers and stops cloning cards the query does not select.

Details:

  • DiscoveryMetadata::filter (lib/runtime) iterates the matching map by reference and clones only the instances the query selects. Same result set for every query kind; filter_instances is folded in. Every list() caller, list_and_watch()'s initial list and its lag resync benefit.
  • KubeDiscoveryClient::list takes a shallow snapshot of list_state (one Arc per worker) under the read lock, releases it, and runs the filter on tokio::task::spawn_blocking. The KubeDiscoveryClient::list returning … for query=… info line is unchanged.
  • ModelDiscoveryController: the periodic reconcile normalizes the listed cards (ControllerHost::normalize: deserialize, legacy-topology and tokenizer normalization, shape validation, mdcsum, LoRA fingerprint, video contract) on spawn_blocking inside the reconciliation task it already spawns; apply_normalized_reconciliation on the controller task only applies the verdicts. The resync_revision discard, the per-instance instance_revisions check, the removal set, retain and the log lines are unchanged. The one behavioural difference is that a card superseded by a newer event is normalized (off-thread) and then dropped, instead of being skipped before normalization; it is neither admitted nor logged, as before. The DiscoveryEvent::Resync path still normalizes inline on the controller task through the kept apply_reconciliation wrapper.
  • ControllerHost::list_instances receives the NamespaceFilter. ModelWatcher lists NamespacedModels { namespace } for NamespaceFilter::Exact (exactly the set normalize can admit under both NamespacePrefixModes) and keeps AllModels for Global and Prefix, since no DiscoveryQuery variant expresses a prefix. Foreign-namespace keys were never admitted; with the narrower list their stale instance_revisions entries are pruned at the next tick instead of lingering.
  • Side effect of the offload: a panic inside normalize or filter now surfaces as a failed reconciliation (warning) or a list() error instead of unwinding the controller task.
  • Tests: metadata.rs checks filter against the matches predicate for every endpoint and model query kind; controller.rs drives start_reconciliation through the spawned task and apply_normalized_reconciliation with an admitted card, a foreign-namespace card, a malformed card, a non-model instance, a card superseded by an in-flight event, and a vanished card, plus the list-error path; watcher.rs checks that reconciliation_query lists every card the filter admits under both prefix modes and exactly those for an exact namespace.

Where should the reviewer start?

  1. lib/llm/src/discovery/controller.rs: start_reconciliation, normalize_reconciliation, apply_reconciliation / apply_normalized_reconciliation (the split), then the test a_periodic_list_is_normalized_off_the_controller_and_applied_as_before.
  2. lib/runtime/src/discovery/kube.rs: list (lock scope).
  3. lib/runtime/src/discovery/metadata.rs: filter.
  4. lib/llm/src/discovery/watcher.rs: reconciliation_query and list_instances.

Summary

Two complementary fixes for the 30 s reconcile stall: run the list filter and the per-card normalization on blocking threads (the latency fix), and clone only matching cards while listing only the namespace the frontend serves when the filter is exact (the CPU fix).

Validation

Commands run on a build host with Rust 1.96.1 (the repo's toolchain), on this branch at 938f4e4 + this diff:

cargo fmt -- --check
cargo clippy --locked -p dynamo-runtime -p dynamo-llm --all-targets --no-deps -- -D warnings
cargo test --locked -p dynamo-runtime --lib discovery::
cargo test --locked -p dynamo-llm --lib discovery::
pre-commit run --files lib/llm/src/discovery/controller.rs lib/llm/src/discovery/watcher.rs lib/runtime/src/discovery/kube.rs lib/runtime/src/discovery/metadata.rs

Results: cargo fmt -- --check clean; clippy clean with -D warnings for both crates; dynamo-runtime discovery:: 100 passed, 0 failed; dynamo-llm discovery:: 241 passed, 0 failed (in one of five runs discovery::model_manager::tests::disaggregated_pair_sharing_an_alias_reports_one_claim_per_role, which this PR does not touch, failed on its captured tracing events and passed on every rerun, patched and unpatched); pre-commit: every applicable hook passed (codespell, case conflicts, merge conflicts, shebangs, line endings, trailing whitespace, AGENTS/CLAUDE pairs).

Measured on a Kubernetes cluster with 341 discovery objects: a frontend-only image built from this branch (--framework triton --target runtime, no engine) swapped onto one frontend of a deployment whose worker stayed on 1.4.1, then a 200 Hz GET /live probe inside the frontend pod for 200 s plus 130 s with a 2 ms threshold, aligned with the frontend log.

frontend ticks observed samples over 2 ms at the 30 s period mean / max of the tick
1.4.1 stock (same cluster, same day) 10 10 6.0 / 8.0 ms
main 7d82c10 (same cluster, same day) 22 13 3.3 / 6.1 ms
this branch (main 938f4e4 + this diff) 13 0 none visible; per-30 s-window max 0.26-0.44 ms

The branch frontend ran 13 reconcile ticks at a 30.000 s period during the two probes (KubeDiscoveryClient::list returning 341-342 instances for query=AllModels, so this deployment uses a namespace prefix and exercised the offload and the clone-only-matches path, not the NamespacedModels narrowing). Probe 1: 48,222 calls, p50 0.08 / p99 0.19 / p99.9 0.25 / max 1.48 ms, no sample over 2 ms (the only 1-2 ms sample is the first request on the new connection). Probe 2: 31,339 calls, p99.9 0.26 / max 1.38 ms, again none over 2 ms. The same probe on main 7d82c10 minutes earlier on the same cluster showed every tick (2.6-6.1 ms). Compatibility with the 1.4.1 worker held: /v1/models lists the model, a classify request returned the same probabilities as on 1.4.1 and main, no WARN or ERROR lines, 0 restarts. (/health on this branch still returns every instance, 271 KB here; that is #16443.)

Related Issues

Summary by CodeRabbit

  • Bug Fixes
    • Discovery results now respect namespace filters, including exact, prefix, and global searches.
    • Periodic updates preserve the latest instance state when newer changes arrive during reconciliation.
    • A discovery listing failure no longer changes the desired state.
    • Filtering work runs without holding shared state locks, helping keep discovery responsive.

Every RECONCILIATION_INTERVAL the frontend's model discovery controller
lists the model cards it can see and re-normalizes each of them. On the
Kubernetes backend that list cloned every model card in the cluster
(card JSON included) under the list_state read lock before the query
predicate ran, and the controller then deserialized and checksummed
every own card inline on its tokio task. Both halves ran on runtime
worker threads, so every request in flight on that frontend waited for
them: a 200 Hz /live probe measures a stall of 3-6 ms per tick on a
342-object cluster, and the per-object cost extrapolates to 40-55 ms per
tick on a 2,400-object cluster, every 30 s, on every frontend. A client
that fans out across many calls per request sees that stall as its p99.

Three changes, each small on its own:

- DiscoveryMetadata::filter iterates the matching map by reference and
  clones only the instances the query selects. The result set is
  unchanged for every query kind; the per-cluster-object clone is gone
  for every caller of list(), list_and_watch()'s initial list and its
  lag resync included.
- KubeDiscoveryClient::list takes a shallow snapshot (one Arc per
  worker) under the read lock, releases it, and runs the filter on
  spawn_blocking.
- The controller's periodic reconcile normalizes the listed cards on
  spawn_blocking inside the reconciliation task it already spawns, and
  the controller task only applies the verdicts. The resync_revision and
  per-instance revision checks, the removal logic and the log lines are
  unchanged; a card superseded by a newer event is now normalized and
  then dropped instead of skipped, which does not change what is
  admitted or logged. ControllerHost::list_instances receives the
  NamespaceFilter so that an exact namespace lists NamespacedModels
  instead of AllModels; Global and Prefix filters still list AllModels,
  because no query expresses a prefix.

The Resync event path keeps normalizing inline on the controller task,
as before. One side effect of the offload: a panic inside normalize or
filter now surfaces as a failed reconciliation (a warning) or a list
error instead of unwinding the controller task.

Closes ai-dynamo#16444

Signed-off-by: Suyang Xu <suyax@amazon.com>
@peter-xxxxx
peter-xxxxx requested a review from a team as a code owner October 15, 2026 00:18
@copy-pr-bot

copy-pr-bot Bot commented Oct 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@peter-xxxxx
peter-xxxxx temporarily deployed to external_collaborator October 15, 2026 00:18 — with GitHub Actions Inactive
@peter-xxxxx
peter-xxxxx temporarily deployed to external_collaborator October 15, 2026 00:18 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi peter-xxxxx! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added perf external-contribution Pull request is from an external contributor labels Oct 15, 2026
@coderabbitai

coderabbitai Bot commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: cd32cdf9-ca60-4f1a-85a3-ac34ac4bee7f
📥 Commits

Reviewing files that changed from the base of the PR and between abc1952 and 34a9ec9.

📒 Files selected for processing: 4
  • lib/llm/src/discovery/controller.rs
  • lib/llm/src/discovery/watcher.rs
  • lib/runtime/src/discovery/kube.rs
  • lib/runtime/src/discovery/metadata.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

Reconciliation now passes namespace filters to discovery listing and normalizes periodic results on a blocking task. Discovery listing and metadata filtering also change how they select and process matching instances.

Changes

Discovery reconciliation

Layer / File(s) Summary
Namespace-filtered reconciliation queries
lib/llm/src/discovery/controller.rs, lib/llm/src/discovery/watcher.rs
ControllerHost::list_instances accepts a namespace filter. The watcher uses NamespacedModels for exact filters and AllModels for global or prefix filters. Tests check query selection and namespace coverage.
Periodic normalization and application
lib/llm/src/discovery/controller.rs
Periodic reconciliation normalizes listed model entries in a blocking task and returns keyed verdicts with the snapshot revision. Application uses those verdicts, protects newer instance revisions, and leaves desired state unchanged on list errors. Tests cover filter forwarding, excluded entries, stale snapshots, and list failures.
Discovery query filtering and listing
lib/runtime/src/discovery/kube.rs, lib/runtime/src/discovery/metadata.rs
Kubernetes listing filters a cloned list-state snapshot in a blocking task after releasing the read lock. Metadata filtering clones only matching instances. Tests check query results across namespaces, components, and endpoints.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 34a9e

This change moves discovery reconciliation work off the runtime and narrows exact-namespace queries. No concrete merge-blocking risk was identified in the supplied context.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 46.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 28 functions across 4 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title is concise, specific, and accurately summarizes the main performance changes: moving reconciliation off the runtime and cloning only matching cards.
Description check Passed The description includes all required sections, explains the changes and behavior, identifies review starting points, references related issues, and documents validation results.
Linked Issues check Passed Issue [#16444] requires the 30-second reconciliation to avoid blocking runtime request handling. The PR moves KubeDiscoveryClient::list filtering to spawn_blocking, moves periodic card normalizati…
Out of Scope Changes check Passed The changes stay within [#16444]. They optimize discovery filtering, periodic reconciliation, namespace query selection, and related tests. The changes do not implement the separate /health work fro…
  • Fix all pre-merge checks with AI
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

card_json: serde_json::json!({}),
model_suffix: None,
};
let filters = [

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This 3×2×5 matrix re-tests unchanged NamespaceFilter and DiscoveryInstance::matches behavior even though reconciliation_query depends only on the filter enum variant, not prefix mode or candidate namespace. The retained exact-query assertion plus one direct assertion each for Global and Prefix protects every changed mapping without these duplicate combinations.

🤖 AI Fix

Replace the nested filter/mode/namespace matrix with direct assertions that Global and Prefix return AllModels, retaining the existing Exact-to-NamespacedModels assertion.

}

/// Filter this metadata by query
/// Filter this metadata by query, cloning only the instances the query selects.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The added “cloning only…” clause narrates this implementation’s iterator order rather than documenting a caller-visible contract or why that ordering must be preserved. The method body already makes the clone-after-filter behavior explicit, so retain only the concise description of what the method returns.

🤖 AI Fix

Remove the implementation-detail clause and leave the doc comment as /// Filter this metadata by query.

@rmccorm4
rmccorm4 temporarily deployed to external_collaborator October 23, 2026 23:06 — with GitHub Actions Inactive
@rmccorm4

Copy link
Copy Markdown
Contributor

/ok to test 4850957

@rmccorm4

Copy link
Copy Markdown
Contributor

CC @tmonty12 @mohammedabdulwahhab on discovery changes

@devin-ai-integration

Copy link
Copy Markdown
Contributor

❌ Dynamo PR CI failed — run 40037879582 (attempt 1) on 4850957e28

Gate checks: ❌ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Framework 1-GPU arm64
vLLM ❌ 1

Failure details

1 job failed: the vLLM arm64 CPU-only pytest stage died with a self-hosted runner container-hook error before pytest reported any results; no test failures were recorded.

❌ vllm-runtime / Test cuda13.0, arm64: runner "custom container implementation failed" during pre_merge_cpu pytest

Job: vllm-runtime / Test cuda13.0, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 120708683031 -R ai-dynamo/dynamo --log-failed

23:34:16 Run bash .github/scripts/pytest-local.sh   (TEST_TYPE=pre_merge_cpu, PYTEST_MARKS="pre_merge and vllm and gpu_0 and not e2e")
▶️ Executing (cwd=/workspace): python -m pytest -n auto --dist=loadscope ...
23:34:49 [Datadog Test Optimization] WARNING ... rerunfailures is installed alongside Datadog Test Optimization ...  (xdist workers starting)
          ... no further output for ~9.5 minutes ...
23:44:16 ##[error]Executing the custom container implementation failed. Please contact your self hosted runner administrator.
...
❌ pre_merge_cpu: step outcome 'failure'
✅ pre_merge_cpu_e2e: pytest exited 0
##[error]A pytest stage did not record a passing result. See this step's log for the stage.

After the xdist workers started, the pre_merge_cpu pytest stage produced no output for about 9.5 minutes. It then ended with the runner's k8s container-hook error, well before the step's 30-minute timeout. No pytest summary, FAILED lines or allure results were written. The later pre_merge_cpu_e2e stage in the same job passed (tests/serve/test_sample.py::test_sample_deployment[aggregated]). This looks like an infrastructure failure: the runner lost its exec into the test container, which can also happen when the pod is killed for running out of resources. Rerunning the failed job is likely enough. If it fails the same way again, look for a hang in that test set.

For agents
{"pr": 16470, "run_id": 40037879582, "run_attempt": 1, "head_sha": "4850957e28931fc1dd9aa1fe88f78b6d7410fd47",
 "failures": [{"job": "vllm-runtime / Test cuda13.0, arm64", "job_id": 120708683031, "failed_step": "Run CPU-only tests (parallelized)", "signature": "Executing the custom container implementation failed (runner container hook) during pre_merge_cpu pytest; no test results recorded", "tests": [], "log_cmd": "gh run view --job 120708683031 -R ai-dynamo/dynamo --log-failed"}]}

Posted automatically by Devin for run 40037879582. Updated on every full-CI run of this PR.

Vinya567 added a commit that referenced this pull request Oct 27, 2026
Overlays a TF32 TensorRT plan on top of a Dynamo frontend base that
carries the FE event-loop stall fixes (#16454 and #16470), unblocking
sidecar p99 on TF32 workloads.

Signed-off-by: Vinya567 <vinyak@nvidia.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — 4850957e Deployed Oct 23, 2026 by rmccorm4 via ok-to-test #38291
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contribution Pull request is from an external contributor perf size/L

Projects

None yet

2 participants