Skip to content

perf(llm): copy-on-write worker-set membership in discovery::Model - #16119

Merged
jthomson04 merged 2 commits into
mainfrom
jthomson04/model-worker-sets-arcswap
Oct 12, 2026
Merged

jthomson04 merged 2 commits into
mainfrom
jthomson04/model-worker-sets-arcswap

Conversation

@jthomson04

@jthomson04 jthomson04 commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Overview:

discovery::Model stores its worker sets in a DashMap, and the request path iterates that map several times per request. Each DashMap iteration read-locks every shard. DashMap::new() sizes its shard count from the CPU count, which gives 2048 shards on a 352-CPU host. So every iteration does thousands of atomic read-lock operations on cache lines that all Tokio workers share.

Membership changes only on discovery events, so this PR makes it copy-on-write with ArcSwap<HashMap<String, Arc<WorkerSet>>>.

Details:

  • Readers load() the current map without writing shared state, then iterate it by reference.
  • Writers (add_worker_set, remove_worker_set) clone the map under a mutex, apply the change and publish the new map.
  • snapshot() now shares the current map in O(1) instead of copying it entry by entry. Later changes publish a new map, so a published snapshot stays isolated from them.
  • Consistency: readiness checks and worker-set selection each run against one loaded map, so a request sees a single consistent membership. Previously select_worker_set_with collected an explicit snapshot to get this.
  • Test: a snapshot is isolated from later membership changes.

Results. Frontend profiles on a 352-CPU, 2-socket Vera node: 352 Tokio workers, 2048 mock workers, concurrency 6144. The two builds differ only in this file. Where two numbers are shown, they come from the on-CPU and off-CPU profiling runs.

before after
DashMap, any frame, CPU-ms/request 6.41 0.17
DashMap leaf, % of frontend samples 12.0% 0.4%
discovery::model::Model, CPU-ms/request 6.26 0.01
arc_swap, CPU-ms/request 0.06 0.09
frontend CPU-ms/request 44.9 / 45.7 40.4 / 41.3
req/s 2724 / 2712 2713 / 2711
mean TTFT, ms 1409 / 1412 1423 / 1422

Frontend CPU drops by about 10%. Throughput and latency are unchanged at this load, because the frontend was not CPU-bound.

Where should the reviewer start?

lib/llm/src/discovery/model.rs: start with update_worker_sets, snapshot() and select_worker_set_with.

Related Issues:

  • None

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Improved consistency while namespace membership changes. Readiness checks, monitoring, and worker selection now use a coherent view of membership, helping prevent inconsistent results during updates.

Model.worker_sets was a DashMap iterated several times per request. Each
iteration read-locks every shard, and DashMap::new() sizes shards from the
CPU count (2048 on a 352-CPU host), so the request path spent ~12% of
frontend CPU in shard read-lock atomics.

Membership only changes on discovery events, so store it as
ArcSwap<HashMap<String, Arc<WorkerSet>>>: readers load the current map,
writers clone, modify and publish under a mutex, and snapshot() shares the
map in O(1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04
jthomson04 requested a review from a team as a code owner October 11, 2026 02:29
@github-actions github-actions Bot added the perf label Oct 11, 2026
@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 0decda55-6886-4488-9b20-e00d9bbcd801

📥 Commits

Reviewing files that changed from the base of the PR and between b208989 and d6bb510.


📒 Files selected for processing (1)
  • lib/llm/src/discovery/model.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.



Walkthrough

Model replaces DashMap membership storage with an ArcSwap-published HashMap. Membership updates publish cloned maps, and membership readers use loaded snapshots. Tests cover snapshot isolation and atomic visibility of concurrent updates.

Changes

Model Membership

Layer / File(s) Summary
Publish membership snapshots
lib/llm/src/discovery/model.rs
Model stores membership in an ArcSwap-published HashMap. A mutex serializes updates that clone and publish the map. Tests verify snapshot isolation and that readers do not observe partial paired updates.
Read membership snapshots
lib/llm/src/discovery/model.rs
Membership queries, readiness, displayability, monitoring, and worker selection read loaded maps. Readiness and worker selection use one map snapshot for related checks.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to d6bb5

No actionable issue is established for this change. It is mergeable after normal checks.

Pre-merge checks | Passed 3 | Failed 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 76.74% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 43 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check Warning The description includes the required overview, details, reviewer guidance, and performance results. However, the required Related Issues section contains "None" instead of an issue reference. Add a valid issue reference, such as "Closes #1234", "Relates to #1234", or a valid Linear issue reference. If no issue exists, create one before updating the description.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly and concisely describes the main change: replacing worker-set membership storage with copy-on-write behavior in discovery::Model.

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Comment thread lib/llm/src/discovery/model.rs Resolved
Comment thread lib/llm/src/discovery/model.rs Resolved
Comment thread lib/llm/src/discovery/model.rs Resolved
@devin-ai-integration

devin-ai-integration Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

❌ Dynamo PR CI failed — run 38277389779 (attempt 1) on 27ef2b8ee7

Gate checks: ❌ backend-status-check · ❌ deploy-status-check · ❌ dynamo-status-check

Other Jobs
dynamo-runtime ⏹️ 1 ✅ 7
frontend ❌ 1 ✅ 2

Failure details

1 job failed: the frontend ACR image copy could not reach the target registry (connection refused at login), an infrastructure error. The cancelled dynamo-runtime / rust-gpu job is not a fail-fast or manual cancel: the self-hosted runner's container implementation failed mid-build (Executing the custom container implementation failed), also infrastructure. A rerun of failed jobs is likely enough.

❌ frontend / Copy to ACR cuda, amd64: skopeo login to ACR — connection refused

Job: frontend / Copy to ACR cuda, amd64 · Failed step: Copy image to target registry · Logs: gh run view --job 114916211076 -R ai-dynamo/dynamo --log-failed

##[group]Run set -euo pipefail
echo "***" | skopeo login "***" --username "***" --password-stdin
##[endgroup]
time="2026-10-12T16:53:24Z" level=fatal msg="authenticating creds for \"***\": pinging container registry ***: Get \"https://***/v2/\": dial tcp 72.147.158.141:443: connect: connection refused"
##[error]Process completed with exit code 1.

The ECR source login succeeded, but the skopeo login to the Azure ACR target registry failed with a TCP connection refused, so the image copy was skipped. This is a registry/network infrastructure error unrelated to the code; a rerun of failed jobs is likely enough.

For agents
{"pr": 16119, "run_id": 38277389779, "run_attempt": 1, "head_sha": "27ef2b8ee7a56fcb5abc4dc0bb17b2481fd1b595", "failures": [{"job": "frontend / Copy to ACR cuda, amd64", "job_id": 114916211076, "failed_step": "Copy image to target registry", "signature": "skopeo login to ACR: connection refused (infrastructure)", "tests": [], "log_cmd": "gh run view --job 114916211076 -R ai-dynamo/dynamo --log-failed"}]}

Posted automatically by Devin for run 38277389779. Updated on every full-CI run of this PR.

Drop two comments that restate the code and the concurrent-readers test,
which only exercised paired updates through the private helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test 27ef2b8

@jthomson04
jthomson04 enabled auto-merge (squash) October 12, 2026 18:37
@jthomson04
jthomson04 merged commit c62addd into main Oct 12, 2026
196 of 203 checks passed
@jthomson04
jthomson04 deleted the jthomson04/model-worker-sets-arcswap branch October 12, 2026 18:40
NurullahMorshed added a commit to NurullahMorshed/dynamo that referenced this pull request Oct 12, 2026
Follow-up to the ported ai-dynamo#11826 commits for APIs that
changed on main and in vLLM after the PR was written:

- Import vLLM's ErrorResponse from vllm.entrypoints.serve.engine.protocol.
  vllm.entrypoints.openai.engine no longer exists in vLLM 0.31.0, so the
  old import made the whole guarded import block fail and the worker
  refuse to start. The test imports it lazily, as test_realtime_handler
  does, so pytest collection does not need vLLM's serving modules.
- Unpack the snapshot engine as main's SnapshotEngineSetupResult
  ((engine setup, StatLoggerFactory)), bind the restored stat logger
  factory to the endpoint and sync the FPM worker id, as the realtime
  worker does.
- Resolve --served-model-name aliases to the primary model in
  /v1/audio/transcriptions after the readiness gate, as /v1/audio/speech
  and /v1/rerank do on main, so alias requests route and are metered
  under the primary name. Add an HTTP test for it.
- Read worker sets through main's copy-on-write snapshot
  (`worker_sets.load().values()`) in `Model::has_transcriptions_engine`,
  as the other engine checks do since ai-dynamo#16119.

Signed-off-by: Nurullah Morshed <nurullah@inworld.ai>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NurullahMorshed added a commit to NurullahMorshed/dynamo that referenced this pull request Oct 15, 2026
Follow-up to the ported ai-dynamo#11826 commits for APIs that
changed on main and in vLLM after the PR was written:

- Import vLLM's ErrorResponse from vllm.entrypoints.serve.engine.protocol.
  vllm.entrypoints.openai.engine no longer exists in vLLM 0.31.0, so the
  old import made the whole guarded import block fail and the worker
  refuse to start. The test imports it lazily, as test_realtime_handler
  does, so pytest collection does not need vLLM's serving modules.
- Unpack the snapshot engine as main's SnapshotEngineSetupResult
  ((engine setup, StatLoggerFactory)), bind the restored stat logger
  factory to the endpoint and sync the FPM worker id, as the realtime
  worker does.
- Resolve --served-model-name aliases to the primary model in
  /v1/audio/transcriptions after the readiness gate, as /v1/audio/speech
  and /v1/rerank do on main, so alias requests route and are metered
  under the primary name. Add an HTTP test for it.
- Read worker sets through main's copy-on-write snapshot
  (`worker_sets.load().values()`) in `Model::has_transcriptions_engine`,
  as the other engine checks do since ai-dynamo#16119.

Signed-off-by: Nurullah Morshed <nurullah@inworld.ai>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants