Repository navigation
perf(llm): copy-on-write worker-set membership in discovery::Model - #16119
Conversation
Model.worker_sets was a DashMap iterated several times per request. Each iteration read-locks every shard, and DashMap::new() sizes shards from the CPU count (2048 on a 352-CPU host), so the request path spent ~12% of frontend CPU in shard read-lock atomics. Membership only changes on discovery events, so store it as ArcSwap<HashMap<String, Arc<WorkerSet>>>: readers load the current map, writers clone, modify and publish under a mutex, and snapshot() shares the map in O(1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
❌ Dynamo PR CI failed — run 38277389779 (attempt 1) on
|
| Other | Jobs |
|---|---|
| dynamo-runtime | ⏹️ 1 ✅ 7 |
| frontend | ❌ 1 ✅ 2 |
Failure details
1 job failed: the frontend ACR image copy could not reach the target registry (connection refused at login), an infrastructure error. The cancelled dynamo-runtime / rust-gpu job is not a fail-fast or manual cancel: the self-hosted runner's container implementation failed mid-build (Executing the custom container implementation failed), also infrastructure. A rerun of failed jobs is likely enough.
❌ frontend / Copy to ACR cuda, amd64: skopeo login to ACR — connection refused
Job: frontend / Copy to ACR cuda, amd64 · Failed step: Copy image to target registry · Logs: gh run view --job 114916211076 -R ai-dynamo/dynamo --log-failed
##[group]Run set -euo pipefail
echo "***" | skopeo login "***" --username "***" --password-stdin
##[endgroup]
time="2026-10-12T16:53:24Z" level=fatal msg="authenticating creds for \"***\": pinging container registry ***: Get \"https://***/v2/\": dial tcp 72.147.158.141:443: connect: connection refused"
##[error]Process completed with exit code 1.
The ECR source login succeeded, but the skopeo login to the Azure ACR target registry failed with a TCP connection refused, so the image copy was skipped. This is a registry/network infrastructure error unrelated to the code; a rerun of failed jobs is likely enough.
For agents
{"pr": 16119, "run_id": 38277389779, "run_attempt": 1, "head_sha": "27ef2b8ee7a56fcb5abc4dc0bb17b2481fd1b595", "failures": [{"job": "frontend / Copy to ACR cuda, amd64", "job_id": 114916211076, "failed_step": "Copy image to target registry", "signature": "skopeo login to ACR: connection refused (infrastructure)", "tests": [], "log_cmd": "gh run view --job 114916211076 -R ai-dynamo/dynamo --log-failed"}]}Posted automatically by Devin for run 38277389779. Updated on every full-CI run of this PR.
Drop two comments that restate the code and the concurrent-readers test, which only exercised paired updates through the private helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
/ok to test 27ef2b8 |
Follow-up to the ported ai-dynamo#11826 commits for APIs that changed on main and in vLLM after the PR was written: - Import vLLM's ErrorResponse from vllm.entrypoints.serve.engine.protocol. vllm.entrypoints.openai.engine no longer exists in vLLM 0.31.0, so the old import made the whole guarded import block fail and the worker refuse to start. The test imports it lazily, as test_realtime_handler does, so pytest collection does not need vLLM's serving modules. - Unpack the snapshot engine as main's SnapshotEngineSetupResult ((engine setup, StatLoggerFactory)), bind the restored stat logger factory to the endpoint and sync the FPM worker id, as the realtime worker does. - Resolve --served-model-name aliases to the primary model in /v1/audio/transcriptions after the readiness gate, as /v1/audio/speech and /v1/rerank do on main, so alias requests route and are metered under the primary name. Add an HTTP test for it. - Read worker sets through main's copy-on-write snapshot (`worker_sets.load().values()`) in `Model::has_transcriptions_engine`, as the other engine checks do since ai-dynamo#16119. Signed-off-by: Nurullah Morshed <nurullah@inworld.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow-up to the ported ai-dynamo#11826 commits for APIs that changed on main and in vLLM after the PR was written: - Import vLLM's ErrorResponse from vllm.entrypoints.serve.engine.protocol. vllm.entrypoints.openai.engine no longer exists in vLLM 0.31.0, so the old import made the whole guarded import block fail and the worker refuse to start. The test imports it lazily, as test_realtime_handler does, so pytest collection does not need vLLM's serving modules. - Unpack the snapshot engine as main's SnapshotEngineSetupResult ((engine setup, StatLoggerFactory)), bind the restored stat logger factory to the endpoint and sync the FPM worker id, as the realtime worker does. - Resolve --served-model-name aliases to the primary model in /v1/audio/transcriptions after the readiness gate, as /v1/audio/speech and /v1/rerank do on main, so alias requests route and are metered under the primary name. Add an HTTP test for it. - Read worker sets through main's copy-on-write snapshot (`worker_sets.load().values()`) in `Model::has_transcriptions_engine`, as the other engine checks do since ai-dynamo#16119. Signed-off-by: Nurullah Morshed <nurullah@inworld.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overview:
discovery::Modelstores its worker sets in aDashMap, and the request path iterates that map several times per request. EachDashMapiteration read-locks every shard.DashMap::new()sizes its shard count from the CPU count, which gives 2048 shards on a 352-CPU host. So every iteration does thousands of atomic read-lock operations on cache lines that all Tokio workers share.Membership changes only on discovery events, so this PR makes it copy-on-write with
ArcSwap<HashMap<String, Arc<WorkerSet>>>.Details:
load()the current map without writing shared state, then iterate it by reference.add_worker_set,remove_worker_set) clone the map under a mutex, apply the change and publish the new map.snapshot()now shares the current map in O(1) instead of copying it entry by entry. Later changes publish a new map, so a published snapshot stays isolated from them.select_worker_set_withcollected an explicit snapshot to get this.Results. Frontend profiles on a 352-CPU, 2-socket Vera node: 352 Tokio workers, 2048 mock workers, concurrency 6144. The two builds differ only in this file. Where two numbers are shown, they come from the on-CPU and off-CPU profiling runs.
discovery::model::Model, CPU-ms/requestarc_swap, CPU-ms/requestFrontend CPU drops by about 10%. Throughput and latency are unchanged at this load, because the frontend was not CPU-bound.
Where should the reviewer start?
lib/llm/src/discovery/model.rs: start withupdate_worker_sets,snapshot()andselect_worker_set_with.Related Issues:
🤖 Generated with Claude Code
Summary by CodeRabbit