Repository navigation
fix(http): make /health status-only by default; list instances only on request - #16454
Conversation
…n request
The frontend /health handler listed every discovered instance and
serialized the list into the response on every request. On large
deployments that is a multi-megabyte body and tens of milliseconds of
work on the async runtime per readiness probe.
/health now returns {"status","stage"} by default. The endpoints and
instances lists are still available with the instances=true query
parameter. In-repo callers that read the lists (Python test helpers and
the kv_router bench) and the docs that describe the body are updated.
Closes ai-dynamo#16443
Signed-off-by: Suyang Xu <suyax@amazon.com>
|
Thanks for the contribution, please resolve the merge conflict |
|
/ok to test f4409aa |
✅ Dynamo PR CI passed — run 39848348994 (attempt 1) on
|
External PR triageBlocker ( Next action: @peter-xxxxx: merge or rebase onto the latest Automated triage, 2026-10-20. This comment is edited in place when the PR changes. |
Resolve health-check documentation and preserve the replacement router tests. Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
|
/ok to test 7601928 |
Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
|
/ok to test 2ec40a2 |
jthomson04
left a comment
There was a problem hiding this comment.
The change is right: a readiness probe shouldn't list discovery.
One correction for the description and #16443: the listing doesn't stall the whole frontend. The handler runs as a single ~40–47 ms poll with no yield point. Kubernetes list reads the in-memory cache, then json! and Json serialize 1.8 MB. So it occupies one tokio worker for that time. The frontend runtime is multi-threaded, with available_parallelism() workers by default, which follows the pod's CPU quota. The other workers keep serving. Tasks already queued on the busy worker wait until it finishes or another worker steals them, which is why this shows up in p99 rather than the median. Could you reword "every in-flight request on that frontend waits" and "every stall delayed the in-flight inference requests" accordingly? The 390/s stall figure counts one long poll per probe, not frontend-wide pauses.
Separately from this PR: if 12 extra /health calls/s moved p50 from 9 to 15 ms, your frontend pods may be running very few workers. The pod CPU limit or DYN_RUNTIME_NUM_WORKER_THREADS would be worth checking.
Non-blocking: the default response no longer includes endpoints or instances. A script that polls /health until endpoints appears (the pattern the old GPT-OSS guide used) will now spin until it times out instead of failing. Please make sure this is called out as a behavior change in the release notes.
This reverts commit b8940be. The pin is no longer needed: the next commit makes the /health checks request ?instances=true, which works on nightlies from before and after upstream ai-dynamo#16454 made /health status-only. Both gpu_tag defaults return to 'latest'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ecks Upstream ai-dynamo#16454 (a789bab, merged 2026-10-23) made the frontend's /health status-only -- {"status":"healthy","stage":"ready"} -- and moved the "endpoints"/"instances" lists behind ?instances=true. The 20261025-1ecc870 nightly behind :latest carries it, so the "Check Service endpoints" step in epd-test-template.yml and the bench pod in every test yaml counted 0 *.generate endpoints and each test failed after its 30-minute wait (run 40370917586), although the workers had registered. Every /health request that reads the endpoint list now asks for ?instances=true: .github/workflows/epd-test-template.yml kubectl get --raw via the apiserver service proxy (the query string is forwarded) .ci/test/dynamo-heterog/k8s/tests/*.yaml bench pod poll and the final dump on timeout, in all 7 tracked tests xpu-heterogeneous-run.yml gets a /health NOTE next to GPU_TAG recording the last nightly before the change (20261023-75d0686, sha256:65242c9b150f...) and the first one seen with it (20261025-1ecc870, sha256:97e5567273c9...). Checked on sc09super16-a100 with real vLLM prefill/decode workers over NIXL (Qwen3-0.6B) in both images: ?instances=true gives the CI count of 2 on each, and older nightlies ignore the parameter and still return the list, so pinning back to 20261023-75d0686 keeps working. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…us-only (#629) * fix(dynamo): fall back to /health?instances=true when /health is status-only Dynamo frontends since ai-dynamo/dynamo#16454 (a789babbe3) answer /health with {"status": "healthy", "stage": "ready"} and list registered instances only for /health?instances=true. srtctl counted workers from that list, so readiness never completed and the job waited until its health-check timeout. DynamoFrontend.probe_ready now reads /health as before and, when the body has no "instances" key, reads /health?instances=true. Older frontends still get a single /health request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynamo): describe the /health instance-listing probe Update the parse_health docstring and the architecture diagrams: Dynamo readiness reads the /health instance listing, asking /health?instances=true when /health is status-only, through DynamoFrontend.probe_ready rather than probe_json_health. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Move the srt-slurm submodule from v2.59.1 (ae3d6877) to v2.65.2 (59920ea1). v2.65.2 adds NVIDIA/srt-slurm#629: Dynamo frontends since ai-dynamo/dynamo#16454 answer /health status-only, so the readiness wait never counted any workers on Dynamo 1.7 images. DynamoFrontend.probe_ready now asks /health?instances=true only when /health has no instances key. 573-participating-gpus.patch still applies, and src/srtctl/core/power is unchanged between the pins, so it stays. Validation: infx generate for every config key writes 2023 recipes that the new srtctl validates, byte-identical to the v2.59.1 output; all patches apply to v2.65.2; infx.matrix.changed lists no config keys. Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Summary
The frontend
/healthhandler lists every instance visible through discovery and serializes that listinto the response on every request. Readiness probes and load balancers only read the status code, but
on a large deployment each probe still pays for the full listing on the async runtime. This PR makes
/healthstatus-only by default. The instance list stays available behind an explicitinstances=truequery parameter.Overview:
On a deployment with about 2,400 discovery instances, one
/healthcall returned a 1.8 MB body andspent 40–47 ms on the Tokio runtime. Load-balancer health checks and Kubernetes readiness probes both
pointed at
/health, so each frontend stalled its event loop about 4 times per second. Every stalldelayed the in-flight inference requests on that frontend.
The instance list was added on purpose (#1312, #2011), and tests and tooling read it. This PR keeps
it and makes it opt-in.
Details:
GET /healthwhile ready returns{"status":"healthy","stage":"ready"}, a 36-byte body.GET /health?instances=truereturns the previousendpointsandinstanceslists, plusstage.503not-ready response,/live, and route registration are unchanged.instancesis parsed as a boolean with axumQuery. A missing value meansfalse. A value otherthan
trueorfalsereturns400 Bad Request.endpointsorinstancesfrom the frontend/healthnow request?instances=true. These are the Python test helpers undertests/and the worker discovery inlib/llm/benches/kv_router_bench.rs.both response forms.
test_health_lists_instances_only_on_requestcovers both forms.Behavior change: a client that parses
endpointsorinstancesfrom the frontend/healthmustadd
?instances=true. Status codes are unchanged, so a probe that checks only the status code is notaffected.
Where should the reviewer start?
lib/llm/src/http/service/health.rsholds the handler change.lib/llm/src/http/service/service_v2.rsholds the new test.tests/utils/engine_process.pyandtests/fault_tolerance/migration/utils.pyare representativecaller updates. The other test files get the same one-line URL change.
Validation
These commands ran on Linux x86_64 with the repository toolchain, Rust 1.96.1. The tree was this
commit on top of
mainat4fe092a39d.cargo fmt -- --checkcargo clippy --locked -p dynamo-llm --all-targets --features kv-router-stress --no-deps -- -D warningskv-router-stressis enabled so the editedkv_router_benchis checkedcargo test --locked -p dynamo-llm --lib http::service::test_health_endpoint_reflects_draining_before_cancellationpre-commit run --files <the 19 changed files>python3 docs/fern/scripts/docs_lint.py --scan docsThe updated Python callers are GPU and end-to-end tests and were not run here. Each change is the
request URL only.
Production measurements on the deployment described above, which runs 100 frontends:
/healthThe stall figure was measured with the health checks moved off the listing
/health.Related Issues
/healthto add instances #1312 and feat(health): extend /health endpoint to include instances (#1312) #2011, which added the instance list🤖 Generated with Claude Code
Summary by CodeRabbit
/healthendpoint now returns readiness status and service stage by default, without listing discovered worker instances or endpoints.?instances=trueto include discovered instances and endpoints. Use this option when you need discovery details; the expanded response is not suitable for readiness probes.