Skip to content

fix(http): make /health status-only by default; list instances only on request - #16454

Merged
rmccorm4 merged 3 commits into
ai-dynamo:mainfrom
peter-xxxxx:suyax/health-status-only
Oct 23, 2026
Merged

rmccorm4 merged 3 commits into
ai-dynamo:mainfrom
peter-xxxxx:suyax/health-status-only

Conversation

@peter-xxxxx

@peter-xxxxx peter-xxxxx commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The frontend /health handler lists every instance visible through discovery and serializes that list
into the response on every request. Readiness probes and load balancers only read the status code, but
on a large deployment each probe still pays for the full listing on the async runtime. This PR makes
/health status-only by default. The instance list stays available behind an explicit
instances=true query parameter.

Overview:

On a deployment with about 2,400 discovery instances, one /health call returned a 1.8 MB body and
spent 40–47 ms on the Tokio runtime. Load-balancer health checks and Kubernetes readiness probes both
pointed at /health, so each frontend stalled its event loop about 4 times per second. Every stall
delayed the in-flight inference requests on that frontend.

The instance list was added on purpose (#1312, #2011), and tests and tooling read it. This PR keeps
it and makes it opt-in.

Details:

  • GET /health while ready returns {"status":"healthy","stage":"ready"}, a 36-byte body.
  • GET /health?instances=true returns the previous endpoints and instances lists, plus stage.
  • The 503 not-ready response, /live, and route registration are unchanged.
  • instances is parsed as a boolean with axum Query. A missing value means false. A value other
    than true or false returns 400 Bad Request.
  • In-repo callers that read endpoints or instances from the frontend /health now request
    ?instances=true. These are the Python test helpers under tests/ and the worker discovery in
    lib/llm/benches/kv_router_bench.rs.
  • The health check reference, the CLI observability guide and the GPT-OSS TensorRT-LLM guide now show
    both response forms.
  • The new unit test test_health_lists_instances_only_on_request covers both forms.

Behavior change: a client that parses endpoints or instances from the frontend /health must
add ?instances=true. Status codes are unchanged, so a probe that checks only the status code is not
affected.

Where should the reviewer start?

  • lib/llm/src/http/service/health.rs holds the handler change.
  • lib/llm/src/http/service/service_v2.rs holds the new test.
  • tests/utils/engine_process.py and tests/fault_tolerance/migration/utils.py are representative
    caller updates. The other test files get the same one-line URL change.

Validation

These commands ran on Linux x86_64 with the repository toolchain, Rust 1.96.1. The tree was this
commit on top of main at 4fe092a39d.

Command Result
cargo fmt -- --check Pass
cargo clippy --locked -p dynamo-llm --all-targets --features kv-router-stress --no-deps -- -D warnings Pass. kv-router-stress is enabled so the edited kv_router_bench is checked
cargo test --locked -p dynamo-llm --lib http::service:: 414 passed, 0 failed. Includes the new test and test_health_endpoint_reflects_draining_before_cancellation
pre-commit run --files <the 19 changed files> All hooks pass and no file is modified
python3 docs/fern/scripts/docs_lint.py --scan docs 0 errors. The 9 warnings are in files this PR does not touch

The updated Python callers are GPU and end-to-end tests and were not run here. Each change is the
request URL only.

Production measurements on the deployment described above, which runs 100 frontends:

Measurement Listing /health Status-only
Response body 1.8 MB 36 bytes
Handler time 40–47 ms 0.5 ms
Event-loop stalls across all frontends 390/s 12/s

The stall figure was measured with the health checks moved off the listing /health.

Related Issues

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Updates
    • The frontend /health endpoint now returns readiness status and service stage by default, without listing discovered worker instances or endpoints.
    • Add ?instances=true to include discovered instances and endpoints. Use this option when you need discovery details; the expanded response is not suitable for readiness probes.
    • Updated health-check guidance to clarify the distinction between readiness checks and discovery.

…n request

The frontend /health handler listed every discovered instance and
serialized the list into the response on every request. On large
deployments that is a multi-megabyte body and tens of milliseconds of
work on the async runtime per readiness probe.

/health now returns {"status","stage"} by default. The endpoints and
instances lists are still available with the instances=true query
parameter. In-repo callers that read the lists (Python test helpers and
the kv_router bench) and the docs that describe the body are updated.

Closes ai-dynamo#16443

Signed-off-by: Suyang Xu <suyax@amazon.com>
@peter-xxxxx
peter-xxxxx requested review from a team as code owners October 14, 2026 22:26
@copy-pr-bot

copy-pr-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@peter-xxxxx
peter-xxxxx temporarily deployed to external_collaborator October 14, 2026 22:26 — with GitHub Actions Inactive
@peter-xxxxx
peter-xxxxx temporarily deployed to external_collaborator October 14, 2026 22:26 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot added fix external-contribution Pull request is from an external contributor documentation Improvements or additions to documentation frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Oct 14, 2026
@coderabbitai

coderabbitai Bot commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 37bb5771-81f8-4de5-9111-24e4ef4fb043
📥 Commits

Reviewing files that changed from the base of the PR and between 4fe092a and f4409aa.

📒 Files selected for processing: 19
  • docs/fern/pages/cli/operations/observability.mdx
  • docs/fern/pages/developer-guide/additional-resources/tensorrt-llm-details/gpt-oss.md
  • docs/fern/pages/reference/observability/health-checks.mdx
  • lib/llm/benches/kv_router_bench.rs
  • lib/llm/src/http/service/health.rs
  • lib/llm/src/http/service/service_v2.rs
  • tests/fault_tolerance/cancellation/test_sglang.py
  • tests/fault_tolerance/cancellation/test_trtllm.py
  • tests/fault_tolerance/cancellation/test_vllm.py
  • tests/fault_tolerance/etcd_ha/test_sglang.py
  • tests/fault_tolerance/etcd_ha/test_trtllm.py
  • tests/fault_tolerance/etcd_ha/test_vllm.py
  • tests/fault_tolerance/migration/test_sglang.py
  • tests/fault_tolerance/migration/utils.py
  • tests/gpu_memory_service/common/runtime.py
  • tests/kvbm_integration/test_consolidator_router_e2e.py
  • tests/router/common.py
  • tests/utils/engine_process.py
  • tests/vllm_self_benchmark/test_self_benchmark_gpu.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The frontend /health endpoint now returns status and stage without discovery details by default. Requests with ?instances=true include discovered endpoints and instances. Callers that rely on those details now request them explicitly.

Changes

Health endpoint and consumers

Layer / File(s) Summary
Health response and documentation
lib/llm/src/http/service/health.rs, lib/llm/src/http/service/service_v2.rs, docs/fern/pages/cli/operations/observability.mdx, docs/fern/pages/reference/observability/health-checks.mdx, docs/fern/pages/developer-guide/additional-resources/tensorrt-llm-details/gpt-oss.md
The handler returns status and stage by default. ?instances=true adds endpoint and instance details. A test checks both responses, and the documentation shows the query and response fields.
Callers request discovery details
lib/llm/benches/kv_router_bench.rs, tests/fault_tolerance/*, tests/gpu_memory_service/common/runtime.py, tests/kvbm_integration/test_consolidator_router_e2e.py, tests/router/common.py, tests/utils/engine_process.py, tests/vllm_self_benchmark/test_self_benchmark_gpu.py
Benchmarks, health checks, and polling helpers that inspect frontend instances now request /health?instances=true. Other health checks remain unchanged.

Priority: ⬆️ High

Estimated code review effort: 3 (Moderate) | ~20 minutes

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to f4409

The default frontend health response avoids discovery-listing work, while callers that need instance details request them explicitly. No actionable merge-blocking risk remains.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 21 functions across 16 files. (3 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: /health is status-only by default and lists instances only when requested.
Description check Passed The description includes all required sections, explains the behavior change and rationale, identifies reviewer starting points, links related issues, and documents validation results.
Linked Issues check Passed Direct issue #16443 requires frontend /health probes to avoid listing and serializing all discovery instances while preserving diagnostic access. lib/llm/src/http/service/health.rs makes the insta…
Out of Scope Changes check Passed The changed files stay connected to #16443. Caller updates preserve consumers that parse endpoints or instances. Documentation updates describe the new request forms. Test updates preserve instanc…
Full details: Docstring Coverage

Explanation

Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 21 functions across 16 files. (3 skipped: 3 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches: 💡 1
⚔️ Resolve merge conflicts: 💡
  • Resolve merge conflict in branch suyax/health-status-only
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@jh-nv

jh-nv commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Thanks for the contribution, please resolve the merge conflict

@jh-nv

jh-nv commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

/ok to test f4409aa

@devin-ai-integration

devin-ai-integration Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 39848348994 (attempt 1) on 2ec40a27a9

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 39848348994. Updated on every full-CI run of this PR.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

External PR triage

Blocker (merge-conflict): The PR has approvals from @tmonty12 (10-19) and @jh-nv (10-20), and trusted CI passed on f4409aa4. It cannot merge because it conflicts with main (GitHub reports dirty). @jh-nv asked for the conflict to be resolved on 10-19.

Next action: @peter-xxxxx: merge or rebase onto the latest main, resolve the conflicts, and push. After that, @jh-nv or @tmonty12 should comment /ok to test <new head sha>, because the trusted-CI approval is per commit, and re-approve if the push dismissed the reviews. The next triage run will audit the new head before that approval.

Automated triage, 2026-10-20. This comment is edited in place when the PR changes.

Written by Devin

Resolve health-check documentation and preserve the replacement router tests.

Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
@rmccorm4
rmccorm4 temporarily deployed to external_collaborator October 22, 2026 19:28 — with GitHub Actions Inactive
@rmccorm4

Copy link
Copy Markdown
Contributor

/ok to test 7601928

Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
@rmccorm4
rmccorm4 temporarily deployed to external_collaborator October 22, 2026 20:20 — with GitHub Actions Inactive
@rmccorm4

Copy link
Copy Markdown
Contributor

/ok to test 2ec40a2

@rmccorm4
rmccorm4 enabled auto-merge (squash) October 23, 2026 00:50

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change is right: a readiness probe shouldn't list discovery.

One correction for the description and #16443: the listing doesn't stall the whole frontend. The handler runs as a single ~40–47 ms poll with no yield point. Kubernetes list reads the in-memory cache, then json! and Json serialize 1.8 MB. So it occupies one tokio worker for that time. The frontend runtime is multi-threaded, with available_parallelism() workers by default, which follows the pod's CPU quota. The other workers keep serving. Tasks already queued on the busy worker wait until it finishes or another worker steals them, which is why this shows up in p99 rather than the median. Could you reword "every in-flight request on that frontend waits" and "every stall delayed the in-flight inference requests" accordingly? The 390/s stall figure counts one long poll per probe, not frontend-wide pauses.

Separately from this PR: if 12 extra /health calls/s moved p50 from 9 to 15 ms, your frontend pods may be running very few workers. The pod CPU limit or DYN_RUNTIME_NUM_WORKER_THREADS would be worth checking.

Non-blocking: the default response no longer includes endpoints or instances. A script that polls /health until endpoints appears (the pattern the old GPT-OSS guide used) will now spin until it times out instead of failing. Please make sure this is called out as a behavior change in the release notes.

@rmccorm4
rmccorm4 merged commit a789bab into ai-dynamo:main Oct 23, 2026
116 of 117 checks passed
aung-san-i added a commit to aung-san-i/dynamo that referenced this pull request Oct 26, 2026
This reverts commit b8940be.

The pin is no longer needed: the next commit makes the /health checks
request ?instances=true, which works on nightlies from before and after
upstream ai-dynamo#16454 made /health status-only. Both gpu_tag
defaults return to 'latest'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aung-san-i added a commit to aung-san-i/dynamo that referenced this pull request Oct 26, 2026
…ecks

Upstream ai-dynamo#16454 (a789bab, merged 2026-10-23) made the
frontend's /health status-only -- {"status":"healthy","stage":"ready"} --
and moved the "endpoints"/"instances" lists behind ?instances=true. The
20261025-1ecc870 nightly behind :latest carries it, so the "Check Service
endpoints" step in epd-test-template.yml and the bench pod in every test
yaml counted 0 *.generate endpoints and each test failed after its
30-minute wait (run 40370917586), although the workers had registered.

Every /health request that reads the endpoint list now asks for
?instances=true:

  .github/workflows/epd-test-template.yml   kubectl get --raw via the
      apiserver service proxy (the query string is forwarded)
  .ci/test/dynamo-heterog/k8s/tests/*.yaml   bench pod poll and the
      final dump on timeout, in all 7 tracked tests

xpu-heterogeneous-run.yml gets a /health NOTE next to GPU_TAG recording
the last nightly before the change (20261023-75d0686,
sha256:65242c9b150f...) and the first one seen with it
(20261025-1ecc870, sha256:97e5567273c9...).

Checked on sc09super16-a100 with real vLLM prefill/decode workers over
NIXL (Qwen3-0.6B) in both images: ?instances=true gives the CI count of
2 on each, and older nightlies ignore the parameter and still return the
list, so pinning back to 20261023-75d0686 keeps working.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vinya567 added a commit that referenced this pull request Oct 27, 2026
Overlays a TF32 TensorRT plan on top of a Dynamo frontend base that
carries the FE event-loop stall fixes (#16454 and #16470), unblocking
sidecar p99 on TF32 workloads.

Signed-off-by: Vinya567 <vinyak@nvidia.com>
richardhuo-nv added a commit to NVIDIA/srt-slurm that referenced this pull request Oct 27, 2026
…us-only (#629)

* fix(dynamo): fall back to /health?instances=true when /health is status-only

Dynamo frontends since ai-dynamo/dynamo#16454 (a789babbe3) answer /health
with {"status": "healthy", "stage": "ready"} and list registered instances
only for /health?instances=true. srtctl counted workers from that list, so
readiness never completed and the job waited until its health-check timeout.

DynamoFrontend.probe_ready now reads /health as before and, when the body
has no "instances" key, reads /health?instances=true. Older frontends still
get a single /health request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(dynamo): describe the /health instance-listing probe

Update the parse_health docstring and the architecture diagrams: Dynamo
readiness reads the /health instance listing, asking /health?instances=true
when /health is status-only, through DynamoFrontend.probe_ready rather than
probe_json_health.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Oseltamivir pushed a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 29, 2026
Move the srt-slurm submodule from v2.59.1 (ae3d6877) to v2.65.2
(59920ea1). v2.65.2 adds NVIDIA/srt-slurm#629: Dynamo frontends since
ai-dynamo/dynamo#16454 answer /health status-only, so the readiness wait
never counted any workers on Dynamo 1.7 images. DynamoFrontend.probe_ready
now asks /health?instances=true only when /health has no instances key.

573-participating-gpus.patch still applies, and src/srtctl/core/power is
unchanged between the pins, so it stays.

Validation: infx generate for every config key writes 2023 recipes that
the new srtctl validates, byte-identical to the v2.59.1 output; all
patches apply to v2.65.2; infx.matrix.changed lists no config keys.

Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — 2ec40a27 Deployed Oct 22, 2026 by rmccorm4 via ok-to-test #37482
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: frontend GET /health lists and serialises every discovery instance in the cell (40–47 ms on the event loop per probe)

6 participants