Skip to content

fix(dynamo): request the instance list from the frontend /health - #627

Open
weireweire wants to merge 2 commits into
NVIDIA:mainfrom
weireweire:fix/dynamo-health-instances
Open

weireweire wants to merge 2 commits into
NVIDIA:mainfrom
weireweire:fix/dynamo-health-instances

Conversation

@weireweire

Copy link
Copy Markdown
Collaborator

Problem

ai-dynamo/dynamo#16454 (a789babbe3, on Dynamo main since 2026-10-24, so in 1.7.0 dev builds) made the frontend /health status-only by default. While ready it now returns {"status":"healthy","stage":"ready"}; the endpoints and instances lists come back only with ?instances=true. Upstream did this because every probe paid for a full discovery listing (a 1.8 MB body and 40-47 ms on the Tokio runtime at about 2,400 instances).

srtctl's readiness wait counts registered generate instances from that instances list (check_dynamo_health). On a Dynamo 1.7.0 dev build (top_of_tree, a git rev after that commit, or a nightly wheel), it never finds the key and polls until the health gate times out, although every worker is up:

Key 'instances' not found in response: {'status': 'healthy', 'stage': 'ready'}

Changes

  • DynamoFrontend.health_endpoint returns /health?instances=true. The base DynamicFrontend keeps /health, so other frontends are unaffected.
  • The property test expects the new path.
  • docs/architecture.md, docs/shadow-engine-recovery.md and examples/features/worker-restart.yaml say to watch the frontend's instance ids on /health?instances=true, since plain /health no longer shows them on newer Dynamo.

Compatibility: every release from v0.5.0 through v1.6.1 has a /health handler that takes no query extractor (lib/llm/src/http/service/health.rs), so older images ignore the parameter and keep returning the list. On main, instances parses as a boolean and true is accepted. Only the startup readiness probe reads this body. The per-worker system-status /health (DYN_SYSTEM_PORT) was not touched upstream and is still probed by status code only.

Validation

  • A disaggregated job (10 prefill workers, 1 DEP16 decode worker) on an image shipping Dynamo 1.7.0.dev20261026 hung in the health wait before this change. With it, the wait counted 10 prefill and 16 decode generate instances and the job proceeded to the benchmark.
  • ruff check and ruff format --check pass on the changed Python file.

Dynamo main (1.7.0-dev) since ai-dynamo/dynamo#16454 (a789babbe3) makes
the frontend /health status-only by default and lists `instances` only
with `?instances=true`. The readiness wait counts registered generate
instances from that list, so on a 1.7.0 dev build (top_of_tree, a git
rev after that commit, or a nightly wheel) it never finds them and polls
"Key 'instances' not found" until the health gate times out, although
every worker is up.

DynamoFrontend now probes /health?instances=true. Every release from
v0.5.0 through v1.6.1 has a /health handler without a query extractor,
so older images ignore the parameter and keep returning the list. Only
the startup readiness probe reads this body; the per-worker system
status /health is unchanged upstream. The docs that tell users to watch
the frontend's instance ids now use the same URL.
@codecov-commenter

codecov-commenter commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.10%. Comparing base (14c3630) to head (6464249).
⚠️ Report is 25 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #627      +/-   ##
==========================================
+ Coverage   85.45%   86.10%   +0.64%     
==========================================
  Files         165      165              
  Lines       24436    25004     +568     
==========================================
+ Hits        20883    21529     +646     
+ Misses       3553     3475      -78     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants