Repository navigation
fix(dynamo): request the instance list from the frontend /health - #627
Open
weireweire wants to merge 2 commits into
Open
weireweire wants to merge 2 commits into
weireweire wants to merge 2 commits into
Conversation
Dynamo main (1.7.0-dev) since ai-dynamo/dynamo#16454 (a789babbe3) makes the frontend /health status-only by default and lists `instances` only with `?instances=true`. The readiness wait counts registered generate instances from that list, so on a 1.7.0 dev build (top_of_tree, a git rev after that commit, or a nightly wheel) it never finds them and polls "Key 'instances' not found" until the health gate times out, although every worker is up. DynamoFrontend now probes /health?instances=true. Every release from v0.5.0 through v1.6.1 has a /health handler without a query extractor, so older images ignore the parameter and keep returning the list. Only the startup readiness probe reads this body; the per-worker system status /health is unchanged upstream. The docs that tell users to watch the frontend's instance ids now use the same URL.
weireweire
requested review from
alec-flowers,
csahithi,
hjjq,
ishandhanani,
kedarpotdar-nv,
kyleliang-nv,
nlevin-ui and
qiching
as code owners
October 27, 2026 11:23
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #627 +/- ##
==========================================
+ Coverage 85.45% 86.10% +0.64%
==========================================
Files 165 165
Lines 24436 25004 +568
==========================================
+ Hits 20883 21529 +646
+ Misses 3553 3475 -78 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…tances # Conflicts: # docs/architecture.md
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ai-dynamo/dynamo#16454 (a789babbe3, on Dynamo main since 2026-10-24, so in 1.7.0 dev builds) made the frontend
/healthstatus-only by default. While ready it now returns{"status":"healthy","stage":"ready"}; theendpointsandinstanceslists come back only with?instances=true. Upstream did this because every probe paid for a full discovery listing (a 1.8 MB body and 40-47 ms on the Tokio runtime at about 2,400 instances).srtctl's readiness wait counts registered generate instances from that
instanceslist (check_dynamo_health). On a Dynamo 1.7.0 dev build (top_of_tree, a git rev after that commit, or a nightly wheel), it never finds the key and polls until the health gate times out, although every worker is up:Changes
DynamoFrontend.health_endpointreturns/health?instances=true. The baseDynamicFrontendkeeps/health, so other frontends are unaffected.docs/architecture.md,docs/shadow-engine-recovery.mdandexamples/features/worker-restart.yamlsay to watch the frontend's instance ids on/health?instances=true, since plain/healthno longer shows them on newer Dynamo.Compatibility: every release from v0.5.0 through v1.6.1 has a
/healthhandler that takes no query extractor (lib/llm/src/http/service/health.rs), so older images ignore the parameter and keep returning the list. On main,instancesparses as a boolean andtrueis accepted. Only the startup readiness probe reads this body. The per-worker system-status/health(DYN_SYSTEM_PORT) was not touched upstream and is still probed by status code only.Validation
ruff checkandruff format --checkpass on the changed Python file.