Repository navigation
feat(planner): add runtime-configurable minimum endpoints - #13038
Conversation
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
WalkthroughThe planner now supports independent prefill and decode endpoint floors, legacy fallback behavior, and a loopback runtime API for atomic updates. Budget, scaling, replay, orchestration, tests, examples, and documentation use the new settings. ChangesPlanner endpoint floors
Estimated code review effort: 4 (Complex) | ~60 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (6)
components/src/dynamo/planner/tests/unit/test_control_api.py (1)
109-110: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMove the
aiohttp.test_utilsimport to module scope.
aiohttpis required by these tests. ImportTestClientandTestServeronce with the other third-party imports.
components/src/dynamo/planner/tests/unit/test_control_api.py#L109-L110: Remove the local import.components/src/dynamo/planner/tests/unit/test_control_api.py#L154-L154: Remove the duplicate local import.Proposed change
from unittest.mock import AsyncMock, patch +from aiohttp.test_utils import TestClient, TestServer import pytest @@ - from aiohttp.test_utils import TestClient, TestServer @@ - from aiohttp.test_utils import TestClient, TestServerAs per coding guidelines, “Keep imports at the top of the file.” As per path instructions,
.ai/python-guidelines.mdrequires imports at module scope.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/src/dynamo/planner/tests/unit/test_control_api.py` around lines 109 - 110, Move the aiohttp TestClient and TestServer import to module scope alongside the other third-party imports in components/src/dynamo/planner/tests/unit/test_control_api.py (anchor lines 109-110). Remove both local or duplicate imports at lines 109-110 and 154-154, leaving the tests to reuse the single module-level import.Sources: Coding guidelines, Path instructions
components/src/dynamo/planner/environment/base.py (1)
457-463: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winReuse this helper for the runtime budget check.
minimum_power_footprint_fitsimplements the startup power-footprint rule. The new runtime path incomponents/src/dynamo/planner/core/base.pyLines 236-265 re-implements the same rule inline: it sumsprefill_min_endpoint * p_wattsanddecode_min_endpoint * d_wattsand compares the total againsttotal_gpu_power_limit.Two implementations of one rule will drift. A change to the tolerance, the rounding, or the
None-watts policy in one place will not reach the other, and aPATCHcould then accept a floor that startup rejects.Call
minimum_power_footprint_fitsfrom the runtime check as well.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/src/dynamo/planner/environment/base.py` around lines 457 - 463, Update the runtime budget check in the core planner path to call minimum_power_footprint_fits instead of duplicating the power-footprint calculation and comparison inline. Pass the runtime budget, prefill_min_endpoint, decode_min_endpoint, p_watts, and d_watts arguments so runtime validation uses the same rule as the startup check.components/src/dynamo/planner/control_api.py (1)
45-51: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winNarrow the exception caught around
request.json().
except Exceptionconverts every failure into a 400.request.json()raisesjson.JSONDecodeErrorfor a malformed body andaiohttp.ContentTypeErrorfor a wrongContent-Type. It also propagates transport errors when the client disconnects mid-body. The current handler reports a disconnect as "request body must be valid JSON", which hides the real cause from the logs.Catch only the decode and content-type errors.
As per coding guidelines: "Prefer failing fast over hiding errors; remove try/except blocks where possible, catch only specific exceptions".
♻️ Proposed refactor
+import json ... async def handle_patch(request: web.Request) -> web.Response: try: payload = await request.json() - except Exception: + except (json.JSONDecodeError, web.HTTPException, ValueError): return _error("request body must be valid JSON", 400)
aiohttp.ContentTypeErrorsubclassesValueError, so the tuple above covers a wrongContent-Typeas well. Confirm the exact aiohttp 3.9 exception hierarchy before you finalize the tuple.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/src/dynamo/planner/control_api.py` around lines 45 - 51, Update handle_patch to catch only the specific JSON parsing and content-type exceptions raised by request.json(), including json.JSONDecodeError and the appropriate aiohttp.ContentTypeError hierarchy for the supported aiohttp version. Preserve the existing 400 response for malformed JSON or incorrect content types, while allowing transport and client-disconnect errors to propagate.Sources: Coding guidelines, Linters/SAST tools
components/src/dynamo/planner/config/defaults.py (1)
40-41: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winConsider defaulting
control_api_portto0.The control API is unauthenticated. It accepts
PATCHrequests that change scaling floors. The bind address127.0.0.1limits access to the pod network namespace, but any sidecar, ephemeral debug container, or process running in that pod can change planner scaling behavior without credentials. A default of0(disabled) makes the surface opt-in and keeps existing deployments unchanged.If default-on is intentional, document the pod-local trust assumption next to this default.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/src/dynamo/planner/config/defaults.py` around lines 40 - 41, Change the default value of control_api_port in the configuration defaults to 0 so the unauthenticated control API is disabled unless explicitly enabled, preserving existing opt-in behavior through configured nonzero values.components/src/dynamo/planner/offline/replay_adapter.py (1)
486-486: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winReplace the index slice with an explicit predicate.
excluded_fields[:6]selects the flat top-level fields and leaves the dotted paths for the structural handling below. The count is correct after this change: indices 0-5 are the six flat entries.The coupling is fragile. A contributor who adds a flat field after index 5, or who adds one without bumping the slice, gets no error. The result is a silently wrong
planner_config_digest, which is the replay identity contract.Select by shape instead of by position.
♻️ Proposed refactor
- for field in excluded_fields[:6]: + # Flat top-level fields only; dotted paths need the structural + # handling below and must not be popped here. + for field in (f for f in excluded_fields if "." not in f): parent = decision_config parts = field.split(".")The
partssplit then always yields one element, so the inner loop becomes a no-op and can be removed in a follow-up.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/src/dynamo/planner/offline/replay_adapter.py` at line 486, Replace the positional slice in the excluded_fields loop with an explicit predicate that selects only flat top-level field names (those without a dot), while leaving dotted paths for the structural handling below. Preserve the existing planner_config_digest replay identity behavior without relying on field ordering or a fixed count.components/src/dynamo/planner/config/planner_config.py (1)
44-50: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
resolve_min_endpointis untyped at both parameters. The shared root cause is the signatureresolve_min_endpoint(config: object, component: str). Theobjectannotation forces every field read throughgetattr, and the barestrlets any value other than"prefill"resolve to the decode floor. One signature change fixes both.
components/src/dynamo/planner/config/planner_config.py#L44-L50: annotateconfigwith aProtocolthat declaresmin_endpoint,prefill_min_endpoint, anddecode_min_endpoint, then replace the threegetattrcalls with direct attribute access.components/src/dynamo/planner/config/planner_config.py#L47-L47: annotatecomponentasLiteral["prefill", "decode"]so a misspelled role fails type checking instead of silently returning the decode floor.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/src/dynamo/planner/config/planner_config.py` around lines 44 - 50, Update components/src/dynamo/planner/config/planner_config.py lines 44-50 in resolve_min_endpoint: define and use a Protocol declaring min_endpoint, prefill_min_endpoint, and decode_min_endpoint, then access those attributes directly instead of getattr. Annotate component at lines 47-47 as Literal["prefill", "decode"] so invalid roles fail type checking; update the config parameter to use the Protocol.Sources: Coding guidelines, Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/src/dynamo/planner/core/base.py`:
- Around line 172-175: Make planner diagnostics finalization idempotent in
_shutdown_runtime: initialize _diagnostics_finalized alongside the other state
fields, and only call self._recorder.finalize() when the flag is false, setting
it before or during the first finalization attempt. Preserve the existing
exception logging while ensuring repeated shutdown calls do not write the report
again.
- Around line 819-820: Bound lock acquisition in both public methods
get_min_endpoints and patch_min_endpoints using a timeout while waiting for
self._config_lock. On timeout, raise a distinct retryable exception and map it
to HTTP 503 in control_api.py; preserve the existing validation error behavior
and next-stable-tick semantics when the lock is acquired.
In
`@docs/fern/pages/developer-guide/knowledge-base/modular-components/planner/planner-guide.md`:
- Line 214: Update the documentation prose in the planner guide to use
“built-in” instead of “builtin” when describing targets, while preserving
identifiers such as builtin_load_propose unchanged.
---
Nitpick comments:
In `@components/src/dynamo/planner/config/defaults.py`:
- Around line 40-41: Change the default value of control_api_port in the
configuration defaults to 0 so the unauthenticated control API is disabled
unless explicitly enabled, preserving existing opt-in behavior through
configured nonzero values.
In `@components/src/dynamo/planner/config/planner_config.py`:
- Around line 44-50: Update
components/src/dynamo/planner/config/planner_config.py lines 44-50 in
resolve_min_endpoint: define and use a Protocol declaring min_endpoint,
prefill_min_endpoint, and decode_min_endpoint, then access those attributes
directly instead of getattr. Annotate component at lines 47-47 as
Literal["prefill", "decode"] so invalid roles fail type checking; update the
config parameter to use the Protocol.
In `@components/src/dynamo/planner/control_api.py`:
- Around line 45-51: Update handle_patch to catch only the specific JSON parsing
and content-type exceptions raised by request.json(), including
json.JSONDecodeError and the appropriate aiohttp.ContentTypeError hierarchy for
the supported aiohttp version. Preserve the existing 400 response for malformed
JSON or incorrect content types, while allowing transport and client-disconnect
errors to propagate.
In `@components/src/dynamo/planner/environment/base.py`:
- Around line 457-463: Update the runtime budget check in the core planner path
to call minimum_power_footprint_fits instead of duplicating the power-footprint
calculation and comparison inline. Pass the runtime budget,
prefill_min_endpoint, decode_min_endpoint, p_watts, and d_watts arguments so
runtime validation uses the same rule as the startup check.
In `@components/src/dynamo/planner/offline/replay_adapter.py`:
- Line 486: Replace the positional slice in the excluded_fields loop with an
explicit predicate that selects only flat top-level field names (those without a
dot), while leaving dotted paths for the structural handling below. Preserve the
existing planner_config_digest replay identity behavior without relying on field
ordering or a fixed count.
In `@components/src/dynamo/planner/tests/unit/test_control_api.py`:
- Around line 109-110: Move the aiohttp TestClient and TestServer import to
module scope alongside the other third-party imports in
components/src/dynamo/planner/tests/unit/test_control_api.py (anchor lines
109-110). Remove both local or duplicate imports at lines 109-110 and 154-154,
leaving the tests to reuse the single module-level import.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9efb9d19-d23f-4705-9f39-f683320d74a7
📒 Files selected for processing (31)
aisimulate/src/aisimulate/sweeper/config.pyaisimulate/tests/sweeper/test_config.pycomponents/src/dynamo/planner/config/defaults.pycomponents/src/dynamo/planner/config/planner_config.pycomponents/src/dynamo/planner/control_api.pycomponents/src/dynamo/planner/core/base.pycomponents/src/dynamo/planner/core/budget.pycomponents/src/dynamo/planner/core/load_scaling.pycomponents/src/dynamo/planner/core/state_machine.pycomponents/src/dynamo/planner/core/throughput_scaling.pycomponents/src/dynamo/planner/environment/base.pycomponents/src/dynamo/planner/offline/replay_adapter.pycomponents/src/dynamo/planner/plugins/orchestrator/engine_adapter.pycomponents/src/dynamo/planner/simulation/provider.pycomponents/src/dynamo/planner/tests/offline/test_replay_adapter_fpm.pycomponents/src/dynamo/planner/tests/plugins/orchestrator/test_engine_adapter.pycomponents/src/dynamo/planner/tests/unit/test_budget.pycomponents/src/dynamo/planner/tests/unit/test_control_api.pycomponents/src/dynamo/planner/tests/unit/test_planner_config.pycomponents/src/dynamo/planner/tests/unit/test_planner_sweep_config_provider.pycomponents/src/dynamo/planner/tests/unit/test_power_budget.pycomponents/src/dynamo/planner/tests/unit/test_throughput_scaling.pydeploy/operator/config/samples/nvidia.com_v1beta1_dynamographdeploymentrequest.yamldocs/fern/pages/developer-guide/knowledge-base/modular-components/planner/overview.mddocs/fern/pages/developer-guide/knowledge-base/modular-components/planner/planner-design.mddocs/fern/pages/developer-guide/knowledge-base/modular-components/planner/planner-guide.mddocs/fern/pages/kubernetes/auto-deployment/dynamo-planner.mdxdocs/fern/pages/reference/components/planner-configuration.mdxdocs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment-request.mdxdocs/fern/translations/zh-CN/pages/developer-guide/knowledge-base/modular-components/planner/planner-design.mddocs/fern/translations/zh-CN/pages/developer-guide/knowledge-base/modular-components/planner/planner-guide.md
This comment has been minimized.
This comment has been minimized.
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
…runtime-min-endpoints Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
jasonqinzhou
left a comment
There was a problem hiding this comment.
COMMENT · 8.5/10 · high confidence
What this PR does
This PR adds independently configurable prefill and decode endpoint floors, plus a loopback runtime API for changing them without restarting the Planner. It carries the effective floors through scaling, GPU and power budgets, rollout reconciliation, replay, AI Simulate, operator examples, and Planner documentation.
The update thoughtfully removes the confusing legacy framing, brings the branch onto the Linux-stable generated references from main, and preserves focused coverage for runtime atomicity, budget infeasibility, cleanup, and replay determinism.
Why this score
This is now strong 8.5/10 work and merits COMMENT rather than changes requested: the prior generated-reference blocker is fixed, all 56 focused current-head tests pass, and the three generated-reference checks are fresh. One non-blocking documentation mismatch remains: the revised min_endpoint wording mentions aggregated and disaggregated modes but omits the still-active fallback in prefill-only and decode-only modes.
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
jthomson04
left a comment
There was a problem hiding this comment.
Approving comment-only change in operator.
Summary
prefill_min_endpointanddecode_min_endpoint. Usemin_endpointfor aggregated deployments, to apply the same minimum to prefill and decode in disaggregated deployments, or to supply the active role in a single-component deployment when its role-specific value is unset; role-specific values replace it for their role.GET/PATCH /v1/min-endpointsruntime API, including aggregated-mode support throughmin_endpoint.Why
Disaggregated deployments need independent prefill and decode replica floors that operators can adjust without restarting the Planner. Aggregated deployments use
min_endpoint; disaggregated deployments can use it to set the same minimum for both roles. In single-component deployments,min_endpointsupplies the active role when its role-specific value is unset.Minimum endpoint footprints are budget-validated at startup. Runtime updates are process-local, atomic with respect to Planner ticks, budget-validated before mutation, and active on the next stable tick.
Developer and user impact
min_endpointremains the aggregated minimum, applies to both prefill and decode in disaggregated deployments, and supplies the active role in single-component deployments when its role-specific value is unset.min_endpoint.127.0.0.1:9086by default and can be disabled withcontrol_api_port: 0.Agentic review
Three independent review passes covered API/lifecycle behavior, scaling and budget invariants, and compatibility/tests/docs. The review found two P1, two P2, and two P3 issues; all were fixed with regression coverage. The final review found no remaining P0-P3 findings.
Where should the reviewer start?
components/src/dynamo/planner/config/planner_config.pycomponents/src/dynamo/planner/control_api.pycomponents/src/dynamo/planner/core/base.pycomponents/src/dynamo/planner/plugins/orchestrator/engine_adapter.pycomponents/src/dynamo/planner/tests/unit/test_control_api.pycomponents/src/dynamo/planner/tests/unit/test_power_budget.pyValidation
1166 passed, 5 skipped— complete Planner test suite80 passed— AI Simulate sweeper and Planner provider tests56 passed— focused configuration and runtime API tests after the documentation update and latest-main mergemypy: no issues in 5 changed Planner source filespython -m compileallgit diff --checkRelated Issues
Summary by CodeRabbit
New Features
min_endpointconfigures aggregated deployments, applies the same minimum to both roles in disaggregated deployments, or supplies the active role in single-component deployments when its role-specific value is unset.Documentation
Bug Fixes