Repository navigation
fix(sidecar): separate engine backend and runtime identity - #15923
Conversation
badc625 to
8166c3e
Compare
|
Updated in 8166c3e and rebased onto upstream as one DCO-signed-off commit. The review fixes preserve engine model flags as a fallback, use supported native sidecar flags for component identity, handle quoted executable paths without matching config-path arguments, and document the confirmed worker-roll migration in the platform upgrade notes. The description includes the before/after hash comparison and the reason native SGLang/TensorRT-LLM executable detection remains outside this PR. All three inline bot findings are addressed. The full dynamo/controller test suites (including envtest), targeted race tests, go vet, and changed-file pre-commit checks pass. Fern configuration and curated-doc lint pass. The full Fern link scan reports the same pre-existing DeepSeek recipe link on this branch and unchanged upstream; details are in Validation. Could a maintainer review the migration and runtime-status fallback and approve full CI with @coderabbitai review |
✅ Action performedReview finished.
|
|
Hi @ai-dynamo/dynamo-operator-codeowners @peilii — could you take a look at this follow-up to #15463 when you have a chance? The review fixes are in, and pre-merge checks, including operator tests, are passing. I'd appreciate a review of the documented upgrade rollout and the engine model-name fallback. Could a maintainer also authorize full CI with |
peilii
left a comment
There was a problem hiding this comment.
Thanks, the runtime-container selection, the tokenized native detection, and the documented one-time roll all look right to me, and the targeted tests pass locally on this head. One regression needs fixing before this merges; details inline on runtime_status.go.
Summary: the new DYN_COMPONENT fallback reads an env var the operator itself injects into the runtime init container at render time (component_common.go getCommonContainer sets DYN_COMPONENT=), and ResolveComponentRuntimeStatus runs on the rendered PodSpec. So for every native sidecar the field is now always populated with the component type. For decode that is wrong: the native sidecars ignore --component/DYN_COMPONENT in disaggregated modes and register under the fixed discovery name, which for decode is backend (lib/backend-common/src/disagg.rs discovery_component). Planner takes the projected name over its correct default (planner/connectors/kubernetes.py _resolve_dgd_service), so native-sidecar decode workers stop matching their MDC entries. On main the field stayed empty for native sidecars and Planner used the right default, so this is a regression in the Planner path this PR targets.
I verified it by rendering a native vLLM sidecar DCD per component type through GenerateBasePodSpecForController and calling ResolveComponentRuntimeStatus on the result: decode -> runtimeComponentName "decode" (sidecar registers "backend"), prefill -> "prefill" (matches), worker -> "worker" (matches). The PR's tests build PodSpecs by hand without the injected env, which is why they do not catch it.
Smaller notes: the branch is 26 commits behind main but merges cleanly; full CI still needs /ok to test. The header comment in lib/sidecar/vllm/deploy/agg.yaml still says main runs the Dynamo worker, which is stale but out of scope here.
8166c3e to
1a41d09
Compare
|
/ok to test 1a41d09 |
peilii
left a comment
There was a problem hiding this comment.
Re-reviewed 1a41d09. The fix resolves the regression: component identity for the native sidecar now follows --disaggregation-mode, then a literal DYN_DISAGGREGATION_MODE, and only aggregated mode projects --component or the injected DYN_COMPONENT. Prefill, decode, and encode leave the field empty so Planner keeps its defaults, which matches discovery_component in lib/backend-common. The envFrom / valueFrom / $() handling is conservative in the right direction, and the new rendered-pod table covers the cases that were missed before, including CLI precedence over the env mode.
Locally on this head: go vet on internal/dynamo and internal/controller is clean and go test ./internal/dynamo passes. Full CI is authorized above; I will approve once it is green.
❌ Dynamo PR CI failed — run 39295636546 (attempt 1) on
|
| Framework | Snapshot |
|---|---|
| vLLM | ❌ 1 ✅ 1 |
| SGLang | ❌ 1 ✅ 1 |
| TRT-LLM | ❌ 1 ✅ 1 |
| Other | Jobs |
|---|---|
| dynamo-runtime | ⏹️ 1 |
Failure details
3 jobs failed the same way: in all three Snapshot Agent Setup jobs, helm install of the snapshot chart timed out after 15m because the snapshot-agent DaemonSet pods stayed Pending (0/14 nodes are available: 1 Insufficient cpu, 13 node(s) didn't satisfy plugin(s) [NodeAffinity]). This is a cluster scheduling/capacity problem, so rerunning the failed jobs is likely enough. The 1 cancelled dynamo-runtime image build is not a fail-fast cancel (see below).
❌ vLLM Snapshot Agent Setup: helm --wait timeout, snapshot-agent pods unschedulable
Job: vLLM Snapshot Agent Setup · Failed step: Install snapshot-agent · Logs: gh run view --job 118216252201 -R ai-dynamo/dynamo --log-failed
wait.go:50: [debug] beginning wait for 15 resources with timeout of 15m0s
ready.go:336: [debug] DaemonSet is not ready: default/snapshot-agent. 1 out of 5 expected pods are ready
wait.go:87: [debug] Error received when checking status of resource snapshot-pvc. Error: 'client rate limiter Wait returned an error: context deadline exceeded'
Error: context deadline exceeded
##[error]Process completed with exit code 1.
--- Debug snapshot-agent failure ---
pod/snapshot-agent-c75r9 0/2 Pending 0 15m
Warning FailedScheduling default-scheduler 0/14 nodes are available: 1 Insufficient cpu, 13 node(s) didn't satisfy plugin(s) [NodeAffinity].
Warning FailedScheduling karpenter Failed to schedule pod, node selector term with matchFields is not supported
helm install of ghcr.io/ai-dynamo/snapshot/snapshot:0.2.0-rc.3 hit its 15m --wait timeout because only 1 of 5 snapshot-agent DaemonSet pods became ready; the other 4 never scheduled (insufficient CPU on the one matching node, and Karpenter would not provision a new node). This is an infrastructure/capacity issue; a rerun of failed jobs is likely enough.
❌ SGLang Snapshot Agent Setup: helm --wait timeout, snapshot-agent pods unschedulable
Job: SGLang Snapshot Agent Setup · Failed step: Install snapshot-agent · Logs: gh run view --job 118216351757 -R ai-dynamo/dynamo --log-failed
wait.go:50: [debug] beginning wait for 15 resources with timeout of 15m0s
ready.go:336: [debug] DaemonSet is not ready: default/snapshot-agent. 0 out of 5 expected pods are ready
Error: context deadline exceeded
##[error]Process completed with exit code 1.
--- Debug snapshot-agent failure ---
Warning FailedScheduling default-scheduler 0/14 nodes are available: 1 Insufficient cpu, 13 node(s) didn't satisfy plugin(s) [NodeAffinity]
Same failure as vLLM: 0 of 5 snapshot-agent pods were scheduled before the 15m helm timeout. This is an infrastructure/capacity issue; a rerun of failed jobs is likely enough.
❌ TRTLLM Snapshot Agent Setup: helm --wait timeout, snapshot-agent pods unschedulable
Job: TRTLLM Snapshot Agent Setup · Failed step: Install snapshot-agent · Logs: gh run view --job 118216379436 -R ai-dynamo/dynamo --log-failed
wait.go:50: [debug] beginning wait for 15 resources with timeout of 15m0s
wait.go:87: [debug] Error received when checking status of resource snapshot-pvc. Error: 'Get "https://127.0.0.1:8443/api/v1/namespaces/default/persistentvolumeclaims/snapshot-pvc": dial tcp 127.0.0.1:8443: connect: connection refused'
ready.go:336: [debug] DaemonSet is not ready: default/snapshot-agent. 0 out of 5 expected pods are ready
Error: context deadline exceeded
##[error]Process completed with exit code 1.
--- Debug snapshot-agent failure ---
Warning FailedScheduling default-scheduler 0/14 nodes are available: 1 Insufficient cpu, 13 node(s) didn't satisfy plugin(s) [NodeAffinity]
Same failure: 0 of 5 snapshot-agent pods were scheduled before the 15m helm timeout. There was also a brief vCluster API connection refused during the wait, which was retried. This is an infrastructure/capacity issue; a rerun of failed jobs is likely enough.
⏹️ dynamo-runtime / image / Build multi-arch cuda13.0 was cancelled during Build and Push Image exactly 60 minutes after it started (17:38 → 18:38 UTC), well after the snapshot failures. This looks like a job time limit or a manual/concurrency cancel, not a fail-fast cancel.
For agents
{"pr": 15923, "run_id": 39295636546, "run_attempt": 1, "head_sha": "7f743a990e4e6ec0414742165ee9a73ff13aee7a",
"failures": [
{"job": "vLLM Snapshot Agent Setup", "job_id": 118216252201, "failed_step": "Install snapshot-agent", "signature": "helm install context deadline exceeded; snapshot-agent DaemonSet pods Pending (FailedScheduling: Insufficient cpu / NodeAffinity)", "tests": [], "log_cmd": "gh run view --job 118216252201 -R ai-dynamo/dynamo --log-failed"},
{"job": "SGLang Snapshot Agent Setup", "job_id": 118216351757, "failed_step": "Install snapshot-agent", "signature": "helm install context deadline exceeded; snapshot-agent DaemonSet pods Pending (FailedScheduling: Insufficient cpu / NodeAffinity)", "tests": [], "log_cmd": "gh run view --job 118216351757 -R ai-dynamo/dynamo --log-failed"},
{"job": "TRTLLM Snapshot Agent Setup", "job_id": 118216379436, "failed_step": "Install snapshot-agent", "signature": "helm install context deadline exceeded; snapshot-agent DaemonSet pods Pending (FailedScheduling: Insufficient cpu / NodeAffinity)", "tests": [], "log_cmd": "gh run view --job 118216379436 -R ai-dynamo/dynamo --log-failed"}
],
"cancelled": [{"job": "dynamo-runtime / image / Build multi-arch cuda13.0", "job_id": 118207519743, "step": "Build and Push Image"}]}Posted automatically by Devin for run 39295636546. Updated on every full-CI run of this PR.
|
@peilii CI is now green on One release question: the upgrade note currently says v1.6.0, but main now targets 1.7. Should this fix be cherry-picked to |
Signed-off-by: Pranjul Gupta <pranjulgupta0203@gmail.com>
e56a0d1 to
7f743a9
Compare
|
Fixed the operator CI failure in 7f743a9. It was a The full CI-matching |
|
Hi @peilii, @sttts has approved the latest revision ( Could you take a final look and merge when ready, or let me know if any additional review or CI is needed? |
|
/ok to test 7f743a9 |
|
@peilii @sttts thanks for the approvals! Could you help with the remaining CI blockers in run 39295636546?
Operator, Operator Integration, Helm, Planner CPU tests, regular backend deployments and pre-merge checks passed. Could you investigate/re-run the failed setup jobs and cancelled runtime build when possible? I tried the failed-jobs rerun API, but the connected integration returned |
…#15923) Signed-off-by: Pranjul Gupta <pranjulgupta0203@gmail.com>
Resolve two conflicts: - connectors/kubernetes.py: main's fallback line from ai-dynamo#15923 is equivalent to 275eb08's; keep main's. - environment/metrics_provider/runtime_provider.py: keep path-based binding with main's start_tracking(max_age_seconds=...), clear both _unjoined_workers and _fpm_paths on teardown, and drop a role's _unjoined_workers entry when only that role is rebound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tailin Lyu <lyutailin@gmail.com>
Since ai-dynamo#15923 the connector uses a DynamoWorkerMetadata CR only when its ownerReferences name a Ready, Running pod labeled with this DGD, and ignores registrations when pods/list is forbidden. Give the matrix's fake pods UIDs, the DGD label and a Ready condition, have each CR own its serving pod, and expect the status/default identity on the pods-forbidden path. Accept start_tracking's max_age_seconds in the fake subscriber, and check that rebinding one role clears its unjoined-worker warning state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tailin Lyu <lyutailin@gmail.com>
Remove the vllm-rs branch that #15923 added to DetectBackendFrameworkFromArgs instead of keeping it for running-Pod classification. Detection recognizes Dynamo launchers only, as it did in 1.6.x, for workload resolution and for vLLM prefill LoRA fallback alike; native engine prefill Pods do not receive LoRA fallback, which #15463 already tracks for sidecar mode. Native engines set backendFramework explicitly. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
…ive engine backend inference An operator-only upgrade from 1.6.x rolls two kinds of unchanged DGDs: - frontendSidecar containers with partially authored probes. Operator 1.6.x field-merged the probe with the frontend defaults; main replaces it as a complete value (#15581), so the rendered template changes on takeover and a timing-only probe renders without a handler. - Native sidecar workers launching vllm-rs directly without backendFramework. Operator 1.6.x resolved an undetected backend; main infers vllm (#15923). The resolved backend is part of the generated DCD spec and therefore of the worker hash, so every such DGD gets a new worker generation on takeover. Pin both behaviors to the immutable operator origin, as #17182 did for the frontend startup probe. Workloads created by operator 1.6.x or earlier keep the 1.6.x rendering and worker hash; workloads created by 1.7.0 or later get the new behavior. The native engine gate lives in determineBackendFrameworkForComponent so DetectBackendFrameworkFromArgs keeps classifying running prefill Pods for LoRA fallback. ValidateSnapshotFailover reads the same origin so admission resolves the backend exactly as rendering does. Unit tests render the fixtures under both origins and pin the worker hashes and merged probes produced by release/1.6.0 for legacy origins. Correct the platform README upgrade notes: describe the origin-pinned backend inference; move the complete-health-probes (#15581) and Grove rollout budget (#15725) notes from v1.6.0 to v1.7.0, where they shipped; restore the v1.6.0 Grove notes to the shipped v0.1.0-alpha.14 text. Update the curated DCD reference for both gates. Related: #16623, #17181 Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Remove the vllm-rs branch that #15923 added to DetectBackendFrameworkFromArgs instead of keeping it for running-Pod classification. Detection recognizes Dynamo launchers only, as it did in 1.6.x, for workload resolution and for vLLM prefill LoRA fallback alike; native engine prefill Pods do not receive LoRA fallback, which #15463 already tracks for sidecar mode. Native engines set backendFramework explicitly. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
…ive engine backend inference An operator-only upgrade from 1.6.x rolls two kinds of unchanged DGDs: - frontendSidecar containers with partially authored probes. Operator 1.6.x field-merged the probe with the frontend defaults; main replaces it as a complete value (#15581), so the rendered template changes on takeover and a timing-only probe renders without a handler. - Native sidecar workers launching vllm-rs directly without backendFramework. Operator 1.6.x resolved an undetected backend; main infers vllm (#15923). The resolved backend is part of the generated DCD spec and therefore of the worker hash, so every such DGD gets a new worker generation on takeover. Pin both behaviors to the immutable operator origin, as #17182 did for the frontend startup probe. Workloads created by operator 1.6.x or earlier keep the 1.6.x rendering and worker hash; workloads created by 1.7.0 or later get the new behavior. The native engine gate lives in determineBackendFrameworkForComponent so DetectBackendFrameworkFromArgs keeps classifying running prefill Pods for LoRA fallback. ValidateSnapshotFailover reads the same origin so admission resolves the backend exactly as rendering does. Unit tests render the fixtures under both origins and pin the worker hashes and merged probes produced by release/1.6.0 for legacy origins. Correct the platform README upgrade notes: describe the origin-pinned backend inference; move the complete-health-probes (#15581) and Grove rollout budget (#15725) notes from v1.6.0 to v1.7.0, where they shipped; restore the v1.6.0 Grove notes to the shipped v0.1.0-alpha.14 text. Update the curated DCD reference for both gates. Related: #16623, #17181 Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Remove the vllm-rs branch that #15923 added to DetectBackendFrameworkFromArgs instead of keeping it for running-Pod classification. Detection recognizes Dynamo launchers only, as it did in 1.6.x, for workload resolution and for vLLM prefill LoRA fallback alike; native engine prefill Pods do not receive LoRA fallback, which #15463 already tracks for sidecar mode. Native engines set backendFramework explicitly. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
…ive engine backend inference An operator-only upgrade from 1.6.x rolls two kinds of unchanged DGDs: - frontendSidecar containers with partially authored probes. Operator 1.6.x field-merged the probe with the frontend defaults; main replaces it as a complete value (#15581), so the rendered template changes on takeover and a timing-only probe renders without a handler. - Native sidecar workers launching vllm-rs directly without backendFramework. Operator 1.6.x resolved an undetected backend; main infers vllm (#15923). The resolved backend is part of the generated DCD spec and therefore of the worker hash, so every such DGD gets a new worker generation on takeover. Pin both behaviors to the immutable operator origin, as #17182 did for the frontend startup probe. Workloads created by operator 1.6.x or earlier keep the 1.6.x rendering and worker hash; workloads created by 1.7.0 or later get the new behavior. The native engine gate lives in determineBackendFrameworkForComponent so DetectBackendFrameworkFromArgs keeps classifying running prefill Pods for LoRA fallback. ValidateSnapshotFailover reads the same origin so admission resolves the backend exactly as rendering does. Unit tests render the fixtures under both origins and pin the worker hashes and merged probes produced by release/1.6.0 for legacy origins. Correct the platform README upgrade notes: describe the origin-pinned backend inference; move the complete-health-probes (#15581) and Grove rollout budget (#15725) notes from v1.6.0 to v1.7.0, where they shipped; restore the v1.6.0 Grove notes to the shipped v0.1.0-alpha.14 text. Update the curated DCD reference for both gates. Related: #16623, #17181 Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Remove the vllm-rs branch that #15923 added to DetectBackendFrameworkFromArgs instead of keeping it for running-Pod classification. Detection recognizes Dynamo launchers only, as it did in 1.6.x, for workload resolution and for vLLM prefill LoRA fallback alike; native engine prefill Pods do not receive LoRA fallback, which #15463 already tracks for sidecar mode. Native engines set backendFramework explicitly. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
…ive engine backend inference An operator-only upgrade from 1.6.x rolls two kinds of unchanged DGDs: - frontendSidecar containers with partially authored probes. Operator 1.6.x field-merged the probe with the frontend defaults; main replaces it as a complete value (#15581), so the rendered template changes on takeover and a timing-only probe renders without a handler. - Native sidecar workers launching vllm-rs directly without backendFramework. Operator 1.6.x resolved an undetected backend; main infers vllm (#15923). The resolved backend is part of the generated DCD spec and therefore of the worker hash, so every such DGD gets a new worker generation on takeover. Pin both behaviors to the immutable operator origin, as #17182 did for the frontend startup probe. Workloads created by operator 1.6.x or earlier keep the 1.6.x rendering and worker hash; workloads created by 1.7.0 or later get the new behavior. The native engine gate lives in determineBackendFrameworkForComponent so DetectBackendFrameworkFromArgs keeps classifying running prefill Pods for LoRA fallback. ValidateSnapshotFailover reads the same origin so admission resolves the backend exactly as rendering does. Unit tests render the fixtures under both origins and pin the worker hashes and merged probes produced by release/1.6.0 for legacy origins. Correct the platform README upgrade notes: describe the origin-pinned backend inference; move the complete-health-probes (#15581) and Grove rollout budget (#15725) notes from v1.6.0 to v1.7.0, where they shipped; restore the v1.6.0 Grove notes to the shipped v0.1.0-alpha.14 text. Update the curated DCD reference for both gates. Related: #16623, #17181 Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Remove the vllm-rs branch that #15923 added to DetectBackendFrameworkFromArgs instead of keeping it for running-Pod classification. Detection recognizes Dynamo launchers only, as it did in 1.6.x, for workload resolution and for vLLM prefill LoRA fallback alike; native engine prefill Pods do not receive LoRA fallback, which #15463 already tracks for sidecar mode. Native engines set backendFramework explicitly. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
…ive engine backend inference An operator-only upgrade from 1.6.x rolls two kinds of unchanged DGDs: - frontendSidecar containers with partially authored probes. Operator 1.6.x field-merged the probe with the frontend defaults; main replaces it as a complete value (#15581), so the rendered template changes on takeover and a timing-only probe renders without a handler. - Native sidecar workers launching vllm-rs directly without backendFramework. Operator 1.6.x resolved an undetected backend; main infers vllm (#15923). The resolved backend is part of the generated DCD spec and therefore of the worker hash, so every such DGD gets a new worker generation on takeover. Pin both behaviors to the immutable operator origin, as #17182 did for the frontend startup probe. Workloads created by operator 1.6.x or earlier keep the 1.6.x rendering and worker hash; workloads created by 1.7.0 or later get the new behavior. The native engine gate lives in determineBackendFrameworkForComponent so DetectBackendFrameworkFromArgs keeps classifying running prefill Pods for LoRA fallback. ValidateSnapshotFailover reads the same origin so admission resolves the backend exactly as rendering does. Unit tests render the fixtures under both origins and pin the worker hashes and merged probes produced by release/1.6.0 for legacy origins. Correct the platform README upgrade notes: describe the origin-pinned backend inference; move the complete-health-probes (#15581) and Grove rollout budget (#15725) notes from v1.6.0 to v1.7.0, where they shipped; restore the v1.6.0 Grove notes to the shipped v0.1.0-alpha.14 text. Update the curated DCD reference for both gates. Related: #16623, #17181 Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Remove the vllm-rs branch that #15923 added to DetectBackendFrameworkFromArgs instead of keeping it for running-Pod classification. Detection recognizes Dynamo launchers only, as it did in 1.6.x, for workload resolution and for vLLM prefill LoRA fallback alike; native engine prefill Pods do not receive LoRA fallback, which #15463 already tracks for sidecar mode. Native engines set backendFramework explicitly. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Summary
Part of #15463.
Native sidecar deployments put the inference engine in
mainand the Dynamo runtime inruntime. The operator did not recognize a directvllm-rsengine launch. Earlier revisions also reconstructed runtime identity from launch arguments, which could hide valid registrations when shell expansion or runtime mode rules produced another name.This revision follows the ownership split requested in review:
main, including standalone DCD rendering. Recognize only directvllm-rsargv[0], including literal executable paths; opaque native launches require explicitbackendFramework. Remove the added native shell-launch heuristic; existing Python detection is unchanged.DynamoWorkerMetadatawith ready, non-terminating Pods through Pod owner name and UID, DGD/component labels, active worker hashes, and runtime namespace. Use registered component, endpoint, model and worker role without filtering by a predicted component name.Modelwrapper withmodel_suffixor non-nullcard_json.lora. Reject conflicting identities across active replicas, including when no active hash is available.Observed registrations do not affect desired worker hashes or engine configuration. The API changes are comments only: no fields, serialization, or conversion behavior change. Regenerated CRD schemas are identical to the base when descriptions are excluded.
Upgrade impact and scope
The upgrade note is under v1.7.0, matching this main-only PR. A 1.6 backport remains a maintainer decision.
Recognizing an implicit
vllm-rsbackend changes the generated DCD backend field and can roll all workers in an affected graph on an operator-only upgrade. Reference manifests explicitly settingbackendFramework: vllmare unaffected by this inference change. Setting that field beforehand moves the migration to an earlier manifest-driven rollout; it does not eliminate it. WithcompilationCache.mountPathconfigured, the engine also gainsVLLM_CACHE_ROOT. Standalone DCDs can roll when cache defaults change the Pod template.Explicit backend configuration does not enable command-based LoRA fallback classification. No native SGLang or TRT-LLM executable detection is added.
Registered native identity is Kubernetes-specific and requires worker-metadata and Pod-list access; the chart already grants these permissions. Legacy status/default fallback cannot discover native custom names, models or capabilities without that access. This does not implement native identity discovery through etcd. Native
servedModelNameandruntimeComponentNamestatus remain absent, so #15463 remains broader than this PR.Validation
Behavioral validation below ran on
e56a0d1. The current head,7f743a990e4e6ec0414742165ee9a73ff13aee7a, changes only import ordering inruntime_status_test.go; the full CI-matching Go linter (golangci-lint v1.62.2) and targeted runtime-status tests pass locally. The base remains upstream70ca8f5231181722f823da7a245ce2f87c760cc5.go test -p 3 ./internal/dynamo/... ./internal/controller/... -count=1passed, including full controller envtests and subpackages.go vet -p 3 ./internal/dynamo/... ./internal/controller/...passed.test_kubernetes_connector.py,test_mdc.py,test_startup_rbac_compatibility.py,test_environment.py, andtest_load_based_scaling.py.$WORKER_NAME/${WORKER_NAME}launches, custom aggregated names, fixed decodebackend, actual Model-form adapters, replica conflicts, recreated Pod UIDs, overlapping rollout revisions/namespaces, Pod-list permission recovery and single-snapshot model refresh. Ten added regression cases failed against the preceding implementation before the fixes.make checkpassed, regenerating Pydantic models, manifests, Kubernetes API references and the platform Helm README with no working-tree changes. CRDs were compared structurally against the base with descriptions excluded.git diff --checkpassed.e56a0d1; checks must rerun on the formatting-only follow-up.Local envtest used Kubernetes 1.30.0 assets with an external localhost advertise-address wrapper. Python tests used mocked Kubernetes APIs and the published
ai-dynamo-runtime==1.5.1wheel for imports; a 1.6.0 wheel was unavailable. No live Kubernetes/GPU serving test was run.One DCO-signed-off commit, authored and committed by Pranjul Gupta. Previous-head green checks do not validate this revision; fresh maintainer-authorized CI and re-review are required.
Reviewer starting points
DetectBackendFrameworkFromArgs,ResolveComponentRuntimeStatus, and Planner's_extract_mdc_entries/_registered_worker_entrycontain the behavior changes.