Skip to content

fix(sidecar): separate engine backend and runtime identity - #15923

Merged
peilii merged 1 commit into
ai-dynamo:mainfrom
Pranjulcr7:pranjul/fix-15463-sidecar-backend
Oct 20, 2026
Merged

peilii merged 1 commit into
ai-dynamo:mainfrom
Pranjulcr7:pranjul/fix-15463-sidecar-backend

Conversation

@Pranjulcr7

@Pranjulcr7 Pranjulcr7 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Part of #15463.

Native sidecar deployments put the inference engine in main and the Dynamo runtime in runtime. The operator did not recognize a direct vllm-rs engine launch. Earlier revisions also reconstructed runtime identity from launch arguments, which could hide valid registrations when shell expansion or runtime mode rules produced another name.

This revision follows the ownership split requested in review:

  • Keep backend inference and mismatch validation on main, including standalone DCD rendering. Recognize only direct vllm-rs argv[0], including literal executable paths; opaque native launches require explicit backendFramework. Remove the added native shell-launch heuristic; existing Python detection is unchanged.
  • Keep compilation-cache configuration and vLLM prefill LoRA classification tied to the engine, independently of runtime wrappers. Avoid mutating command slices during detection.
  • Have Kubernetes Planner associate DynamoWorkerMetadata with ready, non-terminating Pods through Pod owner name and UID, DGD/component labels, active worker hashes, and runtime namespace. Use registered component, endpoint, model and worker role without filtering by a predicted component name.
  • Reuse one metadata/Pod snapshot across prefill and decode model-name refresh. Exclude stale Pod UIDs, candidate revisions, encoders and adapters. Adapter detection matches Rust's actual Model wrapper with model_suffix or non-null card_json.lora. Reject conflicting identities across active replicas, including when no active hash is available.
  • Stop projecting native model/component identity from commands. Preserve legacy status/default fallback when registration is unavailable. Pod-list HTTP 403 warns and falls back for older service accounts; other API failures propagate. Granting Pod-list access takes effect on a subsequent read without restarting Planner.

Observed registrations do not affect desired worker hashes or engine configuration. The API changes are comments only: no fields, serialization, or conversion behavior change. Regenerated CRD schemas are identical to the base when descriptions are excluded.

Upgrade impact and scope

The upgrade note is under v1.7.0, matching this main-only PR. A 1.6 backport remains a maintainer decision.

Recognizing an implicit vllm-rs backend changes the generated DCD backend field and can roll all workers in an affected graph on an operator-only upgrade. Reference manifests explicitly setting backendFramework: vllm are unaffected by this inference change. Setting that field beforehand moves the migration to an earlier manifest-driven rollout; it does not eliminate it. With compilationCache.mountPath configured, the engine also gains VLLM_CACHE_ROOT. Standalone DCDs can roll when cache defaults change the Pod template.

Explicit backend configuration does not enable command-based LoRA fallback classification. No native SGLang or TRT-LLM executable detection is added.

Registered native identity is Kubernetes-specific and requires worker-metadata and Pod-list access; the chart already grants these permissions. Legacy status/default fallback cannot discover native custom names, models or capabilities without that access. This does not implement native identity discovery through etcd. Native servedModelName and runtimeComponentName status remain absent, so #15463 remains broader than this PR.

Validation

Behavioral validation below ran on e56a0d1. The current head, 7f743a990e4e6ec0414742165ee9a73ff13aee7a, changes only import ordering in runtime_status_test.go; the full CI-matching Go linter (golangci-lint v1.62.2) and targeted runtime-status tests pass locally. The base remains upstream 70ca8f5231181722f823da7a245ce2f87c760cc5.

  • Go 1.26.8: go test -p 3 ./internal/dynamo/... ./internal/controller/... -count=1 passed, including full controller envtests and subpackages.
  • Targeted race tests for native sidecars, backend detection, runtime status and LoRA classification passed.
  • go vet -p 3 ./internal/dynamo/... ./internal/controller/... passed.
  • 276 Python tests passed across test_kubernetes_connector.py, test_mdc.py, test_startup_rbac_compatibility.py, test_environment.py, and test_load_based_scaling.py.
  • Regressions cover engine/backend mismatch, standalone cache rendering, runtime-wrapper independence, rendered $WORKER_NAME / ${WORKER_NAME} launches, custom aggregated names, fixed decode backend, actual Model-form adapters, replica conflicts, recreated Pod UIDs, overlapping rollout revisions/namespaces, Pod-list permission recovery and single-snapshot model refresh. Ten added regression cases failed against the preceding implementation before the fixes.
  • make check passed, regenerating Pydantic models, manifests, Kubernetes API references and the platform Helm README with no working-tree changes. CRDs were compared structurally against the base with descriptions excluded.
  • Changed-file pre-commit checks, gofmt and git diff --check passed.
  • Docs lint passed with zero errors and nine warnings; Python/Rust documentation generation passed. Local Fern validation was blocked by automatic approval review because the CLI attempted external telemetry. GitHub's Fern configuration, broken-link and docs-composition checks subsequently passed on e56a0d1; checks must rerun on the formatting-only follow-up.

Local envtest used Kubernetes 1.30.0 assets with an external localhost advertise-address wrapper. Python tests used mocked Kubernetes APIs and the published ai-dynamo-runtime==1.5.1 wheel for imports; a 1.6.0 wheel was unavailable. No live Kubernetes/GPU serving test was run.

One DCO-signed-off commit, authored and committed by Pranjul Gupta. Previous-head green checks do not validate this revision; fresh maintainer-authorized CI and re-review are required.

Reviewer starting points

DetectBackendFrameworkFromArgs, ResolveComponentRuntimeStatus, and Planner's _extract_mdc_entries / _registered_worker_entry contain the behavior changes.

@Pranjulcr7
Pranjulcr7 requested review from a team as code owners October 8, 2026 20:25
@copy-pr-bot

copy-pr-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Pranjulcr7
Pranjulcr7 temporarily deployed to external_collaborator October 8, 2026 20:25 — with GitHub Actions Inactive
@Pranjulcr7
Pranjulcr7 temporarily deployed to external_collaborator October 8, 2026 20:25 — with GitHub Actions Inactive
@github-actions github-actions Bot added the fix label Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor documentation Improvements or additions to documentation deployment::k8s Relates to dynamo deployment in kubernetes labels Oct 8, 2026
@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: d6627d65-7149-426e-b821-0b0b8774f0ab
📥 Commits

Reviewing files that changed from the base of the PR and between 6c7e590 and 8166c3e.

📒 Files selected for processing (11)
  • deploy/helm/charts/platform/README.md
  • deploy/helm/charts/platform/README.md.gotmpl
  • deploy/operator/internal/controller/dynamo_model_controller.go
  • deploy/operator/internal/controller/dynamo_model_controller_unit_test.go
  • deploy/operator/internal/dynamo/component_dynamo_sidecar_test.go
  • deploy/operator/internal/dynamo/graph.go
  • deploy/operator/internal/dynamo/runtime_status.go
  • deploy/operator/internal/dynamo/runtime_status_test.go
  • deploy/operator/internal/dynamo/v1beta1_helpers.go
  • docs/fern/pages/reference/kubernetes-api/dynamo-component-deployment.mdx
  • docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx
🚧 Files skipped from review as they are similar to previous changes (2)
  • docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx
  • docs/fern/pages/reference/kubernetes-api/dynamo-component-deployment.mdx

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The operator selects the Dynamo runtime container for backend inference and runtime status resolution. Prefill LoRA fallback classification also uses this container. Tests and documentation cover native sidecar behavior, cache rendering, identity fallback, and upgrade effects.

Changes

Native Sidecar Runtime Detection

Layer / File(s) Summary
Backend inference and cache rendering
deploy/operator/internal/dynamo/v1beta1_helpers.go, deploy/operator/internal/dynamo/graph.go, deploy/operator/internal/dynamo/component_dynamo_sidecar_test.go, deploy/operator/internal/dynamo/runtime_status_test.go, docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx
The container-selection helper selects the runtime init container when present, otherwise the main container. Backend detection reads the selected launch and recognizes supported native vLLM sidecar commands. Tests cover inference, cache rendering, and input immutability.
Runtime identity and LoRA fallback
deploy/operator/internal/dynamo/runtime_status.go, deploy/operator/internal/dynamo/runtime_status_test.go, deploy/operator/internal/controller/dynamo_model_controller.go, deploy/operator/internal/controller/dynamo_model_controller_unit_test.go
Runtime status derives component and model identity from runtime and engine arguments under the documented fallback conditions. Prefill LoRA fallback classification uses the selected runtime container. Tests cover status resolution and classification.
Sidecar behavior and upgrade guidance
docs/fern/pages/reference/kubernetes-api/dynamo-component-deployment.mdx, docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx, deploy/helm/charts/platform/README.md, deploy/helm/charts/platform/README.md.gotmpl
The documentation describes runtime-based inference, identity fallbacks, and possible worker rollouts. The platform upgrade notes also describe compilation-cache defaults for the engine.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 8166c

No actionable merge-blocking issue was established. The change is ready for normal merge checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 69.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 23 functions across 7 files. (4 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly and concisely describes the main change: separating engine backend inference from runtime identity in sidecar mode.
Description check Passed The description is detailed and covers the change scope, implementation details, reviewer starting points, upgrade impact, validation, and issue reference. It uses a Summary section instead of the tem…
Full details: Description check

Explanation

The description is detailed and covers the change scope, implementation details, reviewer starting points, upgrade impact, validation, and issue reference. It uses a Summary section instead of the template headings and places “Part of #15463” outside a dedicated Related Issues section, but the required information is present.

Full details: Docstring Coverage

Explanation

Docstring coverage is 69.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 23 functions across 7 files. (4 skipped: 4 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

  • Fix all pre-merge checks with AI
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@Pranjulcr7
Pranjulcr7 temporarily deployed to external_collaborator October 8, 2026 20:31 — with GitHub Actions Inactive
@Pranjulcr7
Pranjulcr7 force-pushed the pranjul/fix-15463-sidecar-backend branch from badc625 to 8166c3e Compare October 10, 2026 05:32
@Pranjulcr7
Pranjulcr7 temporarily deployed to external_collaborator October 10, 2026 05:32 — with GitHub Actions Inactive

Copy link
Copy Markdown
Contributor Author

Updated in 8166c3e and rebased onto upstream as one DCO-signed-off commit.

The review fixes preserve engine model flags as a fallback, use supported native sidecar flags for component identity, handle quoted executable paths without matching config-path arguments, and document the confirmed worker-roll migration in the platform upgrade notes. The description includes the before/after hash comparison and the reason native SGLang/TensorRT-LLM executable detection remains outside this PR. All three inline bot findings are addressed.

The full dynamo/controller test suites (including envtest), targeted race tests, go vet, and changed-file pre-commit checks pass. Fern configuration and curated-doc lint pass. The full Fern link scan reports the same pre-existing DeepSeek recipe link on this branch and unchanged upstream; details are in Validation.

Could a maintainer review the migration and runtime-status fallback and approve full CI with /ok to test 8166c3e777018bf1c2cbc950534774dfc488b03f? This commit has a DCO sign-off but no cryptographic signature.

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Contributor Author

Hi @ai-dynamo/dynamo-operator-codeowners @peilii — could you take a look at this follow-up to #15463 when you have a chance? The review fixes are in, and pre-merge checks, including operator tests, are passing. I'd appreciate a review of the documented upgrade rollout and the engine model-name fallback.

Could a maintainer also authorize full CI with /ok to test 8166c3e777018bf1c2cbc950534774dfc488b03f? Thanks!

@peilii peilii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the runtime-container selection, the tokenized native detection, and the documented one-time roll all look right to me, and the targeted tests pass locally on this head. One regression needs fixing before this merges; details inline on runtime_status.go.

Summary: the new DYN_COMPONENT fallback reads an env var the operator itself injects into the runtime init container at render time (component_common.go getCommonContainer sets DYN_COMPONENT=), and ResolveComponentRuntimeStatus runs on the rendered PodSpec. So for every native sidecar the field is now always populated with the component type. For decode that is wrong: the native sidecars ignore --component/DYN_COMPONENT in disaggregated modes and register under the fixed discovery name, which for decode is backend (lib/backend-common/src/disagg.rs discovery_component). Planner takes the projected name over its correct default (planner/connectors/kubernetes.py _resolve_dgd_service), so native-sidecar decode workers stop matching their MDC entries. On main the field stayed empty for native sidecars and Planner used the right default, so this is a regression in the Planner path this PR targets.

I verified it by rendering a native vLLM sidecar DCD per component type through GenerateBasePodSpecForController and calling ResolveComponentRuntimeStatus on the result: decode -> runtimeComponentName "decode" (sidecar registers "backend"), prefill -> "prefill" (matches), worker -> "worker" (matches). The PR's tests build PodSpecs by hand without the injected env, which is why they do not catch it.

Smaller notes: the branch is 26 commits behind main but merges cleanly; full CI still needs /ok to test. The header comment in lib/sidecar/vllm/deploy/agg.yaml still says main runs the Dynamo worker, which is stale but out of scope here.

@Pranjulcr7
Pranjulcr7 force-pushed the pranjul/fix-15463-sidecar-backend branch from 8166c3e to 1a41d09 Compare October 13, 2026 05:26
@Pranjulcr7
Pranjulcr7 temporarily deployed to external_collaborator October 13, 2026 05:26 — with GitHub Actions Inactive
@peilii

peilii commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

/ok to test 1a41d09

@peilii peilii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 1a41d09. The fix resolves the regression: component identity for the native sidecar now follows --disaggregation-mode, then a literal DYN_DISAGGREGATION_MODE, and only aggregated mode projects --component or the injected DYN_COMPONENT. Prefill, decode, and encode leave the field empty so Planner keeps its defaults, which matches discovery_component in lib/backend-common. The envFrom / valueFrom / $() handling is conservative in the right direction, and the new rendered-pod table covers the cases that were missed before, including CLI precedence over the env mode.

Locally on this head: go vet on internal/dynamo and internal/controller is clean and go test ./internal/dynamo passes. Full CI is authorized above; I will approve once it is green.

@devin-ai-integration

devin-ai-integration Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

❌ Dynamo PR CI failed — run 39295636546 (attempt 1) on 7f743a990e

Gate checks: ✅ backend-status-check · ❌ deploy-status-check · ❌ dynamo-status-check

Framework Snapshot
vLLM ❌ 1 ✅ 1
SGLang ❌ 1 ✅ 1
TRT-LLM ❌ 1 ✅ 1
Other Jobs
dynamo-runtime ⏹️ 1

Failure details

3 jobs failed the same way: in all three Snapshot Agent Setup jobs, helm install of the snapshot chart timed out after 15m because the snapshot-agent DaemonSet pods stayed Pending (0/14 nodes are available: 1 Insufficient cpu, 13 node(s) didn't satisfy plugin(s) [NodeAffinity]). This is a cluster scheduling/capacity problem, so rerunning the failed jobs is likely enough. The 1 cancelled dynamo-runtime image build is not a fail-fast cancel (see below).

❌ vLLM Snapshot Agent Setup: helm --wait timeout, snapshot-agent pods unschedulable

Job: vLLM Snapshot Agent Setup · Failed step: Install snapshot-agent · Logs: gh run view --job 118216252201 -R ai-dynamo/dynamo --log-failed

wait.go:50: [debug] beginning wait for 15 resources with timeout of 15m0s
ready.go:336: [debug] DaemonSet is not ready: default/snapshot-agent. 1 out of 5 expected pods are ready
wait.go:87: [debug] Error received when checking status of resource snapshot-pvc. Error: 'client rate limiter Wait returned an error: context deadline exceeded'
Error: context deadline exceeded
##[error]Process completed with exit code 1.
--- Debug snapshot-agent failure ---
pod/snapshot-agent-c75r9   0/2   Pending   0   15m
Warning  FailedScheduling  default-scheduler  0/14 nodes are available: 1 Insufficient cpu, 13 node(s) didn't satisfy plugin(s) [NodeAffinity].
Warning  FailedScheduling  karpenter          Failed to schedule pod, node selector term with matchFields is not supported

helm install of ghcr.io/ai-dynamo/snapshot/snapshot:0.2.0-rc.3 hit its 15m --wait timeout because only 1 of 5 snapshot-agent DaemonSet pods became ready; the other 4 never scheduled (insufficient CPU on the one matching node, and Karpenter would not provision a new node). This is an infrastructure/capacity issue; a rerun of failed jobs is likely enough.

❌ SGLang Snapshot Agent Setup: helm --wait timeout, snapshot-agent pods unschedulable

Job: SGLang Snapshot Agent Setup · Failed step: Install snapshot-agent · Logs: gh run view --job 118216351757 -R ai-dynamo/dynamo --log-failed

wait.go:50: [debug] beginning wait for 15 resources with timeout of 15m0s
ready.go:336: [debug] DaemonSet is not ready: default/snapshot-agent. 0 out of 5 expected pods are ready
Error: context deadline exceeded
##[error]Process completed with exit code 1.
--- Debug snapshot-agent failure ---
Warning  FailedScheduling  default-scheduler  0/14 nodes are available: 1 Insufficient cpu, 13 node(s) didn't satisfy plugin(s) [NodeAffinity]

Same failure as vLLM: 0 of 5 snapshot-agent pods were scheduled before the 15m helm timeout. This is an infrastructure/capacity issue; a rerun of failed jobs is likely enough.

❌ TRTLLM Snapshot Agent Setup: helm --wait timeout, snapshot-agent pods unschedulable

Job: TRTLLM Snapshot Agent Setup · Failed step: Install snapshot-agent · Logs: gh run view --job 118216379436 -R ai-dynamo/dynamo --log-failed

wait.go:50: [debug] beginning wait for 15 resources with timeout of 15m0s
wait.go:87: [debug] Error received when checking status of resource snapshot-pvc. Error: 'Get "https://127.0.0.1:8443/api/v1/namespaces/default/persistentvolumeclaims/snapshot-pvc": dial tcp 127.0.0.1:8443: connect: connection refused'
ready.go:336: [debug] DaemonSet is not ready: default/snapshot-agent. 0 out of 5 expected pods are ready
Error: context deadline exceeded
##[error]Process completed with exit code 1.
--- Debug snapshot-agent failure ---
Warning  FailedScheduling  default-scheduler  0/14 nodes are available: 1 Insufficient cpu, 13 node(s) didn't satisfy plugin(s) [NodeAffinity]

Same failure: 0 of 5 snapshot-agent pods were scheduled before the 15m helm timeout. There was also a brief vCluster API connection refused during the wait, which was retried. This is an infrastructure/capacity issue; a rerun of failed jobs is likely enough.

⏹️ dynamo-runtime / image / Build multi-arch cuda13.0 was cancelled during Build and Push Image exactly 60 minutes after it started (17:38 → 18:38 UTC), well after the snapshot failures. This looks like a job time limit or a manual/concurrency cancel, not a fail-fast cancel.

For agents
{"pr": 15923, "run_id": 39295636546, "run_attempt": 1, "head_sha": "7f743a990e4e6ec0414742165ee9a73ff13aee7a",
 "failures": [
  {"job": "vLLM Snapshot Agent Setup", "job_id": 118216252201, "failed_step": "Install snapshot-agent", "signature": "helm install context deadline exceeded; snapshot-agent DaemonSet pods Pending (FailedScheduling: Insufficient cpu / NodeAffinity)", "tests": [], "log_cmd": "gh run view --job 118216252201 -R ai-dynamo/dynamo --log-failed"},
  {"job": "SGLang Snapshot Agent Setup", "job_id": 118216351757, "failed_step": "Install snapshot-agent", "signature": "helm install context deadline exceeded; snapshot-agent DaemonSet pods Pending (FailedScheduling: Insufficient cpu / NodeAffinity)", "tests": [], "log_cmd": "gh run view --job 118216351757 -R ai-dynamo/dynamo --log-failed"},
  {"job": "TRTLLM Snapshot Agent Setup", "job_id": 118216379436, "failed_step": "Install snapshot-agent", "signature": "helm install context deadline exceeded; snapshot-agent DaemonSet pods Pending (FailedScheduling: Insufficient cpu / NodeAffinity)", "tests": [], "log_cmd": "gh run view --job 118216379436 -R ai-dynamo/dynamo --log-failed"}
 ],
 "cancelled": [{"job": "dynamo-runtime / image / Build multi-arch cuda13.0", "job_id": 118207519743, "step": "Build and Push Image"}]}

Posted automatically by Devin for run 39295636546. Updated on every full-CI run of this PR.

Copy link
Copy Markdown
Contributor Author

@peilii CI is now green on 1a41d09, including Operator Integration and Helm Chart Tests. Thanks for re-reviewing!

One release question: the upgrade note currently says v1.6.0, but main now targets 1.7. Should this fix be cherry-picked to release/1.6.0, or should the note move to the next release heading?

Signed-off-by: Pranjul Gupta <pranjulgupta0203@gmail.com>
@Pranjulcr7
Pranjulcr7 force-pushed the pranjul/fix-15463-sidecar-backend branch from e56a0d1 to 7f743a9 Compare October 17, 2026 16:21
@Pranjulcr7
Pranjulcr7 temporarily deployed to external_collaborator October 17, 2026 16:21 — with GitHub Actions Inactive

Copy link
Copy Markdown
Contributor Author

Fixed the operator CI failure in 7f743a9. It was a goimports ordering error in runtime_status_test.go; this update only moves the config import into the non-standard-library group. I missed this in the earlier local checks.

The full CI-matching golangci-lint v1.62.2 run --timeout=5m and targeted runtime-status tests now pass locally. Kept one signed-off commit. All other executed checks passed on the preceding head, including Fern, generated API references and pre-commit; the new head needs fresh CI.

@Pranjulcr7
Pranjulcr7 requested a review from sttts October 18, 2026 21:03

Copy link
Copy Markdown
Contributor Author

Hi @peilii, @sttts has approved the latest revision (7f743a9), and the pre-merge checks are green, including operator, generated API references and Fern. Thanks again for your guidance on this!

Could you take a final look and merge when ready, or let me know if any additional review or CI is needed?

@peilii
peilii enabled auto-merge (squash) October 19, 2026 17:24

@peilii peilii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sttts

sttts commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

/ok to test 7f743a9

Copy link
Copy Markdown
Contributor Author

@peilii @sttts thanks for the approvals! Could you help with the remaining CI blockers in run 39295636546?

  • vLLM, SGLang and TRT-LLM Snapshot Agent Setup timed out. The logs show pending Pods with insufficient CPU/node-affinity scheduling failures and a snapshot PVC readiness timeout. These appear to be CI infrastructure issues.
  • The Dynamo runtime multi-arch image build was cancelled after roughly an hour, so its dependent tests were skipped.

Operator, Operator Integration, Helm, Planner CPU tests, regular backend deployments and pre-merge checks passed. Could you investigate/re-run the failed setup jobs and cancelled runtime build when possible? I tried the failed-jobs rerun API, but the connected integration returned 403 Resource not accessible by integration. No code changes have been pushed.

@peilii
peilii merged commit 6edd4ee into ai-dynamo:main Oct 20, 2026
198 of 204 checks passed
erezzarum pushed a commit to erezzarum/dynamo that referenced this pull request Oct 23, 2026
…#15923)

Signed-off-by: Pranjul Gupta <pranjulgupta0203@gmail.com>
TailinLyu added a commit to TailinLyu/dynamo that referenced this pull request Oct 31, 2026
Resolve two conflicts:

- connectors/kubernetes.py: main's fallback line from ai-dynamo#15923 is
  equivalent to 275eb08's; keep main's.
- environment/metrics_provider/runtime_provider.py: keep path-based
  binding with main's start_tracking(max_age_seconds=...), clear both
  _unjoined_workers and _fpm_paths on teardown, and drop a role's
  _unjoined_workers entry when only that role is rebound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tailin Lyu <lyutailin@gmail.com>
TailinLyu added a commit to TailinLyu/dynamo that referenced this pull request Oct 31, 2026
Since ai-dynamo#15923 the connector uses a DynamoWorkerMetadata CR only when its
ownerReferences name a Ready, Running pod labeled with this DGD, and
ignores registrations when pods/list is forbidden. Give the matrix's
fake pods UIDs, the DGD label and a Ready condition, have each CR own
its serving pod, and expect the status/default identity on the
pods-forbidden path. Accept start_tracking's max_age_seconds in the
fake subscriber, and check that rebinding one role clears its
unjoined-worker warning state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tailin Lyu <lyutailin@gmail.com>
tmonty12 added a commit that referenced this pull request Nov 12, 2026
Remove the vllm-rs branch that #15923 added to DetectBackendFrameworkFromArgs
instead of keeping it for running-Pod classification. Detection recognizes
Dynamo launchers only, as it did in 1.6.x, for workload resolution and for
vLLM prefill LoRA fallback alike; native engine prefill Pods do not receive
LoRA fallback, which #15463 already tracks for sidecar mode. Native engines
set backendFramework explicitly.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
tmonty12 added a commit that referenced this pull request Nov 13, 2026
…ive engine backend inference

An operator-only upgrade from 1.6.x rolls two kinds of unchanged DGDs:

- frontendSidecar containers with partially authored probes. Operator 1.6.x
  field-merged the probe with the frontend defaults; main replaces it as a
  complete value (#15581), so the rendered template changes on takeover and a
  timing-only probe renders without a handler.
- Native sidecar workers launching vllm-rs directly without backendFramework.
  Operator 1.6.x resolved an undetected backend; main infers vllm (#15923).
  The resolved backend is part of the generated DCD spec and therefore of the
  worker hash, so every such DGD gets a new worker generation on takeover.

Pin both behaviors to the immutable operator origin, as #17182 did for the
frontend startup probe. Workloads created by operator 1.6.x or earlier keep the
1.6.x rendering and worker hash; workloads created by 1.7.0 or later get the
new behavior. The native engine gate lives in determineBackendFrameworkForComponent
so DetectBackendFrameworkFromArgs keeps classifying running prefill Pods for
LoRA fallback. ValidateSnapshotFailover reads the same origin so admission
resolves the backend exactly as rendering does.

Unit tests render the fixtures under both origins and pin the worker hashes and
merged probes produced by release/1.6.0 for legacy origins.

Correct the platform README upgrade notes: describe the origin-pinned backend
inference; move the complete-health-probes (#15581) and Grove rollout budget
(#15725) notes from v1.6.0 to v1.7.0, where they shipped; restore the v1.6.0
Grove notes to the shipped v0.1.0-alpha.14 text. Update the curated DCD
reference for both gates.

Related: #16623, #17181
Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
tmonty12 added a commit that referenced this pull request Nov 13, 2026
Remove the vllm-rs branch that #15923 added to DetectBackendFrameworkFromArgs
instead of keeping it for running-Pod classification. Detection recognizes
Dynamo launchers only, as it did in 1.6.x, for workload resolution and for
vLLM prefill LoRA fallback alike; native engine prefill Pods do not receive
LoRA fallback, which #15463 already tracks for sidecar mode. Native engines
set backendFramework explicitly.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
tmonty12 added a commit that referenced this pull request Nov 13, 2026
…ive engine backend inference

An operator-only upgrade from 1.6.x rolls two kinds of unchanged DGDs:

- frontendSidecar containers with partially authored probes. Operator 1.6.x
  field-merged the probe with the frontend defaults; main replaces it as a
  complete value (#15581), so the rendered template changes on takeover and a
  timing-only probe renders without a handler.
- Native sidecar workers launching vllm-rs directly without backendFramework.
  Operator 1.6.x resolved an undetected backend; main infers vllm (#15923).
  The resolved backend is part of the generated DCD spec and therefore of the
  worker hash, so every such DGD gets a new worker generation on takeover.

Pin both behaviors to the immutable operator origin, as #17182 did for the
frontend startup probe. Workloads created by operator 1.6.x or earlier keep the
1.6.x rendering and worker hash; workloads created by 1.7.0 or later get the
new behavior. The native engine gate lives in determineBackendFrameworkForComponent
so DetectBackendFrameworkFromArgs keeps classifying running prefill Pods for
LoRA fallback. ValidateSnapshotFailover reads the same origin so admission
resolves the backend exactly as rendering does.

Unit tests render the fixtures under both origins and pin the worker hashes and
merged probes produced by release/1.6.0 for legacy origins.

Correct the platform README upgrade notes: describe the origin-pinned backend
inference; move the complete-health-probes (#15581) and Grove rollout budget
(#15725) notes from v1.6.0 to v1.7.0, where they shipped; restore the v1.6.0
Grove notes to the shipped v0.1.0-alpha.14 text. Update the curated DCD
reference for both gates.

Related: #16623, #17181
Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
tmonty12 added a commit that referenced this pull request Nov 13, 2026
Remove the vllm-rs branch that #15923 added to DetectBackendFrameworkFromArgs
instead of keeping it for running-Pod classification. Detection recognizes
Dynamo launchers only, as it did in 1.6.x, for workload resolution and for
vLLM prefill LoRA fallback alike; native engine prefill Pods do not receive
LoRA fallback, which #15463 already tracks for sidecar mode. Native engines
set backendFramework explicitly.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
tmonty12 added a commit that referenced this pull request Nov 13, 2026
…ive engine backend inference

An operator-only upgrade from 1.6.x rolls two kinds of unchanged DGDs:

- frontendSidecar containers with partially authored probes. Operator 1.6.x
  field-merged the probe with the frontend defaults; main replaces it as a
  complete value (#15581), so the rendered template changes on takeover and a
  timing-only probe renders without a handler.
- Native sidecar workers launching vllm-rs directly without backendFramework.
  Operator 1.6.x resolved an undetected backend; main infers vllm (#15923).
  The resolved backend is part of the generated DCD spec and therefore of the
  worker hash, so every such DGD gets a new worker generation on takeover.

Pin both behaviors to the immutable operator origin, as #17182 did for the
frontend startup probe. Workloads created by operator 1.6.x or earlier keep the
1.6.x rendering and worker hash; workloads created by 1.7.0 or later get the
new behavior. The native engine gate lives in determineBackendFrameworkForComponent
so DetectBackendFrameworkFromArgs keeps classifying running prefill Pods for
LoRA fallback. ValidateSnapshotFailover reads the same origin so admission
resolves the backend exactly as rendering does.

Unit tests render the fixtures under both origins and pin the worker hashes and
merged probes produced by release/1.6.0 for legacy origins.

Correct the platform README upgrade notes: describe the origin-pinned backend
inference; move the complete-health-probes (#15581) and Grove rollout budget
(#15725) notes from v1.6.0 to v1.7.0, where they shipped; restore the v1.6.0
Grove notes to the shipped v0.1.0-alpha.14 text. Update the curated DCD
reference for both gates.

Related: #16623, #17181
Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
tmonty12 added a commit that referenced this pull request Nov 13, 2026
Remove the vllm-rs branch that #15923 added to DetectBackendFrameworkFromArgs
instead of keeping it for running-Pod classification. Detection recognizes
Dynamo launchers only, as it did in 1.6.x, for workload resolution and for
vLLM prefill LoRA fallback alike; native engine prefill Pods do not receive
LoRA fallback, which #15463 already tracks for sidecar mode. Native engines
set backendFramework explicitly.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
tmonty12 added a commit that referenced this pull request Nov 16, 2026
…ive engine backend inference

An operator-only upgrade from 1.6.x rolls two kinds of unchanged DGDs:

- frontendSidecar containers with partially authored probes. Operator 1.6.x
  field-merged the probe with the frontend defaults; main replaces it as a
  complete value (#15581), so the rendered template changes on takeover and a
  timing-only probe renders without a handler.
- Native sidecar workers launching vllm-rs directly without backendFramework.
  Operator 1.6.x resolved an undetected backend; main infers vllm (#15923).
  The resolved backend is part of the generated DCD spec and therefore of the
  worker hash, so every such DGD gets a new worker generation on takeover.

Pin both behaviors to the immutable operator origin, as #17182 did for the
frontend startup probe. Workloads created by operator 1.6.x or earlier keep the
1.6.x rendering and worker hash; workloads created by 1.7.0 or later get the
new behavior. The native engine gate lives in determineBackendFrameworkForComponent
so DetectBackendFrameworkFromArgs keeps classifying running prefill Pods for
LoRA fallback. ValidateSnapshotFailover reads the same origin so admission
resolves the backend exactly as rendering does.

Unit tests render the fixtures under both origins and pin the worker hashes and
merged probes produced by release/1.6.0 for legacy origins.

Correct the platform README upgrade notes: describe the origin-pinned backend
inference; move the complete-health-probes (#15581) and Grove rollout budget
(#15725) notes from v1.6.0 to v1.7.0, where they shipped; restore the v1.6.0
Grove notes to the shipped v0.1.0-alpha.14 text. Update the curated DCD
reference for both gates.

Related: #16623, #17181
Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
tmonty12 added a commit that referenced this pull request Nov 16, 2026
Remove the vllm-rs branch that #15923 added to DetectBackendFrameworkFromArgs
instead of keeping it for running-Pod classification. Detection recognizes
Dynamo launchers only, as it did in 1.6.x, for workload resolution and for
vLLM prefill LoRA fallback alike; native engine prefill Pods do not receive
LoRA fallback, which #15463 already tracks for sidecar mode. Native engines
set backendFramework explicitly.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
tmonty12 added a commit that referenced this pull request Nov 16, 2026
…ive engine backend inference

An operator-only upgrade from 1.6.x rolls two kinds of unchanged DGDs:

- frontendSidecar containers with partially authored probes. Operator 1.6.x
  field-merged the probe with the frontend defaults; main replaces it as a
  complete value (#15581), so the rendered template changes on takeover and a
  timing-only probe renders without a handler.
- Native sidecar workers launching vllm-rs directly without backendFramework.
  Operator 1.6.x resolved an undetected backend; main infers vllm (#15923).
  The resolved backend is part of the generated DCD spec and therefore of the
  worker hash, so every such DGD gets a new worker generation on takeover.

Pin both behaviors to the immutable operator origin, as #17182 did for the
frontend startup probe. Workloads created by operator 1.6.x or earlier keep the
1.6.x rendering and worker hash; workloads created by 1.7.0 or later get the
new behavior. The native engine gate lives in determineBackendFrameworkForComponent
so DetectBackendFrameworkFromArgs keeps classifying running prefill Pods for
LoRA fallback. ValidateSnapshotFailover reads the same origin so admission
resolves the backend exactly as rendering does.

Unit tests render the fixtures under both origins and pin the worker hashes and
merged probes produced by release/1.6.0 for legacy origins.

Correct the platform README upgrade notes: describe the origin-pinned backend
inference; move the complete-health-probes (#15581) and Grove rollout budget
(#15725) notes from v1.6.0 to v1.7.0, where they shipped; restore the v1.6.0
Grove notes to the shipped v0.1.0-alpha.14 text. Update the curated DCD
reference for both gates.

Related: #16623, #17181
Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
tmonty12 added a commit that referenced this pull request Nov 16, 2026
Remove the vllm-rs branch that #15923 added to DetectBackendFrameworkFromArgs
instead of keeping it for running-Pod classification. Detection recognizes
Dynamo launchers only, as it did in 1.6.x, for workload resolution and for
vLLM prefill LoRA fallback alike; native engine prefill Pods do not receive
LoRA fallback, which #15463 already tracks for sidecar mode. Native engines
set backendFramework explicitly.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — 7f743a99 Deployed Oct 17, 2026 by Pranjulcr7 via ok-to-test #34692
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deployment::k8s Relates to dynamo deployment in kubernetes documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor fix planner size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants