Skip to content

fix(operator): enable GMS snapshot by default - #12685

Merged
galletas1712 merged 3 commits into
mainfrom
fix/operator-enable-gms-snapshot-default
Aug 7, 2026
Merged

galletas1712 merged 3 commits into
mainfrom
fix/operator-enable-gms-snapshot-default

Conversation

@galletas1712

@galletas1712 galletas1712 commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Remove the temporary GMS Snapshot feature gate and enable ordinary IntraPod Snapshot + GMS through the existing Checkpoint capability.
  • Keep GMS Snapshot Pod replacement under the Checkpoint gate and preserve the existing InterPod restriction.
  • Reject unsupported Snapshot combinations during admission and before reconciliation mutates primary or child resources.
  • Record admission-bypassed stored incompatibilities as terminal failure status without retrying unchanged specifications, while allowing invalid deleting resources to finalize.
  • Temporarily reject Snapshot + active/passive failover; a future PR will enable the initial narrow vLLM configuration.

Implements the second delivery step of #12671.

Where should the reviewer start?

Start with deploy/operator/internal/checkpoint/compatibility.go, then follow its use from webhook validation into the DGD and DCD reconciliation preflight.

Validation

  • go test -count=1 ./internal/checkpoint ./internal/features ./internal/controller ./internal/webhook/validation ./internal/webhook/setup ./cmd
  • go vet ./internal/checkpoint ./internal/features ./internal/controller ./internal/webhook/validation ./internal/webhook/setup ./cmd
  • helm lint deploy/helm/charts/platform/components/operator --set discoveryBackend=kubernetes
  • Rendered the operator Helm subchart with default and custom environment values.

Related Issues

@galletas1712
galletas1712 requested review from a team as code owners August 5, 2026 04:12
@github-actions github-actions Bot added fix documentation Improvements or additions to documentation deployment::k8s Relates to dynamo deployment in kubernetes labels Aug 5, 2026
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Open in Devin Review

@datadog-official

datadog-official Bot commented Aug 5, 2026 •

Copy link
Copy Markdown

Pipelines

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 50.90%

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 1b3550b | Docs | Datadog PR Page | Give us feedback!

@galletas1712
galletas1712 force-pushed the fix/operator-enable-gms-snapshot-default branch from 213fc34 to 681375a Compare August 5, 2026 16:54
@copy-pr-bot

copy-pr-bot Bot commented Aug 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Base automatically changed from fix/operator-enable-gms-snapshot to main August 5, 2026 22:01
@galletas1712
galletas1712 force-pushed the fix/operator-enable-gms-snapshot-default branch 2 times, most recently from afb94f5 to 3a0e204 Compare August 5, 2026 22:02
@coderabbitai

coderabbitai Bot commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The operator removes the temporary GMSSnapshot feature gate. Checkpoint creation and resolution no longer use that gate. A shared compatibility validator rejects unsupported checkpoint combinations during admission and reconciliation.

Changes

Checkpoint and GMS snapshot consolidation

Layer / File(s) Summary
Remove GMSSnapshot configuration and runtime gate
deploy/helm/charts/platform/components/operator/..., deploy/operator/internal/features/..., deploy/operator/cmd/main.go
Removes GMSSnapshot Helm settings, environment injection, feature-gate definitions, and startup registration.
Simplify checkpoint resolution and creation
deploy/operator/internal/checkpoint/..., deploy/operator/internal/controller/dgd_checkpoints_reconciler.go, deploy/operator/internal/controller/dynamocheckpoint_controller_test.go
Removes feature-gate parameters and validation from checkpoint creation and resolution. GMS checkpoint tests now verify acceptance with the Checkpoint gate.
Add checkpoint compatibility preflight
deploy/operator/internal/checkpoint/compatibility.go, deploy/operator/internal/controller/dynamocomponentdeployment_controller.go, deploy/operator/internal/controller/dynamographdeployment_controller.go, deploy/operator/internal/controller/*_test.go
Adds validation for inter-pod GMS and configured failover. Reconciliation aggregates errors, records failure status, avoids active-resource mutation, and finalizes deleting resources first.
Apply compatibility rules to admission
deploy/operator/internal/webhook/validation/...
Uses the shared validator for admission errors. Tests cover create and update paths for accepted intra-pod GMS, rejected inter-pod GMS, rejected active/passive failover, and disabled failover acceptance.

Estimated code review effort: 4 (Complex) | ~45 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 9.52% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the main change: removing the temporary gate and enabling GMS Snapshot through the operator.
Description check ✅ Passed The description explains the changes, reviewer starting point, validation, and related issues, but does not use all template headings or issue-link syntax.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (6)
deploy/operator/cmd/main.go (1)

646-650: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a story comment for this registration block.

Add a one-line comment that states that GMS pod replacement is registered when the Checkpoint gate is enabled.

Proposed change
+	// Register GMS pod replacement when checkpoint support is enabled.
 	if runtimeConfig.Gate.Enabled(features.Checkpoint) {

As per coding guidelines, “In Go code, put a one-line story comment above every multi-line block of logically connected code.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/operator/cmd/main.go` around lines 646 - 650, Add a one-line story
comment immediately above the Checkpoint-gated registration block in the setup
flow, stating that GMS pod replacement is registered when the Checkpoint feature
gate is enabled; leave the existing controller.SetupGMSPodReplacement behavior
unchanged.

Source: Coding guidelines

deploy/operator/internal/checkpoint/checkpoint_test.go (1)

463-482: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add t.Log headings for the new test steps.

  • deploy/operator/internal/checkpoint/checkpoint_test.go#L463-L482: Add headings before checkpoint creation and GMS configuration assertions.
  • deploy/operator/internal/checkpoint/checkpoint_test.go#L786-L792: Add a heading before resolving the enabled configuration without a reference or identity.
  • deploy/operator/internal/controller/dynamocheckpoint_controller_test.go#L574-L612: Add headings before fixture preparation, reconciliation, and status and Job assertions.

As per coding guidelines, “In Go tests, use t.Log to explain the test's story, with one heading before each block implementing a test step.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/operator/internal/checkpoint/checkpoint_test.go` around lines 463 -
482, Add t.Log headings for each test-step block: in
deploy/operator/internal/checkpoint/checkpoint_test.go:463-482, log before
checkpoint creation and before GMS configuration assertions; in
deploy/operator/internal/checkpoint/checkpoint_test.go:786-792, log before
resolving the enabled configuration without a reference or identity; and in
deploy/operator/internal/controller/dynamocheckpoint_controller_test.go:574-612,
log before fixture preparation, reconciliation, and status/Job assertions.

Source: Coding guidelines

deploy/operator/internal/controller/dynamocomponentdeployment_controller.go (1)

169-176: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

The deleted branch is now unreachable.

The new early return at Line 134 handles every object that carries a deletion timestamp. HandleFinalizer can report deleted == true only for a deleting object, so deleted is always false here and Lines 174-176 never run. Drop the unused result to keep the control flow honest.

♻️ Proposed simplification
-	deleted, err := commonController.HandleFinalizer(ctx, dynamoComponentDeployment, r.Client, r)
-	if err != nil {
+	if _, err = commonController.HandleFinalizer(ctx, dynamoComponentDeployment, r.Client, r); err != nil {
 		logs.Error(err, "Failed to handle finalizer")
 		return ctrl.Result{}, err
 	}
-	if deleted {
-		return ctrl.Result{}, nil
-	}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/operator/internal/controller/dynamocomponentdeployment_controller.go`
around lines 169 - 176, Remove the unused deleted result and its unreachable
conditional return from the HandleFinalizer call in the reconcile flow; retain
the existing error handling and return the normal result after successful
finalizer handling.
deploy/operator/internal/webhook/validation/dynamocheckpoint_handler_test.go (1)

166-182: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove the duplicated table row.

This case builds the same fixture as "unprepared pod wiring aggregates in Kubernetes PodSpec declaration order" at Lines 116-132 and expects the identical six errors. After the gmsSnapshot expectation was dropped, the row adds no new behavior. Delete it and keep the earlier row.

As per coding guidelines: "Prefer table tests over one-off Go tests and do not duplicate behavior already covered by a table."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/operator/internal/webhook/validation/dynamocheckpoint_handler_test.go`
around lines 166 - 182, Remove the duplicated “preparation failures aggregate”
table entry from the validation test cases, keeping the earlier “unprepared pod
wiring aggregates in Kubernetes PodSpec declaration order” case and its existing
expectations unchanged.

Source: Coding guidelines

deploy/operator/internal/webhook/validation/dynamocomponentdeployment_validation_envtest_test.go (1)

454-514: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

The "without job" case names describe a distinction the rule does not make. ValidateCheckpointCompatibility never reads checkpoint.job, none of these cases sets Job, and no paired "with job" case exists. The suffix therefore misleads a reader about the rule under test in both files.

  • deploy/operator/internal/webhook/validation/dynamocomponentdeployment_validation_envtest_test.go#L454-L514: drop the "without job" suffix from the four inter-pod GMS case names, for example "v1beta1 checkpoint with inter-pod GMS is rejected on create".
  • deploy/operator/internal/webhook/validation/dynamographdeployment_validation_envtest_test.go#L1182-L1250: drop the same suffix from the four matching case names so both files name the rule identically.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@deploy/operator/internal/webhook/validation/dynamocomponentdeployment_validation_envtest_test.go`
around lines 454 - 514, Rename the four inter-pod GMS test cases in
deploy/operator/internal/webhook/validation/dynamocomponentdeployment_validation_envtest_test.go
lines 454-514 and the four matching cases in
deploy/operator/internal/webhook/validation/dynamographdeployment_validation_envtest_test.go
lines 1182-1250 to remove the misleading “without job” suffix, while preserving
the existing test behavior and expected errors.
deploy/operator/internal/controller/dynamographdeployment_controller_test.go (1)

177-187: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert failure ObservedGeneration semantics.

Assert that ready.ObservedGeneration == dgd.Generation and stored.Status.ObservedGeneration remains zero. Failure conditions identify the failed generation, but status-level ObservedGeneration advances only after error-free reconciliation.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/operator/internal/controller/dynamographdeployment_controller_test.go`
around lines 177 - 187, Extend the failure-status assertions in the test around
ready and stored.Status to verify ready.ObservedGeneration equals
dgd.Generation, while stored.Status.ObservedGeneration remains zero. Keep the
existing failure state, reason, and message assertions unchanged.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/operator/internal/controller/dynamographdeployment_controller.go`:
- Around line 142-150: Stop retrying terminal compatibility failures in both
deploy/operator/internal/controller/dynamographdeployment_controller.go:142-150
and
deploy/operator/internal/controller/dynamocomponentdeployment_controller.go:163-167:
persist the failure condition, return any status-write error, and otherwise
return ctrl.Result{}, nil. In the DCD controller, update Available=False with
the joined compatibility message instead of assigning it to the named err;
update the compatibility tests to expect no reconcile error.

---

Nitpick comments:
In `@deploy/operator/cmd/main.go`:
- Around line 646-650: Add a one-line story comment immediately above the
Checkpoint-gated registration block in the setup flow, stating that GMS pod
replacement is registered when the Checkpoint feature gate is enabled; leave the
existing controller.SetupGMSPodReplacement behavior unchanged.

In `@deploy/operator/internal/checkpoint/checkpoint_test.go`:
- Around line 463-482: Add t.Log headings for each test-step block: in
deploy/operator/internal/checkpoint/checkpoint_test.go:463-482, log before
checkpoint creation and before GMS configuration assertions; in
deploy/operator/internal/checkpoint/checkpoint_test.go:786-792, log before
resolving the enabled configuration without a reference or identity; and in
deploy/operator/internal/controller/dynamocheckpoint_controller_test.go:574-612,
log before fixture preparation, reconciliation, and status/Job assertions.

In `@deploy/operator/internal/controller/dynamocomponentdeployment_controller.go`:
- Around line 169-176: Remove the unused deleted result and its unreachable
conditional return from the HandleFinalizer call in the reconcile flow; retain
the existing error handling and return the normal result after successful
finalizer handling.

In
`@deploy/operator/internal/controller/dynamographdeployment_controller_test.go`:
- Around line 177-187: Extend the failure-status assertions in the test around
ready and stored.Status to verify ready.ObservedGeneration equals
dgd.Generation, while stored.Status.ObservedGeneration remains zero. Keep the
existing failure state, reason, and message assertions unchanged.

In
`@deploy/operator/internal/webhook/validation/dynamocheckpoint_handler_test.go`:
- Around line 166-182: Remove the duplicated “preparation failures aggregate”
table entry from the validation test cases, keeping the earlier “unprepared pod
wiring aggregates in Kubernetes PodSpec declaration order” case and its existing
expectations unchanged.

In
`@deploy/operator/internal/webhook/validation/dynamocomponentdeployment_validation_envtest_test.go`:
- Around line 454-514: Rename the four inter-pod GMS test cases in
deploy/operator/internal/webhook/validation/dynamocomponentdeployment_validation_envtest_test.go
lines 454-514 and the four matching cases in
deploy/operator/internal/webhook/validation/dynamographdeployment_validation_envtest_test.go
lines 1182-1250 to remove the misleading “without job” suffix, while preserving
the existing test behavior and expected errors.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 50de8d55-267a-4019-a093-a2cb9a90ef63

📥 Commits

Reviewing files that changed from the base of the PR and between c5c076a and 3a0e204.

📒 Files selected for processing (27)
  • deploy/helm/charts/platform/components/operator/templates/deployment.yaml
  • deploy/helm/charts/platform/components/operator/values.yaml
  • deploy/operator/cmd/main.go
  • deploy/operator/config/samples/nvidia.com_v1alpha1_dynamocheckpoint.yaml
  • deploy/operator/internal/checkpoint/checkpoint_test.go
  • deploy/operator/internal/checkpoint/compatibility.go
  • deploy/operator/internal/checkpoint/compatibility_test.go
  • deploy/operator/internal/checkpoint/gms_snapshot_gate.go
  • deploy/operator/internal/checkpoint/resolve.go
  • deploy/operator/internal/checkpoint/resource.go
  • deploy/operator/internal/controller/dgd_checkpoints_reconciler.go
  • deploy/operator/internal/controller/dgd_checkpoints_reconciler_test.go
  • deploy/operator/internal/controller/dgd_gms_resource_claims_reconciler_test.go
  • deploy/operator/internal/controller/dynamocheckpoint_controller.go
  • deploy/operator/internal/controller/dynamocheckpoint_controller_test.go
  • deploy/operator/internal/controller/dynamocomponentdeployment_controller.go
  • deploy/operator/internal/controller/dynamocomponentdeployment_controller_test.go
  • deploy/operator/internal/controller/dynamocomponentdeployment_renderer.go
  • deploy/operator/internal/controller/dynamographdeployment_controller.go
  • deploy/operator/internal/controller/dynamographdeployment_controller_test.go
  • deploy/operator/internal/features/gates.go
  • deploy/operator/internal/features/gates_test.go
  • deploy/operator/internal/webhook/validation/dynamocheckpoint.go
  • deploy/operator/internal/webhook/validation/dynamocheckpoint_handler_test.go
  • deploy/operator/internal/webhook/validation/dynamocomponentdeployment_validation_envtest_test.go
  • deploy/operator/internal/webhook/validation/dynamographdeployment_validation_envtest_test.go
  • deploy/operator/internal/webhook/validation/shared_v1beta1.go
💤 Files with no reviewable changes (10)
  • deploy/operator/internal/checkpoint/gms_snapshot_gate.go
  • deploy/operator/internal/controller/dynamocomponentdeployment_renderer.go
  • deploy/operator/config/samples/nvidia.com_v1alpha1_dynamocheckpoint.yaml
  • deploy/helm/charts/platform/components/operator/values.yaml
  • deploy/operator/internal/checkpoint/resource.go
  • deploy/operator/internal/controller/dynamocheckpoint_controller.go
  • deploy/operator/internal/webhook/validation/dynamocheckpoint.go
  • deploy/operator/internal/checkpoint/resolve.go
  • deploy/operator/internal/features/gates_test.go
  • deploy/helm/charts/platform/components/operator/templates/deployment.yaml

@julienmancuso

Copy link
Copy Markdown
Contributor

lgtm, the current guide still says GMS Snapshot is disabled and requires the internal gate removed by this PR. After merge, users will be directed toward nonexistent featureGates.gmsSnapshot configuration. Update snapshot.md to describe ordinary IntraPod support and the remaining InterPod/failover restrictions.

@galletas1712
galletas1712 force-pushed the fix/operator-enable-gms-snapshot-default branch from 79d7343 to e3ff0f5 Compare August 6, 2026 17:01
@galletas1712

Copy link
Copy Markdown
Contributor Author

/ok to test e3ff0f5

Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
@galletas1712
galletas1712 force-pushed the fix/operator-enable-gms-snapshot-default branch from e3ff0f5 to 2446435 Compare August 6, 2026 21:24
@galletas1712

Copy link
Copy Markdown
Contributor Author

/ok to test 2446435

1 similar comment
@galletas1712

Copy link
Copy Markdown
Contributor Author

/ok to test 2446435

@galletas1712
galletas1712 force-pushed the fix/operator-enable-gms-snapshot-default branch from 2446435 to bfb4cee Compare August 6, 2026 23:28
@galletas1712

Copy link
Copy Markdown
Contributor Author

/ok to test bfb4cee

1 similar comment
@galletas1712

Copy link
Copy Markdown
Contributor Author

/ok to test bfb4cee

Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
@galletas1712
galletas1712 force-pushed the fix/operator-enable-gms-snapshot-default branch from bfb4cee to 1b3550b Compare August 6, 2026 23:46
@galletas1712

Copy link
Copy Markdown
Contributor Author

/ok to test 1b3550b

@galletas1712
galletas1712 merged commit 6a4b074 into main Aug 7, 2026
184 of 196 checks passed
@galletas1712
galletas1712 deleted the fix/operator-enable-gms-snapshot-default branch August 7, 2026 01:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deployment::k8s Relates to dynamo deployment in kubernetes documentation Improvements or additions to documentation fix size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants