Skip to content

test(operator): make the Grove pathway reachable from envtest - #13302

Closed
tzulingk wants to merge 1 commit into
tzulingk/elastic-ep-p3-leader-servicefrom
tzulingk/grove-envtest-enablement
Closed

tzulingk wants to merge 1 commit into
tzulingk/elastic-ep-p3-leader-servicefrom
tzulingk/grove-envtest-enablement

Conversation

@tzulingk

Copy link
Copy Markdown
Contributor

Overview:

Makes the operator's Grove pathway reachable from envtest, so Grove behaviour can be covered in CI instead of only by hand on a cluster. Stacked on #13178 — review that one first; this branch is based on it so its first test has a fix to prove.

Tracked as DYN-3987. Opened as a draft: it should land after #13178.

Details:

Raised by @julienmancuso in review on #13178, asking for an envtest covering Owns(&corev1.Service{}). Writing it turned out to be blocked by test infrastructure rather than by the test.

Blocker 1 — no Grove CRD fixtures. internal/controller/testing/grove.io/ held only clustertopologybindings.yaml, so the API server rejected the PodCliqueSet the workloads reconciler creates and the reconcile died before reaching anything else. Added permissive stubs for podcliquesets, podcliques, and podcliquescalinggroups, in the same style as the existing stub. They use x-kubernetes-preserve-unknown-fields, so the schema stays Grove's concern and these fixtures carry no version coupling to Grove releases.

Blocker 2 — the gate was unreachable in practice. The defaulting webhook resolves a DGD's nvidia.com/workload-provider at Create from the gates its own manager was started with. The controller package's shared Env starts from features.Defaults(), where Grove is off, so a DGD admitted there is stamped component regardless of what gates the test gives its own controller — and ensureWorkloadProvider then honours that annotation verbatim:

nvidia.com/enable-grove:true             <- asked for Grove
nvidia.com/workload-provider:component   <- got the component pathway

operatorenv.Options already supports a RuntimeConfig override (env.go:182-184), so no change to operatorenv was needed. Grove tests get their own isolated Env via Env.RunT. Flipping the shared Env instead would silently move every other test in the package onto Grove.

Two further details this turned up, both recorded as comments in the test:

  • the reconcile needs Namespace.Restricted, because cluster-wide mode requires an RBAC manager no envtest runs;
  • it needs a ScaleClient, because the Grove scaler drives PodClique replicas through the scale subresource and fails closed with scale client is nil.

Known limitation: envtest runs no Grove controller and no Grove webhooks, so a PodCliqueSet stays at observedGeneration: 0 and Grove's own admission rules (e.g. the minAvailable: 0 rejection behind grove#676) do not reproduce. This covers our rendering and watch wiring, not Grove's acceptance. Cluster tests are still required for that.

Where should the reviewer start?

  • deploy/operator/internal/controller/dgd_grove_envtest_test.go — the groveEnv declaration and its comment explain why a separate Env is required rather than reusing sharedEnv; the test itself is the first consumer.
  • deploy/operator/internal/controller/testing/grove.io/*.yaml — three stubs; confirm the permissive shape is what we want long term.

Validation:

  • TestGroveDGDRecreatesAnOwnedServiceAfterDeletion passes on this branch (8.1s) and fails on main, which lacks feat(operator): emit a headless leader Service for elastic EP (Phase 3) #13178's watch: the deleted Service was never recreated, so the owned-Service watch is not enqueuing its DGD.
  • Full make envtest green: internal/controller and internal/webhook/validation.
  • make lint clean, go build ./..., gofmt clean.

No envtest could exercise the Grove pathway, so every Grove behaviour in the
operator was verifiable only by hand on a real cluster. Two things were missing.

The Grove types had no envtest fixtures: internal/controller/testing/grove.io
carried only clustertopologybindings.yaml, so the API server rejected the
PodCliqueSet the workloads reconciler creates. Add permissive stubs for
podcliquesets, podcliques, and podcliquescalinggroups, matching the existing
stub's style -- x-kubernetes-preserve-unknown-fields keeps the schema Grove's
concern, so these carry no version coupling to Grove releases.

The gate was also unreachable in practice. The defaulting webhook resolves a
DGD's workload provider at Create from the gates its own manager was started
with, and the controller package's shared Env starts from features.Defaults(),
where Grove is off. A DGD admitted there is stamped "workload-provider:
component" regardless of the gates the test's own controller carries. Options
already supports a RuntimeConfig override, so the fix is an isolated Env for
Grove tests via RunT rather than a change to operatorenv itself; flipping the
shared Env would silently move every other test in the package onto Grove.

The first test to use it covers the DGD controller's owned-Service watch: it
deletes a Service the DGD owns and asserts a new one appears. It passes on this
branch and fails without the watch, which is the coverage asked for in review on
the Phase 3 Service PR.

Two details worth keeping: the reconcile needs Namespace.Restricted set, because
cluster-wide mode requires an RBAC manager no envtest runs, and it needs a
ScaleClient, because the Grove scaler drives PodClique replicas through the
scale subresource and fails closed without one.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
@github-actions github-actions Bot added test deployment::k8s Relates to dynamo deployment in kubernetes labels Aug 15, 2026
@tzulingk
tzulingk marked this pull request as ready for review August 17, 2026 16:05
@tzulingk
tzulingk requested a review from a team as a code owner August 17, 2026 16:05
@tzulingk
tzulingk deleted the branch tzulingk/elastic-ep-p3-leader-service August 19, 2026 23:12
@tzulingk tzulingk closed this Aug 19, 2026
@tzulingk

Copy link
Copy Markdown
Contributor Author

Superseded by #13577. GitHub auto-closed this when #13178 merged and deleted its base branch, and a PR closed that way cannot be reopened. Same branch, same single commit, now based on main.

@tzulingk

Copy link
Copy Markdown
Contributor Author

Superseded by #13577 — same content rebased onto main. This PR auto-closed when #13178 merged and its base branch (tzulingk/elastic-ep-p3-leader-service) was deleted; it can't be reopened, so the rebased work continues in #13577.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deployment::k8s Relates to dynamo deployment in kubernetes size/L test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant