add a new custom backend for extensions?

[jstrachan]

TL;DR allow alternative controllers/operators/steps to populate Secret resources from ExternalSecret CRDs; so add a backendType to the CRD which the external secrets controller ignores and indicates some custom/out of band mechanism does the population.

Background

We're using https://github.com/godaddy/kubernetes-external-secrets by default in Jenkins X 3.x now and its working awesome. However we have a requirement to support just regular kubernetes secrets (i.e. no vault / cloud secret storage) - particularly for low resource usage (e.g. folks trying Jenkins X on minikube).

We'd like to still use ExternalSecret CRDs though but have a backendType string we can use which if the external secrets controller is running its just ignored for that ExternalSecret. Maybe folks could combine custom backend with vault/others too?

This might sound strange; but from a GitOps perspective we want to check in all resources to git - apart from Secrets. So the pipelines in Jenkins X converts all Secret resources generated by helm / helmfile / kustomize / kpt to ExternalSecret resources and check them in.

Then when using vault / (Amazon | Azure | Google) Secret manager then the external secrets controller populates the Secret resources. When not using any of those, we can populate those Secret resources automatically via Jenkins X so we don't need the external secrets controller to do anything. So we basically want a kind of noop backendType.

Currently we've a minor hack ;) - when folks are not using vault / * Secret Manager, we install a modified version of the external secrets CRD which adds a new local backendType (then we don't install the external secrets controller) but it'd be nice to submit a PR on this git repository and add this backendType for real to both the CRD and the controller.

So I thought I'd raise this issue first to see what you thought about the idea. If you're cool with it I'll try create a PR.

While the use case for Jenkins X might seem fairly specific - I could see folks wanting to use custom scripts/pipelines/tools/mechanisms to populate Secret resources - but still reuse the ExternalSecret CRDs to describe the Secret resources that are required.

Aside

Incidentally we've got a little jx-secret CLI too for working with ExternalSecret resources and being able to edit them, import/export them, view which secrets are valid/populated + auto-generate them using a JSON-schema like schema (annotated on the ExternalSecret resources) or wait for the mandatory secrets to be populated etc. See the CLI reference docs here

I'm hoping to do a blog soon on how we're using external secrets; they are awesome!

Questions

What should we call this new noop backendType? I wondered about something like kubernetesSecret or something but maybe we need a name to indicate some out of band / custom process is used to populate them. So maybe custom for the backendType?

[Flydiverny]

Not sure I fully understand the purpose. What do you expect KES to do with an ExternalSecret that has this noop backend? Fully ignore the resource? What would the external secret resource actually contain? How does jenkins x populate the secrets? Eg. does jenkins X also watch KES CRDs and if the noop backend is set, it will create a secret based on the CRD spec getting values from itself?

I guess it somewhat aligns with ideas of standardizing the CRD spec (#477)

[jstrachan]

So we'd like KES to just ignore ExternalSecret resources with backendType noop (or whatever its called) so that the Secret can be populated out of band. In the case of Jenkins X we have an out of band way to make the Secrets.

i.e. we're using the ExternalSecret resource to describe the metadata of how a Secret should look (type, annotations, labels, data entries etc) and want to allow other processes to actually populate the Secret.

e.g. we have a little CLI tool jx secret populate which can generate random passwords or other generators, use templates or default values defined in a schema (populated on the ExternalSecret resources via annotations).

Or folks can explicitly edit a secret in (local k8s Secret / vault / GSM) via jx secret edit

However the Secret is populated we can verify which secrets are populated via jx secret verify which is totally agnostic of how the Secret is populated (KES or some out of band process/tool/approach)

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity.