multiple Owner reference on secret with cluster-api and clusterresourceset

[sebglon]

Is your feature request related to a problem? Please describe.
we deploy some secret generated by vaultstaticsecret with the clusterresourceset (cluster-api).
but VSS fail to sync secret due to 2 reference owner set on the secret

Describe the solution you'd like
VSO have to check if the owner is present on the list but not exact match

Describe alternatives you've considered

Additional context
here is an error sync log

Failed to update k8s secret: invalid ownerReferences, refs=[]v1.OwnerReference{v1.OwnerReference{APIVersion:"secrets.hashicorp.com/v1beta1", Kind:"VaultStaticSecret", Name:"osc-cloud-contr
oller-manager-secret", UID:"8d29d9be-6076-4a43-bb6a-1e33bfb50ea6", Controller:(*bool)(nil), BlockOwnerDeletion:(*bool)(nil)}, v1.OwnerReference{APIVersion:"addons.cluster.x-k8s.io/v1beta1", Kind:"ClusterResourceSet", Name:"osc-cloud-controller-manager", U
ID:"b3f06448-83dd-4a7e-8fa5-7ff58c09340d", Controller:(*bool)(nil), BlockOwnerDeletion:(*bool)(nil)}}
not the owner of the destination Secret devops-b/osc-cloud-controller-manager-secret

[sebglon]

Maybe adding some parameters on the owner like:

  ownerReferences:
  - apiVersion: secrets.hashicorp.com/v1beta1
    kind: VaultStaticSecret
    name: image-credentials-provider
    uid: b23b06af-7423-40eb-9c37-e993deb88dd0
    blockOwnerDeletion: false
    controller: true
  - apiVersion: addons.cluster.x-k8s.io/v1beta1
    kind: ClusterResourceSet
    name: image-credentials
    uid: 840da7a0-fe56-4da7-9c06-2abb6ef42138

Docuemented here: https://github.com/kubernetes/design-proposals-archive/blob/acc25e14ca83dfda4f66d8cb1f1b491f26e78ffe/api-machinery/controller-ref.md

[sebglon]

Here is the context with cluster-api:

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: image-credentials-provider
spec:
  vaultAuthRef: cluster-api-${cluster_env}
  mount: secret
  type: kv-v2
  path: cluster-api/${cluster_env}/image-credentials-provider/
  refreshAfter: 5s
  destination:
    name: image-credentials-provider
    create: true
    overwrite: true
    type: addons.cluster.x-k8s.io/resource-set
    transformation:
      excludeRaw: true
      excludes:
        - .*
      templates:
        secret.yaml:
          text: |
            apiVersion: v1
            kind: Secret
            metadata:
              name: image-credentials-provider
              namespace: kube-system
            data:
              {{- range $key, $value := .Secrets -}}
              {{- printf "\n  %s: %s" $key ($value | b64enc) -}}
              {{ end -}}

But the generated secret as one more owner added by cluster-api.
ownerReference is not only used by controller that create or update resources. but it can be used to design dependency read like used by cluster-api.

One evolution to implement is to not check all ownerRefences but only check with the VSO ref is present or not.

@benashz can you help me to validate the concept. I am available to implement the feature.

[sadoMasupilami]

we would need that as well for confluent kafka operator which writes itself onto secrets as owner as well