Repository navigation
Support for Kubernetes ApplySets #5
Description
Kubernetes "ApplySets" are objects that group resources that are applied together, allowing resources that are removed in a subsequent apply to be pruned or even pruning the entire group of objects.
It would be nice if Nyl supported ApplySets in such a fashion that they are straight forward to use when applying manifests with kubectl, allowing users to easily upgrade applications and pruning removed resources.
In an ideal scenario, one would simply run something like:
$ nyl template . | KUBECTL_APPLYSET=true -f - --pruneThough it may be necessary to reference the apply set with --applyset=kind/name. If the invokation of kubectl is error prone, we could also consider adding a command or option to Nyl to actually run kubectl for you.
$ nyl template . --applyNyl can provide it's own parent resource (e.g. ApplySet.nyl.io/v1). It can automatically detect the presence of the apply set resource in the file, and as part of a source file where manifests are loaded from, Nyl can automatically assign the part-of labels (although that may not be what the ApplySet spec intended/how kubectl implements it, I'm not 100% sure yet).
apiVersion: nyl.io
kind: ApplySet
metadata:
name: my-applysetRelevant resources:
Activity
feature: Add basic `ApplySet` support to `nyl template` command (#5)
NiklasRosenstein commentedon Jul 31, 2024
The current implementation is a bit bumpy.
Nyl detects if there is a
ApplySet.nyl.io/v1resource in a manifest source file. If yes:It ensure that it's
applyset.kubernetes.io/idlabel is set correctly.It sets
applyset.kubernetes.io/toolingtokubectl/1.30applyset.kubernetes.io/contains-group-kindsto the kinds from all resources that belong to the set.That is because kubectl can't create the
ApplySetcustom resource, and complains if the resource does not have these labels:On a subsequent run with
--applyset, it complains that the field manager for these labels don't matchBecause kubectl can't create the custom
ApplySetresource, it must first be applied manually. Otherwise, it complains:error: custom resource ApplySet parents cannot be created automaticallyBut Nyl outputs the ApplySet resource along with the rest, thus the first
--applyset-less apply will contain all other resources as well (hence no resource is getting theapplyset.kubernetes.io/part-oflabel assigned by kubectl).We could set the
applyset.kubernetes.io/part-oflabel on all resources before spitting them out. This works on a normal apply, but with--applyset, kubectl will complain that the input resources already have the label defined.error: ApplySet label "applyset.kubernetes.io/part-of" already set in input dataPresumably this works a bit better with
SecretorConfigMapas the apply set resource, because kubectl can create those. Will have to try it, but they are both namespaces and I think having cluster-scoped apply sets could be quite useful.feature: add `nyl template --apply` option (#5)
NiklasRosenstein commentedon Aug 1, 2024
I've added a
nyl template --applyoption that does the following (assuming there is anApplySetin the manifest source file):--no-applyset-part-ofoption, which will have Nyl not add theapplyset.kubernetes.io/part-oflabel to the generated resources.ApplySetresource exists withkubectl apply --server-side --force-conflicts.kubectl apply --server-side --force-conflicts --applyset=... --pruneWhat's not so pretty yet:
In the beginning, you always get this warning from kubectl:
We need to use
--force-conflict, otherwise we get conflicts with the ApplySet annotations:NiklasRosenstein commentedon Sep 7, 2024
I've added a
nyl template --diffoption in #7, but again it doesn't work well with apply sets and deleted resources. I was expecting thatkubectl diff -f <(echo) -l applyset.kubernetes.io/part-of=...would include all resources that belong to the apply set to show as deleted, but they don't.NiklasRosenstein commentedon Jan 7, 2025
There's also an issue that maybe stems from generating a wrong kind in the annotation for ArgoCD CronWorkflows: