Compose multiple overlays

[nicksnyder]

Thanks for creating kustomize!

At Sourcegraph we deploy our Kubernetes application to customers. We have core services that are deployed to every customer and then a few optional services that each customer can choose to deploy or not (e.g. monitoring, alerting, SSDs, etc.).

A single customer may wish to include multiple optional services that we provide. It would be ideal if they could do a single build that composes multiple overlays together.

This could be accomplished via command line or by defining a new kustomization.yaml that references the overlays to be composed.

More generally, I think I am asking for there to be no distinction between "base" and "overlay". It would be ideal if kustomize could compose an arbitrary number of layers together in a particular order.

Consider the following resources

# frontend/kustomization.yaml
resources:
- frontend.Deployment.yaml

# frontend-replicas/kustomization.yaml
resources:
- frontend-replicas.yaml

# monitoring/kustomization.yaml
resources:
- monitoring.Deployment.yaml

Ideal:

kustomize build frontend/ frontend-replicas/ monitoring/

[paultiplady]

I would like to see multiple overlays as well -- my use-case is a bit more mundane;

I have my base deployment on base/deployment.yaml, and my dev environment config in overlays/development/deployment.yaml, but I want to do a review app overlays/review/deployment.yaml that shares code from that development environment, and also adds new config too.

I think that being able to specify an overlay as a base (i.e review's base is overlays/development) would suffice for my case, but @nicksnyder it sounds like you need to be able to do build A B C or build A C or build A B, i.e. the overlay=>base tree could be dynamic, so I think my suggestion wouldn't work for your case.

[Liujingfang1]

@nicksnyder You can put all these three dirs as bases. Create another kustomizationl.yaml with

bases:
- frontend
- frontend-replicas
- monitoring

[nicksnyder]

I cannot put both frontend and frontend-replicas as bases because they are the same resource. frontend-replicas updates the replica count of frontend.

A given overlay may specify new resources and patch existing resources.

It seems like I would need to have a structure like this:

required-base/
  - frontend.Service.yaml  
  - ... 
optional/
  - monitoring/
    - base/
      - kustomization.yaml
      - monitor.Deployment.yaml
      - ...
    - patches/
      - frontend.Service.yaml
      - ...
  - backup/
    - base/
      - kustomization.yaml
      - backup.Deployment.yaml
      - ...
    - patches/
      - frontend.Service.yaml
      - ...

Then a customer has to write a kustomization file like

bases:
- required-base
- optional/monitoring/base
patches:
- optional/monitoring/patches/frontend.Service.yaml
- ...

Would this work?

It seems a little bit clunky that

1. A customer has to manually list all patch files (seems error prone)
2. I can't "bundle" all the monitoring or backup yaml files together in some way
3. The customer's kustomization is brittle to file additions/removals to monitoring and backup because they are maintaining their own kustomization.

[Liujingfang1]

Your structure will work. To help your customers and reduce the errors, you can provide one or more kustomizations. Put the optional parts in comments and give customers instructions to uncomment for the components they want.