Support of priorityClassName (without overriding entire statefulSet.spec.template.spec.containers field)

[jimmyjamesbaldwin]

Is your feature request related to a problem? Please describe.
We are deploying a rabbitmq cluster using the cluster-operator and want to ensure it has high priority for scheduling within the cluster using a PriorityClass. As per #892, it's possible to override the spec.override.statefulSet.spec.template.spec.priorityClassName field, but Kubernetes does not allow for patching of this value without overriding the entire spec.override.statefulSet.spec.template.spec.containers which is managed by the operator and contains a lot of configuration which I'm hesitant to replicate.

Is there a way to configure the StatefulSet pods to have a PriorityClass set without overriding the entire .spec.containers field?

I'm using the following CRD:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  labels:
    app: rabbitmq-frm
  name: rabbitmq-frm
  namespace: rabbitmq
spec:
  replicas: 3
  image: xxx-registry-1-docker-io.jfrog.io/rabbitmq:3.10.5-management
  service:
    type: ClusterIP
  override:
    statefulSet:
      spec:
        template:
          spec:
            priorityClassName: system-cluster-critical
  persistence:
    storageClassName: managed-premium
    storage: 150Gi
  resources:
    requests:
      cpu: ${rabbitmq_cpu_req_limit}
      memory: ${rabbitmq_memory_req_limit}
    limits:
      cpu: ${rabbitmq_cpu_req_limit}
      memory: ${rabbitmq_memory_req_limit}
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:              
            - store
          topologyKey: "kubernetes.io/hostname"
  rabbitmq:
    additionalPlugins:
      <snip>
    additionalConfig: |
      <snip>

but as can be seen when running Terraform to apply the changes, Kubernetes can't patch this field without providing the whole .spec.containers:

│ Error: rabbitmq/rabbitmq-frm failed to run apply: error when applying patch:
│ <snipped>
│ to:
│ Resource: "rabbitmq.com/v1beta1, Resource=rabbitmqclusters", GroupVersionKind: "rabbitmq.com/v1beta1, Kind=RabbitmqCluster"
│ Name: "rabbitmq-frm", Namespace: "rabbitmq"
│ for: "/tmp/745797788kubectl_manifest.yaml": RabbitmqCluster.rabbitmq.com "rabbitmq-frm" is invalid: spec.override.statefulSet.spec.template.spec.containers: Required value
│ 
│   with kubectl_manifest.rabbitmq_frm[0],
│   on rabbitmq_frm.tf line 1, in resource "kubectl_manifest" "rabbitmq_frm":
│    1: resource "kubectl_manifest" "rabbitmq_frm" {

Thanks!

[ChunyiLyu]

Hi @jimmyjamesbaldwin, there's no need to provide the entire spec.override.statefulSet.spec.template.spec.containers. Instead you can set container definition to an empty list, which satisfies k8s validation api and won't override the default container definition.
For example:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  labels:
    app: rabbitmq-frm
  name: rabbitmq-frm
  namespace: rabbitmq
spec:
  replicas: 3
  image: xxx-registry-1-docker-io.jfrog.io/rabbitmq:3.10.5-management
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers: []
            priorityClassName: system-cluster-critical
  persistence:
    storageClassName: managed-premium
    storage: 150Gi
...

You can see the production example as a reference:

cluster-operator/docs/examples/production-ready/rabbitmq.yaml

Line 38 in 48c9f0e

containers: []

[jimmyjamesbaldwin]

Hi @ChunyiLyu , thanks for posting this! I wasn't aware that setting the container spec to an empty list behaved this way so didn't spot this in the examples, very useful to know! Thanks again

[ChunyiLyu]

Thanks for reporting this issue @jimmyjamesbaldwin! I added an example in the repo so it would help others in the future: https://github.com/rabbitmq/cluster-operator/blob/main/docs/examples/priority-class/priority-class.yaml