issues upgrading to latest

[linkandzelda91]

Describe the bug
I pulled my existing values and tried to upgrade, got an error:

gdewitt@k3s:~$ !872
helm upgrade --install --namespace "ess" ess oci://ghcr.io/element-hq/ess-helm/matrix-stack -f ./essvalues.yaml --wait
Pulled: ghcr.io/element-hq/ess-helm/matrix-stack:26.6.2
Digest: sha256:752ddc775dc7cd5663d114968a37e647c6fa29c3d26501a3a0b00b714404a712
Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
	* job ess-synapse-check-config failed: BackoffLimitExceeded

gdewitt@k3s:~$ !872
helm upgrade --install --namespace "ess" ess oci://ghcr.io/element-hq/ess-helm/matrix-stack -f ./essvalues.yaml --wait
Pulled: ghcr.io/element-hq/ess-helm/matrix-stack:26.6.2
Digest: sha256:752ddc775dc7cd5663d114968a37e647c6fa29c3d26501a3a0b00b714404a712
Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
	* job ess-synapse-check-config failed: BackoffLimitExceeded

gdewitt@k3s:~$ kubectl logs -n ess job/ess-synapse-check-config -c render-config
failed to open file: open /secrets/ess-generated/SYNAPSE_EXTRA: no such file or directory

gdewitt@k3s:~$ kubectl get secret ess-generated -n ess
NAME            TYPE     DATA   AGE
ess-generated   Opaque   12     306d
gdewitt@k3s:~$ kubectl get jobs -n ess
NAME                          STATUS     COMPLETIONS   DURATION   AGE
ess-deployment-markers-post   Complete   1/1           4s         293d
ess-deployment-markers-pre    Complete   1/1           7s         5m1s
ess-init-secrets              Complete   1/1           9s         4m53s
ess-synapse-check-config      Failed     0/1           4m39s      4m39s
gdewitt@k3s:~$ kubectl describe secret ess-generated -n ess
Name:         ess-generated
Namespace:    ess
Labels:       app.kubernetes.io/component=matrix-tools
              app.kubernetes.io/instance=ess-init-secrets
              app.kubernetes.io/managed-by=matrix-tools-init-secrets
              app.kubernetes.io/name=init-secrets
              app.kubernetes.io/part-of=matrix-stack
              app.kubernetes.io/version=0.8.0
Annotations:  <none>

Type:  Opaque

Data
====
SYNAPSE_REGISTRATION_SHARED_SECRET:               32 bytes
MAS_ENCRYPTION_SECRET:                            64 bytes
MAS_RSA_PRIVATE_KEY:                              2374 bytes
POSTGRES_ADMIN_PASSWORD:                          32 bytes
POSTGRES_MATRIX_AUTHENTICATION_SERVICE_PASSWORD:  32 bytes
POSTGRES_SYNAPSE_PASSWORD:                        32 bytes
SYNAPSE_EXTRA:                                    114 bytes
SYNAPSE_MACAROON:                                 32 bytes
SYNAPSE_SIGNING_KEY:                              54 bytes
ELEMENT_CALL_LIVEKIT_SECRET:                      32 bytes
MAS_ECDSA_PRIME256V1_PRIVATE_KEY:                 138 bytes
MAS_SYNAPSE_SHARED_SECRET:                        32 bytes

a lot of this troubleshooting is from AI guidance, I do not use k8s frequently so if it's gone wrong sorry.

I'm on ubuntu if it matters

gdewitt@k3s:~$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=25.04
DISTRIB_CODENAME=plucky
DISTRIB_DESCRIPTION="Ubuntu 25.04"
gdewitt@k3s:~$ 

[linkandzelda91]

here's my essvalues.yaml, removed my oauth2 secret

gdewitt@k3s:~$ cat essvalues.yaml 
elementWeb:
  ingress:
    host: chat.mayogirl.com
ingress:
  tlsEnabled: false
matrixAuthenticationService:
  additional:
    1-custom-config:
      config: |
        upstream_oauth2:
          providers:
            - id: "01K4DY4T64MFS95MQBQPQC87DH"
              human_name: Authentik
              issuer: "https://authentik.mayogirl.com/application/o/matrix/"
              client_id: "removed"
              client_secret: "removed"
              token_endpoint_auth_method: client_secret_basic
              scope: "openid profile email"
              claims_imports:
                localpart:
                  action: require
                  template: "{{ user.preferred_username }}"
                displayname:
                  action: suggest
                  template: "{{ user.name }}"
                email:
                  action: suggest
                  template: "{{ user.email }}"
                  set_email_verification: always
  ingress:
    host: account.mayogirl.com
matrixRTC:
  ingress:
    host: mrtc.mayogirl.com
  sfu:
    additional:
      10-external-ip.yml:
        config: |
          rtc:
            use_external_ip: false
            # To workaround https://github.com/livekit/livekit/issues/2088
            # Any IP address is acceptable, it doesn't need to be a correct one,
            # it just needs to be present to get LiveKit to skip checking all local interfaces
            # We assign here a TEST-NET IP which is
            # overridden by the NODE_IP env var at runtime
            node_ip: 198.51.100.1
serverName: mayogirl.com
synapse:
  ingress:
    host: mayogirl.com
elementAdmin:
  enabled: true
  ingress:
    host: essadmin.mayogirl.com
gdewitt@k3s:~$ 

[benbz]

• What version were you upgrading from?
• What Helm version are you using?
• Can you provide the output of kubectl get -n ess job/ess-synapse-check-config -o yaml when you're performing an upgrade that's failing

[linkandzelda91]

• What version were you upgrading from?
• What Helm version are you using?
• Can you provide the output of kubectl get -n ess job/ess-synapse-check-config -o yaml when you're performing an upgrade that's failing

Thank you.
I was on version matrix-stack-25.9.1
Helm is Version:"v3.18.6"

gdewitt@k3s:~$ kubectl get -n ess job/ess-synapse-check-config -o yaml
apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "0"
  creationTimestamp: "2026-07-08T02:56:51Z"
  generation: 1
  labels:
    app.kubernetes.io/component: matrix-server
    app.kubernetes.io/instance: ess-synapse-check-config
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: synapse-check-config
    app.kubernetes.io/part-of: matrix-stack
    app.kubernetes.io/version: v1.155.0
    helm.sh/chart: matrix-stack-26.6.2
    k8s.element.io/postgres-password-synapse-hash: dbd8665cfb39383754f02ce18502f9979ad8a74d
    k8s.element.io/synapse-config-hash: db3ac926ca443a0f1f9296c7f6edc60b994500ed
    k8s.element.io/synapse-instance: ess-synapse-check-config
    k8s.element.io/synapse-secret-hash: 0f710c6d017350fb3da93e921a169898a26fbf08
  name: ess-synapse-check-config
  namespace: ess
  resourceVersion: "52386166"
  uid: 85c77209-e9df-4b99-b01d-ca74199a3de4
spec:
  backoffLimit: 0
  completionMode: NonIndexed
  completions: 1
  manualSelector: false
  parallelism: 1
  podReplacementPolicy: TerminatingOrFailed
  selector:
    matchLabels:
      batch.kubernetes.io/controller-uid: 85c77209-e9df-4b99-b01d-ca74199a3de4
  suspend: false
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: matrix-server
        app.kubernetes.io/instance: ess-synapse-check-config
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: synapse-check-config
        app.kubernetes.io/part-of: matrix-stack
        app.kubernetes.io/version: v1.155.0
        batch.kubernetes.io/controller-uid: 85c77209-e9df-4b99-b01d-ca74199a3de4
        batch.kubernetes.io/job-name: ess-synapse-check-config
        controller-uid: 85c77209-e9df-4b99-b01d-ca74199a3de4
        job-name: ess-synapse-check-config
        k8s.element.io/postgres-password-synapse-hash: dbd8665cfb39383754f02ce18502f9979ad8a74d
        k8s.element.io/synapse-config-hash: db3ac926ca443a0f1f9296c7f6edc60b994500ed
        k8s.element.io/synapse-instance: ess-synapse-check-config
        k8s.element.io/synapse-secret-hash: 0f710c6d017350fb3da93e921a169898a26fbf08
    spec:
      automountServiceAccountToken: false
      containers:
      - command:
        - python3
        - -m
        - synapse.config
        - -c
        - /conf/homeserver.yaml
        env:
        - name: LD_PRELOAD
          value: libjemalloc.so.2
        image: oci.element.io/synapse:v1.155.0
        imagePullPolicy: Always
        name: synapse
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 50m
            memory: 50Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /conf/homeserver.yaml
          name: rendered-config
          readOnly: true
          subPath: homeserver.yaml
        - mountPath: /secrets/ess-synapse-hook
          name: secret-8d6c4f5d7d48
          readOnly: true
        - mountPath: /secrets/ess-generated
          name: secret-eda259c754a3
          readOnly: true
        - mountPath: /conf/log_config.yaml
          name: plain-config
          subPath: log_config.yaml
        - mountPath: /media
          name: media
        - mountPath: /tmp
          name: tmp
      dnsPolicy: ClusterFirst
      initContainers:
      - args:
        - render-config
        - -output
        - /conf/homeserver.yaml
        - /config-templates/01-homeserver-underrides.yaml
        - /secrets/ess-generated/SYNAPSE_EXTRA
        - /config-templates/04-homeserver-overrides.yaml
        - /config-templates/05-main.yaml
        env:
        - name: APPLICATION_NAME
          value: '{{ hostname }}'
        - name: SYNAPSE_POSTGRES_PASSWORD
          value: '{{ readfile "/secrets/ess-generated/POSTGRES_SYNAPSE_PASSWORD" |
            quote }}'
        image: ghcr.io/element-hq/ess-helm/matrix-tools:0.8.0
        imagePullPolicy: Always
        name: render-config
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 50m
            memory: 50Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /config-templates
          name: plain-config
          readOnly: true
        - mountPath: /secrets/ess-synapse-hook
          name: secret-8d6c4f5d7d48
          readOnly: true
        - mountPath: /secrets/ess-generated
          name: secret-eda259c754a3
          readOnly: true
        - mountPath: /conf
          name: rendered-config
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 10091
        runAsGroup: 10091
        runAsNonRoot: true
        runAsUser: 10091
        seccompProfile:
          type: RuntimeDefault
      serviceAccount: ess-synapse-check-config
      serviceAccountName: ess-synapse-check-config
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/instance: ess-synapse-check-config
        maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
      volumes:
      - configMap:
          defaultMode: 420
          name: ess-synapse-hook
        name: plain-config
      - name: secret-8d6c4f5d7d48
        secret:
          defaultMode: 420
          secretName: ess-synapse-hook
      - name: secret-eda259c754a3
        secret:
          defaultMode: 420
          secretName: ess-generated
      - emptyDir:
          medium: Memory
        name: rendered-config
      - emptyDir:
          medium: Memory
        name: media
      - emptyDir:
          medium: Memory
        name: tmp
status:
  conditions:
  - lastProbeTime: "2026-07-08T02:56:57Z"
    lastTransitionTime: "2026-07-08T02:56:57Z"
    message: Job has reached the specified backoff limit
    reason: BackoffLimitExceeded
    status: "True"
    type: FailureTarget
  - lastProbeTime: "2026-07-08T02:56:57Z"
    lastTransitionTime: "2026-07-08T02:56:57Z"
    message: Job has reached the specified backoff limit
    reason: BackoffLimitExceeded
    status: "True"
    type: Failed
  failed: 1
  ready: 0
  startTime: "2026-07-08T02:56:51Z"
  terminating: 0
  uncountedTerminatedPods: {}

[benbz]

The Job YAML all looks correct.

For my 2nd question I meant the output of helm version.

For now, can you try kubectl -n ess delete job/ess-synapse-check-config and retrying the upgrade.

I'd like to highlight that as per the 26.4.1 release notes as you've jumped from 25.9.1 -> 26.6.2 without stopping at a release between 25.12.2 & 26.4.0, https://github.com/element-hq/ess-helm/blob/main/docs/maintenance.md#manually-running-the-event-resigning-background-job applies to your deployment