Intermittent hang at clickhouse extract-from-config on NFS-backed storage (v25.8)

[pinkyjpainadath]

Description:
We are experiencing an intermittent issue where ClickHouse pods get stuck during startup at the extract-from-config phase when using NFS-mounted storage. The issue resolves itself after multiple restarts, but it blocks pod readiness meanwhile.

Environment:

ClickHouse version: 25.8.9.20 (official build)
Docker image: clickhouse/clickhouse-server:25.8
Kubernetes: v1.28.x
NFS storage: v3, options: rw,noatime,sync,vers=3,rsize=8192,wsize=8192,hard,proto=tcp,timeo=600,retrans=2,sec=sys
CPU / Memory: 6 cores / 16Gi
CI/CD: ArgoCD deployment

Symptoms:
Pod starts, but clickhouse extract-from-config hangs indefinitely.
No logs are produced in /var/log/clickhouse-server.
Issue occurs even with CLICKHOUSE_SKIP_USER_SETUP=1 and local paths configured.
NFS is healthy and writable; file creation as ClickHouse user succeeds.
After several restarts (sometimes hours), pod starts successfully.

Pod state example:

Containers:
  clickhouse:
    State: Running
    Ready: False
    Restart Count: 1
    Command:
      clickhouse extract-from-config --config-file /etc/clickhouse-server/config.xml --key=path
    Limits:
      cpu: 8
      memory: 16Gi

Relevant logs / outputs:

# ps aux
root          1  99.3  0.0 556000  2448 ?        Rsl  01:14   1:10 /usr/bin/clickhouse-server --config-file=/etc/clickhouse-server/config.xml
root         22  0.0  0.0 10500  2836 ?        S    00:49   0:00 /bin/bash /entrypoint.sh
root         23 96.0  0.0 556016 18332 ?        Rl   00:49   0:21 clickhouse extract-from-config --config-file /etc/clickhouse-server/config.xml --key=path

# df -hT /var/lib/clickhouse
Filesystem                           Type  Size  Used Avail Use% Mounted on
10.74.41.50:/k8s_xdi_dev_clickhouse_ins00 nfs 101G  920M 100G  1% /var/lib/clickhouse

# ls -l /var/lib/clickhouse/preprocessed_configs
-rw-r-----  1 clickhouse clickhouse 92021 Nov  5 23:17 config.xml
-rw-r-----  1 clickhouse clickhouse 10072 Nov  5 23:17 users.xml

Additional Notes:

The hang seems independent of write activity; extract-from-config is read-light.
Multiple restarts eventually succeed.

Request:
Could the team investigate potential NFS-related hangs in extract-from-config, especially for first-time setups or pre-existing volumes? Any recommended workarounds or startup options would be appreciated.

Below is the chi

apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"clickhouse.altinity.com/v1","kind":"ClickHouseInstallation","metadata":{"annotations":{"prometheus.io/port":"7000","prometheus.io/scrape":"true"},"labels":{"app.kubernetes.io/instance":"xdi-dev_clickhouse-ins-test"},"name":"clickhouse-xdi-dev-ins","namespace":"xdi-dev"},"spec":{"configuration":{"clusters":[{"layout":{"replicasCount":1,"shardsCount":1},"name":"ins","templates":{"dataVolumeClaimTemplate":"chi-data-xdi-dev-ins","podTemplate":"chi-pod-xdi-dev-ins"}}],"files":{"kafka_ssl.xml":"\u003cclickhouse\u003e\n 
      \u003ckafka\u003e\n   
      \u003ckafka_auto_offset_reset\u003eearliest\u003c/kafka_auto_offset_reset\u003e\n   
      \u003csecurity_protocol\u003eSSL\u003c/security_protocol\u003e\n   
      \u003cssl_ca_location\u003e/etc/clickhouse-client/certs/ca.pem\u003c/ssl_ca_location\u003e\n   
      \u003cssl_certificate_location\u003e/etc/clickhouse-client/certs/client_cert.pem\u003c/ssl_certificate_location\u003e\n   
      \u003cssl_key_location\u003e/etc/clickhouse-client/certs/client_key.key\u003c/ssl_key_location\u003e\n 
      \u003c/kafka\u003e\n\u003c/clickhouse\u003e\n"},"users":{"admin/access_management":1,"admin/k8s_secret_password":"xdi-dev/clickhouse-ins-creds/admin","admin/networks/ip":"0.0.0.0/0","user/k8s_secret_password":"xdi-dev/clickhouse-ins-creds/user","user/networks/ip":"0.0.0.0/0","user/readonly":1}},"defaults":{"templates":{"dataVolumeClaimTemplate":"chi-data-xdi-dev-ins","podTemplate":"chi-pod-xdi-dev-ins"}},"stop":"no","taskID":"1","templates":{"podTemplates":[{"name":"chi-pod-xdi-dev-ins","spec":{"containers":[{"env":[{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"s3-clickhouse-credentials"}}},{"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"s3-clickhouse-credentials"}}},{"name":"CLICKHOUSE_SKIP_USER_SETUP","value":"1"}],"image":"clickhouse/clickhouse-server:25.8","imagePullPolicy":"IfNotPresent","name":"clickhouse","resources":{"limits":{"cpu":8,"memory":"16Gi"},"requests":{"cpu":6,"memory":"16Gi"}},"volumeMounts":[{"mountPath":"/var/lib/clickhouse","name":"chi-data-xdi-dev-ins"},{"mountPath":"/etc/clickhouse-client/certs","name":"kafka-certs","readOnly":true}]}],"volumes":[{"name":"kafka-certs","secret":{"secretName":"kafka-certs"}}]}}],"volumeClaimTemplates":[{"name":"chi-data-xdi-dev-ins","reclaimPolicy":"Retain","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"100Gi"}},"storageClassName":"xdi-dev-chi-ins"}}]},"troubleshoot":"no"}}
    prometheus.io/port: '7000'
    prometheus.io/scrape: 'true'
  creationTimestamp: '2025-11-06T01:14:40Z'
  finalizers:
    - finalizer.clickhouseinstallation.altinity.com
  generation: 2
  labels:
    app.kubernetes.io/instance: xdi-dev_clickhouse-ins-test
  name: clickhouse-xdi-dev-ins
  namespace: xdi-dev
  resourceVersion: '4917548229'
  uid: b45a7cc7-3d27-4000-bda0-faae3b28d4d1
spec:
  configuration:
    clusters:
      - layout:
          replicasCount: 1
          shardsCount: 1
        name: ins
        templates:
          dataVolumeClaimTemplate: chi-data-xdi-dev-ins
          podTemplate: chi-pod-xdi-dev-ins
    files:
      kafka_ssl.xml: |
        <clickhouse>
          <kafka>
            <kafka_auto_offset_reset>earliest</kafka_auto_offset_reset>
            <security_protocol>SSL</security_protocol>
            <ssl_ca_location>/etc/clickhouse-client/certs/ca.pem</ssl_ca_location>
            <ssl_certificate_location>/etc/clickhouse-client/certs/client_cert.pem</ssl_certificate_location>
            <ssl_key_location>/etc/clickhouse-client/certs/client_key.key</ssl_key_location>
          </kafka>
        </clickhouse>
    users:
      admin/access_management: 1
      admin/k8s_secret_password: xdi-dev/clickhouse-ins-creds/admin
      admin/networks/ip: 0.0.0.0/0
      user/k8s_secret_password: xdi-dev/clickhouse-ins-creds/user
      user/networks/ip: 0.0.0.0/0
      user/readonly: 1
  defaults:
    templates:
      dataVolumeClaimTemplate: chi-data-xdi-dev-ins
      podTemplate: chi-pod-xdi-dev-ins
  stop: 'no'
  taskID: '1'
  templates:
    podTemplates:
      - name: chi-pod-xdi-dev-ins
        spec:
          containers:
            - env:
                - name: AWS_ACCESS_KEY_ID
                  valueFrom:
                    secretKeyRef:
                      key: AWS_ACCESS_KEY_ID
                      name: s3-clickhouse-credentials
                - name: AWS_SECRET_ACCESS_KEY
                  valueFrom:
                    secretKeyRef:
                      key: AWS_SECRET_ACCESS_KEY
                      name: s3-clickhouse-credentials
                - name: CLICKHOUSE_SKIP_USER_SETUP
                  value: '1'
              image: 'clickhouse/clickhouse-server:25.8'
              imagePullPolicy: IfNotPresent
              name: clickhouse
              resources:
                limits:
                  cpu: 8
                  memory: 16Gi
                requests:
                  cpu: 6
                  memory: 16Gi
              volumeMounts:
                - mountPath: /var/lib/clickhouse
                  name: chi-data-xdi-dev-ins
                - mountPath: /etc/clickhouse-client/certs
                  name: kafka-certs
                  readOnly: true
          volumes:
            - name: kafka-certs
              secret:
                secretName: kafka-certs
    volumeClaimTemplates:
      - name: chi-data-xdi-dev-ins
        reclaimPolicy: Retain
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 100Gi
          storageClassName: xdi-dev-chi-ins
  troubleshoot: 'no'

[Slach]

Issue is not related to clickhouse-operator itself, but let's try to figure out

Could you upgrade clickhouse to 25.8.11.66? I see you already use 25.8.
Let's temporarily change imagePullPolicy: IfNotPresent to imagePullPolicy: Always

Could you provide startup logs kubectl logs -n xdi-dev --container=clickhouse --since=24h chi-clickhouse-xdi-dev-ins-0-0-0

Moreover, could you share kubectl get pod -n xdi-dev chi-clickhouse-xdi-dev-ins-0-0-0 -o yaml

[pinkyjpainadath]

@Slach Currently pod is running. Had the above issue today twice. When the issue happens there are no logs in the container. clickhouse server process will not be running in the container. I will change the imagepullpolicy to Always and try. this is the yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 611f65d340ec460e749fb2a9a731240f2da751e12cb2e7c6d0c193ddab7d831d
    cni.projectcalico.org/podIP: xxxx
    cni.projectcalico.org/podIPs: xxx
    prometheus.io/port: "7000"
    prometheus.io/scrape: "true"
  creationTimestamp: "2025-11-06T01:26:00Z"
  generateName: chi-clickhouse-xdi-dev-ins-ins-0-0-
  labels:
    app.kubernetes.io/instance: xdi-dev_clickhouse-ins-test
    clickhouse.altinity.com/app: chop
    clickhouse.altinity.com/chi: clickhouse-xdi-dev-ins
    clickhouse.altinity.com/cluster: ins
    clickhouse.altinity.com/namespace: xdi-dev
    clickhouse.altinity.com/ready: "yes"
    clickhouse.altinity.com/replica: "0"
    clickhouse.altinity.com/shard: "0"
    controller-revision-hash: chi-clickhouse-xdi-dev-ins-ins-0-0-7866876976
    statefulset.kubernetes.io/pod-name: chi-clickhouse-xdi-dev-ins-ins-0-0-0
  name: chi-clickhouse-xdi-dev-ins-ins-0-0-0
  namespace: xdi-dev
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: chi-clickhouse-xdi-dev-ins-ins-0-0
    uid: a5acbe28-ed47-485b-ac55-f4ea0be9b336
  resourceVersion: "4917547293"
  uid: d06a41f4-151a-46d2-88ca-85f8ece6ef6b
spec:
  containers:
  - env:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          key: AWS_ACCESS_KEY_ID
          name: s3-clickhouse-credentials
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          key: AWS_SECRET_ACCESS_KEY
          name: s3-clickhouse-credentials
    - name: CLICKHOUSE_SKIP_USER_SETUP
      value: "1"
    image: clickhouse/clickhouse-server:25.8
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 10
      httpGet:
        path: /ping
        port: http
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 3
      successThreshold: 1
      timeoutSeconds: 1
    name: clickhouse
    ports:
    - containerPort: 9000
      name: tcp
      protocol: TCP
    - containerPort: 8123
      name: http
      protocol: TCP
    - containerPort: 9009
      name: interserver
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /ping
        port: http
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 3
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "8"
        memory: 16Gi
      requests:
        cpu: "6"
        memory: 16Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/clickhouse
      name: chi-data-xdi-dev-ins
    - mountPath: /etc/clickhouse-client/certs
      name: kafka-certs
      readOnly: true
    - mountPath: /etc/clickhouse-server/config.d/
      name: chi-clickhouse-xdi-dev-ins-common-configd
    - mountPath: /etc/clickhouse-server/users.d/
      name: chi-clickhouse-xdi-dev-ins-common-usersd
    - mountPath: /etc/clickhouse-server/conf.d/
      name: chi-clickhouse-xdi-dev-ins-deploy-confd-ins-0-0
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-djm97
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostAliases:
  - hostnames:
    - chi-clickhouse-xdi-dev-ins-ins-0-0
    ip: 127.0.0.1
  hostname: chi-clickhouse-xdi-dev-ins-ins-0-0-0
  nodeName: pudc-oakw-115.networks.in.telstra.com.au
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: chi-clickhouse-xdi-dev-ins-ins-0-0
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: chi-data-xdi-dev-ins
    persistentVolumeClaim:
      claimName: chi-data-xdi-dev-ins-chi-clickhouse-xdi-dev-ins-ins-0-0-0
  - name: kafka-certs
    secret:
      defaultMode: 420
      secretName: kafka-certs
  - configMap:
      defaultMode: 420
      name: chi-clickhouse-xdi-dev-ins-common-configd
    name: chi-clickhouse-xdi-dev-ins-common-configd
  - configMap:
      defaultMode: 420
      name: chi-clickhouse-xdi-dev-ins-common-usersd
    name: chi-clickhouse-xdi-dev-ins-common-usersd
  - configMap:
      defaultMode: 420
      name: chi-clickhouse-xdi-dev-ins-deploy-confd-ins-0-0
    name: chi-clickhouse-xdi-dev-ins-deploy-confd-ins-0-0
  - name: kube-api-access-djm97
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-11-06T01:26:00Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2025-11-06T01:26:13Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2025-11-06T01:26:13Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2025-11-06T01:26:00Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://ab06b8df38c9301a0721ac12907aa23fa86cf28cbf9ab58b2fec071261c2b4d4
    image: docker.io/clickhouse/clickhouse-server:25.8
    imageID: sha256:b29f0bf648bd9a22b7b9df6ef033cd7d2a3f19aa48ebca3d1dd12e4bcc483a5c
    lastState: {}
    name: clickhouse
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2025-11-06T01:26:03Z"
  hostIP: 10.41.40.190
  phase: Running
  podIP: 10.244.45.19
  podIPs:
  - ip: 10.244.45.19
  qosClass: Burstable
  startTime: "2025-11-06T01:26:00Z"

[Slach]

ok. let's change CHI and skip entrypoint completelly

spec:
   templates:
    podTemplates:
      - name: chi-pod-xdi-dev-ins
        spec:
          containers:
              name: clickhouse
              image: 'clickhouse/clickhouse-server:25.8'
              command:
              - /usr/bin/clickhouse-server
              - --config-file=/etc/clickhouse-server/config.xml

[pinkyjpainadath]

even with entrypoint ommited, container still hang. now it hangs at clickhouse-server process. If i try 4 or 5 times, once it would go through.

[Slach]

could you provide logs for clickhouse container?

What show in /var/log/clichouse-server/clickhouse-server.log?

[Slach]

let's change kind: ClickHouseInstallation

spec:
  troubleshoot: "yes" 

in this case liveness and readinessProbes should be skipped during reconcile and you can attach to container for debug

[pinkyjpainadath]

@Slach when this issue is there there is no container log at all. even with troubleshoot , logs are not there. I can see the process is hanging, even though liveness and readiness probes are not failing

[Slach]

if container start
check /var/log/clickhouse-server/clickhouse-server.log is this file empty?