Ingress NGINX v1.10.2 & v1.11.0 throw core dumps

[auwaerter]

What happened:

I was updating from ingress-nginx 1.10.1 to 1.11.0 using helm upgrade ingress-nginx /helm/ingress-nginx --install -f /helm/ingress-nginx/custom-values.yaml

All five ingresses assigned to that ingress-nginx instance went unresponsive, came up a few times but then crashed constantly.

When opening the ingress hosts in the browser, i see lots of messages such as:

2024/07/09 07:00:34 [alert] 23#23: worker process 229 exited on signal 11 (core dumped)
2024/07/09 07:00:35 [alert] 23#23: worker process 29 exited on signal 11 (core dumped)
2024/07/09 07:02:17 [alert] 23#23: worker process 328 exited on signal 11 (core dumped)
2024/07/09 07:02:22 [alert] 23#23: worker process 295 exited on signal 11 (core dumped)

A fallback to 1.10.1 fixed the issue.

What you expected to happen:

A normal (rolling) deployment with responsive hosts and no core dumps.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller
Release: v1.11.0
Build: 96dea88
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.25.5

Kubernetes version (use kubectl version):

Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.6

Environment:

• Cloud provider or hardware configuration: AWS

• OS (e.g. from /etc/os-release): Ubuntu 20.04.6 LTS / containerd://1.7.18

• Kernel (e.g. uname -a): 5.15.0-107-generic

• Basic cluster related info:

  • kubectl version: Already described, see above...
  • kubectl get nodes -o wide: Already described, see above...

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress

ingress-nginx           ingress-nginx           19              2024-07-09 09:12:31.294492684 +0200 CEST        deployed        ingress-nginx-4.11.0                    1.11.0
ingress-nginx-internet  ingress-nginx-internet  15              2024-05-02 08:55:37.339164187 +0200 CEST        deployed        ingress-nginx-4.10.1                    1.10.1

• If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>
  values.json

• if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses
    ingressclasses.json
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
    getallwide.log
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
    pod-describe.log
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
    svc-describe.log
  • kubectl -n <ingresscontrollernamespace> logs <ingresscontrollerservicename>
    ingress-nginx-controller-5bd44bf869-4k9kf.log

How to reproduce this issue:

• Have a ingress-nginx controller running in 1.10.1.
• Update to 1.11.0 using helm with the given configuration.
• Check the logs / ingress hosts assigned.

Anything else we need to know:

Attached logs:

values.json
ingressclasses.json
getallwide.log
pod-describe.log
svc-describe.log
ingress-nginx-controller-5bd44bf869-4k9kf.log

[zeeZ]

Cluster getting core dumps with 1.11.0 (OVH):

• K8S v1.29.3
• Ubuntu 22.04.4 LTS
• 5.15.0-107-generic
• containerd://1.6.32

Cluster with zero dumps so far (self-hosted):

• K8S v1.27.10
• Flatcar Container Linux by Kinvolk 3815.2.5 (Oklo)
• 6.1.96-flatcar
• containerd://1.7.11

Sadly the cluster where I could easily access core dumps is the one that's stable.

[thomaspeitz]

Setup:

• EKS / AWS
• 1.29

Which makes the problem in our case worse, ist that it completely filled our nodes disks.
So not only ingress controller stopped working but nodes completely got destroyed.
On cloud no issue, they get rotated but I am afraid of those getting the issue on onprem.

We already have 9 upvotes on this one.
Could you put a warning into the release nodes till this gets resolved?
Is there an option to remove the release from release page till it is properly debugged?

[ConnorJC3]

Kernel version appears to affect the bug - I'm unsure of the exact cutoff but all working nodes have version 6.x and all broken have 5.x (typically 5.15) in my testing.

[bootc]

Kernel version appears to affect the bug - I'm unsure of the exact cutoff but all working nodes have version 6.x and all broken have 5.x (typically 5.15) in my testing.

Sorry to rain on this, but my cluster with kernel 6.9.7 was failing in the same way. Debian Trixie (testing), RKE2, K8s 1.30.2, containerd 1.7.17.

[Gacko]

Does the same happen with v1.10.2? That would help narrowing down the root cause.

[bootc]

Does the same happen with v1.10.2? That would help narrowing down the root cause.

Yes, it did on my cluster. 1.10.1 OK, 1.10.2 and 1.11.0 both bad.

[zeeZ]

Does the same happen with v1.10.2? That would help narrowing down the root cause.

Yes.

[Gacko]

Ok, thanks! That really helps as we "only" introduced patches to v1.10.2.

[rouke-broersma]

I have the following environments:

1 (onprem):
Ingress-nginx: 1.11.0
Kubernetes: Talos v1.30.0
Kernel: 6.6.29-talos
Issue: Yes, however due to my specific setup I think my replicas are not exiting at the same time which means I still have inbound connectivity
Error Message: worker process 2362 exited on signal 11

2 (onprem, not exactly the same as 1 however no significant differences..):
Ingress-nginx: 1.10.2
Kubernetes: Talos v1.30.0
Kernel: 6.6.29-talos
Issue: Yes, and there is no inbound connectivity due to exiting at the same time
Error Message: worker process 2362 exited on signal 11

3 (Azure):
Ingress-nginx: 1.11.0
Kubernetes: AKS v1.29.2
Kernel: 5.15.158.2-1.cm2
Issue: No errors at all

[strongjz]

thanks, I'll discuss pulling the release on github with @Gacko but we can not remove the images on 1.10.2 and 1.11.0 in the kubernetes registry. .

[koehn]

I’m seeing the same when using 1.11.0 on my hybrid ARM64/AMD64 cluster running k3s 1.29.6, kernel 6.1:

k3s-01   Ready    control-plane,etcd,master   240d   v1.29.6+k3s2   10.0.1.234    <none>        Armbian 24.5.1 bookworm          6.1.43-vendor-rk35xx   containerd://1.7.17-k3s1
k3s-02   Ready    control-plane,etcd,master   90d    v1.29.6+k3s2   10.0.1.235    <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-22-amd64         containerd://1.7.17-k3s1
k3s-03   Ready    control-plane,etcd,master   45d    v1.29.6+k3s2   10.0.1.236    <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-21-amd64         containerd://1.7.17-k3s1