0.25.0 crashes at backend reload maybe due to recent OpenResty changes

[tailtwo]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.): yes

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): illegal opcode resty

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version: 0.25.0

Kubernetes version (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: PowerEdge R805
• OS (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
• Kernel (e.g. uname -a): Linux server1 4.19.0-5-amd64 Basic structure  #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64 GNU/Linux
• Install tools: Kubernetes installed with kubeadm, nginx-ingress installed with deploy files
• Others:

What happened:
Upgrading from Nginx-ingress 0.24.1 to 0.25.0 make the Nginx controller crash on one specific server of my Kubernetes cluster. Other servers have a different hardware/OS.

$ kubectl -n ingress-nginx logs -f nginx-ingress-controller-66b5db5886-r96cp
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.25.0
  Build:      git-1387f7b7e
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

W0711 12:31:57.434293       7 flags.go:221] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0711 12:31:57.450169       7 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0711 12:31:57.450526       7 main.go:183] Creating API client for https://10.96.0.1:443
I0711 12:31:57.468721       7 main.go:227] Running in Kubernetes cluster version v1.15 (v1.15.0) - git (clean) commit e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529 - platform linux/amd64
I0711 12:31:57.475744       7 main.go:91] Validated ingress-nginx/default-http-backend as the default backend.
I0711 12:31:57.818615       7 main.go:102] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
E0711 12:31:57.821325       7 main.go:131] v1.15.0
I0711 12:31:57.902011       7 nginx.go:275] Starting NGINX Ingress controller
I0711 12:31:58.046448       7 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"2c97a076-a91c-46bb-bc2d-389fa1d1941d", APIVersion:"v1", ResourceVersion:"12628", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I0711 12:31:58.047034       7 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"13a4c919-61ff-40ee-85f0-83ddff941d52", APIVersion:"v1", ResourceVersion:"12626", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I0711 12:31:58.047136       7 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"1d20f74c-f549-425a-b587-d548423a9f7c", APIVersion:"v1", ResourceVersion:"2344118", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration
E0711 12:31:58.985418       7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0711 12:31:59.110694       7 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"**redacted-ns**", Name:"**redacted-app**", UID:"2c633b1c-c671-4a61-a7fb-d3402a91807c", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1030315", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress **redacted-ns**/**redacted-app**
W0711 12:31:59.111023       7 backend_ssl.go:46] Error obtaining X.509 certificate: no object matching key "**redacted-ns**/ldap.**redacted**.com-tls" in local store
I0711 12:31:59.111220       7 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"**redacted-ns**", Name:"www.**redacted**.com-institutionnel", UID:"4a04b1da-a195-4d74-a53f-60a968a11635", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"2011288", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress **redacted-ns**/www.**redacted**.com
I0711 12:31:59.112157       7 backend_ssl.go:66] Adding Secret "**redacted-ns**/www.**redacted**.com-tls" to the local store
I0711 12:31:59.112370       7 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"**redacted-ns**", Name:"maven-geotoolkit", UID:"6ac6c584-6fa0-4c23-8e4d-d43e9a2144df", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"2565951", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress **redacted-ns**/**redacted-app**
I0711 12:31:59.113206       7 backend_ssl.go:66] Adding Secret "**redacted-ns**/maven.geotoolkit.org-tls" to the local store
I0711 12:31:59.113423       7 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"**redacted-ns**", Name:"**redacted-app**", UID:"2cb1939d-c450-4239-9940-c962c116c491", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1047149", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress **redacted-ns**/**redacted-app**
I0711 12:31:59.114220       7 backend_ssl.go:66] Adding Secret "**redacted-ns**/matrix.**redacted**.com-tls" to the local store
I0711 12:31:59.114532       7 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"**redacted-ns**", Name:"**redacted**.com-**redacted**", UID:"171133c4-0f2c-45a2-9a59-ffc48ef68836", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"2011284", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress **redacted-ns**/**redacted**.com-**redacted**
I0711 12:31:59.115294       7 backend_ssl.go:66] Adding Secret "**redacted-ns**/**redacted**.com-tls" to the local store
I0711 12:31:59.115712       7 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"**redacted-ns**", Name:"**redacted-app**", UID:"ab171ba4-009c-43ce-ac8e-e58c85492cb7", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"2011754", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress **redacted-ns**/**redacted-app**
I0711 12:31:59.116333       7 backend_ssl.go:66] Adding Secret "**redacted-ns**/tourisme.**redacted**.com-tls" to the local store
I0711 12:31:59.116554       7 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"**redacted-ns**", Name:"**redacted-app**", UID:"a107b519-1b75-4c02-9c14-4f99ca3c92af", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"2344026", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress **redacted-ns**/**redacted-app**
I0711 12:31:59.117504       7 backend_ssl.go:66] Adding Secret "**redacted-ns**/**redacted-app**.**redacted**.com-tls" to the local store
I0711 12:31:59.118473       7 backend_ssl.go:66] Adding Secret "**redacted-ns**/**redacted-app**.**redacted**.com-tls" to the local store
I0711 12:31:59.119408       7 backend_ssl.go:66] Adding Secret "**redacted-ns**/**redacted-app**.**redacted**.com-tls" to the local store
I0711 12:31:59.120326       7 backend_ssl.go:66] Adding Secret "**redacted-ns**/**redacted-app**.**redacted**.com-tls" to the local store
I0711 12:31:59.120348       7 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"**redacted-ns**", Name:"**redacted-app**", UID:"9ab752b9-f875-42c5-bce1-20a6025af9f6", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"2011280", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress **redacted-ns**/**redacted-app**
I0711 12:31:59.120403       7 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"**redacted-ns**", Name:"**redacted-app**", UID:"b5dfa150-c868-4f39-b390-5e438abcbde3", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1020014", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress **redacted-ns**/**redacted-app**
I0711 12:31:59.120430       7 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"**redacted-ns**", Name:"**redacted-app**", UID:"06c17071-bd4a-4f80-87a2-a336fad0f8f9", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"2011278", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress **redacted-ns**/**redacted-app**
I0711 12:31:59.120530       7 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"**redacted-ns**", Name:"**redacted-app**", UID:"c05c6133-e725-4a62-978f-84fd700d30ed", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"2011282", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress **redacted-ns**/**redacted-app**
I0711 12:31:59.121295       7 backend_ssl.go:66] Adding Secret "**redacted-ns**/sonar.**redacted**.com-tls" to the local store
I0711 12:31:59.121485       7 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"**redacted-ns**", Name:"**redacted-app**", UID:"0558b051-b513-4ece-bfb9-5b726a113181", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1960548", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress **redacted-ns**/**redacted-app**
I0711 12:31:59.122282       7 backend_ssl.go:66] Adding Secret "**redacted-ns**/didro.io-tls" to the local store
I0711 12:31:59.203047       7 nginx.go:319] Starting NGINX process
I0711 12:31:59.203071       7 leaderelection.go:235] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
W0711 12:31:59.207021       7 controller.go:1101] Error getting SSL certificate "**redacted-ns**/ldap.**redacted**.com-tls": local SSL certificate **redacted-ns**/ldap.**redacted**.com-tls was not found. Using default certificate
I0711 12:31:59.208233       7 controller.go:133] Configuration changes detected, backend reload required.
W0711 12:31:59.282508       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:31:59.282549       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:31:59.282560       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
W0711 12:31:59.282569       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
E0711 12:31:59.324860       7 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:31:59.324993       7 queue.go:130] requeuing initial-sync, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I0711 12:31:59.442406       7 status.go:86] new leader elected: nginx-ingress-controller-777cd75786-tlrq9
W0711 12:32:02.538047       7 controller.go:1101] Error getting SSL certificate "**redacted-ns**/ldap.**redacted**.com-tls": local SSL certificate **redacted-ns**/ldap.**redacted**.com-tls was not found. Using default certificate
I0711 12:32:02.538644       7 controller.go:133] Configuration changes detected, backend reload required.
W0711 12:32:02.597077       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:02.597183       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:02.597197       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
W0711 12:32:02.597207       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
E0711 12:32:02.643467       7 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:32:02.643516       7 queue.go:130] requeuing **redacted-ns**/**redacted-app**, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:32:05.871031       7 controller.go:1101] Error getting SSL certificate "**redacted-ns**/ldap.**redacted**.com-tls": local SSL certificate **redacted-ns**/ldap.**redacted**.com-tls was not found. Using default certificate
I0711 12:32:05.871495       7 controller.go:133] Configuration changes detected, backend reload required.
W0711 12:32:05.941151       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:05.941493       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:05.941607       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
W0711 12:32:05.941689       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
E0711 12:32:05.981765       7 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:32:05.982296       7 queue.go:130] requeuing **redacted-ns**/**redacted-app**, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E0711 12:32:08.979529       7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W0711 12:32:09.204806       7 controller.go:1101] Error getting SSL certificate "**redacted-ns**/ldap.**redacted**.com-tls": local SSL certificate **redacted-ns**/ldap.**redacted**.com-tls was not found. Using default certificate
I0711 12:32:09.205253       7 controller.go:133] Configuration changes detected, backend reload required.
W0711 12:32:09.271074       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:09.271110       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:09.271120       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
W0711 12:32:09.271131       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
E0711 12:32:09.311692       7 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:32:09.311754       7 queue.go:130] requeuing kube-system/kube-controller-manager, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
2019/07/11 12:32:09 Get http+unix://nginx-status/nginx_status: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W0711 12:32:09.826013       7 nginx_status.go:172] unexpected error obtaining nginx status info: Get http+unix://nginx-status/nginx_status: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W0711 12:32:12.538222       7 controller.go:1101] Error getting SSL certificate "**redacted-ns**/ldap.**redacted**.com-tls": local SSL certificate **redacted-ns**/ldap.**redacted**.com-tls was not found. Using default certificate
I0711 12:32:12.538737       7 controller.go:133] Configuration changes detected, backend reload required.
W0711 12:32:12.608690       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:12.608741       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:12.608753       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
W0711 12:32:12.608898       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
E0711 12:32:12.653866       7 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:32:12.654342       7 queue.go:130] requeuing **redacted-ns**/**redacted-app**, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E0711 12:32:15.130272       7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W0711 12:32:15.871638       7 controller.go:1101] Error getting SSL certificate "**redacted-ns**/ldap.**redacted**.com-tls": local SSL certificate **redacted-ns**/ldap.**redacted**.com-tls was not found. Using default certificate
I0711 12:32:15.872374       7 controller.go:133] Configuration changes detected, backend reload required.
W0711 12:32:15.945510       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:15.945547       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:15.945558       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
W0711 12:32:15.945568       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
E0711 12:32:15.997947       7 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:32:15.998071       7 queue.go:130] requeuing **redacted-ns**/**redacted-app**, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E0711 12:32:18.978922       7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W0711 12:32:19.204543       7 controller.go:1101] Error getting SSL certificate "**redacted-ns**/ldap.**redacted**.com-tls": local SSL certificate **redacted-ns**/ldap.**redacted**.com-tls was not found. Using default certificate
I0711 12:32:19.205105       7 controller.go:133] Configuration changes detected, backend reload required.
W0711 12:32:19.276926       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:19.277047       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:19.277076       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
W0711 12:32:19.277094       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
E0711 12:32:19.327344       7 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:32:19.327384       7 queue.go:130] requeuing ingress-nginx/ingress-nginx, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:32:22.538363       7 controller.go:1101] Error getting SSL certificate "**redacted-ns**/ldap.**redacted**.com-tls": local SSL certificate **redacted-ns**/ldap.**redacted**.com-tls was not found. Using default certificate
I0711 12:32:22.538617       7 controller.go:133] Configuration changes detected, backend reload required.
W0711 12:32:22.628456       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:22.628493       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:22.628504       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
W0711 12:32:22.628513       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
E0711 12:32:22.675789       7 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:32:22.675828       7 queue.go:130] requeuing **redacted-ns**/**redacted-app**, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E0711 12:32:25.129184       7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W0711 12:32:25.871419       7 controller.go:1101] Error getting SSL certificate "**redacted-ns**/ldap.**redacted**.com-tls": local SSL certificate **redacted-ns**/ldap.**redacted**.com-tls was not found. Using default certificate
I0711 12:32:25.871822       7 controller.go:133] Configuration changes detected, backend reload required.
W0711 12:32:25.936798       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:25.937177       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:25.937217       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
W0711 12:32:25.937245       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
E0711 12:32:25.983728       7 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:32:25.983788       7 queue.go:130] requeuing **redacted-ns**/**redacted-app**, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E0711 12:32:28.978715       7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W0711 12:32:29.204957       7 controller.go:1101] Error getting SSL certificate "**redacted-ns**/ldap.**redacted**.com-tls": local SSL certificate **redacted-ns**/ldap.**redacted**.com-tls was not found. Using default certificate
I0711 12:32:29.205366       7 controller.go:133] Configuration changes detected, backend reload required.
W0711 12:32:29.275267       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:29.275306       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:29.275317       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
W0711 12:32:29.275326       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
E0711 12:32:29.324430       7 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:32:29.324485       7 queue.go:130] requeuing **redacted-ns**/**redacted-app**, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:32:32.538309       7 controller.go:1101] Error getting SSL certificate "**redacted-ns**/ldap.**redacted**.com-tls": local SSL certificate **redacted-ns**/ldap.**redacted**.com-tls was not found. Using default certificate
I0711 12:32:32.538969       7 controller.go:133] Configuration changes detected, backend reload required.
W0711 12:32:32.605955       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:32.605995       7 nginx.go:1165] Already exists an Ingress with "www.**redacted**.com" hostname. Skipping creation of redirection from "www.**redacted**.com" to "**redacted**.com".
W0711 12:32:32.606012       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
W0711 12:32:32.606025       7 nginx.go:1165] Already exists an Ingress with "**redacted**.com" hostname. Skipping creation of redirection from "**redacted**.com" to "www.**redacted**.com".
E0711 12:32:32.651064       7 controller.go:145] Unexpected reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0711 12:32:32.651206       7 queue.go:130] requeuing kube-system/kube-scheduler, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E0711 12:32:35.130939       7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0711 12:32:35.192594       7 main.go:154] Received SIGTERM, shutting down
I0711 12:32:35.194784       7 nginx.go:402] Shutting down controller queues
I0711 12:32:35.194823       7 status.go:117] updating status of Ingress rules (remove)
I0711 12:32:35.206737       7 nginx.go:418] Stopping NGINX process
I0711 12:32:35.215955       7 main.go:158] Error during shutdown: signal: illegal instruction (core dumped)
I0711 12:32:35.216019       7 main.go:162] Handled quit, awaiting Pod deletion
E0711 12:32:38.979667       7 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0711 12:32:45.216508       7 main.go:165] Exiting with 1

System logs :

$ journalctl -xef
[...]
juil. 11 14:41:16 serv1 kernel: traps: nginx[16760] trap invalid opcode ip:7ff6bf40ecf2 sp:7ffe7da9b570 error:0 in libluajit-5.1.so.2.1.0[7ff6bf40e000+73000]
[...]

What you expected to happen:
Nginx controller init without issues.

How to reproduce it (as minimally and precisely as possible):
Upgrade from 0.24.1 to 0.25.0 on a specific server hardware.

Anything else we need to know:
The culpit may be Openresty. This issue seems to fit this case : openresty/docker-openresty#39

[aledbf]

@tailtwo you are right. This issue is related to missing instructions in your PowerEdge R805 (eleven years since the release date)

We have no plans to support old hardware (or the resources to start adding these exceptions and new images).

For that reason:

1. You can use a nodeSelector in your deployment to NOT use this node.
2. Build your own image of the ingress controller, removing sse from openresty
   process exit with code 132 openresty/docker-openresty#39 (comment)

[towolf]

@aledbf I get the same kind of crashes.

Are you saying our CPUs are too old?

[7588496.737816] traps: nginx[28636] trap invalid opcode ip:7f55d239acf2 sp:7fff272af690 error:0 in libluajit-5.1.so.2.1.0[7f55d239a000+73000]                                                                      
[7588504.921061] traps: nginx[28817] trap invalid opcode ip:7fd9a3e24cf2 sp:7fff86a6fc80 error:0 in libluajit-5.1.so.2.1.0[7fd9a3e24000+73000]                                                                      
[7588507.145871] traps: nginx[28947] trap invalid opcode ip:7fb8dd9cfcf2 sp:7ffe24acf1d0 error:0 in libluajit-5.1.so.2.1.0[7fb8dd9cf000+73000]                                                                      
[7588507.202233] traps: nginx[28967] trap invalid opcode ip:7f6afc025cf2 sp:7fffb8ad6930 error:0 in libluajit-5.1.so.2.1.0[7f6afc025000+73000]                                                                      
[7588510.529822] traps: nginx[29045] trap invalid opcode ip:7f2fcb406cf2 sp:7ffe967edf90 error:0 in libluajit-5.1.so.2.1.0[7f2fcb406000+73000]                                                                      
[7588513.868688] traps: nginx[29091] trap invalid opcode ip:7fe925578cf2 sp:7ffe59e12570 error:0 in libluajit-5.1.so.2.1.0[7fe925578000+73000]                                                                      
[7588517.196087] traps: nginx[29269] trap invalid opcode ip:7ff468e74cf2 sp:7fffade55620 error:0 in libluajit-5.1.so.2.1.0[7ff468e74000+73000]                                                                      
[7588520.529491] traps: nginx[29303] trap invalid opcode ip:7ff87d62ecf2 sp:7fff3cdbf2c0 error:0 in libluajit-5.1.so.2.1.0[7ff87d62e000+73000]                                                                      
[7588523.868384] traps: nginx[29352] trap invalid opcode ip:7fdd0f8d2cf2 sp:7fffde9f0b30 error:0 in libluajit-5.1.so.2.1.0[7fdd0f8d2000+73000]                                                                      
[7588527.196596] traps: nginx[29515] trap invalid opcode ip:7fdb7d074cf2 sp:7ffc5b695890 error:0 in libluajit-5.1.so.2.1.0[7fdb7d074000+73000]                                                                      
[7588529.328189] traps: nginx[29652] trap invalid opcode ip:7f33611cacf2 sp:7fff4adf4830 error:0 in libluajit-5.1.so.2.1.0[7f33611ca000+73000]                                                                      
[7588530.528741] traps: nginx[29741] trap invalid opcode ip:7fb97e955cf2 sp:7ffe84e84d50 error:0 in libluajit-5.1.so.2.1.0[7fb97e955000+73000]                                                                      
[7588532.408816] traps: nginx[29864] trap invalid opcode ip:7f1c84ce4cf2 sp:7ffe4dbdd320 error:0 in libluajit-5.1.so.2.1.0[7f1c84ce4000+73000]                                                                      

# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           2
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               15
Model name:          Intel(R) Xeon(R) CPU           E5320  @ 1.86GHz
Stepping:            7
CPU MHz:             1866.725
BogoMIPS:            3733.40
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            4096K
NUMA node0 CPU(s):   0-7
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca lahf_lm pti tpr_shadow dtherm

[towolf]

@aledbf OpenResty itself now officially supports old hardware by having two variants of each image:

[aledbf]

@towolf yes, it seems the same issue. The release date of E5320 was on Nov 14, 2006

[aledbf]

#4311 (comment)

You should build a custom version to remove the sse42 version.

[towolf]

@aledbf don't we exclude a lot of architectures then? What about ARM and Rpi and so on and so forth?

[Julusian]

@towolf This doesnt really affect ARM as SSE is a x86 specific extension.

I just hit this myself for Intel E5345, and it took me a while to figure out how to build a working version, so for anyone else needing it this is what I did: 9fdae37
I don't know if both changes are required, but it works with them both.

@aledbf It would be really great if there could be an official 'amd64-legacy' build for those of us who are stuck on older hardware.

[kvncrw]

I just ran into this on my Dell R815 server. Sure, it's old, but it's also 48 real cores w/ 256 GB and perfectly capable of running modern workloads.

👍 amd64-legacy

[aledbf]

Maybe this issue can be fixed if this PR openresty/luajit2#75 is merged

[rzr]

Even if this issue is closed,
meanwhile the above luatjit fix is released
I would expect a check in some scripts that would throw an alert to user

Can you suggest a file to insert something like

 grep  sse4_2  /proc/cpuinfo || echo "ERROR: CPU is not supporting sse-4.2"

Thanks

Note for the record it is also affecting microk8s v1.16.0 with CPU "AMD Athlon(tm) 64 X2 Dual Core Processor 4600+"

Relate-to:

openresty/luajit2#20
openresty/luajit2#75
openresty/luajit2#82