CRD can not be created sucessful in installlation of OLM v.0.24.0 version (same bug as #2778: Applying OLM CRDs fails due to last-applied-configuration annotation) #2968
Comments
|
Hi @shaojini, thanks for the issue.
which is displayed when the The underlying condition is the The easiest resolution is manual, cleaning up old resources and using kind delete clusters kind
kind create cluster
export olm_release=v0.24.0
kubectl create -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/${olm_release}/crds.yamlI'll try to get this bug in the agenda for discussion since we've evidently passed the threshold where users start encountering this and it's not great UX. Could you update this issue with the source of your instructions that got you started here? It may be that we need to update them as well. Thanks! |
|
Follow-up: I stuck this on the community meeting on 23rd May: agenda. |
Thanks @grokspawn reply. I have removed all resources used OLM, removed the cluster totally, removed old version and reinstalled new version of kubeadm, kubectl, kubelet, created a new cluster. After installing the "cert-manager" via I have following results according to your instructions: root@K8s-master: root@K8s-master:~# kubectl create -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/${olm_release}/olm.yaml It seems the OLM has been installed successfully, however, it is not: root@K8s-master:~# kubectl get apiservices.apiregistration.k8s.io v1.packages.operators.coreos.com root@K8s-master:~# kubectl get crds The packetserver service is fail in installing. What is wrong? Thanks. |
The pods of packageserver have been created: However, CRD of the packageserver had not been created successfully. |
|
Hi @shaojini. This is why I asked where your instruction source was. That is just one step towards installing OLM. In order to get the ability to query the packagemanifests api, you have to install it via kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.24.0/olm.yamlIf you delete/create your kind cluster and follow the quickstart at https://olm.operatorframework.io/docs/getting-started/ then it includes the CRDs, APIservices, default catalog sources to be fully ready. It's a much smoother user experience. |
|
Created #2969 to document how to perform an upgrade. |
|
Hi @grokspawn . I really use the link given by you to install the OLM. Now the main problem is the "apiServices" can not be installed successfully. I try to find the problem with the following command: root@K8s-master:~# kubectl describe csv packageserver -n olm Name: packageserver ......... ......... Events: Normal RequirementsUnknown 27m operator-lifecycle-manager requirements not yet checked The result shows that it seems there is an image to be download from docker hub. The main problem might be the Docker Hub are limiting to pull images for the public Decker registry currently (I have met the same problem for creating a pot when pulling images from Docker hub). In order to resolve the problem of image pulling from the docker hub, it needs to provide the credential for pulling with k8s pod specification. I don't know how the problem can be resolved for the apiServices installing. |
|
This can be partially avoided if you force the kubectl commands to use server-side rendering which avoid adding the last known configuration field inside of the metadata. However, for those attempting to apply these manifests via GitOps--you will face kubernetes-sigs/structured-merge-diff#130 where the manifests will never align. The manifests in this repository are missing the ContainerPort protocol and caused ArgoCD to barf. There really needs to be more thought into the ease of use of this project. There is no supported way for deploying this in a declarative manner. No helm chart support. No way to deploy in a GitOps method. ARM installation works when you use their shell script.... but still required to do a manual edit to one of the deployments. |
|
Hi @acelinkit Thanks for pointing out GitOps considerations. That's a really good point.
@shaojini I think that your initial request has been addressed. The slack channel may be a better place to help you with troubleshooting the apiservice issue. The images are pulled from quay.io not dockerhub. Looking at the logs of the olm-operator may help. |
|
@acelinkit checking the latest version the port protocol has been added to the manifests, also here, last month. It should be part of the next release |
|
Hey @fgiloux. Here is some additional context on my remarks. I attempted to apply the helm chart directly inside this repo from ArgoCD using the v0.24.0 tag. The manifests missing port issue came up after doing server-side apply and would not apply as the manifests were rejected. I could have probably forced somerhing to eventually work, but I gave up because charts are not supported or the direction this project aims to support. #829 So I followed the docs.The base Readme https://github.com/operator-framework/operator-lifecycle-manager/tree/master#installation points to installation docs https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/install/install.md. the installation docs then defer to the release page for installing a specific release. https://github.com/operator-framework/operator-lifecycle-manager/releases. The release notes have you download and run https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.24.0/install.sh At that point I could go deconstruct the shell script to identify what it's doing. Then write a declarative way for what it's doing. So I just run the shell script and move on. When I go to install an operator I faced another OLM problem and opened an issue. #2971. After applying a workaround my experience with operators was not great. ArgoCD operator does not have a ARM set of images. MetalLB operator had ARM images but I couldn't get it to work. Just trying to share my user experience. Other hobbiests I have talked to outright avoid OLM entirely. |
|
@acelinkit thanks for taking the time and providing this valuable feedback. |
I am so many hours into realizing that this is a bug and not my noobery. i am also on ARM and have found that operators themselves work well enough but any attempts at using OLM are a nightmare of endless architecture related errors. |
|
Hi. It seems the new version of v.0.25.0 has fixed the problem of the"OLM fails to install packageserver with FailedDiscoveryCheck error". root@k8s-master:~# kubectl get apiservices.apiregistration.k8s.io v1.packages.operators.coreos.com |
|
Encountered the same problem, using version v0.25.0. root@ubuntu:~/k8/olm# ll
total 205656
-rw-r--r-- 1 root root 773165 Sep 14 15:35 crds.yaml
-rw-r--r-- 1 root root 1886 Sep 14 15:41 install.sh
-rw-r--r-- 1 root root 10730 Sep 14 15:41 olm.yaml
-rw-r--r-- 1 root root 121468017 Sep 14 15:42 operator-lifecycle-manager_0.25.0_linux_amd64.tar.gz
-rwxr-xr-x 1 root root 88330149 Sep 14 15:29 operator-sdk_linux_amd64
root@ubuntu:~/k8/olm# kubectl apply -f crds.yaml
customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/installplans.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/olmconfigs.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operatorconditions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operatorgroups.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operators.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/subscriptions.operators.coreos.com created
The CustomResourceDefinition "clusterserviceversions.operators.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
root@ubuntu:~/k8/olm# vim crds.yaml
root@ubuntu:~/k8/olm# vim crds.yaml
root@ubuntu:~/k8/olm# vim crds.yaml
root@ubuntu:~/k8/olm# kubectl get deployments -n olm
No resources found in olm namespace.
root@ubuntu:~/k8/olm# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:34:27Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:27:46Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
|
|
This works for me. root@ubuntu:~/k8/olm# kubectl apply -f crds.yaml --server-side=true
customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/clusterserviceversions.operators.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/installplans.operators.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/olmconfigs.operators.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/operatorconditions.operators.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/operatorgroups.operators.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/operators.operators.coreos.com serverside-applied
customresourcedefinition.apiextensions.k8s.io/subscriptions.operators.coreos.com serverside-applied |
|
This issue seems to have returned in v0.26.0 |
|
This issue is present in latest version as well v.0.31.0 |
Bug Report
What did you do?
A clear and concise description of the steps you took (or insert a code snippet).
root@K8s-master:
# docker login# export olm_release=v0.24.0root@K8s-master:
root@K8s-master:~# kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/${olm_release}/crds.yaml
What did you expect to see?
A clear and concise description of what you expected to happen (or insert a code snippet).
Successful installation
What did you see instead? Under which circumstances?
A clear and concise description of what you expected to happen (or insert a code snippet).
customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/installplans.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/olmconfigs.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operatorconditions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operatorgroups.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operators.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/subscriptions.operators.coreos.com created
The CustomResourceDefinition "clusterserviceversions.operators.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
When I switched into "operator-sdk" to install OLM, it did not show any error for the installation of OLM, however, the packageserver service is in "false":
root@K8s-master:~# operator-sdk olm install
INFO[0000] Fetching CRDs for version "latest"
INFO[0000] Fetching resources for resolved version "latest"
I0519 20:39:28.599145 64106 request.go:690] Waited for 1.01264153s due to client-side throttling, not priority and fairness, request: GET:https://192.168.0.7:6443/apis/cilium.io/v2?timeout=32s
INFO[0008] Creating CRDs and resources
INFO[0008] Creating CustomResourceDefinition "catalogsources.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "clusterserviceversions.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "installplans.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "olmconfigs.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "operatorconditions.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "operatorgroups.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "operators.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "subscriptions.operators.coreos.com"
INFO[0009] Creating Namespace "olm"
INFO[0009] Creating Namespace "operators"
INFO[0009] Creating ServiceAccount "olm/olm-operator-serviceaccount"
INFO[0009] Creating ClusterRole "system:controller:operator-lifecycle-manager"
INFO[0009] Creating ClusterRoleBinding "olm-operator-binding-olm"
INFO[0009] Creating OLMConfig "cluster"
I0519 20:39:38.648832 64106 request.go:690] Waited for 1.447733128s due to client-side throttling, not priority and fairness, request: GET:https://192.168.0.7:6443/apis/operators.coreos.com/v1alpha2?timeout=32s
INFO[0012] Creating Deployment "olm/olm-operator"
INFO[0012] Creating Deployment "olm/catalog-operator"
INFO[0012] Creating ClusterRole "aggregate-olm-edit"
INFO[0012] Creating ClusterRole "aggregate-olm-view"
INFO[0012] Creating OperatorGroup "operators/global-operators"
INFO[0012] Creating OperatorGroup "olm/olm-operators"
INFO[0012] Creating ClusterServiceVersion "olm/packageserver"
INFO[0012] Creating CatalogSource "olm/operatorhubio-catalog"
INFO[0012] Waiting for deployment/olm-operator rollout to complete
INFO[0012] Waiting for Deployment "olm/olm-operator" to rollout: 0 of 1 updated replicas are available
INFO[0014] Deployment "olm/olm-operator" successfully rolled out
INFO[0014] Waiting for deployment/catalog-operator rollout to complete
INFO[0014] Deployment "olm/catalog-operator" successfully rolled out
INFO[0014] Waiting for deployment/packageserver rollout to complete
INFO[0014] Waiting for Deployment "olm/packageserver" to appear
INFO[0015] Waiting for Deployment "olm/packageserver" to rollout: 0 of 2 updated replicas are available
INFO[0028] Deployment "olm/packageserver" successfully rolled out
INFO[0028] Successfully installed OLM version "latest"
NAME NAMESPACE KIND STATUS
catalogsources.operators.coreos.com CustomResourceDefinition Installed
clusterserviceversions.operators.coreos.com CustomResourceDefinition Installed
installplans.operators.coreos.com CustomResourceDefinition Installed
olmconfigs.operators.coreos.com CustomResourceDefinition Installed
operatorconditions.operators.coreos.com CustomResourceDefinition Installed
operatorgroups.operators.coreos.com CustomResourceDefinition Installed
operators.operators.coreos.com CustomResourceDefinition Installed
subscriptions.operators.coreos.com CustomResourceDefinition Installed
olm Namespace Installed
operators Namespace Installed
olm-operator-serviceaccount olm ServiceAccount Installed
system:controller:operator-lifecycle-manager ClusterRole Installed
olm-operator-binding-olm ClusterRoleBinding Installed
cluster OLMConfig Installed
olm-operator olm Deployment Installed
catalog-operator olm Deployment Installed
aggregate-olm-edit ClusterRole Installed
aggregate-olm-view ClusterRole Installed
global-operators operators OperatorGroup Installed
olm-operators olm OperatorGroup Installed
packageserver olm ClusterServiceVersion Installed
operatorhubio-catalog olm CatalogSource Installed
However,
root@K8s-master:~# kubectl get apiservices.apiregistration.k8s.io v1.packages.operators.coreos.com
E0519 21:20:02.374158 88834 memcache.go:287] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.375416 88834 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.377921 88834 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.380392 88834 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.405270 88834 memcache.go:287] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.427846 88834 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.433485 88834 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.436921 88834 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
NAME SERVICE AVAILABLE AGE
v1.packages.operators.coreos.com olm/packageserver-service False (FailedDiscoveryCheck) 40m
The "packages.operators.coreos.com" is not functioning:
root@K8s-master:~# kubectl get crd | grep operators.coreos.com
E0519 21:28:58.773330 94129 memcache.go:287] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:28:58.774865 94129 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:28:58.777895 94129 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:28:58.781929 94129 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
catalogsources.operators.coreos.com 2023-05-19T17:39:35Z
clusterserviceversions.operators.coreos.com 2023-05-19T17:39:35Z
installplans.operators.coreos.com 2023-05-19T17:39:35Z
olmconfigs.operators.coreos.com 2023-05-19T17:39:35Z
operatorconditions.operators.coreos.com 2023-05-19T17:39:35Z
operatorgroups.operators.coreos.com 2023-05-19T17:39:35Z
operators.operators.coreos.com 2023-05-19T17:39:35Z
subscriptions.operators.coreos.com 2023-05-19T17:39:35Z
Environment
root@K8s-master:~# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.4", GitCommit:"f89670c3aa4059d6999cb42e23ccb4f0b9a03979", GitTreeState:"clean", BuildDate:"2023-04-12T12:05:35Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}
root@K8s-master:~# kubectl cluster-info
E0519 21:39:38.159210 100380 memcache.go:287] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:39:38.166851 100380 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:39:38.169478 100380 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:39:38.172656 100380 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
Kubernetes control plane is running at https://192.168.0.7:6443
CoreDNS is running at https://192.168.0.7:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Possible Solution
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: