[Bug][Investigation] Examine reconciliation backoff when there is an issue creating pods

[DmitriGekhtman]

Search before asking

• I searched the issues and found no similar issues.

KubeRay Component

ray-operator

What happened + What you expected to happen

The issue comment here indicates there may be a problem with reconciliation backoff when there's a persistent issue creating a pod:
#639 (comment)

Investigate and fix!

Reproduction script

Try applying a resource quota or concocting some other condition that would prevent a worker pod from being created. Check if reconciliation is too frequent with KubeRay master.

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[sihanwang41]

TLDR: I can't reproduce the issue.

I am using Kuberay on Kind to test it locally. I am using github.com/ray-project/kuberay/ray-operator/config/default?ref=master&timeout=90s and provision a ray cluster that exceeds the quota. I can see the backoff reconciling happens.

2022-11-06T23:01:30.511Z	ERROR	controller.raycluster-controller	Reconciler error	{"reconciler group": "ray.io", "reconciler kind": "RayCluster", "name": "raycluster-mini", "namespace": "kuberay-debug", "error": "pods \"raycluster-mini-head-cgpsq\" is forbidden: exceeded quota: test-cpu-quota, requested: limits.cpu=500m,requests.cpu=500m, used: limits.cpu=0,requests.cpu=0, limited: limits.cpu=300m,requests.cpu=200m"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227
2022-11-06T23:01:50.994Z	INFO	controllers.RayCluster	reconciling RayCluster	{"cluster name": "raycluster-mini"}
2022-11-06T23:01:50.996Z	INFO	controllers.RayCluster	reconcileServices 	{"headService service found": "raycluster-mini-head-svc"}
2022-11-06T23:01:50.996Z	INFO	controllers.RayCluster	reconcilePods 	{"creating head pod for cluster": "raycluster-mini"}
2022-11-06T23:01:50.997Z	INFO	RayCluster-Controller	Setting pod namespaces	{"namespace": "kuberay-debug"}
2022-11-06T23:01:50.997Z	INFO	controllers.RayCluster	head pod labels	{"labels": {"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","groupName":"headgroup","ray.io/cluster":"raycluster-mini","ray.io/cluster-dashboard":"raycluster-mini-dashboard","ray.io/group":"headgroup","ray.io/identifier":"raycluster-mini-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head","rayCluster":"raycluster-sample","rayNodeType":"head"}}
2022-11-06T23:01:50.998Z	INFO	RayCluster-Controller	Head pod container with index 0 identified as Ray container.
2022-11-06T23:01:50.998Z	INFO	controllers.RayCluster	createHeadPod	{"head pod with name": "raycluster-mini-head-"}
2022-11-06T23:01:51.024Z	ERROR	controller.raycluster-controller	Reconciler error	{"reconciler group": "ray.io", "reconciler kind": "RayCluster", "name": "raycluster-mini", "namespace": "kuberay-debug", "error": "pods \"raycluster-mini-head-wsvp7\" is forbidden: exceeded quota: test-cpu-quota, requested: limits.cpu=500m,requests.cpu=500m, used: limits.cpu=0,requests.cpu=0, limited: limits.cpu=300m,requests.cpu=200m"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227
2022-11-06T23:02:32.011Z	INFO	controllers.RayCluster	reconciling RayCluster	{"cluster name": "raycluster-mini"}
2022-11-06T23:02:32.017Z	INFO	controllers.RayCluster	reconcileServices 	{"headService service found": "raycluster-mini-head-svc"}
2022-11-06T23:02:32.115Z	INFO	controllers.RayCluster	reconcilePods 	{"creating head pod for cluster": "raycluster-mini"}
2022-11-06T23:02:32.117Z	INFO	RayCluster-Controller	Setting pod namespaces	{"namespace": "kuberay-debug"}
2022-11-06T23:02:32.117Z	INFO	controllers.RayCluster	head pod labels	{"labels": {"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","groupName":"headgroup","ray.io/cluster":"raycluster-mini","ray.io/cluster-dashboard":"raycluster-mini-dashboard","ray.io/group":"headgroup","ray.io/identifier":"raycluster-mini-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head","rayCluster":"raycluster-sample","rayNodeType":"head"}}
2022-11-06T23:02:32.118Z	INFO	RayCluster-Controller	Head pod container with index 0 identified as Ray container.
2022-11-06T23:02:32.120Z	INFO	controllers.RayCluster	createHeadPod	{"head pod with name": "raycluster-mini-head-"}
2022-11-06T23:02:32.811Z	ERROR	controller.raycluster-controller	Reconciler error	{"reconciler group": "ray.io", "reconciler kind": "RayCluster", "name": "raycluster-mini", "namespace": "kuberay-debug", "error": "pods \"raycluster-mini-head-tskg9\" is forbidden: exceeded quota: test-cpu-quota, requested: limits.cpu=500m,requests.cpu=500m, used: limits.cpu=0,requests.cpu=0, limited: limits.cpu=300m,requests.cpu=200m"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227
2022-11-06T23:03:53.310Z	INFO	controllers.RayCluster	reconcile RayCluster Event	{"event name": "kuberay-operator-77c6cf9455-zmmtt.172520edebc5a5d2"}
2022-11-06T23:03:53.311Z	INFO	controllers.RayCluster	pod not found or no valid annotations	{"pod name": "kuberay-operator-77c6cf9455-zmmtt"}
2022-11-06T23:03:54.732Z	INFO	controllers.RayCluster	reconciling RayCluster	{"cluster name": "raycluster-mini"}
2022-11-06T23:03:54.743Z	INFO	controllers.RayCluster	reconcileServices 	{"headService service found": "raycluster-mini-head-svc"}
2022-11-06T23:03:54.744Z	INFO	controllers.RayCluster	reconcilePods 	{"creating head pod for cluster": "raycluster-mini"}
2022-11-06T23:03:54.744Z	INFO	RayCluster-Controller	Setting pod namespaces	{"namespace": "kuberay-debug"}
2022-11-06T23:03:54.744Z	INFO	controllers.RayCluster	head pod labels	{"labels": {"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","groupName":"headgroup","ray.io/cluster":"raycluster-mini","ray.io/cluster-dashboard":"raycluster-mini-dashboard","ray.io/group":"headgroup","ray.io/identifier":"raycluster-mini-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head","rayCluster":"raycluster-sample","rayNodeType":"head"}}
2022-11-06T23:03:54.745Z	INFO	RayCluster-Controller	Head pod container with index 0 identified as Ray container.
2022-11-06T23:03:54.745Z	INFO	controllers.RayCluster	createHeadPod	{"head pod with name": "raycluster-mini-head-"}
2022-11-06T23:03:54.815Z	ERROR	controller.raycluster-controller	Reconciler error	{"reconciler group": "ray.io", "reconciler kind": "RayCluster", "name": "raycluster-mini", "namespace": "kuberay-debug", "error": "pods \"raycluster-mini-head-mbgdc\" is forbidden: exceeded quota: test-cpu-quota, requested: limits.cpu=500m,requests.cpu=500m, used: limits.cpu=0,requests.cpu=0, limited: limits.cpu=300m,requests.cpu=200m"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227
2022-11-06T23:06:38.663Z	INFO	controllers.RayCluster	reconciling RayCluster	{"cluster name": "raycluster-mini"}
2022-11-06T23:06:38.664Z	INFO	controllers.RayCluster	reconcileServices 	{"headService service found": "raycluster-mini-head-svc"}
2022-11-06T23:06:38.710Z	INFO	controllers.RayCluster	reconcilePods 	{"creating head pod for cluster": "raycluster-mini"}
2022-11-06T23:06:38.711Z	INFO	RayCluster-Controller	Setting pod namespaces	{"namespace": "kuberay-debug"}
2022-11-06T23:06:38.711Z	INFO	controllers.RayCluster	head pod labels	{"labels": {"app.kubernetes.io/created-by":"kuberay-operator","app.kubernetes.io/name":"kuberay","groupName":"headgroup","ray.io/cluster":"raycluster-mini","ray.io/cluster-dashboard":"raycluster-mini-dashboard","ray.io/group":"headgroup","ray.io/identifier":"raycluster-mini-head","ray.io/is-ray-node":"yes","ray.io/node-type":"head","rayCluster":"raycluster-sample","rayNodeType":"head"}}
2022-11-06T23:06:38.711Z	INFO	RayCluster-Controller	Head pod container with index 0 identified as Ray container.
2022-11-06T23:06:38.712Z	INFO	controllers.RayCluster	createHeadPod	{"head pod with name": "raycluster-mini-head-"}
2022-11-06T23:06:39.321Z	ERROR	controller.raycluster-controller	Reconciler error	{"reconciler group": "ray.io", "reconciler kind": "RayCluster", "name": "raycluster-mini", "namespace": "kuberay-debug", "error": "pods \"raycluster-mini-head-5ddnc\" is forbidden: exceeded quota: test-cpu-quota, requested: limits.cpu=500m,requests.cpu=500m, used: limits.cpu=0,requests.cpu=0, limited: limits.cpu=300m,requests.cpu=200m"}

[DmitriGekhtman]

Thanks! Closing this, then.

But it would be nice to get #639 merged...

Would you mind trying it out again with that PR branch? Do you see the issue reported by the author?

PR branch operator can be deployed by

• checking out the branch
• cd'ing to kuberay/ray-operator
• running the following commands

IMG=gekho/kuberay-operator make docker-build && docker push gekho/kuberay-operator

IMG=gekho/kuberay-operator make deploy

substituting the relevant image name.