Scaling issue: kube apiserver throttles external-provisioner when 100 PVCs created at same time

[saad-ali]

Reported by @cduchesne

The CSI external-provisioner has scaling issues.

When 100 PVCs are created at same time, the CSI external-provisioner hammers the kube apiserver with requests and gets throttled, causing all sorts of issues.

[sbezverk]

@cduchesne Please try the same test using this PR for external provisioner:
#66
I am curious if you see the same scalability issue.

[orainxiong]

@sbezverk, I have hit the same issue. After getting throttled, it will take over 30mins to create a Volume. I will try your PR.

[orainxiong]

@sbezverk

The problem is still not resolved. I am not sure if I understand correctly, the root cause is that external-provisioner excessively talk against Kube-API-Server.

Every time external-provisioner perform the operations of provisioning and deleting Volume, a sudden surge of requests comes in Kube-API-Server, and then the external-provisioner get throttled. As a result, all operations will takes unpredictable times and cause errors.

We can go through the logic of provisioning to figure out more details.

• Watch both Events and PersistentVolumeClaim to detect whether the operation of provisioning is successful or failure:

	successCh, err := ctrl.watchProvisioning(claim, stopCh)

	pvcWatch, err := ctrl.claimSource.Watch(options)
	if err != nil {
		return nil, err
	}

	failWatch, err := ctrl.getPVCEventWatch(claim, v1.EventTypeWarning, "ProvisioningFailed")
	if err != nil {
		pvcWatch.Stop()
		return nil, err
	}

	successWatch, err := ctrl.getPVCEventWatch(claim, v1.EventTypeNormal, "ProvisioningSucceeded")
	if err != nil {
		failWatch.Stop()
		pvcWatch.Stop()
		return nil, err
	}

• List PersistentVolumeClaim to get leadership information to determine whether

	oldLeaderElectionRecord, err := le.config.Lock.Get()

• Update PersistentVolumeClaim's annotation to acquire an exclusive lock before perform the operation of provisioning

	if err = le.config.Lock.Update(leaderElectionRecord); err != nil {
		glog.Errorf("Failed to update lock %s: %v", le.config.Lock.Describe(), err)
		return false
	}

Once external-provisioner gets throttled, the operation of watching Events will take over 10 seconds.

I0606 09:48:19.214596       1 request.go:480] Throttling request took 12.833541313s, request: GET:https://10.96.0.1:443/api/v1/namespaces/default/events?fieldSelector=involvedObject.name%3Dqcfs-pvc-only-114%2CinvolvedObject.namespace%3Ddefault%2CinvolvedObject.kind%3DPersistentVolumeClaim%2CinvolvedObject.uid%3D516852a4-6927-11e8-b1b9-525400e717a6%2Ctype%3DWarning%2Creason%3DProvisioningFailed
I0606 09:48:29.614493       1 request.go:480] Throttling request took 10.389247175s, request: GET:https://10.96.0.1:443/api/v1/namespaces/default/events?fieldSelector=involvedObject.namespace%3Ddefault%2CinvolvedObject.kind%3DPersistentVolumeClaim%2CinvolvedObject.uid%3D516852a4-6927-11e8-b1b9-525400e717a6%2CinvolvedObject.name%3Dqcfs-pvc-only-114%2Ctype%3DNormal%2Creason%3DProvisioningSucceeded

Worse, because of the policy of timeout of LeaderElector and provision, external-provisioner will retry periodically and produce more requests.

It will make things worse.

[saad-ali]

Thanks for surfacing this @orainxiong.

Problem sounds like the external-provisioner (and probably external-attacher and driver-registrar), open connections to kube API server without thinking twice. If you provision a lot of volumes at once, the kube API server gets mad and throttles everything.

We should do the following:

• Create a worker pool so that no more then N operations can be carried out against the API server at once.
• Allow N to be configured via a flag to the binary.
• Incorporate Kubernetes informer cache in to sidecar containers so that reads/writes are optimized.

[orainxiong]

@saad-ali Many thanks for your reply.

Incorporate Kubernetes informer cache in to sidecar containers so that reads/writes are optimized

As you mentioned, external-provisioner should leverage Informer cache to avoid talking directly against Kube-API-Server. And perhaps, I think we could use the issue CSI attach/detach should use shared informer for reference.

If possible, I hope to have the opportunity to ask you @saad-ali some views on the granularity of lock of LeaderElector. The granularity of lock which implemented with ProvisionPVCLock is very fine and allows multiple external-provisioner work parallel to improve efficiency. However, the disadvantage is the improvement comes with increasing the complexity of logic and overhead of Kube-API-Server. From my point view, the disadvantage outweights the advantage. I am wondering why we don't copy the convention way like existingControl Plane which allows one controller active and others passive.

[saad-ali]

Spoke with Jan. He agrees with @orainxiong: Major source of issues is probably the leader election logic -- each provisioner tries to write an annotation trying to be leader, if not successful each will retry after some time.

We should focus on making leader election scalable.

[saad-ali]

Temporary workaround may be to disable leader election per PV and instead use leader election per provisioner.

[wongma7]

Yes n pvcs * m provisioners is unacceptably bad. The original code is ancient, I will copy how HA is done in kube. It is only like this for the rare case where a person wants multiple provisioner instances (pointing to different storage sources w/ same properties) to serve the same storage class. But that isn't a common use-case and this is a lazy and inefficient way of achieving it anyway.

An alternative was sharding but I don't think it's possible

[orainxiong]

@saad-ali Thanks for your reply.

As we discussed above, this issue is mainly due to the leadership election and the abuse of API Server. I try to fix the problem through the following steps which requires the substantial changes to external-provisioner and external-storage. I am not sure whether it is right or not. Looking forward to your suggestions.

• Implement leader election per provisioner as Kube-Controller-Manager does. Meanwhile, the role of external-provisioner-runner should be added more permissions to access both configmaps, endpoints and secrets.
• Delete both function lockProvisionClaimOperation and related codes within external-storage.
  Note: Storage provisioner should perform lock election with themselves rather than relying on external-storage.

But, these are not enough, both of external-storage and external-provisioner are supposed to use informer cache instead of talking directly against API Server.

There is a PR show more details which mainly involve following files :

• cmd/csi-provisioner/csi-provisioner.go
• pkg/controller/controller.go
• vendor/github.com/kubernetes-incubator/external-storage/lib/controller/controller.go

These changes resulted in significantly reducing the time-consuming of provision and delete. In my environment, the time we take for provisioning 100 CSI PVC is reduced from more than 50 mins to 100 seconds.

[vladimirvivien]

Where are we with this? Is @orainxiong PR the work around we want to pursue ?

[vladimirvivien]

@orainxiong can you please create a proper PR so we can comment on it. Thanks.

[orainxiong]

@vladimirvivien Here is PR #104

If any questions, please let me know. Many thanks.