Switch leader election to lease

[timebertt]

How to categorize this issue?

/area scalability
/kind enhancement
/priority normal

What would you like to be added:

Once kubernetes-sigs/controller-runtime#1147 has been released and vendored, we should come back to the default leader election settings again and rather switch to the lease object for leader election.

Why is this needed:

We have seen significant scalability problems, when running more than 100 Shoot grm instances in one Seed each doing leader election on configmaps every 2s.
We were able to workaround this issue with #72 and gardener/gardener#2668 but we should switch to leases for having a long-term solution.

[timebertt]

Good news, kubernetes-sigs/controller-runtime#1147 was merged in the meantime and c-r@v0.7.0-alpha.1 just got released (including the configurable leader election lock), so we will probably be able to vendor the new release soon :)

[rfranzke]

ℹ️ Depends on revendoring sigs.k8s.io/controller-runtime@v0.7 in gardener/gardener.

[rfranzke]

I guess this can be done now after #105, right @timebertt? I think the default is now configmapsleases already, so is this issue only about setting the default to leases then?

[timebertt]

I think the default is now configmapsleases already, so is this issue only about setting the default to leases then?

Yes, exactly.
We probably have to do some migration logic in g/g in order to fully migrate to leases.
I guess, we can't simply change the default here, as users might skip releases of g/g or clusters might be hibernated for a long time, so that they skip a grm deployment using the configmapsleases lock.