Add a new metrics to expose local cached items size

[halfcrazy]

Backgroud
In controller-runtime, the default behaviour is to cache all data using client.Get unless using a read-only client or custom cache policies, e.g., ClientDisableCacheFor, NewCache. Therefore, using client.Get on a specific pod in a large cluster with many pods may lead to high memory usage.

Proposal
Add a new metric to expose the local cached items' size. Then we can analyze the high memory usage caused by the misuse cache policy.

[halfcrazy]

#3054

[krisztianfekete]

@halfcrazy, looks like this will be done by #3202

[halfcrazy]

@halfcrazy, looks like this will be done by #3202

I'm afraid #3202 cannot resolve this issue. AFAIK, until kubernetes/kubernetes#129160 is merged. We need to register the informer metric too.

[xigang]

/cc @sbueringer

[sbueringer]

Yeah, we're going to wait for this to be implemented in k/k. Then we can probably pick it up (if it's safe from a metric cardinality point of view)

[xigang]

Thanks @sbueringer ! Could you help review this PR kubernetes/kubernetes#129160? It would be great to keep things moving forward.

[sbueringer]

Huge backlog unfortunately at the moment. I'll try to get to it, but I can't promise it.

[xigang]

@sbueringer Thank you.