Memory leak in v2.1.0

[janartodesk]

This is a slice of the average memory usage plot for the liveness probe container to illustrate the issue (Y-axis in MiB):

We had been running the driver with a short 2 second liveness check interval to increase our chances discovering any issues with it (a new cluster and CSI is an uncharted territory for us), racking up ~1.5GiB of memory usage per liveness probe container in around two weeks.

Otherwise the liveness probe chugs along happily with no anomalies in logs or other metrics for the probe itself or any of the related workloads.

[msau42]

/kind bug

[msau42]

/assign @Jiawei0227

[Jiawei0227]

A quick google search lead me to this and it does seem like what happens in this case:

https://stackoverflow.com/questions/21080642/memory-leak-in-go-http-standard-library

I will investigate more on this next week.

[Jiawei0227]

Hmm, actually I might be wrong... This does not seem related. I will need more checking on the code

[Jiawei0227]

Issue is at this line:

livenessprobe/cmd/livenessprobe/main.go

Line 52 in e63e79e

conn, err := acquireConnection(ctx, metrics.NewCSIMetricsManager(""))

Everytime there is a new request, it will create a new metricsManager which seems unnecessary. This should be accountable for the memory leaking as well.

[Jiawei0227]

Steps to triage:

1. Start a livenessprobe container inside a driver.
2. Add _ "net/http/pprof" to enable pprof support.
3. curl -sK -v http://10.128.15.211:9808/debug/pprofeap > heap.out to get the heap
4. go tool pprof heap.out to analyze the heap

It is quite straight forward to see that: NewCSIMetricsManager() is taking a lot memory usage.

(pprof) top
Showing nodes accounting for 5158.14kB, 100% of 5158.14kB total
Showing top 10 nodes out of 51
      flat  flat%   sum%        cum   cum%
 1024.16kB 19.86% 19.86%  3079.02kB 59.69%  github.com/kubernetes-csi/csi-lib-utils/metrics.NewCSIMetricsManager
  532.26kB 10.32% 30.17%   532.26kB 10.32%  google.golang.org/protobuf/internal/strs.(*Builder).AppendFullName
  522.70kB 10.13% 40.31%   522.70kB 10.13%  google.golang.org/protobuf/internal/filedesc.(*File).initDecls
  518.65kB 10.06% 50.36%   518.65kB 10.06%  k8s.io/component-base/metrics.newKubeRegistry
  512.14kB  9.93% 60.29%  1024.16kB 19.86%  github.com/prometheus/client_golang/prometheus.(*metricMap).getOrCreateMetricWithLabelValues
  512.10kB  9.93% 70.22%   512.10kB  9.93%  github.com/prometheus/client_golang/prometheus.(*Registry).Register
  512.05kB  9.93% 80.15%   512.05kB  9.93%  github.com/prometheus/client_golang/prometheus.NewDesc
  512.05kB  9.93% 90.07%   512.05kB  9.93%  k8s.io/component-base/metrics.NewHistogramVec (inline)
  512.02kB  9.93%   100%   512.02kB  9.93%  github.com/prometheus/client_golang/prometheus.makeLabelPairs
         0     0%   100%  1024.16kB 19.86%  github.com/container-storage-interface/spec/lib/go/csi.(*identityClient).Probe

[Jiawei0227]

#94 should fix this.

[Jiawei0227]

/close

[k8s-ci-robot]

@Jiawei0227: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.