ruler has hundreds of goroutines

[bboreham]

I'm guessing we create some unnecessary ones per tennant.

Excerpt:

goroutine 9051 [select]:
github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery.(*TargetSet).Run(0xc4221965a0, 0x302d2e0, 0xc421b2cc80)
	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery/discovery.go:203 +0x2b3
created by github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/notifier.(*Notifier).ApplyConfig
	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/notifier/notifier.go:267 +0x3e0

goroutine 96109984 [select]:
github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc42221e3f0, 0xc420413260, 0xc451415540, 0xc42685efd0, 0xc42685ef98, 0x5, 0x3010160, 0xc429463340)
	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery/discovery.go:273 +0x369
created by github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery/discovery.go:252 +0x2e0

goroutine 5761 [select]:
github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders.func1(0xc42219dc20, 0xc420f0a000, 0xc42043d810, 0xc420b9bdf0, 0xc420b9bde0, 0x5, 0x3010160, 0xc421d62700)
	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery/discovery.go:273 +0x369
created by github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery/discovery.go:252 +0x2e0

goroutine 5794 [select, 1272 minutes]:
github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/notifier.(*Notifier).Run(0xc421422ea0)
	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/notifier/notifier.go:309 +0x100
created by github.com/weaveworks/cortex/pkg/ruler.(*Ruler).getOrCreateNotifier
	/go/src/github.com/weaveworks/cortex/pkg/ruler/ruler.go:231 +0x1c9

goroutine 6673 [select]:
github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery/dns.(*Discovery).Run(0xc421a09c00, 0x302d2e0, 0xc421a09c80, 0xc421380a80)
	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery/dns/dns.go:106 +0x1b2
created by github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/discovery/discovery.go:250 +0x26f

goroutine 8497 [select]:
github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/notifier.(*Notifier).Run(0xc420446ea0)
	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/prometheus/notifier/notifier.go:309 +0x100
created by github.com/weaveworks/cortex/pkg/ruler.(*Ruler).getOrCreateNotifier
	/go/src/github.com/weaveworks/cortex/pkg/ruler/ruler.go:231 +0x1c9

[bboreham]

getOrCreateNotifier() calls notifier.ApplyConfig()which creates a goroutine callingRun()on everyAlertManagerConfig.TargetSetwhich creates two goroutines for everyTargetProvider`.

So not a bug, really, but we could consider condensing them.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.