alertmanager has thousands of goroutines

[bboreham]

Similar to #664 - maybe we need some per tennant but ones like logging and mesh gossip are questionable.

First number (e.g. 2 @) is the count per tennant:

4 @ 0x42e23c 0x43e4a8 0x9697bb 0x45d541
#	0x9697ba	github.com/weaveworks/cortex/vendor/github.com/weaveworks/mesh.(*gossipSender).run+0x11a	/go/src/github.com/weaveworks/cortex/vendor/github.com/weaveworks/mesh/gossip.go:101

2 @ 0x42e23c 0x42e32e 0x4055b4 0x40525b 0x9c8765 0x45d541
#	0x9c8764	github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/provider/mem.(*Alerts).Subscribe.func1+0x194	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/provider/mem/mem.go:114

1 @ 0x42e23c 0x42e32e 0x4055b4 0x40525b 0x9c4b56 0x9c58f7 0x45d541
#	0x9c4b55	github.com/weaveworks/cortex/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run+0xf5			/go/src/github.com/weaveworks/cortex/vendor/github.com/oklog/oklog/pkg/group/group.go:43
#	0x9c58f6	github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/inhibit.(*Inhibitor).Run+0x366	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/inhibit/inhibit.go:125

1 @ 0x42e23c 0x43e4a8 0x13ddc0f 0x45d541
#	0x13ddc0e	github.com/weaveworks/cortex/pkg/alertmanager.New.func5+0xde	/go/src/github.com/weaveworks/cortex/pkg/alertmanager/alertmanager.go:137

1 @ 0x42e23c 0x43e4a8 0x984c0d 0x45d541
#	0x984c0c	github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/nflog.(*nlog).run+0x1cc	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/nflog/nflog.go:317

1 @ 0x42e23c 0x43e4a8 0x9967a4 0x13dda7e 0x45d541
#	0x9967a3	github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/silence.(*Silences).Maintenance+0x1a3	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/silence/silence.go:293
#	0x13dda7d	github.com/weaveworks/cortex/pkg/alertmanager.New.func3+0xcd							/go/src/github.com/weaveworks/cortex/pkg/alertmanager/alertmanager.go:108

1 @ 0x42e23c 0x43e4a8 0x9b7aa8 0x9b69c9 0x45d541
#	0x9b7aa7	github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/dispatch.(*Dispatcher).run+0x207	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/dispatch/dispatch.go:178
#	0x9b69c8	github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/dispatch.(*Dispatcher).Run+0x158	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/dispatch/dispatch.go:70

1 @ 0x42e23c 0x43e4a8 0x9c5088 0x9c70ac 0x9c4c87 0x45d541
#	0x9c5087	github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/inhibit.(*Inhibitor).runGC+0x147		/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/inhibit/inhibit.go:60
#	0x9c70ab	github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/inhibit.(*Inhibitor).Run.func1+0x3b	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/inhibit/inhibit.go:113
#	0x9c4c86	github.com/weaveworks/cortex/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1+0x26			/go/src/github.com/weaveworks/cortex/vendor/github.com/oklog/oklog/pkg/group/group.go:38

1 @ 0x42e23c 0x43e4a8 0x9c5251 0x9c714c 0x9c4c87 0x45d541
#	0x9c5250	github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/inhibit.(*Inhibitor).run+0x180		/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/inhibit/inhibit.go:76
#	0x9c714b	github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/inhibit.(*Inhibitor).Run.func3+0x3b	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/inhibit/inhibit.go:119
#	0x9c4c86	github.com/weaveworks/cortex/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1+0x26			/go/src/github.com/weaveworks/cortex/vendor/github.com/oklog/oklog/pkg/group/group.go:38

1 @ 0x42e23c 0x43e4a8 0x9c784f 0x45d541
#	0x9c784e	github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/provider/mem.(*Alerts).runGC+0x1ee	/go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/provider/mem/mem.go:54

[bboreham]

This isn't helped by alertmanager creating a worker for every config including deleted ones.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.