panic: [gossip] duplicate channel nflog:10130 after restart alertmanager

[lelenanam]

panic: [gossip] duplicate channel nflog:10130

goroutine 42 [running]:
github.com/weaveworks/cortex/vendor/github.com/weaveworks/mesh.(*Router).NewGossip(0xc42028c000, 0xc445340c93, 0xb, 0xc35460, 0xc443b2f480, 0x0, 0x0)
    /go/src/github.com/weaveworks/cortex/vendor/github.com/weaveworks/mesh/router.go:136 +0x399
github.com/weaveworks/cortex/pkg/alertmanager.(*gossipFactory).newGossip(0xc4202609e0, 0xc445340c93, 0xb, 0xc35460, 0xc443b2f480, 0x0, 0x0)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/mesh.go:182 +0xf6
github.com/weaveworks/cortex/pkg/alertmanager.New.func1(0xc35460, 0xc443b2f480, 0xb78600, 0xc445624150)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/alertmanager.go:72 +0x59
github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/nflog.WithMesh.func1(0xc443b2f480, 0xc443b2f480, 0x410928)
    /go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/nflog/nflog.go:173 +0x3b
github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/nflog.New(0xc42044b9c0, 0x6, 0x6, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/nflog/nflog.go:252 +0xdb
github.com/weaveworks/cortex/pkg/alertmanager.New(0xc4455e6b40, 0xc4455e6b40, 0x5, 0xbac2bd)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/alertmanager.go:70 +0x5c4
github.com/weaveworks/cortex/pkg/alertmanager.(*MultitenantAlertmanager).newAlertmanager(0xc420268680, 0xc42046a850, 0x5, 0xc44539cf50, 0xc44539cf50, 0x0, 0x0)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/multitenant.go:476 +0x113
github.com/weaveworks/cortex/pkg/alertmanager.(*MultitenantAlertmanager).setConfig(0xc420268680, 0xc42046a850, 0x5, 0x0, 0xc4201bf380, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/multitenant.go:457 +0x63b
github.com/weaveworks/cortex/pkg/alertmanager.(*MultitenantAlertmanager).addNewConfigs(0xc420268680, 0xc4200b4690)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/multitenant.go:386 +0x280
github.com/weaveworks/cortex/pkg/alertmanager.(*MultitenantAlertmanager).Run(0xc420268680)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/multitenant.go:298 +0x85
created by main.main
    /go/src/github.com/weaveworks/cortex/cmd/alertmanager/main.go:50 +0x250

It happed before (July 12 for example):

[bboreham]

Note the mesh code will be removed if we address #793

[bboreham]

The incidence of this went down dramatically after #1558 was merged, suggesting it was linked to high numbers of gossip channels.