Alertmanager intermittent crash at startup

[bboreham]

Seems that one or two of our alertmanagers will bounce a few times on startup, then eventually get going.

Note the mesh code will be removed if we address #793

Symptom:

level=info ts=2018-08-06T10:07:04.474111574Z caller=multitenant.go:434 msg="blank Alertmanager configuration; using fallback" user_id=9230
panic: [gossip] duplicate channel nflog:9230

goroutine 29 [running]:
github.com/weaveworks/cortex/vendor/github.com/weaveworks/mesh.(*Router).NewGossip(0xc42028c0e0, 0xc44f87be64, 0xa, 0xc9d200, 0xc4559b9700, 0x0, 0x0)
    /go/src/github.com/weaveworks/cortex/vendor/github.com/weaveworks/mesh/router.go:136 +0x399
github.com/weaveworks/cortex/pkg/alertmanager.(*gossipFactory).newGossip(0xc4201d8c80, 0xc44f87be64, 0xa, 0xc9d200, 0xc4559b9700, 0x0, 0x0)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/mesh.go:182 +0xf6
github.com/weaveworks/cortex/pkg/alertmanager.New.func1(0xc9d200, 0xc4559b9700, 0xbd8ac0, 0xc4527f1c20)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/alertmanager.go:72 +0x59
github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/nflog.WithMesh.func1(0xc4559b9700, 0xc4559b9700, 0x410928)
    /go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/nflog/nflog.go:173 +0x3b
github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/nflog.New(0xc4201c19c0, 0x6, 0x6, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/weaveworks/cortex/vendor/github.com/prometheus/alertmanager/nflog/nflog.go:252 +0xdb
github.com/weaveworks/cortex/pkg/alertmanager.New(0xc44f016be0, 0xc44f016be0, 0x4, 0xc101a2)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/alertmanager.go:70 +0x5c4
github.com/weaveworks/cortex/pkg/alertmanager.(*MultitenantAlertmanager).newAlertmanager(0xc420286780, 0xc421a4f000, 0x4, 0xc455ddefc0, 0xc455ddefc0, 0x0, 0x0)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/multitenant.go:476 +0x113
github.com/weaveworks/cortex/pkg/alertmanager.(*MultitenantAlertmanager).setConfig(0xc420286780, 0xc421a4f000, 0x4, 0x0, 0xc421a545a0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/multitenant.go:457 +0x63b
github.com/weaveworks/cortex/pkg/alertmanager.(*MultitenantAlertmanager).addNewConfigs(0xc420286780, 0xc4200c2690)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/multitenant.go:386 +0x280
github.com/weaveworks/cortex/pkg/alertmanager.(*MultitenantAlertmanager).Run(0xc420286780)
    /go/src/github.com/weaveworks/cortex/pkg/alertmanager/multitenant.go:298 +0x85
created by main.main
    /go/src/github.com/weaveworks/cortex/cmd/alertmanager/main.go:49 +0x212

[bboreham]

Duplicate of #899