Skip to content

Conversation

jtlisi
Copy link
Contributor

@jtlisi jtlisi commented Feb 11, 2020

After rebasing #1686, I noticed a major bug in the Alertmanager. The notify.NewPipelineBuilder function is called every time a change is applied causing a panic:

--- FAIL: TestMultitenantAlertmanager_loadAllConfigs (0.02s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16c72e6]

goroutine 15 [running]:
testing.tRunner.func1(0xc0001bca00)
	/usr/local/Cellar/go/1.13.6/libexec/src/testing/testing.go:874 +0x3a3
panic(0x1a2fbe0, 0x2440fe0)
	/usr/local/Cellar/go/1.13.6/libexec/src/runtime/panic.go:679 +0x1b2
github.com/prometheus/alertmanager/dispatch.(*Dispatcher).Stop(0x0)
	/Users/jtlisi/go/pkg/mod/github.com/prometheus/[email protected]/dispatch/dispatch.go:219 +0x26
github.com/cortexproject/cortex/pkg/alertmanager.(*Alertmanager).ApplyConfig(0xc00036b200, 0x1b69ac3, 0x5, 0xc00039a3f0, 0x6, 0x0)
	/Users/jtlisi/GrafanaLabs/projects/cortex/pkg/alertmanager/alertmanager.go:192 +0x263
github.com/cortexproject/cortex/pkg/alertmanager.(*MultitenantAlertmanager).newAlertmanager(0xc000091800, 0x1b69ac3, 0x5, 0xc00039a3f0, 0x0, 0x0, 0xc00051ba40)
	/Users/jtlisi/GrafanaLabs/projects/cortex/pkg/alertmanager/multitenant.go:434 +0x21e
github.com/cortexproject/cortex/pkg/alertmanager.(*MultitenantAlertmanager).setConfig(0xc000091800, 0x1b69ac3, 0x5, 0x1b9a6c1, 0x34, 0x247cd18, 0x0, 0x0, 0x501de00, 0x100)
	/Users/jtlisi/GrafanaLabs/projects/cortex/pkg/alertmanager/multitenant.go:401 +0xad1
github.com/cortexproject/cortex/pkg/alertmanager.(*MultitenantAlertmanager).syncConfigs(0xc000091800, 0xc00051b770)
	/Users/jtlisi/GrafanaLabs/projects/cortex/pkg/alertmanager/multitenant.go:289 +0x2cd
github.com/cortexproject/cortex/pkg/alertmanager.(*MultitenantAlertmanager).updateConfigs(0xc000091800, 0xc0003bc000, 0xc0000406c0)
	/Users/jtlisi/GrafanaLabs/projects/cortex/pkg/alertmanager/multitenant.go:271 +0x5c
github.com/cortexproject/cortex/pkg/alertmanager.TestMultitenantAlertmanager_loadAllConfigs(0xc0001bca00)
	/Users/jtlisi/GrafanaLabs/projects/cortex/pkg/alertmanager/multitenant_test.go:77 +0x4f8
testing.tRunner(0xc0001bca00, 0x1c646f0)
	/usr/local/Cellar/go/1.13.6/libexec/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
	/usr/local/Cellar/go/1.13.6/libexec/src/testing/testing.go:960 +0x350
FAIL	github.com/cortexproject/cortex/pkg/alertmanager	0.380s
FAIL
Error: Tests failed.

This PR fixes the issue and builds a single PipelineBuilder for each per-tenant alertmanager.

Checklist

  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Copy link
Contributor

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@pracucci pracucci merged commit 3327661 into master Feb 12, 2020
@pracucci pracucci deleted the 20200211_fix_registry_panic branch February 12, 2020 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants