Fix flapping acceptance test

[QuentinBisson]

This PR fixes the flapping TestRetry acceptance test:

for i in {1..20}
do
  go test . -run TestRetry -count 1
done
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.514s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.515s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.513s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.514s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.512s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.513s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.514s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.515s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.512s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.514s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.511s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.513s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.514s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.514s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.513s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.514s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.513s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.515s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.514s
ok      github.com/prometheus/alertmanager/test/with_api_v2/acceptance  6.515s

[QuentinBisson]

cc @roidelapluie as we talked about it in #1682

I am not sure why the checks are not reported though

[roidelapluie]

Is it only flapping in go 1.16 or did we miss it?

[QuentinBisson]

I cannot reproduce it with go 1.15.11. I tested the code with both group_interval: 1s and group_interval: 2s and the test are running successfully in both cases

[roidelapluie]

Can you please also update the .circleci.yml to use go 1.16 for the acceptance tests?

Thanks!!

[QuentinBisson]

This is causing other tests to fail now :(

[roidelapluie]

I have relaunched the tests and they fail with another error..

[QuentinBisson]

After some trials, I see the text passing but the build is failing because go.sum is changing after the make proto step

[roidelapluie]

prometheus/prometheus@75e505b#diff-beb85cf1fc0ad937994d001f06b66b2a0e87a9416cc722aa97849b0d6ba17cb0

I copy back and forth the go.sum in prometheus. We could do the same.

[QuentinBisson]

Oh interesting :)

[QuentinBisson]

This seems to have fixed it :D

[beorn7]

Just a tiny nit.

I'm missing the context to say if the go.sum hack is kosher and if the increased group interval makes sense. But in doubt, I'd say let's merge this and release. We can always improve/revert/…

[beorn7]

Thanks @QuentinBisson .

@roidelapluie @simonpasquier I'm merging this now, to unblock the much needed release. I won't have time to shepherd it before end of next week. Will any of you be able to?