Build Cop: bot is a little spammy

[nnegrey]

GoogleCloudPlatform/python-docs-samples#2805

[tbpg]

A few options for how we could improve this:

1. Don't comment when the test fails again.
2. Only comment when it's a new build ID.
3. Add a feature where you can add a buildcop: silence or buildcop: ignore label.

[nnegrey]

I like 1 and 2. I wonder if 3 could cause problems and some things could get missed accidentally, but not sure how those flags could be used.

[tbpg]

#285 will help with this for flaky tests. But, that won't change the behavior when the test fails many times in a row.

@bcoe specifically mentioned liking having a comment for every failure (option 1).

Maybe there is a good in between? Don't comment more than N times? Or maybe option 2 would be enough.

[tbpg]

@nnegrey, have the latest couple changes helped? The bot only comments once per build/commit ID failure (the second option above). It's also more proactive about marking issues as flaky when they pass & fail for the same build ID.

Should we do more? The next step would be something like:

• If there are more than 15 comments, add buildcop: quiet and comment describing the next bullet.
• If there is the buildcop: quiet label, don't comment, but close the issue if the test passes (with the usual flaky logic).

[nnegrey]

a buildcop: quiet label would be awesome, mostly I would probably add it once I'm aware what the issue is and have a PR that way the conversation on the bug doesn't get cluttered.

[nnegrey]

Would it be able to re-open the issue and start chattering away even if the quiet label was added? Like Oh I thought I fixed it, but it popped back up later due to some other change in the API

[tbpg]

I would expect it to ignore the quiet label when re-opening. So, it would re-open the issue, leave one comment explaining why it was re-opened, mark the issue flaky, and not comment anymore. Does that make sense to you?

[nnegrey]

Yep!