Auto-retry jasmine test on CI

[etpinard]

Splitting the jasmine tests into two CI jobs did help alleviate some of our intermittent test failures, but these still happen in about 1 run out of 2.

As Circle CI 2.0 doesn't have an auto-retry setting yet, we currently have to painfully navigate the CircleCI UI to restart our failed jobs. This is of course a waste time. So, I propose this in-house solution.

@alexcjohnson @dfcreative what do you think?

[etpinard]

See example here where the first npm run test-jasmine call failed, but the second succeeded.

[alexcjohnson]

Mostly I love it, but I'm a little concerned about two things:

• If there really is a test failure you didn't notice locally, it will take 5 times as long before you see it
• At some point we should really fix these tests so they are robust, rather than just continuing to throw extra computing power at them. This solution hides the failures so it'll be hard to figure out where they are.

There was some discussion about how this could be done within jasmine, to just retry failing tests, they left it as the not very helpful "I think you should be able to build something that can do this with the existing custom reporter and filter function" but if we could get this to work that would be the best of all jasmine/jasmine#960

One idea though that I think would satisfy my concerns: can we make a tag for known unreliable tests, split those out into a separate task, and only run those ones in the retry loop? That way we would have a place to look to see which tests these are (and fix them when we get a chance), and other test failures wouldn't get slowed down.

[etpinard]

Mostly I love it

I'll take it. I was expecting this PR to be somewhat controversial. 😌

---

A few comments on your comments:

If there really is a test failure you didn't notice locally, it will take 5 times as long before you see it

That's not 100% true, one could still open up the CircleCI test page and stop the run when a true failure occurs. But yeah, this isn't ideal.

At some point we should really fix these tests so they are robust, rather than just continuing to throw extra computing power at them. This solution hides the failures so it'll be hard to figure out where they are.

Of course, that would be best. But as making tests robust on CI is very time consuming and sometimes frustrating, I hope this auto-retry solution is a step in the right direction.

There was some discussion about how this could be done within jasmine, to just retry failing tests

That would decrease our average total test time, but that may hide failures that occur because of bad test teardown. For example, if two tests ran in succession count the number of some DOM node and the first doesn't call destroyGraphDiv after completion, the second might fail in a full run, and pass in an isolated run.

One idea though that I think would satisfy my concerns: can we make a tag for known unreliable tests, split those out into a separate task, and only run those ones in the retry loop?

I like this solution too. I'm thinking of adding @flaky spec tag. Commit coming soon.

[etpinard]

Can you think of any other flaky tests worth adding to the list @alexcjohnson ?

[alexcjohnson]

Not sure, I think you got the most common ones, but now it's super easy to add @flaky if another test fails spuriously. Lets just keep an eye out for failures over the next few days after merging this.

[alexcjohnson]

Lovely, this should really help our productivity! 💃

[dy]

Brilliant solution @etpinard, thanks

[etpinard]

Flaky test alert: