Allow `@RepeatedTest` to stop after a certain number of failures

[YuanPeiqi]

This idea is similar to issue #2119

I think the function mentioned in issue #2119 could be extended to allow users to define tolerable number of failures, by introduce something like tolerableFailureNumber attribute (which is an integer)

@RepeatedTest(10, tolerableFailureNumber = 2)	// this test will terminate after five failures

This can not only fulfill the requirements in issue #2119 by setting tolerableFailureNumber to 1, but also provide richer functionality.

[mpkorstanje]

I'm not part of the JUnit team. I'm commenting as a contributor to both Cucumber and JUnit. Cucumber has had this feature in some implementations and it is decidedly less trivial then it seems.

Fundamentally it opens three questions, 1) what is considered a failure of a test, and 2) how are the attempts reported, and 3) how does a test continue with the next attempt.

Given a test is repeated N times:

• It can be considered past if all attempts did not fail. This effectively makes each of the N attempst an individual test.
• It can be considered failed when at least 1 of the attempts failed.
• It can be considered failed as soon as 1 of the attempts is passed.
• It can be considered passed when X% of the test passed.
• It can be considered passed or failed according to some decision function

Reporting on the attempts should then include:

• Only passed attempt(s)
• Only the failed attempt(s)
• All executed attempts (passed and failed)
• All attempts (passed, failed, canceled).
• Any of the above in combination with motivational output from the decision function.

And when a test can be decided to have passed or or failed based on the attempts it executed the remaining attempts should be:

• Cancelled
• Continued

Depending on the exact usecase any combination of these 3 is desirable.

Currently JUnit 5 does not have anything to accurately model or represent attempts. While representing attempts as tests may be tempting at first. This conceptual mismatch soon leaks out into reporting and test pass/fail decisions.

Conceptually attempts are difficult because they do not exist in the hierarchial test description, but they do exist as part of the execution and reporting. Additionally reporting entities need to be able to understand if a result should be included and the correct semantics vy which it should be presented.

So tread carefully.

[sbrannen]

Team Decision: Implement support for skipping remaining repetitions after a specified failure threshold has been exceeded.

@RepeatedTest(value = 10, failureThreshold = 2)

For example, if a repeated test begins to fail at the 6th repetition, the result would look similar to the following.