Support retry-on-failure for only the failing @Test cases

[swhitty]

Motivation

xcodebuild supports the option -retry-tests-on-failure which can be passed to automatically retry tests that fail. Swift Testing is partially supported with this option, but all Swift Testing @Test are retried, even if they had already passed.

Consider the following tests;

struct SwiftTestingA {
  @Test
  func successA(){
    #expect(Bool(true))
  }

 @Test
  func failureA() {
    #expect(Bool(false))
  }
}

struct SwiftTestingB {
  @Test
  func successB() {
    #expect(Bool(true))
  }
}

final class XCTestA: XCTestCase {

  func testSuccess() {
    XCTAssertTrue(true)
  }

  func testFailure() {
    XCTAssertTrue(false)
  }
}

When executing with xcodebuild test ...

• XCTestA.testSuccess() is executed and passes 1 time only (expected).
• SwiftTestingA.success() is executed and passes 3 times (1 expected).
• SwiftTestingB.success() is executed and passes 3 times (1 expected).

% xcodebuild test -scheme TestLibrary -retry-tests-on-failure

Test suite 'XCTestA' started
Test suite 'SwiftTestingB' started
Test suite 'SwiftTestingA' started
Test case 'SwiftTestingB/successB()' passed (0.000 seconds)
Test case 'SwiftTestingA/successA()' passed (0.000 seconds)
Test case 'XCTestA.testFailure()' failed (0.284 seconds)
Test case 'XCTestA.testFailure()' failed (0.001 seconds)
Test case 'XCTestA.testFailure()' failed (0.001 seconds)
Test case 'SwiftTestingA/failureA()' failed (0.000 seconds)
Test suite 'SwiftTestingB' started
Test suite 'SwiftTestingA' started
Test case 'SwiftTestingB/successB()' passed (0.000 seconds)
Test case 'SwiftTestingA/successA()' passed (0.000 seconds)
Test case 'SwiftTestingA/failureA()' failed (0.000 seconds)
Test suite 'SwiftTestingB' started
Test suite 'SwiftTestingA' started
Test case 'SwiftTestingA/successA()' passed (0.000 seconds)
Test case 'SwiftTestingB/successB()' passed (0.000 seconds)
Test case 'SwiftTestingA/failureA()' failed (0.000 seconds)
Test case 'XCTestA.testSuccess()' passed (0.001 seconds)

Proposed solution

With xcodebuild aside, it would be useful for swift test to natively support an option like -retry-tests-on-failure to automatically retry tests that fail;

% swift test -retry-tests-on-failure

◇ Suite SwiftTestingA started.
◇ Suite SwiftTestingB started.
✔ Test successA() passed after 0.001 seconds.
✔ Test successB() passed after 0.001 seconds.
✔ Suite SwiftTestingB passed after 0.001 seconds.
✘ Test failureA() recorded an issue at Tests.swift:17:5: Expectation failed: Bool(false)
✘ Test failureA() iteration: 1 failed after 0.001 seconds with 1 issue.
✘ Test failureA() recorded an issue at Tests.swift:17:5: Expectation failed: Bool(false)
✘ Test failureA() iteration: 2 failed after 0.001 seconds with 1 issue.
✘ Test failureA() recorded an issue at Tests.swift:17:5: Expectation failed: Bool(false)
✘ Test failureA() iteration: 3 failed after 0.001 seconds with 1 issue.
✘ Suite SwiftTestingA failed after 0.001 seconds with 3 issues.
✘ Test run with 3 tests failed after 0.001 seconds with 3 issues.

Tests that pass successA() / successB() would not be retried and would only be executed once.

While parametrized tests could potentially retry just the parameters that fail — the CLI output reports these as a single test so it may be fine to just retry all parameters if a single parameter fails.

Alternatives considered

No response

Additional information

https://forums.swift.org/t/retry-tests-on-failure-retries-test-that-have-already-passed/83031

[LarsPetersHH]

I would like to add a big "yes, please!" to this request. The current behavior is less than ideal when running time consuming tests, e.g. when testing against a real backend (end to end tests).

[foxware00]

Similar context on this one, if one test fails/flakes it causes all the Swift Testing tests of which we have 1000s to re-run rather than the single test.

[grynspan]

Swift Testing already has command-line support for running multiple iterations (with the same lack of filtering as is noted here) but that option hasn't been propagated up to Swift Package Manager.

[stmontgomery]

Tracked by rdar://130508488

[Velin92]

Please this is definitely very important