Support running Swift Testing tests repeatedly

[Stannieman]

Disclaimer
This issue is mainly for tracking for now. I think it can only be implemented if additional things are added to Swift Testing itself.

Context
Currently it is impossible to run Swift Testing tests repeatedly because the counter for the xxx.1.yyy, xxx.2.yyy naming does not get reset between test iterations. I tried a few things within SnapshotTesting testing to fix this, but nothing I can come up with is reliable.

What I tried

1. Look for something similar to XCTestObserver to reset the counter when a test starts/ends. No such thing seems to exist.
2. Check Test.current and see if, for a given test function, it is different than the previous time that test function ran. This does not work because Test.current is identical for each iteration.
3. Even though Test is a struct, we can still take an UnsafePointer to Test.current. We could check if it is different than the previous time that test function ran. As expected the struct frequently has the exact same address as the previous iteration so this is not a usable indicator. (It would anyway be unreliable at best.)
4. Check the current #line of the assertSnapshot call. When there are multiple calls in 1 test then the line will always increase. If the line did not increase since the previous call for a given test function they we can assume the test ran again and reset the counter. This does not work when the test is calling assertSnapshot inside a loop or for parametrized tests, so it is not usable as indication either.
5. Check "Relaunch Tests for Each Epetition" when running the test in Xcode. It has no effect.

I am out of ideas, but feel free to suggest something else I can try. I feel like the only reliable solution is that the Test struct contains an iteration count that keeps track of this.

I created a request for Swift Testing to add such property. It can be tracked here: swiftlang/swift-testing#1041.

Caveat for implementation
Just something to keep in mind when implementing this.
For XCTest the counterMap is cleared when a test ends.
For Swift Testing we cannot do this because tests run in parallel. We can only remove/reset the count for the current test, otherwise we might accidentally reset the counter for other tests that are still running.

[stephencelis]

This should be possible to fix in Xcode 16.3 with Swift Testing's new TestScoping trait. We plan on looking into this soon, but are you interested in giving it a shot?

[Stannieman]

It looks like that would require the user adding a trait to the test. Is that desirable?
I think it would definitely be better if it can happen automagically once Swift Testing gives us the tool to do so. (I have good faith that it will be added.)

A scenario that I predict will happening at our company all the time:

• Tests are not by default written with running repeatedly in mind.
• Only when a test turns out to be flaky (e.g. a race condition is involved) we run it 1000 times for debugging.
• This will always fail the first time.
• User remembers to add the trait.

So I feel like a solution without trait results in a nicer experience.

That being said, maybe a public function can be exposed to reset the counter? Then users can still write their own trait in 16.3 as a temporary fix until we can fix it properly. Of course the drawback here is that this is a bit of internal plumbing that gets exposed.

[mbrandonw]

It looks like that would require the user adding a trait to the test. Is that desirable? I think it would definitely be better if it can happen automagically once Swift Testing gives us the tool to do so. (I have good faith that it will be added.)

Test traits are the best way for this to work, and I do not think it's certain that Swift Testing will provide an alternative in the near future. TestScoping traits are the tool we have today.

So I feel like a solution without trait results in a nicer experience.

That being said, maybe a public function can be exposed to reset the counter? Then users can still write their own trait in 16.3 as a temporary fix until we can fix it properly. Of course the drawback here is that this is a bit of internal plumbing that gets exposed.

If we provided such a tool it would only cause other problems. It may fix running tests repeatedly, but it will break running tests in parallel because one test resetting that global state will cause another parallel test's counter to also reset.

The best way to handle this would be for counterMap to be upgraded to a @TaskLocal that holds onto a reference box of a counter. And then each test would be give its own counter that is isolated from all other tests, allowing it to work with in-process parallel tests, as well as parameterized tests and repeatedly run tests.

[Stannieman]

With traits we have the same problem AFAIK. From the docs it sounds like they are executed for each iteration AND for each case in a parametrized test. So we still cannot tell the difference between these 2 scenarios, and we need to be able to do so because for different cases the counter should not reset.

You're on to something with the TaskLocal variable though, that's definitely the better way to track the count.

[Stannieman]

Hold on, we're getting somewhere with the snippet below.

What works

• Non-parametrized test + single assertSnapshot call.
• Non-parametrized test + multiple assertSnapshot calls.
• Parametrized test with multiple cases + single assertSnapshot call.
• Parametrized test with multiple cases + multiple assertSnapshot calls.
• Parametrized test with single case + single assertSnapshot call.
• Parametrized test with single case + multiple assertSnapshot callss.

What doesn't work

• Multiple assertSnapshot calls on 1 line. But who does that?
• Parametrized tests with duplicate cases. But IMO this is acceptable as it makes no sense to have duplicates.

Possible improvement

• The "should we reset counter" logic can be rework a bit so we have a func inferCurrentTestIteration() which can later be replaced with a count provided by Swift Testing if/when they add it to the API.
• Also use #column to allow multiple assertSnapshot calls on 1 line.

What I hate

• The use of debugDescription as test case id. Unfortunately the API does not currently expose a better way to uniquely identify cases.

If this approach is something acceptable for you then I can prepare a PR + some tests in the coming days.

Followup discovery

It seems that this TaskLocal is still shared across multiple tests so we probably still need the dictionary to keep multiple tests apart.

private class SnapshotCount {
    var previousAssertLine: UInt = 0
    var encounteredCaseIds = [String]()
    var count = 0
}

private class SnapshotCounts {
    private var snapshotCounts = [SourceLocation: SnapshotCount]()
    func getOrCreate(for test: Testing.Test) -> SnapshotCount {
        let key = test.sourceLocation
        guard let snapshotCount = snapshotCounts[key] else {
            let new = SnapshotCount()
            snapshotCounts[key] = new
            return new
        }
        return snapshotCount
    }
}

private enum TestContext {
    @TaskLocal
    static var snapshotCounts = SnapshotCounts()
}

private let snapshotCountQueue = DispatchQueue(label: "Snapshot count queue")

struct RepeatingSnapshots {
    private func resetCounterIfNeeded(
        test: Testing.Test,
        assertLine: UInt,
        currentCaseId: String
    ) {
        let snapshotCount = TestContext.snapshotCounts.getOrCreate(for: test)
        
        guard assertLine <= snapshotCount.previousAssertLine else {
            // If the current line increases it is a different snapshot within the same test case.
            // In this case we should not reset the counter.
            return
        }
        
        func resetCounter() {
            snapshotCount.previousAssertLine = 0
            snapshotCount.encounteredCaseIds.removeAll()
            snapshotCount.count = 0
        }
        
        guard test.isParameterized else {
            // If we are not in a parametrized test and the line decreases or remains the same,
            // then this means we started a new iteration of the same test.
            // In this case we should reset the counter.
            //
            // This workaround fails when multiple assertSnapshot calls are put on the same line,
            // but maybe we can use column too to make it more robust.
            resetCounter()
            return
        }
        
        if snapshotCount.encounteredCaseIds.contains(currentCaseId) {
            // If we are in a parametrized test we cannot rely on the line trick alone
            // because it would reset for every test case.
            // Instead we also check if we already encountered the same test case before.
            // If so, that means we started a new iteration of the same test.
            // In this case we should reset the counter too.
            resetCounter()
            return
        }
    }
    
    private func count(assertLine: UInt) -> Int {
        guard let currentTest = Testing.Test.current else {
            fatalError("XCTest support needs to be fixed.")
        }
        
        let currentCaseId = Testing.Test.Case.current.debugDescription
        
        return snapshotCountQueue.sync {
            resetCounterIfNeeded(
                test: currentTest,
                assertLine: assertLine,
                currentCaseId: currentCaseId
            )
            
            let snapshotCount = TestContext.snapshotCounts.getOrCreate(for: currentTest)
            snapshotCount.previousAssertLine = assertLine
            snapshotCount.encounteredCaseIds.append(currentCaseId)
            snapshotCount.count += 1
            
            return snapshotCount.count
        }
    }
    
    private func assertSnapshot(line: UInt = #line) {
        print("Snapshot count: \(count(assertLine: line))")
    }
    
    @Test
    func tracksCount_withoutParams_oneSnapshot() {
        assertSnapshot()
    }
    
    @Test
    func tracksCount_withoutParams_multipleSnapshots() {
        assertSnapshot()
        assertSnapshot()
    }
    
    @Test(.serialized, arguments: ["a"])
    func tracksCount_withSingleParam_oneSnapshot(param: String) {
        assertSnapshot()
    }
    
    @Test(.serialized, arguments: ["a"])
    func tracksCount_withSingleParam_multipleSnapshots(param: String) {
        assertSnapshot()
        assertSnapshot()
    }
    
    @Test(.serialized, arguments: ["a", "b", "c"])
    func tracksCount_withMultipleParams_oneSnapshot(param: String) {
        assertSnapshot()
    }
    
    @Test(.serialized, arguments: ["a", "b", "c"])
    func tracksCount_withMultipleParams_multipleSnapshots(param: String) {
        assertSnapshot()
        assertSnapshot()
    }
}

[mbrandonw]

With traits we have the same problem AFAIK. From the docs it sounds like they are executed for each iteration AND for each case in a parametrized test. So we still cannot tell the difference between these 2 scenarios, and we need to be able to do so because for different cases the counter should not reset.

The counter should be reset for all of these situations. But for parameterized tests you should be using the named: argument of assertSnapshot in order to incorporate the data of the parameterization into the name of the artifact.

I should also mention that the named: argument is a work around for all of this even before we get proper support with a @TaskLocal. It allows you to provide a unique name for the artifact and then the counter is not used at all.

It seems that this TaskLocal is still shared across multiple tests so we probably still need the dictionary to keep multiple tests apart.

Each test/suite will need a .snapshots trait applied so that it can construct a fresh counter reference when each test run:

@Suite(.snapshots)
struct FeatureTests {
  …
}

In order to enforce this the @TaskLocal should probably start out nil. That would force the trait to be applied in order to get the counter going.

[Stannieman]

Ahah ok. I thought that the not-resetting of the counter for each parameter was a feature, but it's just the same issue as run repeatedly and should behave the same then. If using a named snapshot is the preferred solution in that case then that's fine and would simplify the counter book keeping a lot indeed.

If making the test fail if the trait is not set then using a trait also sounds ok to me. My main concern was that people would forget it and then only after writing 1000 such tests realize that they cannot run repeatedly. This will be a breaking change though, but probably not too major.

For the TaskLocal, not sure how to keep that to the scope of a single test. You'd somehow need to do My.$taskLocalValue.withValue(Counter()) { // run test body in here } or it will not stay isolated for that test. But then again I have not experimented with custom traits yet, so maybe it's trivial.

[mbrandonw]

If making the test fail if the trait is not set then using a trait also sounds ok to me. My main concern was that people would forget it and then only after writing 1000 such tests realize that they cannot run repeatedly. This will be a breaking change though, but probably not too major.

I was not suggesting to fail the test if the trait is not used. That would need to wait for a 2.0 release.

For the TaskLocal, not sure how to keep that to the scope of a single test. You'd somehow need to do My.$taskLocalValue.withValue(Counter()) { // run test body in here } or it will not stay isolated for that test. But then again I have not experimented with custom traits yet, so maybe it's trivial.

This is what Swift 6.1's TestScoping is for.

[stephencelis]

@Stannieman I've pushed a PR for this here: #966

Can you give it a spin? Note that you'll have to annotate the test/suite with the .snapshots test trait for it to work.

@Suite(.snapshots())
struct MySnapshotTests {
  …
}

[Stannieman]

Thanks! And Xcode 16.3 just came out, so will it today.
Btw, why does it need to store the counter in a dictionary for XCTest? Isn't there always only 1 test running per test process?