Add support for Swift Testing in SwiftSyntaxMacrosTestsSupport

[0xTim]

• Explanation:
  As described in Adopt Swift-Testing in test utils such as SwiftSyntaxMacrosTestSupport #2720 SwiftSyntaxMacrosTestsSupport does not work with Swift Testing, which is an issue given Swift Testing is the way to test projects and XCTest is deprecated. If produces false positives (where tests pass even if they shouldn't) which is a major issue, especially as there are no warnings.

This PR adds support for Swift Testing so that tests fail correctly. This does not introduce an issue with a circular dependency on Swift Testing. There is a circular dependency at the package level, but this is allowed due to swiftlang/swift-package-manager#7530. There is no circular dependency between targets.

• Scope: No changes to existing code unless tests are passing when they shouldn't be. Only affects SwiftSyntaxMacrosTestsSupport
• Issues: Adopt Swift-Testing in test utils such as SwiftSyntaxMacrosTestSupport #2720
• Risk: Low risk
• Testing: Tests written to ensure it fails as expected
• Reviewers:

[grynspan]

This is not the correct test to determine which library is in use because code can run in a detached task. See swiftlang/swift-testing#475

[0xTim]

What is the correct way? Should we split it out into a expect function instead?

[grynspan]

There is no correct way at this time, which is why that issue is still open. Jerry's work should allow us to just call #expect() here and have it work under all configurations.

[0xTim]

The pitch looks like it will solve the issue, but still require work in the library to migrate over to Swift Testing APIs. What I propose is that we land this now, as it solves a problem that exists for users today (and potentially provide a release in the next monthly Linux release/Swift patch release) and then fix forward when the proposal lands. Given it's still in the pitch phase it likely won't be landed until 6.4 and waiting 10 months for a solution seems like a bad idea.

Regarding the Test.current issue - from my understanding this works in all instances apart from those running in a detached task. For this specific case, I can't see a scenario when a user would be using the assertMacroExpansion from a detached task so we can fix this for the majority of the users and those attempting to use it from a detached task will see no change in behaviour.

[grynspan]

@stmontgomery Your take? You okay with this presumably being nonfunctional with the package build?

For this specific case, I can't see a scenario when a user would be using the assertMacroExpansion from a detached task so we can fix this for the majority of the users and those attempting to use it from a detached task will see no change in behaviour.

Let's at least document it as unsupported in the symbol's Markup?

[grynspan]

If we can agree here that SwiftSyntax can provide Testing support by depending on the toolchain's Testing module can we unblock the PR on this matter?

I'd be morally okay with saying "the features in the swift-syntax repository are dependent on the built-in copy of Swift Testing even if you include a package dependency" however this will break builds on non-Apple platforms with flat linker namespaces due to duplicate symbols at link time.

I think there is still the larger question on how this is currently implemented. The if Test.current != nil check in this method is certainly less than ideal. Instead I would propose that we add an entirely new method called expectMacroExpansion that is based on Testing unconditionally. So existing users of assertMacroExpansion can continue to use it with XCTest and Testing users can start adopting the new one. How does that sound to everyone?

Once @jerryjrchen's work on the interop feature lands, it will be possible to implement this in a way that depends on neither XCTest nor Swift Testing. It may be a better idea to just wait until that work is done and revisit the problem at that point.

[FranzBusch]

I'd be morally okay with saying "the features in the swift-syntax repository are dependent on the built-in copy of Swift Testing even if you include a package dependency" however this will break builds on non-Apple platforms with flat linker namespaces due to duplicate symbols at link time.

Only if there is both Testing from the toolchain and from the package right? Which I thought we agreed is only really valid in development environments.

[grynspan]

Only if there is both Testing from the toolchain and from the package right? Which I thought we agreed is only really valid in development environments.

All testing environments are development environments.

If swift-syntax explicitly links the copy of Swift Testing in the toolchain, that will break developers who want to test their macros and have a package dependency on Swift Testing.

In addition, if they are using the package copy of Swift Testing and you are using the toolchain's copy, your calls to e.g. #expect() won't be routed to the infrastructure the developer's test target links against, so a failure will be invisible.

Jerry's work should give us an escape hatch for this problem, so we should wait until it lands and then make the necessary changes here.

[0xTim]

My concern with waiting is that it could be a year until that's shipped, meanwhile users have no indications their tests are passing incorrectly, whereas we could fix it today for the majority of use cases

[grynspan]

I appreciate the concern, but there are technical blockers here. We must not cause build failures for teams using Swift Testing as a package.

[0xTim]

Ok all, I've introduced a separate function to be used from Swift testing and added some documentation to the assert functions. Let me know what you think!

cc @FranzBusch

[grynspan]

I think this is still ultimately a moot point with @jerryjrchen's upcoming work?

[FranzBusch]

I think this is still ultimately a moot point with @jerryjrchen's upcoming work?

I thought we identified in the other thread that this is not the case. While the interoperability will allow the existing XCTest based method to work. It won't work with the strict mode which seems limiting. I think the new documentation in the swift-testing makes it pretty clear that the primary use-case of swift-testing as a package is for local development.

[0xTim]

To add some more context as well, I was talking to a couple of maintainers at FOSDEM who have both hit this issue with false positives

I think this is still ultimately a moot point with @jerryjrchen's upcoming work?

This work is still ongoing correct? Because it won't land until 6.4 at the earliest I'd assume

[grynspan]

Right but that doesn't mean we should ship new API that immediately becomes obsolete with the next Swift release.

[grynspan]

While the interoperability will allow the existing XCTest based method to work. It won't work with the strict mode which seems limiting.

The end result of Jerry's work is, as I understand it, that the existing API in this library can adopt #expect() in place of XCTAssertTrue() (etc. etc.) and it'll work in both libraries without problems.

I think the new documentation in the swift-testing makes it pretty clear that the primary use-case of swift-testing as a package is for local development.

I'm not sure I understand how that relates to my comment, but maybe you're referring to something earlier in the conversation I missed?

[jerryjrchen]

While the interoperability will allow the existing XCTest based method to work. It won't work with the strict mode which seems limiting.

We can resolve the strict mode restriction against using XCTest API in a Swift Testing test by adopting #expect in place of XCTest API (I can see XCTAssert* and XCTFail from a quick search). Interop also ensures the conversion to #expect still works as expected if the helpers are being called from an XCTest test case.

[stmontgomery]

Hi folks, I want to suggest a path forward here which I hope will unblock progress on this PR and allow it to proceed in the short term (if accepted by the repo's code owners, of course!) while laying out a plan for how it will align with our anticipated and fast-approaching interoperability work.

1. Keep just one "assert" function, with the existing name assertMacroExpansion()
   • …instead of introducing a separate function expectMacroExpansion() and having one for XCTest, one for Swift Testing.
   • This is to avoid bifurcation between the XCTest and Swift Testing worlds.
2. Modify assertMacroExpansion() so that it calls both Issue.record() (Swift Testing) and XCTFail() (XCTest), one after the other.
   • For right now, this is sufficient to ensure failures are surfaced regardless of which testing library is active.
   • Note: to achieve full reporting fidelity, the function will need to gain fileID: and column: parameters with default values and plumb those through to the "generic" funnel point so Swift Testing can reference them.
3. (Important) Communicate the potential effects of this change to SwiftSyntax users
   • This could happen via a detailed release note in the tag this change lands in. See the 602.0.0 notes for example of the format of the last release.
   • This note should do several things:
     • First, celebrate the useful new functionality—Swift Testing support for macro testing!
     • Then, explain the known caveat(s). The main one being that if you have a package dependency on Swift Testing, you upgrade to this (major) version of SwiftSyntax, and you attempt to validate macros from a Swift Testing test, then it's possible failures may not be surfaced correctly.
     • Link to my new document in the Swift Testing repo about this topic, and emphasize that it's due to using a non-built in copy.
     • Offer two solutions for anyone affected by this:
       • Simplest: Remove the package dependency on Swift Testing and switch to using a built-in copy, as the document recommends.
       • Otherwise, switch from importing SwiftSyntaxMacrosTestSupport to SwiftSyntaxMacrosGenericTestSupport and specify your own failureHandler: closure in which you call Issue.record() yourself. That should work even when using a package copy of Swift Testing.
4. Later, in a subsequent PR, adjust this logic for Swift Testing interoperability
   • If/when the pitched Interoperability feature arrives, assertMacroExpansion() will need to be adjusted to avoid redundant test issues being recorded due to it calling both Issue.record() and XCTFail().
   • To avoid that, at that point in time assertMacroExpansion() can be modified to stop calling XCTFail() starting in the Swift version where interoperability is first enabled.
   • (The call to XCTFail() shouldn't be removed altogether though, since SwiftSyntax could presumably still be built and used from older Swift versions.)

I've assembled a branch (atop this PR branch) with my suggested code changes for this PR in a commit you can see here: stmontgomery@c772dc6. Thoughts?

[stmontgomery]

There are two other points above I wanted to respond to:

While the interoperability will allow the existing XCTest based method to work. It won't work with the strict mode which seems limiting.

Building on what @jerryjrchen said, the semantics of strict mode will be that in any situation where Swift Testing offers an API, that should be preferred over the analogous XCTest variant. It does not mandate that you must only call the APIs corresponding to the active test framework, in case that's something folks are concerned about here. I do think/hope strict mode can eventually be used in conjunction with this work, if end users choose to enable it.

We must not cause build failures for teams using Swift Testing as a package.

I recognize that this plan does have the potential to break users who have a package dependency on Swift Testing. I suspect the practical impact of that is likely small due to few users relying on it, and I think the tradeoff here is very worthwhile since we'll be delivering a frequently requested feature. The document I wrote about distribution locations of Swift Testing was intended to help clarify our recommendations regarding use of the package copy, and frankly, to discourage its use more strongly than we have in the past.

This change will roll out in a new major tag of SwiftSyntax, meaning (per SemVer) it will be an explicit action users must take so the timing of the upgrade will be in their control. And I believe we can mitigate and help users make the transition gracefully with a clearly-worded release note (discussed above).

[0xTim]

@stmontgomery looks good to me - happy to pull those changes into my branch as I have done (just some extra changes to add if you agree around the can import stuff and expanding the test info) or have you submit the PR