Skip to content

Initial support for Swift macros#6185

Merged
neonichu merged 1 commit into
mainfrom
macros
Feb 28, 2023
Merged

Initial support for Swift macros#6185
neonichu merged 1 commit into
mainfrom
macros

Conversation

@neonichu

@neonichu neonichu commented Feb 24, 2023

Copy link
Copy Markdown
Contributor
  • Add a new module CompilerPluginSupport which vends the macro target type
  • Represent macro targets in the package model
  • Build system support for macros

In order to be able to test this end-to-end before compiler support has landed, we're building macros as dylibs using the BUILD_MACROS_AS_DYLIBS flag since that is already supported by the Swift compiler in main.

The macro API is behind the 999.0 tools version since this has not gone through Swift evolution yet.

@neonichu neonichu self-assigned this Feb 24, 2023
@neonichu

Copy link
Copy Markdown
Contributor Author

@swift-ci please smoke test

Comment thread Sources/Build/BuildPlan.swift
@neonichu

Copy link
Copy Markdown
Contributor Author

@swift-ci please smoke test

Comment thread Sources/Build/BuildDescription/ProductBuildDescription.swift
Comment thread Sources/Build/BuildDescription/SwiftTargetBuildDescription.swift Outdated
Comment thread Sources/CompilerPluginSupport/TargetExtensions.swift
Comment thread Sources/PackageModel/Manifest/TargetDescription.swift
@neonichu neonichu marked this pull request as ready for review February 27, 2023 18:51
@neonichu neonichu requested a review from abertelrud as a code owner February 27, 2023 18:51
@neonichu

Copy link
Copy Markdown
Contributor Author

@swift-ci please smoke test

Comment thread Package.swift
// Enable building macros as dynamic libraries by default for bring-up.
for target in package.targets.filter({ $0.type == .regular || $0.type == .test }) {
target.swiftSettings = (target.swiftSettings ?? []) + [ .define("BUILD_MACROS_AS_DYLIBS") ]
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This essentially flips the logic for now since dylib macros are the only ones which can be tested end-to-end right now.

@neonichu neonichu changed the title WIP: Swift macros Initial support for Swift macros Feb 27, 2023
@neonichu

Copy link
Copy Markdown
Contributor Author

Looks like the self-hosted Linux toolchain understands the flag but doesn't contain the host swift-syntax? Or potentially the search paths are not correct for the Linux toolchain...

Test Suite 'Selected tests' started at 2023-02-27 18:54:29.954Test Suite 'MacroTests' started at 2023-02-27 18:54:29.956Test Case 'MacroTests.testMacrosBasic' started at 2023-02-27 18:54:29.956**** FAILURE EXECUTING SUBPROCESS ****
output: Building for debugging...
error: emit-module command failed with exit code 1 (use -v to see invocation)
[1/7] Emitting module MacroImpl
/tmp/Macros.ZNzbdr/MacroPackage/Sources/MacroImpl/macro.swift:1:8: error: no such module 'SwiftSyntax'
import SwiftSyntax
       ^

@neonichu

Copy link
Copy Markdown
Contributor Author

Ah no, I commented out the guard for the new test 🙈

@neonichu

Copy link
Copy Markdown
Contributor Author

@swift-ci please smoke test

@neonichu

Copy link
Copy Markdown
Contributor Author

Looks like at least one functional test was broken during the time they were disabled.

@neonichu

Copy link
Copy Markdown
Contributor Author

@swift-ci please smoke test

@neonichu

Copy link
Copy Markdown
Contributor Author

@swift-ci please smoke test

@neonichu

Copy link
Copy Markdown
Contributor Author
main/swiftpm/Sources/Build/BuildDescription/SwiftTargetBuildDescription.swift:392:9: error: call can throw but is not marked with 'try'
        self.requiredMacroProducts.forEach { macro in
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/swiftpm/Sources/Build/BuildDescription/SwiftTargetBuildDescription.swift:392:44: note: call is to 'rethrows' function, but argument function can throw
        self.requiredMacroProducts.forEach { macro in

Interesting, I did not see this locally, but it does sound correct.

@neonichu

Copy link
Copy Markdown
Contributor Author

@swift-ci please smoke test

@neonichu

Copy link
Copy Markdown
Contributor Author

Ah this is now accidentally always thrown

Basics/Errors.swift:20: Fatal error: building macros is not supported yet

@neonichu

Copy link
Copy Markdown
Contributor Author

@swift-ci please smoke test

@neonichu

Copy link
Copy Markdown
Contributor Author

Looks like the new library isn't visible in bootstrapped builds?

stderr: error: 'macropackage': Invalid manifest (compiled with: ["/home/build-user/build/buildbot_incremental/toolchain-linux-x86_64/usr/bin/swiftc", "-vfsoverlay", "/tmp/TemporaryDirectory.5FSAMh/vfs.yaml", "-L", "/home/build-user/build/buildbot_incremental/swiftpm-linux-x86_64/x86_64-unknown-linux-gnu/lib/swift/pm/ManifestAPI", "-lPackageDescription", "-Xlinker", "-rpath", "-Xlinker", "/home/build-user/build/buildbot_incremental/swiftpm-linux-x86_64/x86_64-unknown-linux-gnu/lib/swift/pm/ManifestAPI", "-swift-version", "5", "-I", "/home/build-user/build/buildbot_incremental/swiftpm-linux-x86_64/x86_64-unknown-linux-gnu/lib/swift/pm/ManifestAPI", "-package-description-version", "999.0.0", "-module-cache-path", "/home/build-user/build/buildbot_incremental/swiftpm-linux-x86_64/x86_64-unknown-linux-gnu/release", "/tmp/Macros.wvB2Uq/MacroPackage/Package.swift", "-Xfrontend", "-disable-implicit-concurrency-module-import", "-Xfrontend", "-disable-implicit-string-processing-module-import", "-o", "/tmp/TemporaryDirectory.9Jjuzi/macropackage-manifest"])
/tmp/Macros.wvB2Uq/MacroPackage/Package.swift:3:8: error: no such module 'CompilerPluginSupport'
import CompilerPluginSupport
       ^

@neonichu

Copy link
Copy Markdown
Contributor Author

Looks like the new library isn't visible in bootstrapped builds?

Can repro locally with ./Utilities/bootstrap test --filter FunctionalTests.MacroTests.testMacrosBasic -- not entirely sure how to fix yet

@neonichu

Copy link
Copy Markdown
Contributor Author

Looks like the new library isn't visible in bootstrapped builds?

Ah right, the test portion of bootstrapped builds only sees ManifestAPI/PluginAPI modules built by the CMake stage

@neonichu

Copy link
Copy Markdown
Contributor Author

@swift-ci please smoke test

Comment thread Utilities/bootstrap

if platform.system() == 'Darwin':
call(["sed", "-i", "", "s/macosx10.10/macosx%s/" % (g_macos_deployment_target), "build.ninja"], cwd=build_dir)
call(["sed", "-i", "", "s/CompilerPluginSupport.swiftinterface .*pm\/ManifestAPI\/PackageDescription.swiftinterface/CompilerPluginSupport.swiftinterface/", "build.ninja"], cwd=build_dir)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the way we're building PackageDescription with CMake is fairly broken, we are forcing it to produce a Swift interface, but CMake actually doesn't know about them. We haven't noticed the breakage until now, but since CompilerPluginSupport needs to link it, it surfaces here and needs to be worked around.

@neonichu

Copy link
Copy Markdown
Contributor Author

@swift-ci please smoke test Windows platform

@neonichu

Copy link
Copy Markdown
Contributor Author

Just noticed the Windows CI job was missing, it seems like maybe it doesn't automatically happen anymore with the standard smoke test? cc @shahmishal

@neonichu

Copy link
Copy Markdown
Contributor Author
======UPDATE FAILURES======
C:\Users\swift-ci\jenkins\workspace\swiftpm-PR-windows\llvm-project failed (ret=128): ['git', 'checkout', 'stable/20220421']
fatal: Unable to create 'C:/Users/swift-ci/jenkins/workspace/swiftpm-PR-windows/llvm-project/.git/index.lock': File exists.

That didn't take long :)

@neonichu

Copy link
Copy Markdown
Contributor Author

Looks like there's a remaining issue with the bootstrap install process:

FileNotFoundError: [Errno 2] No such file or directory: '/home/build-user/build/buildbot_incremental/swiftpm-linux-x86_64/x86_64-unknown-linux-gnu/release/libCompilerPluginSupport.so'

- Add a new module `CompilerPluginSupport` which vends the macro target type
- Represent macro targets in the package model
- Build system support for macros

In order to be able to test this end-to-end before compiler support has landed, we're building macros as dylibs using the `BUILD_MACROS_AS_DYLIBS` flag since that is already supported by the Swift compiler in `main`.

The macro API is behind the 999.0 tools version since this has not gone through Swift evolution yet.
@neonichu

Copy link
Copy Markdown
Contributor Author

@swift-ci please smoke test

@neonichu

Copy link
Copy Markdown
Contributor Author

CI is passing now, I am going to disable the functional tests again.

@neonichu

Copy link
Copy Markdown
Contributor Author

@swift-ci please smoke test

@neonichu neonichu enabled auto-merge (squash) February 28, 2023 06:06
@neonichu neonichu merged commit d733140 into main Feb 28, 2023
@neonichu neonichu deleted the macros branch February 28, 2023 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants