Android support

[marcprux]

As discussed in #83, and now that the Android SDK nightly-6.1 supports Swift Testing (swiftlang/swift-package-manager#8362), this PR adds experimental support for cross-compiling Swift packages for Android and running their tests on an Android emulator. It is only run when the enable_android_checks input is set to true, making it opt-in for now.

It uses swift-android-action to set up the environment and run the tests, which is the same workflow that powers https://swift-everywhere.org and various other Swift CI workflows. This action, in turn, delegates to the reactivecircus/android-emulator-runner workflow to set up and run the emulator based in the inputs passed to the workflow (android_api_level, android_channel, android_profile, and android_target). Support for transferring specific files to the Android emulator (with android_copy_files) and setting custom test environment variables (with android_test_env) enable customization of the test environment (for example, see https://github.com/jpsim/Yams/blob/main/.github/workflows/swiftpm.yml#L146).

This works towards the one of the goals of the Android Workgroup to provide continuous integration support for Android. A successful run of this workflow can be seen at https://github.com/marcprux/swift-algorithms/actions/runs/13838259992

Closes #83.

[marcprux]

@shahmishal There's a pending review for the codeowner "swiftlang/github-workflows-contributors", but I don't know who that group consists of. At yesterday's Android WG meeting, we (myself, @compnerd, @finagolfin, @Joannis, and @andriydruk) discussed how me might get some discussion going around this.

As documented in the action's README, the action will transition to using an official Android SDK build once it is published on swiftlang, but getting something going sooner rather than later will help keep packages building on Android and prevent regressions (like swiftlang/swift-tools-support-core#499, swiftlang/swift-package-manager#8372, and swiftlang/swift-format#963). And the action is already in use by a bunch of popular projects (like Yams, Vapor, GraphQL, and Alamofire), so building on this strong base of projects will help improve the Android coverage for everyone.

[shahmishal]

Thank you for creating a pull request to add Android workflow.

As documented in the action's README, the action will transition to using an official Android SDK build once it is published on swiftlang, but getting something going sooner rather than later will help keep packages building on Android and prevent regressions (like swiftlang/swift-tools-support-core#499, swiftlang/swift-package-manager#8372, and swiftlang/swift-format#963). And the action is already in use by a bunch of popular projects (like Yams, Vapor, GraphQL, and Alamofire), so building on this strong base of projects will help improve the Android coverage for everyone.

Let's work on getting the official Android SDK set up on ci.swift.org and start publishing on swift.org

@marcprux, could you please share the documentation or steps for producing it on Linux? It would be helpful if we could create a Dockerfile that includes the necessary NDKs and tools for building the Android SDK. cc: @finagolfin

@shahmishal There's a pending review for the codeowner "swiftlang/github-workflows-contributors", but I don't know who that group consists of. At yesterday's Android WG meeting, we (myself, @compnerd, @finagolfin, @Joannis, and @andriydruk) discussed how me might get some discussion going around this.

I would be happy to discuss this further in the next Android WG meeting. However, I would like to have the .artifactbundle available on swift.org as soon as possible.

[marcprux]

@marcprux, could you please share the documentation or steps for producing it on Linux?

The SDK that I'm using is built with @finagolfin's actions at https://github.com/finagolfin/swift-android-sdk, which uses scripts, patches, and external libraries built with https://github.com/finagolfin/swift-android-sdk/blob/main/.github/workflows/sdks.yml and https://github.com/finagolfin/swift-android-sdk/blob/main/get-packages-and-swift-source.swift. I grab the nightlies and publish them at https://github.com/skiptools/swift-android-toolchain/releases, which is what the swift-android-action installs and uses.

I would like to have the .artifactbundle available on swift.org as soon as possible.

So would we! We've been discussing the best way to hammer the process into something that can be incorporated into something official.

[finagolfin]

Let's work on getting the official Android SDK set up on ci.swift.org and start publishing on swift.org

@marcprux, could you please share the documentation or steps for producing it on Linux? It would be helpful if we could create a Dockerfile that includes the necessary NDKs and tools for building the Android SDK. cc: @finagolfin

That's great news! I don't think I ever asked about producing an official Android SDK yet, as I simply assumed the first step would be to get it on the official CI and declared an officially supported platform first, which I have delayed setting up as I worked on other issues I felt were more pressing for the upcoming 6.1 release and 6.2 branch.

However, in the meantime, @compnerd told us at the latest community Android WG meeting that TBC has started shipping Android SDKs in the nightly Windows toolchain and they have enabled running non-executable tests from the compiler validation suite for Android on the Windows CI, as can be seen in the meeting notes @Joannis posted this weekend. So since Android is now being tested as part of the official Windows CI, I see no obstacle to moving forward with an official Android SDK also. 😃

If the official Windows toolchain is already shipping official Android SDKs, the fastest way to do this would be to grab those from the Windows toolchain and package them as separate Android SDK bundles also. We discussed this with Saleem at the Android WG meeting, but he said the Windows toolchain does not support separate SDK bundles yet, hence bundling everything together on Windows.

I think it would be a great testament to Swift's cross-platform support and the Windows toolchain specifically if we can produce these official Android SDK bundles initially from Windows. 😸 I will follow that up with pulls to get build-script building Android SDKs on linux and macOS too, so it is easy for people using those platforms to build it themselves and to make sure the official CI notifies us if anything breaks when building SDKs.

@marcprux, do you have time now to get started by looking into simply repackaging the prebuilt Android SDKs from the official nightly Windows toolchain as Android SDK bundles? The first step would be to download those and create an SDK bundle from them locally, then test cross-compiling some packages against it and make sure it works. If it does, you can simply script what you did in the official swift-docker repo too, as Alastair did to produce the static linux SDK, but with much fewer steps since you're just repackaging prebuilt SDKs from the Windows toolchain. I'll join you once 6.1 ships and 6.2 branches.

@shahmishal and @compnerd, let us know what you think of my plan above to get some separate SDK bundles up quickly. I may have missed something, particularly since I've never used the Windows Swift toolchain, so let me know if I did.

[marcprux]

@marcprux, could you please share the documentation or steps for producing it on Linux?

I've put up a draft PR at swiftlang/swift-docker#467 which contains stand-alone build scripts for a Swift 6.1 Android SDK. There are still some issues being discussed (swiftlang/swift#80788), but the build process itself creates an installable .artifactbundle that is tested as part of the CI.

It would be helpful if we could create a Dockerfile that includes the necessary NDKs and tools for building the Android SDK. cc: @finagolfin

It is not yet dockerized. The SDK itself includes the subset of the NDK sysroot that is needed for the build.

[jakepetroules]

As part of this could we get the ability to control the host as well?

Ideally we'd be able to test macos-android and windows-android in addition to linux-android. For SwiftBuild/SwiftPM that sort of coverage can be quite important.

[jakepetroules]

The SDK itself includes the subset of the NDK sysroot that is needed for the build.

Maybe a bit off topic... I think I mentioned this to @compnerd elsewhere, but could we get the ability to have the SDK reference an externally provided NDK, similar to how Windows works? I don't think we should be copying the content in, because even if the Android NDK license allows this, other toolchains may not, so it would be good not to set a precedent.

[marcprux]

As part of this could we get the ability to control the host as well?

Ideally we'd be able to test macos-android and windows-android in addition to linux-android.

That's a good idea, and the swift-android-action does support running on macOS as well as Ubuntu, but macOS is currently limited to GH's Intel runners, as the M-series didn't support nested virtualization until recently, so none of the free GH-hosted runners after macos-13 can run the test cases (see ReactiveCircus/android-emulator-runner#350 for the whole saga). If GH upgrades their macos runners to M4 machines in the future, then we might be able to run the tests on those.

Windows would be nice, but they don't support SDK .artifactbundles yet (@compnerd please correct me if I am wrong, or if is something being worked on). We might be able to cobble together the right build flags or synthesize an old-style destination.json file for Windows, but I don't have enough experience on the platform to assess the level of effort that would require.

For SwiftBuild/SwiftPM that sort of coverage can be quite important.

FWIW, it has been exceedingly rare in my experience for a package to successfully cross-compile from Linux to Android and fail from macOS to Android. And even less rare (maybe never) that test cases would have different outcomes when run from different hosts, since the testing is performed on the Android QEMU emulator, and so the host machine shouldn't make any difference.

[marcprux]

The SDK itself includes the subset of the NDK sysroot that is needed for the build.

Maybe a bit off topic... I think I mentioned this to @compnerd elsewhere, but could we get the ability to have the SDK reference an externally provided NDK, similar to how Windows works? I don't think we should be copying the content in, because even if the Android NDK license allows this, other toolchains may not, so it would be good not to set a precedent.

That's something we are working on — I was just chatting with @finagolfin about it. Separating the SDK and NDK from each other will involve some work (e.g., see swiftlang/swift#76381 and swiftlang/swift#79621 and swiftlang/swift-driver#1822).

But that just affects the ultimate SDK distribution bundle; for the purposes of this GH workflow PR, it is all abstracted behind the swift-android-action, which will be updated to accommodate the separated SDK/NDK once we have released it.

Personally, I'd like to see the validation enabled by this workflow become available sooner rather than later. Every few days some Android-breaking change slips into some package, and finding/fixing these after the fact is considerably more difficult than having it caught up front, especially when the breakage slips into a release and triggers a cascade of Android failures in dependent packages.

[finagolfin]

It is not yet dockerized. The SDK itself includes the subset of the NDK sysroot that is needed for the build.

That's not what he's referring to. Pretty much all the official toolchains and SDKs are built by Dockerfiles. Mishal'd like us to do the same for the official Android SDK, ie build it in Docker. We'll have to create a linux Dockerfile with the NDK installed that runs build-script and your libcurl/xml build scripts, as I don't think Mishal wants the Android build invoked the way your current pull does it, ie without Docker.

macOS is currently limited to GH's Intel runners, as the M-series didn't support nested virtualization until recently, so none of the free GH-hosted runners after macos-13 can run the test cases

We could have the Android workflow only build the packages and tests on all macOS runners, and skip running the tests in the emulator and report that if it detects that the particular macOS runner doesn't support running the Android emulator.

Windows would be nice, but they don't support SDK .artifactbundles yet (@compnerd please correct me if I am wrong, or if is something being worked on). We might be able to cobble together the right build flags or synthesize an old-style destination.json file for Windows, but I don't have enough experience on the platform to assess the level of effort that would require.

Yes, Saleem told us that SDK bundles are not supported by the Windows toolchain yet, but I doubt destination.json files are not, so we could probably unpack the SDK bundle on Windows and use it that way, as the bundle simply comes with its own destination file.

Personally, I'd like to see the validation enabled by this workflow become available sooner rather than later.

I agree that we should get your SDK bundle pull in once it builds 6.2 and 6.3 in a linux Docker and we've tried using it with an external NDK instead. Everything else mentioned above is nice to have, but not necessary IMO to get a first testing version up. Of course, that decision of what to accept for an initial version of the SDK bundle is not up to me, but likely Mishal and maybe a couple others.