Support building for target catalyst

[dpwiese]

As of macOS 10.15 target catalyst can be used to build an iPad app to run on macOS. Existing .Framework built with gomobile produces the following error when targetting catalyst:

error: Building for Mac Catalyst, but the linked framework 'Sample.framework' was built for iOS + iOS Simulator.
You may need to restrict the platforms for which this framework should be linked in the target editor, or replace it with an XCFramework that supports both platforms. (in target 'MySampleApp' from project 'MySampleApp')

This PR adds a case for catalyst (although architecture is just amd64) when configuring the environment for each architecture, providing the flags needed to enable the built .Framework to be used with catalyst.

Fixes golang/go#36856

Update 18-Jan-2021

This PR has been updated to provide nominally working support of .xcframework generation with go1.15, which has removed support for some 32 bit architectures. See: dpwiese#1 (comment). In addition, unlike dpwiese#1 this PR doesn't require manual thinning or combining of the generated .frameworks for each of the various "architectures" - this is now done automatically.

Todo
Running go test fails with the following.

bind_test.go:321: gomobile bind failed: exit status 1
    /var/folders/rx/h40bjl5n0s1fvr5h2w278tm80000gn/T/gomobile-test335293790/gomobile: darwin-catalyst: go build -tags ios -buildmode=c-archive -o /var/folders/rx/h40bjl5n0s1fvr5h2w278tm80000gn/T/gomobile-work-224727912/Cgopkg-amd64.a ./gobind failed: exit status 1
    go: cannot determine module path for source directory /private/var/folders/rx/h40bjl5n0s1fvr5h2w278tm80000gn/T/gomobile-work-224727912/src (outside GOPATH, module path must be specified)

Note: on current master branch TestIOSBuild fails

[gopherbot]

This PR (HEAD: a7f75e2) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/mobile/+/226817 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

[gopherbot]

Message from Gobot Gobot:

Patch Set 1:

Congratulations on opening your first change. Thank you for your contribution!

Next steps:
Within the next week or so, a maintainer will review your change and provide
feedback. See https://golang.org/doc/contribute.html#review for more info and
tips to get your patch through code review.

Most changes in the Go project go through a few rounds of revision. This can be
surprising to people new to the project. The careful, iterative review process
is our way of helping mentor contributors and ensuring that their contributions
have a lasting impact.

During May-July and Nov-Jan the Go project is in a code freeze, during which
little code gets reviewed or merged. If a reviewer responds with a comment like
R=go1.11, it means that this CL will be reviewed as part of the next development
cycle. See https://golang.org/s/release for more details.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/226817.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Hyang-Ah Hana Kim:

Patch Set 1: Run-TryBot+1

(4 comments)

How do you expect users to specify the 'catalyst' as the target os? (-target flag?)

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/226817.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Gobot Gobot:

Patch Set 1:

TryBots beginning. Status page: https://farmer.golang.org/try?commit=155879fb

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/226817.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Gobot Gobot:

Patch Set 1: TryBot-Result+1

TryBots are happy.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/226817.
After addressing review feedback, remember to publish your drafts!

[dpwiese]

After having opened this (draft) PR implementing the suggestion in golang/go#36856, I realized I was quite naive and the solution will be more involved than what I'd previously and optimistically hoped. I managed to spend a few moments looking back at this, and here's what I've learned.

Each of the different architectures is used to build a corresponding static library which is ultimately combined with lipo and placed in the .framework output, as below:

lipo -create -arch armv7 sample-arm.a -arch arm64 sample-arm64.a -arch i386 sample-386.a -arch x86_64 sample-amd64.a -arch x86_64 sample-amd64.a -o Sample.framework/Versions/A/Sample

It is simple enough to create a separate amd64 static archive with the flags needed for catalyst, but multiple archives of the same architecture cannot be combined with lipo. So I started looking at creating two separate amd64 static archives, each with the flags needed for the iOS simulator and catalyst, and then combining them with libtool or ar. My first attempt resulted in the following error when attempting to use the generated .framework:

In ios/Sample.framework/Sample(000006.o), building for Mac Catalyst, but linking in object file built for iOS Simulator, file 'ios/Sample.framework/Sample' for architecture x86_64

I think this approach should work, and I probably just need to spend some time learning libtool/ar better, as I may not be invoking them correctly to combine the static archives. I hope to look back at this again soon.

[aki-xavier]

After having opened this (draft) PR implementing the suggestion in golang/go#36856, I realized I was quite naive and the solution will be more involved than what I'd previously and optimistically hoped. I managed to spend a few moments looking back at this, and here's what I've learned.

Each of the different architectures is used to build a corresponding static library which is ultimately combined with lipo and placed in the .framework output, as below:

lipo -create -arch armv7 sample-arm.a -arch arm64 sample-arm64.a -arch i386 sample-386.a -arch x86_64 sample-amd64.a -arch x86_64 sample-amd64.a -o Sample.framework/Versions/A/Sample

It is simple enough to create a separate amd64 static archive with the flags needed for catalyst, but multiple archives of the same architecture cannot be combined with lipo. So I started looking at creating two separate amd64 static archives, each with the flags needed for the iOS simulator and catalyst, and then combining them with libtool or ar. My first attempt resulted in the following error when attempting to use the generated .framework:

In ios/Sample.framework/Sample(000006.o), building for Mac Catalyst, but linking in object file built for iOS Simulator, file 'ios/Sample.framework/Sample' for architecture x86_64

I think this approach should work, and I probably just need to spend some time learning libtool/ar better, as I may not be invoking them correctly to combine the static archives. I hope to look back at this again soon.

I think Apple recently introduced .xcframework to solve exactly this packaging problem.

[dpwiese]

I think Apple recently introduced .xcframework to solve exactly this packaging problem.

I looked into this quickly last night and it does seem to be a promising, and probably the best, solution forwards. I was able to quickly generate a working .xcframework by creating a separate .frameworks for each architecture with gomobile bind, and then combining the architectures with xcodebuild -create-xcframework.

This was not a smooth process though, with the first attempt giving:

error: unable to find any architecture information in the binary at 'Sample.framework/Sample'

This was easily solved by converting Sample to a non-FAT binary with lipo -thin. Then, attempting to combine all the various architectures with xcodebuild -create-xcframework gave:

Both ios-armv7 and ios-arm64 represent two equivalent library definitions.
Both ios-i386-simulator and ios-x86_64-simulator represent two equivalent library definitions.

so I need to better understand what is going on and change how the .frameworks are being generated and combined.
A cursory search didn't turn up much, but I did notice in https://github.com/firebase/firebase-ios-sdk/pull/4737/files for example:

xcframework doesn't support legacy architectures: armv7, i386.

But I haven't yet verified that this is (or is not) truly the case. Regardless, using xcodebuild -create-xcframework has resulted in a satisfactory solution for the moment (at least for the platforms I wish to support), but more care will be needed to figure out the best way to update the mobile tools. I'll try to look at this more when I have time.

[constantinevassil]

See: golang/go#36856

As of macOS 10.15 target catalyst can be used to build an iPad app to run on macOS. Existing .Framework built with gomobile produces the following error when targetting catalyst:

error: Building for Mac Catalyst, but the linked framework 'Sample.framework' was built for iOS + iOS Simulator.
You may need to restrict the platforms for which this framework should be linked in the target editor, or replace it with an XCFramework that supports both platforms. (in target 'MySampleApp' from project 'MySampleApp')

This PR adds a case for catalyst (although architecture is just amd64) when configuring the environment for each architecture, providing the flags needed to enable the built .Framework to be used with catalyst.

Fixes golang/go#36856

I implemented the commit locally:
a7f75e2

How to invoke it?
This is not working:
gomobile bind -target=iOS/catalyst

[dpwiese]

What is in a7f75e2 will not work. See the discussion above: #45 (comment). If you need to get a solution you can see: dpwiese#1 - it's WIP but should get the job done. It will allow you to generate a separate .framework for each architecture and combine them with xcodebuild -create-xcframework resulting in an .xcframework that works on iOS, simulator, and catalyst.

[thstart]

What is in a7f75e2 will not work. See the discussion above: #45 (comment). If you need to get a solution you can see: dpwiese#1 - it's WIP but should get the job done. It will allow you to generate a separate .framework for each architecture and combine them with xcodebuild -create-xcframework resulting in an .xcframework that works on iOS, simulator, and catalyst.

Actually it worked.
Here is what I did:

1. manually applied the patch
   a7f75e2
   to env.go in /Users//go/src/golang.org/x/mobile/cmd/gomobile

2. go get /Users//go/src/golang.org/x/mobile/cmd/gomobile

3. /Users//go/bin/gomobile init

4. /Users//go/bin/gomobile bind -target=ios -v Gomobile
   It build all possible targets.

Then deleted the Xcode cache, checked the macOS box and build the macOS target.
Then ran the code in the macBook. The UI showed up, etc. Now I have to deal with file
permissions - I have defaults, files written in Documents folder etc. This is different than iOS
and need to see how to make it work.

[dpwiese]

@thstart - it "works" in that you'll get a build for architecture amd64 that will work on catalyst, but with the few changes in a7f75e2 it simply overwrites the existing amd64 build with the flags that were for the iOS simulator. It is discussed above: #45 (comment)

If sacrificing iOS simulator support is an acceptable compromise for you to be able to support Catalyst, then you'll be just fine. But that was not at all the intention of this, and a problem that is nominally solved in dpwiese#1 with .xcframework.

[thstart]

@thstart - it "works" in that you'll get a build for architecture amd64 that will work on catalyst, but with the few changes in a7f75e2 it simply overwrites the existing amd64 build with the flags that were for the iOS simulator. It is discussed above: #45 (comment)

If sacrificing iOS simulator support is an acceptable compromise for you to be able to support Catalyst, then you'll be just fine. But that was not at all the intention of this, and a problem that is nominally solved in dpwiese#1 with .xcframework.

I'm testing only on physical devices so simulator doesn't matter. I will look at .xcframework solution, but don't see steps in what to do.

But after testing it on a Mac - the performance is very slow. Are there a way optimize it?

[thstart]

@thstart - it "works" in that you'll get a build for architecture amd64 that will work on catalyst, but with the few changes in a7f75e2 it simply overwrites the existing amd64 build with the flags that were for the iOS simulator. It is discussed above: #45 (comment)

If sacrificing iOS simulator support is an acceptable compromise for you to be able to support Catalyst, then you'll be just fine. But that was not at all the intention of this, and a problem that is nominally solved in dpwiese#1 with .xcframework.

Any update on this feature?

[dpwiese]

@thstart - I've done no work on this since what has been in dpwiese#1 and the comment dpwiese#1 (comment) which I hope is helpful. I don't have any insight at the moment regarding performance on Mac.

[gopherbot]

Message from Go Bot:

Patch Set 1:

Congratulations on opening your first change. Thank you for your contribution!

Next steps:
Within the next week or so, a maintainer will review your change and provide
feedback. See https://golang.org/doc/contribute.html#review for more info and
tips to get your patch through code review.

Most changes in the Go project go through a few rounds of revision. This can be
surprising to people new to the project. The careful, iterative review process
is our way of helping mentor contributors and ensuring that their contributions
have a lasting impact.

During May-July and Nov-Jan the Go project is in a code freeze, during which
little code gets reviewed or merged. If a reviewer responds with a comment like
R=go1.11, it means that this CL will be reviewed as part of the next development
cycle. See https://golang.org/s/release for more details.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/226817.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Go Bot:

Patch Set 1:

TryBots beginning. Status page: https://farmer.golang.org/try?commit=155879fb

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/226817.
After addressing review feedback, remember to publish your drafts!

[gopherbot]

Message from Go Bot:

Patch Set 1: TryBot-Result+1

TryBots are happy.

---

Please don’t reply on this GitHub thread. Visit golang.org/cl/226817.
After addressing review feedback, remember to publish your drafts!

[imWildCat]

Anything I can help to move this one forward?

[dpwiese]

@imWildCat - I think the main effort needed here is better defining the spec. Is it acceptable to drop support for .frameworks and armv7 and i386 architectures and generate only .xcframework as in the tentative solution in dpwiese#1 that I've been using? Or should the ability to generate .xcframeworks be optional, with current generation of .framework untouched? Need support for Apple Silicon be considered? Basically I think if there were clarity on what needs to be done the implementation would be quite straightforward.

[imWildCat]

@dpwiese thanks for your reply!

I think generating xcframeork is the only way to do, especially for Xcode 12.3 or newer version:

https://developer.apple.com/forums/thread/669411

Fat frameworks are no longer supported.

[dpwiese]

@imWildCat great, and thanks for the helpful link. I'll try to have a look at this on the weekend.

[dpwiese]

@imWildCat I updated this in dpwiese#2 and it is working again, now with go1.15 which has dropped support for some 32-bit architectures. I've got to fix the tests so when I'm able to do that, I'll open a PR into upstream and have updated this PR with my latest changes.

[gopherbot]

This PR (HEAD: 6cad7ff) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/mobile/+/226817 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

[jerson]

I think Apple recently introduced .xcframework to solve exactly this packaging problem.

I looked into this quickly last night and it does seem to be a promising, and probably the best, solution forwards. I was able to quickly generate a working .xcframework by creating a separate .frameworks for each architecture with gomobile bind, and then combining the architectures with xcodebuild -create-xcframework.

This was not a smooth process though, with the first attempt giving:

error: unable to find any architecture information in the binary at 'Sample.framework/Sample'

This was easily solved by converting Sample to a non-FAT binary with lipo -thin. Then, attempting to combine all the various architectures with xcodebuild -create-xcframework gave:

Both ios-armv7 and ios-arm64 represent two equivalent library definitions.
Both ios-i386-simulator and ios-x86_64-simulator represent two equivalent library definitions.

so I need to better understand what is going on and change how the .frameworks are being generated and combined.
A cursory search didn't turn up much, but I did notice in https://github.com/firebase/firebase-ios-sdk/pull/4737/files for example:

xcframework doesn't support legacy architectures: armv7, i386.

But I haven't yet verified that this is (or is not) truly the case. Regardless, using xcodebuild -create-xcframework has resulted in a satisfactory solution for the moment (at least for the platforms I wish to support), but more care will be needed to figure out the best way to update the mobile tools. I'll try to look at this more when I have time.

in addition to this you can create a valid xcframework doing this

mkdir arm
gomobile bind -ldflags="-w -s" -target=ios/arm64,ios/arm -o ./arm/Openpgp.framework github.com/jerson/openpgp-mobile/openpgp

mkdir amd64
gomobile bind -ldflags="-w -s" -target=ios/amd64 -o ./amd64/Openpgp.framework github.com/jerson/openpgp-mobile/openpgp

xcodebuild -create-xcframework \
	-framework ./arm/Openpgp.framework \
	-framework ./amd64/Openpgp.framework \
	-output Openpgp.xcframework

the final xcframework should looks like this

a ./Openpgp.xcframework
a ./Openpgp.xcframework/ios-x86_64-simulator
a ./Openpgp.xcframework/Info.plist
a ./Openpgp.xcframework/ios-arm64_armv7
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Openpgp
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Resources
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Versions
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Headers
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Modules
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Versions/A
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Versions/Current
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Versions/A/Openpgp
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Versions/A/Resources
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Versions/A/Headers
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Versions/A/Modules
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Versions/A/Modules/module.modulemap
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Versions/A/Headers/Openpgp.objc.h
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Versions/A/Headers/ref.h
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Versions/A/Headers/Openpgp.h
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Versions/A/Headers/Universe.objc.h
a ./Openpgp.xcframework/ios-arm64_armv7/Openpgp.framework/Versions/A/Resources/Info.plist
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Openpgp
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Resources
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Versions
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Headers
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Modules
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Versions/A
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Versions/Current
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Versions/A/Openpgp
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Versions/A/Resources
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Versions/A/Headers
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Versions/A/Modules
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Versions/A/Modules/module.modulemap
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Versions/A/Headers/Openpgp.objc.h
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Versions/A/Headers/ref.h
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Versions/A/Headers/Openpgp.h
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Versions/A/Headers/Universe.objc.h
a ./Openpgp.xcframework/ios-x86_64-simulator/Openpgp.framework/Versions/A/Resources/Info.plist

I'm using go 1.14.13
and gomobile: golang.org/x/mobile/cmd/gomobile@33b80540585f2b31e503da24d6b2a02de3c53ff5

[dpwiese]

Closing. It seems #65 represents the current evolution of this work.