Build a static library version of xctest #190

spevans · 2017-05-22T15:17:07Z

libXCTest.a can optionally be installed with
--static-library-install-path

This is required for working towards getting Foundation to be statically linkable on Linux

parkera · 2017-05-22T17:51:36Z

I'm not sure a 100% statically linked Foundation is possible or desirable on Linux. We'll always have ICU as an external dependency, for example.

aciidgh · 2017-05-23T09:02:54Z

We definitely want an option to statically link all the libraries present in the toolchain. This will, for e.g., allow server side Swift developers to create a portable binary. True that there will be some external dependencies like ICU and libcurl but installing them is much easier than installing the exact toolchain a binary was built with. Note that it is already possible on macOS to create portable Swift binaries using the package manager.

Relevant SwiftPM bugs:
https://bugs.swift.org/browse/SR-648
https://bugs.swift.org/browse/SR-2280

briancroom · 2017-05-23T19:41:13Z

@swift-ci please test

briancroom · 2017-05-23T19:48:55Z

Hey @spevans, I don't see any immediate issues with this patch, but I'd like to better understand how this fits into the static linking effort. How does a static Corelibs-XCTest build assist with producing a static Corelibs-Foundation?

Could we get some additional bugs filed to help document the roadmap?

spevans · 2017-05-25T12:33:00Z

Originally I was working on fully static binaries, which I did get working but has some downsides in requiring libicu and libcurl to be recompiled.

After talking with @aciidb0mb3r we decided that making the -static-stdlib option work fully on Linux would be the best solution and because this is creating a dynamic binary but linking in static libs we need to produce .a files for each swift library. When linking with -static-stdlib, swiftc points the linker at lib/swift_static/linux instead of lib/swift/linux so if libXCTest needs to be linked in then it needs a file in that directory.

So the steps to get this all working are:

For each of libdispatch, foundation, xctest produce a .a library file (done)
During the build process, make sure these .a files go in a different build dir to the .so files so that linking the TestFoundationStatic in 3) uses the correct libs (not done)
Add a test_static target to corelibs-foundation that tests steps 1 & 2 and also validates the libs can be linked together which includes libXCTest.a. (done on my machine will open PR after 2 )
This will also act as a test during build time.
Work out how to use swift-autolink-extract and some -Xlinker args to somehow pass the extra linker arguments that libFoundation.a needs when building packages using swiftpm (not done yet - will probably needs some discussion on swift-dev)

Extra fixes

Find a better solution so we can remove the -defsym=__CFConstantStringClassReference=... linker argument (done, needs a PR)
Add a proper solution to eliminate the need for objc_retainAutoreleasedReturnValue in foundation and dispatch which means that the --allow-multiple-definition linker argument can be removed.

So Im currently working on part 2). For now, dont merge this PR as I will add another commit to put the lib in a different build dir. And the test failure on macOS is due to corelibs-foundation needing some swift4 fixes but they are also being worked on.

The current status I have it at is:

$ ~/swift-test/usr/bin/swift build                                                                                                                                                                                  
Compile Swift Module 'StaticExe' (1 sources)
Linking ./.build/debug/StaticExe
$ ldd ./.build/debug/StaticExe
        linux-vdso.so.1 =>  (0x00007ffcfd519000)
        libswiftCore.so => /home/spse/swift-test/usr/lib/swift/linux/libswiftCore.so (0x00007fbac7025000)
        libswiftSwiftOnoneSupport.so => /home/spse/swift-test/usr/lib/swift/linux/libswiftSwiftOnoneSupport.so (0x00007fbac6fb9000)
        libFoundation.so => /home/spse/swift-test/usr/lib/swift/linux/libFoundation.so (0x00007fbac649d000)
        libswiftGlibc.so => /home/spse/swift-test/usr/lib/swift/linux/libswiftGlibc.so (0x00007fbac6497000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbac6265000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fbac6061000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fbac5e5d000)
        libdispatch.so => /home/spse/swift-test/usr/lib/swift/linux/libdispatch.so (0x00007fbac5dc3000)
        libXCTest.so => /home/spse/swift-test/usr/lib/swift/linux/libXCTest.so (0x00007fbac5d78000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fbac59f6000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fbac56ed000)
[more dynamic libs...]

v

$ ~/swift-test/usr/bin/swift build  -Xswiftc -static-stdlib -Xlinker -lcurl -Xlinker -lxml2 -Xlinker -lbsd -Xlinker   --allow-multiple-definition                                                   Compile Swift Module 'StaticExe' (1 sources)
Linking ./.build/debug/StaticExe
$ ldd ./.build/debug/StaticExe                                                                                                                                                                      
        linux-vdso.so.1 =>  (0x00007ffdccd84000)
        libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f2582841000)
        libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f2582486000)
        libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f2582270000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2582053000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f2581e50000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2581c4b000)
        libatomic.so.1 => /usr/lib/x86_64-linux-gnu/libatomic.so.1 (0x00007f2581a43000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f25816c1000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f25813b7000)
[more dynamic libs but none of them are swift...]

spevans · 2017-05-30T23:18:52Z

@briancroom Does the above answer your questions? This PR is required for swiftlang/swift#9958 which should satisfy steps 1 & 2 above

briancroom · 2017-05-31T03:50:32Z

build_script.py

@@ -206,6 +206,11 @@ def build(args):
                build_dir=build_dir,
                foundation_build_dir=foundation_build_dir))

+        # Build the static library
+        run("mkdir -p {build_dir}/static".format(build_dir=build_dir))


It looks like there are three locations in this patch that are building {build_dir}/static path. It seems worthwhile to extract a helper for producing this path, similar to the existing core_foundation_build_dir method.

briancroom · 2017-05-31T03:59:59Z

Hey @spevans, sorry for the delay! Things are pretty busy around here right now :)

Thanks for the writeup with your progress! This looks like an intriguing effort. I'd recommend putting what you wrote into one or more issues on bugs.swift.org to make it easier to track progress as you push things forward.

This patch looks good to me, though I left one minor suggestion.

@parkera do you have anything more to add here?

briancroom · 2017-05-31T04:00:02Z

@swift-ci please test

libXCTest.a can optionally be installed with --static-library-install-path - Put the static libXCTest.a into a static/ subdir when building - This simplifies linking in static libraries over shared libraries when using -static-stdlib during build testing.

spevans · 2017-05-31T16:01:31Z

@briancroom I updated https://bugs.swift.org/browse/SR-648 with the current status and added a static_lib_build_dir() method - and then did a rebase and commit squash.

briancroom · 2017-05-31T18:11:47Z

@swift-ci please test

briancroom · 2017-06-06T08:20:41Z

@swift-ci please test

spevans · 2017-06-06T15:14:25Z

@briancroom the OSX tests failed due to corelibs-foundation not compiling - this is an unrelated issue but there are some PRs waiting to be merged that fix the issues

spevans · 2017-06-10T15:39:14Z

@parkera Could you rerun the test on this PR please, I think the Foundation Xcode fixes should make it work now.

parkera · 2017-06-10T15:39:47Z

Sure

parkera · 2017-06-10T15:39:52Z

@swift-ci please test

parkera · 2017-06-10T18:26:27Z

  "__T015SwiftFoundation21__CFSwiftGetBaseClasss9AnyObject_pXpyF", referenced from:
      ___CFInitialize in libCoreFoundation.a(CFRuntime.o)
ld: symbol(s) not found for architecture x86_64

I think we're still trying to figure out where and when the mangling changes take effect...

parkera · 2017-06-10T18:27:36Z

@swift-ci please test

spevans · 2017-06-13T15:20:02Z

@modocache Could you rerun the tests on this PR please, hopefully the recent fixes should help

modocache · 2017-06-13T15:28:25Z

@swift-ci please test

briancroom · 2017-06-13T18:13:58Z

Great! Let's get this landed.

spevans mentioned this pull request May 27, 2017

SR-648: Allow swiftpm to statically link binaries on Linux swiftlang/swift#9958

Merged

briancroom reviewed May 31, 2017

View reviewed changes

spevans force-pushed the pr_static_xctest branch from 6c92851 to 192a7ec Compare May 31, 2017 14:50

briancroom merged commit 860d868 into swiftlang:master Jun 13, 2017

spevans mentioned this pull request May 31, 2017

[SR-648] swift package manager should have the option to produce statically linked binaries swiftlang/swift-package-manager#5432

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build a static library version of xctest #190

Build a static library version of xctest #190

spevans commented May 22, 2017

parkera commented May 22, 2017

aciidgh commented May 23, 2017

briancroom commented May 23, 2017

briancroom commented May 23, 2017

spevans commented May 25, 2017

spevans commented May 30, 2017

briancroom May 31, 2017

briancroom commented May 31, 2017

briancroom commented May 31, 2017

spevans commented May 31, 2017

briancroom commented May 31, 2017

briancroom commented Jun 6, 2017

spevans commented Jun 6, 2017

spevans commented Jun 10, 2017

parkera commented Jun 10, 2017

parkera commented Jun 10, 2017

parkera commented Jun 10, 2017

parkera commented Jun 10, 2017

spevans commented Jun 13, 2017

modocache commented Jun 13, 2017

briancroom commented Jun 13, 2017

Build a static library version of xctest #190

Build a static library version of xctest #190

Conversation

spevans commented May 22, 2017

parkera commented May 22, 2017

aciidgh commented May 23, 2017

briancroom commented May 23, 2017

briancroom commented May 23, 2017

spevans commented May 25, 2017

spevans commented May 30, 2017

briancroom May 31, 2017

Choose a reason for hiding this comment

briancroom commented May 31, 2017

briancroom commented May 31, 2017

spevans commented May 31, 2017

briancroom commented May 31, 2017

briancroom commented Jun 6, 2017

spevans commented Jun 6, 2017

spevans commented Jun 10, 2017

parkera commented Jun 10, 2017

parkera commented Jun 10, 2017

parkera commented Jun 10, 2017

parkera commented Jun 10, 2017

spevans commented Jun 13, 2017

modocache commented Jun 13, 2017

briancroom commented Jun 13, 2017