[SR-648] swift package manager should have the option to produce statically linked binaries

[swift-ci]

Previous ID	SR-648
Radar	rdar://problem/32618121
Original Reporter	jaybuff (JIRA User)
Type	Improvement

Additional Detail from JIRA

Votes	34
Component/s	Package Manager
Labels	Improvement
Assignee	None
Priority	Medium

md5: a97136565bbf5fd9b15515ab552f61b8

is blocked by:

• SR-2280 swiftc -static-stdlib option fails on Ubuntu 14.04 & 15.10

is duplicated by:

• SR-727 (emscripten support) Compile Swift code with statically linked stdlib
• SR-10624 SwiftPM doesn't hand --swift-static-stdlib through

relates to:

• SR-2280 swiftc -static-stdlib option fails on Ubuntu 14.04 & 15.10
• SR-10625 tests for -static-*
• SR-730 Flag to statically link Swift standard library

Issue Description:

When I do swift build for a simple hello world swift program it ldd tells me that it is linked against libstdc++.so.6, libswiftCore.so and other .so files. I would like to ask swift build to produce a single file that is not dynamically linked against any shared objects.

[swift-ci]

Comment by Kostiantyn Koval (JIRA)

can you please provide your build output, and your "Hello world" program. Are you running it on Linux or Mac OS ?
I dont see that "libstdc" is included in my examples (not for exe or lib)

Example:

//Exe target
print("Hi Main")

//Lib target
func a() {
  print("Hi Main")
}

//Exe target 
swift build -vv
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swiftc --driver-mode=swift -I /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/lib/swift/pm -L /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/lib/swift/pm -lPackageDescription -target x86_64-apple-macosx10.10 /Users/konstantin/Work/spmExample/Simple/Package.swift
/usr/bin/xcrun --sdk macosx --show-sdk-path
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swift-build-tool -v -f /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/llbuild.yaml
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swiftc -module-name Simple -incremental -emit-dependencies -emit-module -emit-module-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.swiftmodule -output-file-map /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/output-file-map.json -c /Users/konstantin/Work/spmExample/Simple/main.swift -I /Users/konstantin/Work/spmExample/Simple/.build/debug -j8 -Onone -g -v -target x86_64-apple-macosx10.10 -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I /usr/local/include -D SWIFT_PACKAGE
Apple Swift version 3.0-dev (LLVM f95d47afa7, Clang f66c5bb67b, Swift b745691a38)
Target: x86_64-apple-macosx10.10
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swift -frontend -c -primary-file /Users/konstantin/Work/spmExample/Simple/main.swift -target x86_64-apple-macosx10.10 -enable-objc-interop -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I /Users/konstantin/Work/spmExample/Simple/.build/debug -I /usr/local/include -g -D SWIFT_PACKAGE -emit-module-doc-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/main~partial.swiftdoc -Onone -module-name Simple -emit-module-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/main~partial.swiftmodule -emit-dependencies-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/main.d -emit-reference-dependencies-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/main.swiftdeps -o /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/main.swift.o
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swift -frontend -emit-module /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/main~partial.swiftmodule -parse-as-library -target x86_64-apple-macosx10.10 -enable-objc-interop -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I /Users/konstantin/Work/spmExample/Simple/.build/debug -I /usr/local/include -g -D SWIFT_PACKAGE -emit-module-doc-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.swiftdoc -module-name Simple -o /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.swiftmodule
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swiftc -o /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/main.swift.o -target x86_64-apple-macosx10.10 -Xlinker -all_load -g -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -L/usr/local/lib


//lib target 
swift build -vv
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swiftc --driver-mode=swift -I /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/lib/swift/pm -L /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/lib/swift/pm -lPackageDescription -target x86_64-apple-macosx10.10 /Users/konstantin/Work/spmExample/Simple/Package.swift
/usr/bin/xcrun --sdk macosx --show-sdk-path
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swift-build-tool -v -f /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/llbuild.yaml
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swiftc -module-name Simple -incremental -emit-dependencies -emit-module -emit-module-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.swiftmodule -output-file-map /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/output-file-map.json -parse-as-library -c /Users/konstantin/Work/spmExample/Simple/A.swift -I /Users/konstantin/Work/spmExample/Simple/.build/debug -j8 -Onone -g -v -target x86_64-apple-macosx10.10 -enable-testing -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I /usr/local/include -D SWIFT_PACKAGE
Apple Swift version 3.0-dev (LLVM f95d47afa7, Clang f66c5bb67b, Swift b745691a38)
Target: x86_64-apple-macosx10.10
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swift -frontend -c -primary-file /Users/konstantin/Work/spmExample/Simple/A.swift -target x86_64-apple-macosx10.10 -enable-objc-interop -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I /Users/konstantin/Work/spmExample/Simple/.build/debug -I /usr/local/include -enable-testing -g -D SWIFT_PACKAGE -emit-module-doc-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/A~partial.swiftdoc -Onone -parse-as-library -module-name Simple -emit-module-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/A~partial.swiftmodule -emit-dependencies-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/A.d -emit-reference-dependencies-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/A.swiftdeps -o /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/A.swift.o
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2016-01-25-a.xctoolchain/usr/bin/swift -frontend -emit-module /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/Simple/A~partial.swiftmodule -parse-as-library -target x86_64-apple-macosx10.10 -enable-objc-interop -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I /Users/konstantin/Work/spmExample/Simple/.build/debug -I /usr/local/include -enable-testing -g -D SWIFT_PACKAGE -emit-module-doc-path /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.swiftdoc -module-name Simple -o /Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.swiftmodule
/bin/sh -c "rm -f '/Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.a'; ar cr '/Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.a' '/Users/konstantin/Work/spmExample/Simple/.build/debug/Simple.o/A.swift.o'" 

[swift-ci]

Comment by Kostiantyn Koval (JIRA)

Also I've verified that generated `Simple.a` static library don't have any dynamic loaded dependencies

otool -L Simple.a
Archive : Simple.a
Simple.a(A.swift.o): 

[swift-ci]

Comment by Jay Buffington (JIRA)

This is in Ubuntu 15.10

$ ls -la
total 20
drwxrwxr-x   4 501 dialout  136 Feb  1 19:31 .
drwxr-xr-x 292 501 dialout 9928 Feb  1 19:28 ..
-rw-rw-r--   1 501 dialout    0 Feb  1 19:28 Package.swift
drwxrwxr-x   3 501 dialout  102 Feb  1 19:29 Sources

$ cat Sources/main.swift
print("hello world")

$ swift build
Compiling Swift Module 'swiftstatic' (1 sources)
Linking Executable:  .build/debug/swift-static

$ ldd  .build/debug/swift-static
    linux-vdso.so.1 =>  (0x00007fff3fb40000)
    libswiftCore.so => /home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/lib/swift/linux/libswiftCore.so (0x00007fa3b782e000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fa3b74a2000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa3b719a000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fa3b6f83000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa3b6bb8000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa3b699a000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa3b6796000)
    libicuuc.so.55 => /usr/lib/x86_64-linux-gnu/libicuuc.so.55 (0x00007fa3b6401000)
    libicui18n.so.55 => /usr/lib/x86_64-linux-gnu/libicui18n.so.55 (0x00007fa3b5f9e000)
    libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007fa3b5d8e000)
    /lib64/ld-linux-x86-64.so.2 (0x000055e1b014c000)
    libicudata.so.55 => /usr/lib/x86_64-linux-gnu/libicudata.so.55 (0x00007fa3b42d6000)

[swift-ci]

Comment by Kostiantyn Koval (JIRA)

Here is the output I get for the MacOS. It includes 3 dynamic libraries.

otool -L .build/debug/Simple
.build/debug/Simple:
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
    @rpath/libswiftCore.dylib (compatibility version 0.0.0, current version 0.0.0)

Also the package manager doesn't control this behaviour. I think it's more related to the LLBD (the build system).

[mxcl]

Discussion of this on the lists suggests this isn't yet possible, so we need to open a bug on Swift itself so that it gains the option to link statically on Linux.

[ddunbar]

This should be unblocked, it would be great to get this in the Swift 3.0 timeframe.

[abertelrud]

I'm checking to see if this is working now (this issue hasn't been updated for a while now). If it works, great, but if not, it seems to me that we are out of time for Swift 3 for this.

[abertelrud]

When I try `swift build -Xswiftc -static-stdlib` on Ubuntu 14.0.4, I get this:

/usr/bin/ld: cannot find -lbsd
/usr/bin/ld: cannot find -licui18n
/usr/bin/ld: cannot find -licuuc
clang: error: linker command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: build had 1 command failures
error: exit(1): /home/user/Documents/usr/bin/swift-build-tool -f /home/user/Documents/SR648/.build/debug.yaml -v
user@ubuntu:~/Documents/SR648$ gedit /home/user/Documents/SR648/.build/debug.yaml

[abertelrud]

It isn't that clear to me where this comes from. SwiftPM doesn't seem to be passing it down.

[abertelrud]

On macOS it works, so this might be a toolchain incompatibility with Ubuntu 14.0.4. I will try on Ubuntu 15.10 and file a JIRA if it's incompatible.

[abertelrud]

I can reproduce this in a small test case, and I have filed SR-2280.

[abertelrud]

I am removing the swift-3.0 label from this, since it is blocked by SR-2280.

[spevans]

Should a static executable option be added to swiftc or the swift-build-tool? I think an `-emit-static-executable' or '-emit-executable -static' option added to swiftc might be the best approach rather than the package manager so it can be used directly

I have a solution for this for Linux/ELF which uses a small stub to fix up the linker issues which I can turn into a PR

static_stub.c

#if defined(__LINUX__) && defined(__ELF__)
#error "This only works on Linux/ELF"  // Needs testing on other ELF platforms
#else
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>

#define __USE_GNU
#include <link.h>

// This forces resolving of these weak symbols but keeps them hidden
// externally
const void *unused1 __attribute__ ((unused, visibility("internal"))) = pthread_self;
const void *unused2 __attribute__ ((unused, visibility("internal"))) = pthread_key_create;
const void *unused3 __attribute__ ((unused, visibility("internal"))) = pthread_once;


// libdl emulation to load the protocol and metadata sections, these get fixed
// using the --defsym linker commands below
extern void *__swift2_protocol_conformances_start;
extern void *__swift2_type_metadata_start;


// Fake a handle if opening NULL (this binary) else abort
void *
dlopen(const char *filename, int flag)
{
    if (filename != NULL) {
        fprintf(stderr, "dlopen called with filename=%s\n", filename);
        abort();
    }
    return (void *)1;
}


// Hardcoded to allow lookup of 2 known symbols, enough for now
void *
dlsym(void *handle, const char *symbol)
{
    if (handle != (void *)1) {
        fprintf(stderr, "dlsym(): bad handle: %p\n", handle);
        abort();
    }
    if (!strcmp(symbol, ".swift2_protocol_conformances_start")) {
        return &__swift2_protocol_conformances_start;
    } else if (!strcmp(symbol, ".swift2_type_metadata_start")) {
        return &__swift2_type_metadata_start;
    } else {
        return NULL;
    }
}


int
dladdr(const void *addr, Dl_info *info)
{
    // FIXME: Dummy implementation for now, returns `not found'
    fprintf(stderr, "dladdr() required for %p\n", addr);

    // info declared as nonnull in dlfcn.h
    info->dli_fname = NULL;
    info->dli_fbase = NULL;
    info->dli_sname = NULL;
    info->dli_saddr = NULL;
    
    return 0;
}


int
dlclose(void *handle)
{
    if (handle != (void *)1) {
        fprintf(stderr, "dlclose(): bad handle: %p", handle);
    }
    return 0;
}


// Hardcoded to return an empty structure, the caller of this only
// cares about the filename anyway and can deal with it being NULL
int
dl_iterate_phdr(int (*callback) (struct dl_phdr_info *info,
                                 size_t size, void *data), void *data)
{
    // Dummy empty structure for dl_iterate_phdr
    struct dl_phdr_info phdr_info = { .dlpi_addr = 0 };

    return callback(&phdr_info, sizeof(struct dl_phdr_info), data);
}


// These commands will get extracted by `swift-autolink-extract'
// and passed to clang++
#define AUTOLINK_ENTRY(x) asm(".asciz \"" x "\"")

asm(".section .swift1_autolink_entries");
AUTOLINK_ENTRY("-static");
AUTOLINK_ENTRY("-Xlinker --defsym=__swift2_protocol_conformances_start=.swift2_protocol_conformances_start");
AUTOLINK_ENTRY("-Xlinker --defsym=__swift2_type_metadata_start=.swift2_type_metadata_start");
AUTOLINK_ENTRY("-licui18n");
AUTOLINK_ENTRY("-licuuc");
AUTOLINK_ENTRY("-licudata");
AUTOLINK_ENTRY("-lpthread");

#endif // linux && ELF

$ clang -O -Wall -Werror -c static_stub.c -o static_stub.o
$ swiftc -static-stdlib -emit-executable -o main main.swift static_stub.o
$ file main
main: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=41e9e4c815e695b8369a03239db50e06d4f00085, not stripped
$ ldd main
    not a dynamic executable
$ ./main
Hello there

This method takes advantage of the auto link section so the static_stub can provide its own linker commands
$ swift-autolink-extract static_stub.o 
-static
-Xlinker --defsym=__swift2_protocol_conformances_start=.swift2_protocol_conformances_start
-Xlinker --defsym=__swift2_type_metadata_start=.swift2_type_metadata_start
-licui18n
-licuuc
-licudata
-lpthread

I have a slightly different solution that patches swift/stdlib which reduces the size of the libdl emulation code and makes it simpler to understand which might be a better approach if its decided to add the option to swiftc rather than swiftpm

[spevans]

Proposed fix https:// github.com/swiftlang/swift/pull/4754

Updated fix that doesnt use any libdl hacks but instead builds its own version of libicu:
swiftlang/swift#5394

[ddunbar]

Is this done now?