SwiftPM doesn't work in the Linux build #713

MaxDesiatov · 2020-04-19T15:30:08Z

SwiftPM works fine in the macOS build of our toolchain, but it fails in the Linux build as can be seen in this WASFoundation CI run:

+ swift build
/github/workspace: error: manifest parse error(s):
<unknown>:0: error: cannot load underlying module for 'Dispatch'

The text was updated successfully, but these errors were encountered:

This is apparently needed for SwiftPM to cross-compile on Linux, which I think causes #713.

MaxDesiatov · 2020-04-24T16:51:30Z

I got some progress here, turns out we don't copy enough .modulemap and header files for Dispatch from the upstream snapshot, which I attempted to fix in #719. Unfortunately, after those are copied, there's a different error:

# swift build -v
lsb_release -r
/usr/bin/swiftc -print-target-info
/usr/bin/swiftc -L /usr/lib/swift/pm/4_2 -lPackageDescription -Xlinker -rpath -Xlinker /usr/lib/swift/pm/4_2 -swift-version 5 -I /usr/lib/swift/pm/4_2 -package-description-version 5.0.0 /Runtime/Package.swift -o /tmp/TemporaryDirectory.QkYZlG/runtime-manifest
/Runtime: error: manifest parse error(s):
/usr/bin/ld.gold: error: cannot open Scrt1.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crti.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtn.o: No such file or directory
/usr/bin/ld.gold: error: cannot find -lpthread
/usr/bin/ld.gold: error: cannot find -lutil
/usr/bin/ld.gold: error: cannot find -ldl
/usr/bin/ld.gold: error: cannot find -lm
/usr/bin/ld.gold: error: cannot find -lc
/tmp/Package-d81280.o:Package-d81280.o:function $s18PackageDescription6TargetC10DependencyOWOb: error: undefined reference to 'memcpy'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)

Now after pasting that swiftc invocation (which I assume was kicked off by the Swift driver) with a -v flag, it produces this:

# /usr/bin/swiftc -v -L /usr/lib/swift/pm/4_2 -lPackageDescription -Xlinker -rpath -Xlinker /usr/lib/swift/pm/4_2 -swift-version 5 -I /usr/lib/swift/pm/4_2 -package-description-version 5.0.0 /Runtime/Package.swift -o /tmp/TemporaryDirectory.QkYZlG/runtime-manifest
Swift version 5.3-dev (LLVM 620270b41e, Swift 8060135cbc)
Target: x86_64-unknown-linux-gnu
/usr/bin/swift -frontend -c -primary-file /Runtime/Package.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -color-diagnostics -I /usr/lib/swift/pm/4_2 -swift-version 5 -package-description-version 5.0.0 -module-name main -o /tmp/Package-3c750c.o
/usr/bin/swift-autolink-extract /tmp/Package-3c750c.o -o /tmp/Package-ab907d.autolink
/usr/bin/clang -target x86_64-unknown-linux-gnu -fuse-ld=gold -pie -Xlinker -rpath -Xlinker /usr/lib/swift/linux /usr/lib/swift/linux/x86_64/swiftrt.o /tmp/Package-3c750c.o @/tmp/Package-ab907d.autolink -L /usr/lib/swift/linux -lswiftCore --target=x86_64-unknown-linux-gnu -v -L /usr/lib/swift/pm/4_2 -lPackageDescription -Xlinker -rpath -Xlinker /usr/lib/swift/pm/4_2 -o /tmp/TemporaryDirectory.QkYZlG/runtime-manifest
clang version 10.0.0 (https://github.com/swiftwasm/llvm-project 9b67bbb5065b3d5619ebeac670c3149ba95a97da)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/5
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/5.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/8
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/usr/bin/ld.gold" --sysroot=/opt/wasi-sdk/share/wasi-sysroot -pie -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /tmp/TemporaryDirectory.QkYZlG/runtime-manifest Scrt1.o crti.o /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/crtbeginS.o -L/usr/lib/swift/linux -L/usr/lib/swift/pm/4_2 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0 -L/opt/wasi-sdk/share/wasi-sysroot/lib -rpath /usr/lib/swift/linux /usr/lib/swift/linux/x86_64/swiftrt.o /tmp/Package-3c750c.o -lswiftCore -lswiftGlibc -lpthread -lutil -ldl -lm -lFoundation -lswiftDispatch -ldispatch -lBlocksRuntime -lswiftSwiftOnoneSupport -lswiftCore -lPackageDescription -rpath /usr/lib/swift/pm/4_2 -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/crtendS.o crtn.o
/usr/bin/ld.gold: error: cannot open Scrt1.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crti.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtn.o: No such file or directory
/usr/bin/ld.gold: error: cannot find -lpthread
/usr/bin/ld.gold: error: cannot find -lutil
/usr/bin/ld.gold: error: cannot find -ldl
/usr/bin/ld.gold: error: cannot find -lm
/usr/bin/ld.gold: error: cannot find -lc
/usr/bin/ld.gold: fatal error: /tmp/TemporaryDirectory.QkYZlG/runtime-manifest: open: No such file or directory
clang: error: linker command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)

The most peculiar bit here is "/usr/bin/ld.gold" --sysroot=/opt/wasi-sdk/share/wasi-sysroot, even though it is being linked for the x86_64 host. It has to be linked for x86_64 even when cross-compiling for WASI, as it needs to "interpret" the package manifest in the host environment to pull dependencies before the cross-compilation process is started.

I've spent a lot of time trying to understand where this --sysroot path comes from, it seems to be hardcoded somewhere, probably embedded when the toolchain is built. I'm still not sure if it's coming from our LLVM fork or the LLVM bits from the wasi-sdk distribution, and that's crucial. We need to understand where it's set to be able to fix it.

kateinoigakukun · 2020-04-24T23:50:29Z

@MaxDesiatov Thanks for update.

I've spent a lot of time trying to understand where this --sysroot path comes from, it seems to be hardcoded somewhere, probably embedded when the toolchain is built. I'm still not sure if it's coming from our LLVM fork or the LLVM bits from the wasi-sdk distribution, and that's crucial. We need to understand where it's set to be able to fix it.

The default sysroot path is hardcoded in https://github.com/swiftwasm/wasi-sdk/blob/swiftwasm/Makefile#L6.
So you need to pass sysroot path explicitly from SwiftPM like when building swiftlang/swift-package-manager@master...swiftwasm:swiftwasm#diff-b1d92ba41a0ed8ddabdf8c2604a41504R828-R839

MaxDesiatov · 2020-04-27T18:38:22Z

This doesn't seem to be an issue with SwiftPM, but an issue with our build of clang. In our Linux build both swift build and swift build --triple wasm32-unknown-wasi are broken.

After I copy clang from the https://swift.org/download nightly toolchain like this

cp /swift-DEVELOPMENT-SNAPSHOT-2020-04-23-a-ubuntu18.04/usr/bin/clang* /usr/bin/

swift build works again, but obviously swift build --triple wasm32-unknown-wasi now fails for a different reason. It's now able to parse the package manifest, but when compiling a module, the upstream clang doesn't know how to handle this triple.

MaxDesiatov · 2020-04-27T18:45:01Z

Or, to paraphrase it, the problem is that our build of clang uses /opt/wasi-sdk/share/wasi-sysroot no matter what triple is passed, which fails when linking the package manifest. While the upstream build of clang uses / as sysroot no matter what triple is passed, which succeeds when linking the package manifest, but obviously fails when compiling for the wasi triple. But yeah, there should be a way to override it, either from the Swift driver or from SwiftPM, investigating it now.

MaxDesiatov · 2020-04-28T11:15:20Z

This is definitely a problem with our llvm-project build and not related to SwiftPM, even a simple file such as this:

#include <stdio.h>

int main() {
	return 0;
}

fails to compile:

# clang test.c -o test
test.c:1:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.

Passing sysroot explicitly doesn't help when invoked this way clang --sysroot=/ test.c -o test. Investigating further...

kateinoigakukun · 2020-04-28T12:50:42Z

@MaxDesiatov clang --sysroot=/ test.c should fail because / is a sysroot for the host environment.
If you want to compile it to wasi-unknown-wasm32, you should pass wasi-sysroot directory as sysroot.

clang test.c -target wasi-unknown-wasm32 --sysroot=/Library/Developer/Toolchains/swift-wasm-DEVELOPMENT-SNAPSHOT-2020-04-27-a.xctoolchain/usr/share/wasi-sysroot

MaxDesiatov · 2020-04-28T12:57:32Z

I understand that, but the problem as I described above is that we need clang working for the host environment to be able to parse Package.swift package manifest, that's why I'm explicitly trying to understand whether compiling for the host environment works at all (and it seems it's broken). swift build calls swiftc which compiles for the host environment, which then delegates to clang with the target explicitly specified as x86_64-unknown-linux-gnu, this invocation can be seen in swiftc -v logs

/usr/bin/clang \
  -target x86_64-unknown-linux-gnu 
  -fuse-ld=gold -pie -Xlinker -rpath -Xlinker /usr/lib/swift/linux \
  /usr/lib/swift/linux/x86_64/swiftrt.o /tmp/Package-3c750c.o \
  @/tmp/Package-ab907d.autolink -L /usr/lib/swift/linux -lswiftCore \
  --target=x86_64-unknown-linux-gnu -v -L /usr/lib/swift/pm/4_2 \
  -lPackageDescription -Xlinker -rpath -Xlinker /usr/lib/swift/pm/4_2 \
  -o /tmp/TemporaryDirectory.QkYZlG/runtime-manifest

MaxDesiatov · 2020-04-28T12:59:52Z

To clarify, swift build --triple wasm32-unknown-wasi does exactly the same thing, because the package manifest needs to be compiled for the host environment first, even when cross-compiling. It seems like our clang always cross-compiles and is not able to compile for the host environment.

kateinoigakukun · 2020-04-28T13:01:03Z

@MaxDesiatov Oh, sorry I misunderstood what you said 🙇
I'll take a look around this issue

MaxDesiatov · 2020-04-28T13:04:45Z

Here's what happens with our clang when you try to build for the host architecture with both target and sysroot specified explicitly:

# clang -v --sysroot=/usr --target=x86_64-unknown-linux-gnu test.c -o test
clang version 10.0.0 (https://github.com/swiftwasm/llvm-project 9b67bbb5065b3d5619ebeac670c3149ba95a97da)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/5
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/5.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/5
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/5.5.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/usr/bin/clang" -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name test.c -mrelocation-model static -mthread-model posix -mframe-pointer=all -fmath-errno -fno-rounding-math -masm-verbose -mconstructor-aliases -munwind-tables -target-cpu x86-64 -dwarf-column-info -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /usr/lib/clang/10.0.0 -isysroot /usr -internal-isystem /usr/usr/local/include -internal-isystem /usr/lib/clang/10.0.0/include -internal-externc-isystem /usr/include -internal-externc-isystem /usr/usr/include -fdebug-compilation-dir /test -ferror-limit 19 -fmessage-length 0 -fgnuc-version=4.2.1 -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -faddrsig -o /tmp/test-f6b9af.o -x c test.c
clang -cc1 version 10.0.0 based upon LLVM 10.0.0git default target wasm32-wasi
ignoring nonexistent directory "/usr/usr/local/include"
ignoring nonexistent directory "/usr/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/clang/10.0.0/include
 /usr/include
End of search list.
In file included from test.c:1:
/usr/include/stdio.h:27:10: fatal error: 'bits/libc-header-start.h' file not found
#include <bits/libc-header-start.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

kateinoigakukun · 2020-04-28T13:33:52Z

@MaxDesiatov I found the reason for this issue.
We use clang built by wasi-sdk but it's only targeting for wasm due to LLVM_TARGETS_TO_BUILD=WebAssembly https://github.com/swiftwasm/wasi-sdk/blob/swiftwasm/Makefile#L26-L27

So we need to build the clang for X64, or don't use the wasi-sdk version clang and copy clang from build/Ninja-ReleaseAssert/x86_64-linux-gnu/bin/clang when packaging toolchain

MaxDesiatov · 2020-04-28T13:36:13Z

Yeah, that's my theory as well 👍
I'm currently verifying it in #719 by not copying clang binaries from wasi-sdk, I assume that won't overwrite our clang binaries from build/Ninja-ReleaseAssert/x86_64-linux-gnu/bin/clang already included in the package. If they're not included, I'll add them.

kateinoigakukun · 2020-04-28T13:42:22Z

lld is not included in the build bin directory, so please add flag -DLLVM_ENABLE_PROJECTS="clang;compiler-rt;clang-tools-extra;lld" for build-script invocation as extra-cmake-options

MaxDesiatov · 2020-04-28T13:44:17Z

Maybe we could copy just lld from wasi-sdk then, but nothing else? I don't see why that wouldn't work. Would save some CI time on not bulding it from scratch. lld is not used for x86_64 builds anyway, as far as I understand.

kateinoigakukun · 2020-04-28T13:58:14Z

Maybe we could copy just lld from wasi-sdk then

Yes, we can copy them from wasi-sdk. It's OK at this time.

But honestly saying, I want to remove wasi-sdk dependency from this project because it's hard to sync llvm version between them.
I know that CI time will be longer than now, but versions mismatch of llvm would be a potential problem.

If we don't need to copy binaries from wasi-sdk, we only need to copy wasi-libc's wasi-sysroot to the toolchain. It's simpler than current build steps.

P.S.
lld is necessary because wasm-ld is a symbolic link to lld. So please copy wasm-ld also https://github.com/swiftwasm/swift/pull/719/files#diff-6ab42cf11ecd931ea357fa8ca0faeb4cL64

MaxDesiatov · 2020-04-28T14:04:21Z

Awesome, thanks! I'll keep the possible LLVM version incompatibility in mind.

MaxDesiatov added bug Something isn't working in progress The issue is currently being worked on labels Apr 19, 2020

MaxDesiatov self-assigned this Apr 19, 2020

MaxDesiatov mentioned this issue Apr 19, 2020

Print SDK path when MissingUnderlyingModule #717

Closed

MaxDesiatov added a commit that referenced this issue Apr 19, 2020

Attempt to build libdispatch on Linux

dbf98c5

This is apparently needed for SwiftPM to cross-compile on Linux, which I think causes #713.

This was referenced Apr 19, 2020

Attempt to build libdispatch on Linux #718

Closed

Copy Dispatch and Clang from upstream toolchain #719

Merged

Build XCTest for the wasm32-unknown-wasi triple #597

Closed

MaxDesiatov removed the in progress The issue is currently being worked on label Apr 22, 2020

MaxDesiatov removed their assignment Apr 22, 2020

MaxDesiatov closed this as completed in #719 May 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SwiftPM doesn't work in the Linux build #713

SwiftPM doesn't work in the Linux build #713

MaxDesiatov commented Apr 19, 2020 •

edited

Loading

MaxDesiatov commented Apr 24, 2020 •

edited

Loading

kateinoigakukun commented Apr 24, 2020

MaxDesiatov commented Apr 27, 2020 •

edited

Loading

MaxDesiatov commented Apr 27, 2020

MaxDesiatov commented Apr 28, 2020

kateinoigakukun commented Apr 28, 2020

MaxDesiatov commented Apr 28, 2020 •

edited

Loading

MaxDesiatov commented Apr 28, 2020 •

edited

Loading

kateinoigakukun commented Apr 28, 2020

MaxDesiatov commented Apr 28, 2020

kateinoigakukun commented Apr 28, 2020 •

edited

Loading

MaxDesiatov commented Apr 28, 2020

kateinoigakukun commented Apr 28, 2020

MaxDesiatov commented Apr 28, 2020

kateinoigakukun commented Apr 28, 2020 •

edited

Loading

MaxDesiatov commented Apr 28, 2020

SwiftPM doesn't work in the Linux build #713

SwiftPM doesn't work in the Linux build #713

Comments

MaxDesiatov commented Apr 19, 2020 • edited Loading

MaxDesiatov commented Apr 24, 2020 • edited Loading

kateinoigakukun commented Apr 24, 2020

MaxDesiatov commented Apr 27, 2020 • edited Loading

MaxDesiatov commented Apr 27, 2020

MaxDesiatov commented Apr 28, 2020

kateinoigakukun commented Apr 28, 2020

MaxDesiatov commented Apr 28, 2020 • edited Loading

MaxDesiatov commented Apr 28, 2020 • edited Loading

kateinoigakukun commented Apr 28, 2020

MaxDesiatov commented Apr 28, 2020

kateinoigakukun commented Apr 28, 2020 • edited Loading

MaxDesiatov commented Apr 28, 2020

kateinoigakukun commented Apr 28, 2020

MaxDesiatov commented Apr 28, 2020

kateinoigakukun commented Apr 28, 2020 • edited Loading

MaxDesiatov commented Apr 28, 2020

MaxDesiatov commented Apr 19, 2020 •

edited

Loading

MaxDesiatov commented Apr 24, 2020 •

edited

Loading

MaxDesiatov commented Apr 27, 2020 •

edited

Loading

MaxDesiatov commented Apr 28, 2020 •

edited

Loading

MaxDesiatov commented Apr 28, 2020 •

edited

Loading

kateinoigakukun commented Apr 28, 2020 •

edited

Loading

kateinoigakukun commented Apr 28, 2020 •

edited

Loading