Add support for Amazon Linux on x86_64

[MaxDesiatov]

I hope we can use Amazon Linux Docker image on CI. The only concern is whether it will fit wihin disk size constraints of GitHub Actions agents.

[hassan-shahbazi]

I spent the last few hours to work on this issue, but wasmer looks broken on the latest version of Amazon Linux 2. wasmerio/wasmer#1970.
Also, I think the support for new targets should be added to wasi-sdk as well.

[MaxDesiatov]

Thanks for the investigation, much appreciated! I'm mostly busy with the Apple Silicon build recently, but will be able to focus on this one after Apple Silicon build is ready.

[hassan-shahbazi]

Thanks Max. I try to continue working on it as soon as I get some clues about that issue with wasmer.

[kateinoigakukun]

Is Wasmer required only for running test suites? I think it's ok to skip test suites on amazonlinux2 at the moment 👍

[hassan-shahbazi]

Skipping wasmer and tests was a good idea, but building still fails. From what I can see, the script build-foundation.sh and particularly libdispatch are the problem.

-- Build files have been written to: /host-build/Ninja-Release/libdispatch-linux-x86_64
+ popd
/swift
+ ///host-build/cmake-linux-x86_64/bin/cmake --build ///host-build/Ninja-Release/libdispatch-linux-x86_64 -- -j12 all
[57/82][ 69%][2.685s] Linking Swift shared library libswiftDispatch.so
FAILED: libswiftDispatch.so src/swift/CMakeFiles/swiftDispatch.dir/Block.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Data.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Dispatch.swift.o src/swift/CMakeFiles/swiftDispatch.dir/IO.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Private.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Queue.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Source.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Time.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Wrapper.swift.o src/swift/swift/Dispatch.swiftmodule 
: && //host-build/Ninja-Release/swift-linux-x86_64/bin/swiftc -output-file-map src/swift/CMakeFiles/swiftDispatch.dir/output-file-map.json -incremental -j 12 -emit-library -o libswiftDispatch.so -module-name Dispatch -module-link-name swiftDispatch -emit-module -emit-module-path src/swift/swift/Dispatch.swiftmodule -emit-dependencies -DswiftDispatch_EXPORTS -module-cache-path "///host-build/Ninja-Release/libdispatch-linux-x86_64/module-cache" -O   -Xcc -fblocks -Xcc -fmodule-map-file=//swift-corelibs-libdispatch/dispatch/module.modulemap -Xcc -I//swift-corelibs-libdispatch -I //swift-corelibs-libdispatch/src/BlocksRuntime -I . -I //swift-corelibs-libdispatch -I //swift-corelibs-libdispatch/src -I src //swift-corelibs-libdispatch/src/swift/Block.swift //swift-corelibs-libdispatch/src/swift/Data.swift //swift-corelibs-libdispatch/src/swift/Dispatch.swift //swift-corelibs-libdispatch/src/swift/IO.swift //swift-corelibs-libdispatch/src/swift/Private.swift //swift-corelibs-libdispatch/src/swift/Queue.swift //swift-corelibs-libdispatch/src/swift/Source.swift //swift-corelibs-libdispatch/src/swift/Time.swift //swift-corelibs-libdispatch/src/swift/Wrapper.swift -no-toolchain-stdlib-rpath -Xlinker -soname -Xlinker libswiftDispatch.so  -L /host-build/Ninja-Release/libdispatch-linux-x86_64/src/swift  -L /host-build/Ninja-Release/libdispatch-linux-x86_64  -L /host-build/Ninja-Release/libdispatch-linux-x86_64   -L /usr/lib/gcc/x86_64-redhat-linux/7   -L /host-build/Ninja-Release/llvm-linux-x86_64/lib -Xlinker -rpath -Xlinker /host-build/Ninja-Release/libdispatch-linux-x86_64:  src/swift/libDispatchStubs.a  libdispatch.so  libBlocksRuntime.so  -lstdc++  -lm  -lgcc_s  -lgcc  -lc  -lgcc_s  -lgcc && :
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "dispatch.h"
         ^
//swift-corelibs-libdispatch/dispatch/dispatch.h:32:10: error: 'os/generic_unix_base.h' file not found
#include <os/generic_unix_base.h>
         ^
//swift-corelibs-libdispatch/src/swift/Block.swift:13:8: error: could not build C module 'CDispatch'
import CDispatch
       ^

[58/82][ 70%][601.467s] Building C object tests/CMakeFiles/bsdtests.dir/bsdtests.c.o
ninja: build stopped: subcommand failed.
ERROR: command terminated with a non-zero exit status 1, aborting

ERROR: command terminated with a non-zero exit status 1, aborting

[kateinoigakukun]

@hassan-shahbazi Hmm, it's weired that build-foundation.sh tries to build libdispatch 🤔 AFAIK we now skip building libdispatch.

[hassan-shahbazi]

Since when are you skipping libdispatch? My fork is branched from master last updated on Dec 24th.

[kateinoigakukun]

I think we've skipped from the beginning. Could you share your Dockerfile if you have?

[hassan-shahbazi]

I don't have Dockerfile, but you can checkout my fork https://github.com/hassan-shahbazi/swift/tree/amazonlinux in docker and try building it. (hint: it took around 2 hours on a 12-core MacBook)

[kateinoigakukun]

Ah, I see. The libdispatch building is for host environment, not wasm environment, so that's correct behavior to build it.

[kateinoigakukun]

@hassan-shahbazi Please try with this Dockerfile

FROM amazonlinux:2

WORKDIR /home/ec2-user

# The build needs a package from the EPEL repo so that needs to be enabled.
# https://www.tecmint.com/install-epel-repository-on-centos/
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

# Update and install needed build packages
RUN yum -y update
RUN yum -y group install "development tools"
RUN yum -y install \
      git cmake clang python swig uuid-devel libicu-devel libedit-devel \
      libxml2-devel sqlite-devel ncurses-devel pkgconfig python-devel \
      python-pkgconfig libbsd-devel libuuid-devel pexpect curl-devel \
      tzdata rsync wget which python-six ninja-build

# Grab an updated version of cmake
RUN wget https://cmake.org/files/v3.16/cmake-3.16.5-Linux-x86_64.tar.gz \
      && tar xvzf cmake-3.16.5-Linux-x86_64.tar.gz \
      && rm cmake-3.16.5-Linux-x86_64.tar.gz

# Add updated version of cmake to path 
ENV PATH="/home/ec2-user/cmake-3.16.5-Linux-x86_64/bin/:${PATH}"


# Bootstrap the swift source and do a full checkout
RUN git clone https://github.com/swiftwasm/swift.git
WORKDIR /home/ec2-user/swift
RUN ./utils/update-checkout --clone --scheme wasm

RUN ./utils/webassembly/build-toolchain.sh

[hassan-shahbazi]

With some modifications, I tried the Dockerfile, but it yet fails. I would retry later to see if I can find the root cause.

make[1]: *** [out/build/icudt65l/metadata.res] Bus error (core dumped)
make[1]: *** Waiting for unfinished jobs....
../bin/gendict: done writing	./out/build/icudt65l/brkitr/cjdict.dict (0s).
genrb number of files: 514
number of shared strings: 5424
16-bit units for strings:  95038 = 190076 bytes
16-bit units saved:       268574 = 537148 bytes
16-bit units not saved:     2967 =   5934 bytes
genrb number of files: 800
number of shared strings: 4416
16-bit units for strings:  65321 = 130642 bytes
16-bit units saved:       326043 = 652086 bytes
16-bit units not saved:    13861 =  27722 bytes
make[1]: Leaving directory `/home/ec2-user/host-build/Ninja-Release/libicu-linux-x86_64/data'
make: *** [install-recursive] Error 2
ERROR: command terminated with a non-zero exit status 2, aborting

Building the standard library for: 
ERROR: command terminated with a non-zero exit status 1, aborting

[MaxDesiatov]

@kateinoigakukun I'm still seeing CoreFoundation failures in jobs such as this one https://github.com/swiftwasm/swiftwasm-buildbot/runs/1884279645

Would it make sense to keep the issue open until this is resolved?

[kateinoigakukun]

I think it's a random failure for module caches. It's ok to keep this open until the job succeed

[kateinoigakukun]

Today's snapshot succeed to release amazonlnux2 toolchain 🎉 https://github.com/swiftwasm/swift/releases/tag/swift-wasm-DEVELOPMENT-SNAPSHOT-2021-02-14-a