Clang picks wrong linker/assembler

[agners]

When using the a system installation of Clang (e.g. with Ubuntu 18.04 or Arch Linux), and a cross compiler toolchain from Linaro, building fails with the following errors:

export CROSS_COMPILE=arm-linux-gnueabihf-
export ARCH=arm
export PATH="/home/user/gcc-linaro-7.2.1-2017.11-x86_64_arm-linux-gnueabihf/bin/:$PATH"

$ make CC=clang HOSTCC=clang
  SYSHDR  arch/arm/include/generated/uapi/asm/unistd-common.h
  SYSHDR  arch/arm/include/generated/uapi/asm/unistd-oabi.h
  SYSHDR  arch/arm/include/generated/uapi/asm/unistd-eabi.h
  UPD     include/config/kernel.release
  UPD     include/generated/utsrelease.h
  HOSTCC  scripts/basic/fixdep
  SYSNR   arch/arm/include/generated/asm/unistd-nr.h
  GEN     arch/arm/include/generated/asm/mach-types.h
  SYSTBL  arch/arm/include/generated/calls-oabi.S
  SYSTBL  arch/arm/include/generated/calls-eabi.S
  CC      kernel/bounds.s
clang-6.0: warning: argument unused during compilation: '-Wa,-W' [-Wunused-command-line-argument]
  CC      arch/arm/kernel/asm-offsets.s
clang-6.0: warning: argument unused during compilation: '-Wa,-W' [-Wunused-command-line-argument]
  CALL    scripts/checksyscalls.sh
clang-6.0: warning: argument unused during compilation: '-Wa,-W' [-Wunused-command-line-argument]
<stdin>:1332:2: warning: syscall io_pgetevents not implemented [-W#warnings]
#warning syscall io_pgetevents not implemented
 ^
1 warning generated.
  HOSTCC  scripts/dtc/dtc.o
  HOSTCC  scripts/dtc/flattree.o
  HOSTCC  scripts/dtc/fstree.o
  HOSTCC  scripts/dtc/data.o
  HOSTCC  scripts/dtc/livetree.o
  HOSTCC  scripts/dtc/treesource.o
  HOSTCC  scripts/dtc/srcpos.o
  HOSTCC  scripts/dtc/checks.o
  HOSTCC  scripts/dtc/util.o
  LEX     scripts/dtc/dtc-lexer.lex.c
  YACC    scripts/dtc/dtc-parser.tab.h
  HOSTCC  scripts/dtc/dtc-lexer.lex.o
  YACC    scripts/dtc/dtc-parser.tab.c
  HOSTCC  scripts/dtc/dtc-parser.tab.o
  HOSTLD  scripts/dtc/dtc
  HOSTCC  scripts/mod/mk_elfconfig
  CC      scripts/mod/empty.o
clang-6.0: error: unsupported argument '-W' to option 'Wa,'
make[2]: *** [scripts/Makefile.build:306: scripts/mod/empty.o] Error 1
make[1]: *** [scripts/Makefile.build:546: scripts/mod] Error 2
make: *** [Makefile:1075: scripts] Error 2

A similar issue can also be observed with arm64:

export CROSS_COMPILE=aarch64-linux-gnu-
export ARCH=arm64
export PATH="/home/user/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/:$PATH"


$ make CC=clang HOSTCC=clang
arch/arm64/Makefile:40: LSE atomics not supported by binutils
arch/arm64/Makefile:48: Detected assembler with broken .inst; disassembly will be unreliable
  CALL    scripts/checksyscalls.sh
  VDSOA   arch/arm64/kernel/vdso/gettimeofday.o
arch/arm64/kernel/vdso/gettimeofday.S: Assembler messages:
arch/arm64/kernel/vdso/gettimeofday.S:28: Error: no such instruction: `vdso_data .req x6'
arch/arm64/kernel/vdso/gettimeofday.S:29: Error: no such instruction: `seqcnt .req w7'
arch/arm64/kernel/vdso/gettimeofday.S:30: Error: no such instruction: `w_tmp .req w8'
arch/arm64/kernel/vdso/gettimeofday.S:31: Error: no such instruction: `x_tmp .req x8'
arch/arm64/kernel/vdso/gettimeofday.S:139: Error: no such instruction: `adr vdso_data,_vdso_data'
...
arch/arm64/kernel/vdso/gettimeofday.S:316: Error: too many memory references for `mov'
arch/arm64/kernel/vdso/gettimeofday.S:320: Error: expecting operand after ','; got nothing
arch/arm64/kernel/vdso/gettimeofday.S:321: Error: no such instruction: `svc '
clang-6.0: error: assembler command failed with exit code 1 (use -v to see invocation)
make[1]: *** [arch/arm64/kernel/vdso/Makefile:54: arch/arm64/kernel/vdso/gettimeofday.o] Error 1
make: *** [arch/arm64/Makefile:158: vdso_prepare] Error 2

In both cases Clang chooses the wrong version of external utilities. In the arm32 case its the wrong linker which seems to cause problems first, in the arm64 case the assembler (using -v reveals /usr/bin/ld and /usr/bin/as instead of the assembler/linker provided by the cross compiler).

[agners]

Passing -B (described as Add <dir> to search path for binaries and object files used implicitly) seems to help:

make CC="clang -B /home/ags/gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu/bin" HOSTCC=clang

Alternatively, in both cases symlinking clang into the cross compiler directory helps. Clang than also claims its install directory is the cross compilers directory.

$ clang --version
clang version 6.0.1 (tags/RELEASE_601/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /home/user/gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu/bin

I guess this is an expected behavior of Clang? Should/can we let the Linux build system pass a reasonable search dir?

[agners]

This seems to do the trick:

--- a/Makefile
+++ b/Makefile
@@ -495,10 +495,11 @@ endif
 ifeq ($(cc-name),clang)
 ifneq ($(CROSS_COMPILE),)
 CLANG_TARGET   := --target=$(notdir $(CROSS_COMPILE:%-=%))
-GCC_TOOLCHAIN  := $(realpath $(dir $(shell which $(LD)))/..)
+GCC_TOOLCHAIN_BIN := $(dir $(shell which $(LD)))
+GCC_TOOLCHAIN  := $(realpath $(GCC_TOOLCHAIN_BIN))
 endif
 ifneq ($(GCC_TOOLCHAIN),)
-CLANG_GCC_TC   := --gcc-toolchain=$(GCC_TOOLCHAIN)
+CLANG_GCC_TC   := --gcc-toolchain=$(GCC_TOOLCHAIN) -B $(GCC_TOOLCHAIN_BIN)
 endif
 KBUILD_CFLAGS += $(CLANG_TARGET) $(CLANG_GCC_TC)
 KBUILD_AFLAGS += $(CLANG_TARGET) $(CLANG_GCC_TC)

Does that seem sensible? Hope this breaks no other use cases...

[nickdesaulniers]

Let me speak with @stephenhines in person. I wonder if your install of Clang is configured to only support x86_64, so it always picks the wrong target toolchain?

[nickdesaulniers]

Speaking more with @stephenhines and @pirama-arumuga-nainar , it sounds like -B is the way to go if you do not have your cross tools in a short list of paths that Clang is built to search. There's also a --sysroot flag which is similar. It sounds like you can compile with clang -v <args> path/to/c_with_inline_assembly.c to get a print out of the search paths.

I would recommend against the patch in #78 (comment), since it sounds like you may have your cross tools in a non standard search path. That said, I still think Clang should obey $PATH.

[agners]

Building with ARM64 with clang -v leads to this output

"/usr/bin/clang-6.0" -cc1 -triple aarch64--linux-gnu -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name gettimeofday.S -mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -no-integrated-as -mconstructor-aliases -fuse-init-array -target-cpu generic -target-feature +neon -target-abi aapcs -fallow-half-arguments-and-returns -dwarf-column-info -debugger-tuning=gdb -v -coverage-notes-file /home/ags/projects/toradex/imx/linux-vanilla-llvm/arch/arm64/kernel/vdso/gettimeofday.gcno -nostdsysteminc -nobuiltininc -resource-dir /usr/lib/clang/6.0.1 -dependency-file arch/arm64/kernel/vdso/.gettimeofday.o.d -MT arch/arm64/kernel/vdso/gettimeofday.o -sys-header-deps -isystem /usr/lib/clang/6.0.1/include -include ./include/linux/kconfig.h -I ./arch/arm64/include -I ./arch/arm64/include/generated -I ./include -I ./arch/arm64/include/uapi -I ./arch/arm64/include/generated/uapi -I ./include/uapi -I ./include/generated/uapi -D __KERNEL__ -D __ASSEMBLY__ -D CONFIG_BROKEN_GAS_INST=1 -fno-dwarf-directory-asm -fdebug-compilation-dir /home/ags/projects/toradex/imx/linux-vanilla-llvm -ferror-limit 19 -fmessage-length 211 -stack-protector 2 -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -o /tmp/gettimeofday-e3be40.s -x assembler-with-cpp arch/arm64/kernel/vdso/gettimeofday.S
clang -cc1 version 6.0.1 based upon LLVM 6.0.1 default target x86_64-pc-linux-gnu
#include "..." search starts here:
#include <...> search starts here:
 ./arch/arm64/include
 ./arch/arm64/include/generated
 ./include
 ./arch/arm64/include/uapi
 ./arch/arm64/include/generated/uapi
 ./include/uapi
 ./include/generated/uapi
 /usr/lib/clang/6.0.1/include
End of search list.
 "/usr/bin/as" -I ./arch/arm64/include -I ./arch/arm64/include/generated -I ./include -I ./arch/arm64/include/uapi -I ./arch/arm64/include/generated/uapi -I ./include/uapi -I ./include/generated/uapi -gdwarf-2 -o arch/arm64/kernel/vdso/gettimeofday.o /tmp/gettimeofday-e3be40.s
arch/arm64/kernel/vdso/gettimeofday.S: Assembler messages:
arch/arm64/kernel/vdso/gettimeofday.S:28: Error: no such instruction: `vdso_data .req x6'

[agners]

Well yes, I do have my cross compiler toolchain in a non standard path. That is why I have to add it to the PATH. We do add --gcc-toolchain already, so why can't we not make sure that also linker/assembler get picked up from the right path?

[stephenhines]

As mentioned above, this cross-compiler doesn't seem to have been set up with a known sysroot for ARM tools. Thus it is falling back to /usr/bin/as, which isn't useful. The easiest solution is to pass -B with the path to your cross-compiler tools.

[agners]

With this cross-compiler you refer to Clang/LLVM?

So the Clang/LLVM I am using is a stock build from Ubuntu or ArchLinux. Of course this has not been built with a known sysroot for an ARM toolchain...

[agners]

FWIW, this is how I can reproduce it on a vanilla Ubuntu 18.04:

$ apt-get update && apt-get install -y build-essential git clang wget bison flex bc
$ wget https://releases.linaro.org/components/toolchain/binaries/latest/armv8l-linux-gnueabihf/gcc-linaro-7.3.1-2018.05-x86_64_armv8l-linux-gnueabihf.tar.xz
....
$ tar xJf gcc-linaro-7.3.1-2018.05-x86_64_armv8l-linux-gnueabihf.tar.xz
....
$ export CROSS_COMPILE=aarch64-linux-gnu-
$ export ARCH=arm64
$ export PATH="${HOME}/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/:$PATH"
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
....
$ cd linux
$ make CC=clang HOSTCC=clang defconfig && make CC=clang HOSTCC=clang

We are at a point where developers can use LLVM/Clang to build Linux without the need of building LLVM/Clang from scratch. We should support that as far as possible.

[nickdesaulniers]

Wouldn't Ubuntu/Debian devs install via:
sudo apt install binutils-aarch64-linux-gnu gcc-aarch64-linux-gnu
which would put them in /usr/bin? This works for me.

So your example, you're still installing your cross toolchain to a non standard dir. If Clang doesn't respect $PATH for these, then your install is non-standard and needs -B or manual symlinks in /usr/bin to /non/standard/path/aarch64-linux-gnu-as. Let me test the -B patch and see if it hurts at all.

[nickdesaulniers]

The diff in #78 (comment) works for me. You can add my reviewed by tag to send it upstream. Might want to change GCC_TOOLCHAIN_BIN to GCC_TOOLCHAIN_DIR, but doesn't matter.

I also think clang should respect $PATH.

[agners]

@nickdesaulniers hm, I guess that is true, using Clang from distro and gcc from Linaro might actually be a rather unusual case. It still seems a valid use case to me.

Ok, thanks, will send a patch upstream. I will use --prefix= though, since we should avoid using space characters when passing arguments to clang (according to bb3f38c "kbuild: clang: fix build failures with sparse check").

[nickdesaulniers]

Can you share the patch before sending it? I don't understand what you mean.

@stephenhines has convinced me that maybe Clang should not check $PATH for these, as it's possible to confuse which other build utilities we select.

[nickdesaulniers]

oh, nvm I understand.

[agners]

Sorry, wasn't very clear above. For the records, I meant replacing -B <path> with --prefix=<path>, which should be equivalent according to https://clang.llvm.org/docs/ClangCommandLineReference.html

Will send a patch upstream.

[nickdesaulniers]

patch: https://lore.kernel.org/r/[email protected]/