test PPC with LLVM_IAS=1

[nickdesaulniers]

@aik sent a series for enabling LLVM_IAS=1. @mpe accepted it, so it should be in linux-next soon. The cover letter mentions we need https://reviews.llvm.org/D115419 that @nemanjai is working on. As soon as that lands, we should be able to start testing probably our ppc64le configs on linux-next with clang-14 with LLVM_IAS=1 enabled.

[mpe]

It also needs the hwsync suport which is already merged in LLVM.

[nathanchance]

With D115419 applied and Linux at commit 112450df61b7 ("Merge branch 'i2c/for-mergewindow' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux"), we get closer to building a powernv_defconfig kernel with LLVM=1:

$ make -skj"$(nproc)" ARCH=powerpc LLVM=1 powernv_defconfig all
/usr/sbin/ld: unrecognised emulation mode: elf64lppc
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu elf_l1om elf_k1om
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [arch/powerpc/kernel/vdso64/Makefile:44: arch/powerpc/kernel/vdso64/vdso64.so.dbg] Error 1
...

This is ClangBuiltLinux/linux#774. If I quickly hack around that error:

diff --git a/arch/powerpc/kernel/vdso64/Makefile b/arch/powerpc/kernel/vdso64/Makefile
index 3c5baaa6f1e7..45dc62ba1541 100644
--- a/arch/powerpc/kernel/vdso64/Makefile
+++ b/arch/powerpc/kernel/vdso64/Makefile
@@ -39,6 +39,8 @@ asflags-y := -D__VDSO64__ -s
 targets += vdso64.lds
 CPPFLAGS_vdso64.lds += -P -C -U$(ARCH)

+vdsold-$(CONFIG_LD_IS_LLD) := -fuse-ld=lld
+
 # link rule for the .so file, .lds has to be first
 $(obj)/vdso64.so.dbg: $(src)/vdso64.lds $(obj-vdso64) $(obj)/vgettimeofday.o FORCE
        $(call if_changed,vdso64ld_and_check)
@@ -53,4 +55,4 @@ include/generated/vdso64-offsets.h: $(obj)/vdso64.so.dbg FORCE

 # actual build commands
 quiet_cmd_vdso64ld_and_check = VDSO64L $@
-      cmd_vdso64ld_and_check = $(CC) $(c_flags) -o $@ -Wl,-T$(filter %.lds,$^) $(filter %.o,$^); $(cmd_vdso_check)
+      cmd_vdso64ld_and_check = $(CC) $(c_flags) $(vdsold-y) -o $@ -Wl,-T$(filter %.lds,$^) $(filter %.o,$^); $(cmd_vdso_check)

I get another one:

ld.lld: warning: cannot find entry symbol _start; not setting start address
objcopy: Unable to recognise the format of the input file `vmlinux'
make[2]: *** [arch/powerpc/boot/Makefile:391: arch/powerpc/boot/zImage.pseries] Error 1
objcopy: Unable to recognise the format of the input file `vmlinux'
make[2]: *** [arch/powerpc/boot/Makefile:391: arch/powerpc/boot/zImage.epapr] Error 1
make[2]: Target 'arch/powerpc/boot/zImage' not remade because of errors.
make[1]: *** [arch/powerpc/Makefile:283: zImage] Error 2

Looks like arch/powerpc/boot/wrapper has hardcoded references to the GNU tools (e.g. ${CROSS}objcopy), so we cannot omit CROSS_COMPILE for now. After adding CROSS_COMPILE=powerpc64le-linux-gnu- back to the command line, I get a kernel that boots in QEMU:

$ boot-qemu.sh -a ppc64le -k . -t 30s
QEMU location: /qemu/bin

QEMU version: QEMU emulator version 6.2.50 (v6.2.0-39-g0da6106883)
...
[    0.000000][    T0] Linux version 5.16.0-09670-g112450df61b7-dirty (nathan@archlinux-ax161) (ClangBuiltLinux clang version 14.0.0 (https://github.com/llvm/llvm-project 1344e35c9822db4998d9eec7f219ecab700ecaa1), LLD 14.0.0) #1 SMP Fri Jan 14 17:53:20 UTC 2022
...

ppc64le_defconfig builds as well.

Very close!

[nickdesaulniers]

I get a kernel that boots in QEMU:
Very close!

That's great!

Looks like arch/powerpc/boot/wrapper has hardcoded references to the GNU tools (e.g. ${CROSS}objcopy)

Should be an easy fix?

[nathanchance]

Looks like arch/powerpc/boot/wrapper has hardcoded references to the GNU tools (e.g. ${CROSS}objcopy)

Should be an easy fix?

When I looked last time I explored this for hermetic builds a couple of years ago, it was not super apparent what the fix should be because it sounds like arch/powerpc/boot/wrapper is used in other projects so we cannot just convert those references over to ${OBJCOPY} or the like (@mpe can correct me if I am wrong). It looks like @gwelymernans touched this area in 5.11, perhaps he could shed some light on how we might work around this.

[mpe]

Do you have a link to the discussion from a couple of years ago?

AFAIK the wrapper is not used in any other projects, although that was the idea of it originally.

It is potentially called by other scripts from outside the kernel tree. I think in the old days maybe some distro scripts called it, but I think the modern distros don't. Possibly some 32-bit embedded stuff still uses it.

Also eg. see torvalds/linux@528229d, which talks about "the signing tool" calling the wrapper script. I'm not sure that signing tool was ever released, and it's basically dead now, but that's the kind of thing we have to keep in mind, external scripts that call the wrapper and expect it to behave a certain way.

So the main thing is to make any change backward compatible, if possible. ie. so that an existing caller will get the same behaviour it used to.

What is the LLVM alternative to objcopy ?

[nathanchance]

Do you have a link to the discussion from a couple of years ago?

I have tried to find it but I do not recall ever bringing this up to upstream, so I could be misremembering something... shrugs

AFAIK the wrapper is not used in any other projects, although that was the idea of it originally.

It is potentially called by other scripts from outside the kernel tree. I think in the old days maybe some distro scripts called it, but I think the modern distros don't. Possibly some 32-bit embedded stuff still uses it.

Also eg. see torvalds/linux@528229d, which talks about "the signing tool" calling the wrapper script. I'm not sure that signing tool was ever released, and it's basically dead now, but that's the kind of thing we have to keep in mind, external scripts that call the wrapper and expect it to behave a certain way.

So the main thing is to make any change backward compatible, if possible. ie. so that an existing caller will get the same behaviour it used to.

Ack. I have hacked something up that works for powernv_defconfig when building zImage.epapr:

https://git.kernel.org/pub/scm/linux/kernel/git/nathan/linux.git/log/?h=wip/llvm-1-powerpc-boot-wrapper

$ make -skj"$(nproc)" ARCH=powerpc LLVM=1 distclean powernv_defconfig zImage.epapr
ld.lld: warning: cannot find entry symbol _start; not setting start address

$ boot-qemu.sh -a ppc64le -k . -t 30s
QEMU location: /usr/bin

QEMU version: QEMU emulator version 6.2.0
...
[    0.000000][    T0] Linux version 5.16.0-11415-gc1a97611aaa2 ([email protected]) (ClangBuiltLinux clang version 14.0.0 (https://github.com/llvm/llvm-project d4ffb6ec65d7173ee63fa0e70eedd049944ceeff), LLD 14.0.0) #1 SMP Fri Jan 21 12:16:52 MST 2022
...

However, the build fails when building zImage.pseries (included in the all target, obviously related to the new use of ld.lld):

$ make -skj"$(nproc)" ARCH=powerpc LLVM=1 distclean powernv_defconfig all
...
ld.lld: error: relocation R_PPC64_ADDR64 cannot be used against symbol '_start'; recompile with -fPIC
>>> referenced by crt0.o:(.text+0x8) in archive arch/powerpc/boot/wrapper.a

ld.lld: error: relocation R_PPC64_ADDR64 cannot be used against symbol '_etext'; recompile with -fPIC
>>> referenced by crt0.o:(.text+0x10) in archive arch/powerpc/boot/wrapper.a

ld.lld: error: relocation R_PPC64_ADDR64 cannot be used against symbol '__bss_start'; recompile with -fPIC
>>> referenced by crt0.o:(.text+0x18) in archive arch/powerpc/boot/wrapper.a

ld.lld: error: relocation R_PPC64_ADDR64 cannot be used against symbol '_end'; recompile with -fPIC
>>> referenced by crt0.o:(.text+0x20) in archive arch/powerpc/boot/wrapper.a
make[2]: *** [arch/powerpc/boot/Makefile:400: arch/powerpc/boot/zImage.pseries] Error 1
...

If I apply the same fix as commit 4c078c86b4a4 ("powerpc/boot/wrapper: Add "-z notext" flag to disable diagnostic") to the pseries case in arch/powerpc/boot/wrapper, zImage.pseries builds successfully and I can boot it in QEMU, just like zImage.epapr.

There was one other issue I noticed with ppc44x_defconfig (the 32-bit config that we test and boot) related to relro. I solved this by adding -z norelro.

What is the LLVM alternative to objcopy ?

llvm-objcopy :)

[nathanchance]

I realized that tuxsuite provides CROSS_COMPILE so we can actually enable this configuration with LLVM_IAS=1 as soon as https://reviews.llvm.org/D115419 is merged, which will allow us to workaround ClangBuiltLinux/linux#774 and ClangBuiltLinux/linux#1601.

[nickdesaulniers]

just noting that https://reviews.llvm.org/D115419 is merged.

[nathanchance]

I am reopening this due to #324 and ClangBuiltLinux/linux#1581. I build tested it locally but did not boot test.

I will likely just build these configurations with ld.bfd in the meantime but I am open to other suggestions.

[nathanchance]

So we have a few different ways to work around ClangBuiltLinux/linux#1581:

1. Disable the boot tests for the distribution configurations.
2. Disable CONFIG_RELOCATABLE=y for the distribution configurations.
3. Use ld.bfd instead of ld.lld.

@nickdesaulniers do you have a preference?

[nickdesaulniers]

3

[mpe]

IMHO ld.lld having an "imprecise" value for DT_RELACOUNT is just plainly a bug in ld.lld.

Our relocation code is primitive, it runs very very early in boot, and no doubt it could be improved, but it has worked for the last ~13 years with no issue.