Segfault in 1.47.0 and newer stable when building `firecracker` on `aarch64`

[acatangiu]

Hi!

I'm seeing an aarch64 compilation regression coming from 1.46.0 to newer (1.47.0, 1.48.0).

When building the mmds crate of the firecracker project, rustc fails with SIGSEGV.

Code

firecracker/src/mmds# cargo build --target aarch64-unknown-linux-gnu
   Compiling mmds v0.1.0 (/firecracker/src/mmds)
error: could not compile `mmds`

Caused by:
  process didn't exit successfully: `rustc --crate-name mmds --edition=2018 src/mmds/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C panic=abort -C embed-bitcode=no -C debuginfo=2 -C metadata=a83966860f21f0c3 -C extra-filename=-a83966860f21f0c3 --out-dir /firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps --target aarch64-unknown-linux-gnu -C incremental=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/incremental -L dependency=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps -L dependency=/firecracker/build/cargo_target/debug/deps --extern dumbo=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libdumbo-ea66873127194d79.rmeta --extern lazy_static=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/liblazy_static-0925de59eeb12781.rmeta --extern logger=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/liblogger-bddc603ca5b94da7.rmeta --extern micro_http=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libmicro_http-7d059f1cd22f8e20.rmeta --extern serde_json=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libserde_json-ce294f118ad2c211.rmeta --extern snapshot=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libsnapshot-74eb108aec90b439.rmeta --extern utils=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libutils-745e285d9d355f39.rmeta --extern versionize=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libversionize-ef8bd22d261d724f.rmeta --extern versionize_derive=/firecracker/build/cargo_target/debug/deps/libversionize_derive-997f2c850e518d2b.so -C link-arg=-lgcc -C link-arg=-lfdt` (signal: 11, SIGSEGV: invalid memory reference)

I will try to create an MVE for it when I get the chance.

Meta

Works fine:

rustc 1.46.0 (04488afe3 2020-08-24)
binary: rustc
commit-hash: 04488afe34512aa4c33566eb16d8c912a3ae04f9
commit-date: 2020-08-24
host: aarch64-unknown-linux-gnu
release: 1.46.0
LLVM version: 10.0

Crashes:

rustc 1.47.0 (18bf6b4f0 2020-10-07)
binary: rustc
commit-hash: 18bf6b4f01a6feaf7259ba7cdae58031af1b7b39
commit-date: 2020-10-07
host: aarch64-unknown-linux-gnu
release: 1.47.0
LLVM version: 11.0

Crashes

rustc 1.48.0 (7eac88abb 2020-11-16)
binary: rustc
commit-hash: 7eac88abb2e57e752f3302f02be5f3ce3d7adfb4
commit-date: 2020-11-16
host: aarch64-unknown-linux-gnu
release: 1.48.0
LLVM version: 11.0

Works fine:

rustc 1.50.0-nightly (0f6f2d681 2020-12-06)
binary: rustc
commit-hash: 0f6f2d681b39c5f95459cd09cb936b6ceb27cd82
commit-date: 2020-12-06
host: aarch64-unknown-linux-gnu
release: 1.50.0-nightly

Same behavior with both aarch64-unknown-linux-gnu and aarch64-unknown-linux-musl targets.

Reproduces only on debug, building --release works on all of the above rustc versions.

Error output

process didn't exit successfully: `rustc --crate-name mmds --edition=2018 src/mmds/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C panic=abort -C embed-bitcode=no -C debuginfo=2 -C metadata=a83966860f21f0c3 -C extra-filename=-a83966860f21f0c3 --out-dir /firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps --target aarch64-unknown-linux-gnu -C incremental=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/incremental -L dependency=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps -L dependency=/firecracker/build/cargo_target/debug/deps --extern dumbo=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libdumbo-ea66873127194d79.rmeta --extern lazy_static=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/liblazy_static-0925de59eeb12781.rmeta --extern logger=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/liblogger-bddc603ca5b94da7.rmeta --extern micro_http=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libmicro_http-7d059f1cd22f8e20.rmeta --extern serde_json=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libserde_json-ce294f118ad2c211.rmeta --extern snapshot=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libsnapshot-74eb108aec90b439.rmeta --extern utils=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libutils-745e285d9d355f39.rmeta --extern versionize=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libversionize-ef8bd22d261d724f.rmeta --extern versionize_derive=/firecracker/build/cargo_target/debug/deps/libversionize_derive-997f2c850e518d2b.so -C link-arg=-lgcc -C link-arg=-lfdt` (signal: 11, SIGSEGV: invalid memory reference)

GDB Backtrace

Thread 11 "rustc" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xffffe058fc30 (LWP 26541)]
0x0000fffff1ad1488 in llvm::MVT::getSizeInBits() const () from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
(gdb) bt
#0  0x0000fffff1ad1488 in llvm::MVT::getSizeInBits() const () from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#1  0x0000fffff28f2418 in llvm::CallLowering::handleAssignments(llvm::CCState&, llvm::SmallVectorImpl<llvm::CCValAssign>&, llvm::MachineIRBuilder&, llvm::SmallVectorImpl<llvm::CallLowering::ArgInfo>&, llvm::CallLowering::ValueHandler&) const () from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#2  0x0000fffff28f3a80 in llvm::CallLowering::handleAssignments(llvm::MachineIRBuilder&, llvm::SmallVectorImpl<llvm::CallLowering::ArgInfo>&, llvm::CallLowering::ValueHandler&) const () from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#3  0x0000fffff27b6024 in llvm::AArch64CallLowering::lowerFormalArguments(llvm::MachineIRBuilder&, llvm::Function const&, llvm::ArrayRef<llvm::ArrayRef<llvm::Register> >) const
    () from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#4  0x0000fffff290e140 in llvm::IRTranslator::runOnMachineFunction(llvm::MachineFunction&) ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#5  0x0000fffff2d5d280 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) [clone .part.40] [clone .constprop.41] ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#6  0x0000fffff37ecc10 in llvm::FPPassManager::runOnFunction(llvm::Function&) ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#7  0x0000fffff37ed260 in llvm::FPPassManager::runOnModule(llvm::Module&) ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#8  0x0000fffff37ed5a8 in llvm::legacy::PassManagerImpl::run(llvm::Module&) ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#9  0x0000fffff1aa0e10 in LLVMRustWriteOutputFile () from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#10 0x0000fffff1a1fb2c in rustc_codegen_llvm::back::write::write_output_file ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#11 0x0000fffff1a22f1c in rustc_codegen_llvm::back::write::codegen ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#12 0x0000fffff1a0a3f8 in rustc_codegen_ssa::back::write::finish_intra_module_work ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#13 0x0000fffff1a059e8 in rustc_codegen_ssa::back::write::execute_work_item ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#14 0x0000fffff19219b8 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#15 0x0000fffff198439c in core::ops::function::FnOnce::call_once{{vtable-shim}} ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#16 0x0000fffff0c02c84 in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once ()
    at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/boxed.rs:1042
#17 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/boxed.rs:1042
#18 std::sys::unix::thread::Thread::new::thread_start () at library/std/src/sys/unix/thread.rs:87
#19 0x0000fffff08f4088 in start_thread (arg=0xffffe1160e8f) at pthread_create.c:463
#20 0x0000fffff0adaffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

My system info:

$ uname -a
Linux cd2a1aab8486 4.15.0-1088-aws #93~16.04.1-Ubuntu SMP Wed Nov 18 01:34:22 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

[dillona]

cargo-bisect-rustc shows the regression was in:

commit 7ce71c362be9a89e7897ac066aba6e3e6f747800
Merge: e482c86b9de b450c0c86cc
Author: bors <bors@rust-lang.org>
Date:   Sun Aug 23 04:02:41 2020 +0000

    Auto merge of #73526 - cuviper:rust-llvm11, r=nikic

    Upgrade to LLVM 11 (rc2)

    This builds on #73525 to try actually moving rust-lang/llvm-project to LLVM 11.

[apiraino]

Assigning P-high as discussed as part of the Prioritization Working Group procedure and removing I-prioritize.

[nagisa]

SIGSEGV in LLVM is usually an indication of an LLVM assert firing. MVCE is one way to narrow this down, using LLVM with assertions enabled to obtain the assertion message may be another way to sufficiently narrow this down.