Skip to content

Segfault in 1.47.0 and newer stable when building firecracker on aarch64 #79789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
acatangiu opened this issue Dec 7, 2020 · 5 comments
Closed

Segfault in 1.47.0 and newer stable when building firecracker on aarch64 #79789

acatangiu opened this issue Dec 7, 2020 · 5 comments
Labels
A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. C-bug Category: This is a bug. E-needs-mcve Call for participation: This issue has a repro, but needs a Minimal Complete and Verifiable Example I-crash Issue: The compiler crashes (SIGSEGV, SIGABRT, etc). Use I-ICE instead when the compiler panics. O-Arm Target: 32-bit Arm processors (armv6, armv7, thumb...), including 64-bit Arm in AArch32 state P-high High priority regression-from-stable-to-stable Performance or correctness regression from one stable version to another. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@acatangiu
Copy link

Hi!

I'm seeing an aarch64 compilation regression coming from 1.46.0 to newer (1.47.0, 1.48.0).

When building the mmds crate of the firecracker project, rustc fails with SIGSEGV.

Code

firecracker/src/mmds# cargo build --target aarch64-unknown-linux-gnu
   Compiling mmds v0.1.0 (/firecracker/src/mmds)
error: could not compile `mmds`

Caused by:
  process didn't exit successfully: `rustc --crate-name mmds --edition=2018 src/mmds/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C panic=abort -C embed-bitcode=no -C debuginfo=2 -C metadata=a83966860f21f0c3 -C extra-filename=-a83966860f21f0c3 --out-dir /firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps --target aarch64-unknown-linux-gnu -C incremental=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/incremental -L dependency=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps -L dependency=/firecracker/build/cargo_target/debug/deps --extern dumbo=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libdumbo-ea66873127194d79.rmeta --extern lazy_static=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/liblazy_static-0925de59eeb12781.rmeta --extern logger=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/liblogger-bddc603ca5b94da7.rmeta --extern micro_http=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libmicro_http-7d059f1cd22f8e20.rmeta --extern serde_json=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libserde_json-ce294f118ad2c211.rmeta --extern snapshot=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libsnapshot-74eb108aec90b439.rmeta --extern utils=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libutils-745e285d9d355f39.rmeta --extern versionize=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libversionize-ef8bd22d261d724f.rmeta --extern versionize_derive=/firecracker/build/cargo_target/debug/deps/libversionize_derive-997f2c850e518d2b.so -C link-arg=-lgcc -C link-arg=-lfdt` (signal: 11, SIGSEGV: invalid memory reference)

I will try to create an MVE for it when I get the chance.

Meta

Works fine:

rustc 1.46.0 (04488afe3 2020-08-24)
binary: rustc
commit-hash: 04488afe34512aa4c33566eb16d8c912a3ae04f9
commit-date: 2020-08-24
host: aarch64-unknown-linux-gnu
release: 1.46.0
LLVM version: 10.0

Crashes:

rustc 1.47.0 (18bf6b4f0 2020-10-07)
binary: rustc
commit-hash: 18bf6b4f01a6feaf7259ba7cdae58031af1b7b39
commit-date: 2020-10-07
host: aarch64-unknown-linux-gnu
release: 1.47.0
LLVM version: 11.0

Crashes

rustc 1.48.0 (7eac88abb 2020-11-16)
binary: rustc
commit-hash: 7eac88abb2e57e752f3302f02be5f3ce3d7adfb4
commit-date: 2020-11-16
host: aarch64-unknown-linux-gnu
release: 1.48.0
LLVM version: 11.0

Works fine:

rustc 1.50.0-nightly (0f6f2d681 2020-12-06)
binary: rustc
commit-hash: 0f6f2d681b39c5f95459cd09cb936b6ceb27cd82
commit-date: 2020-12-06
host: aarch64-unknown-linux-gnu
release: 1.50.0-nightly

Same behavior with both aarch64-unknown-linux-gnu and aarch64-unknown-linux-musl targets.

Reproduces only on debug, building --release works on all of the above rustc versions.

Error output

process didn't exit successfully: `rustc --crate-name mmds --edition=2018 src/mmds/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C panic=abort -C embed-bitcode=no -C debuginfo=2 -C metadata=a83966860f21f0c3 -C extra-filename=-a83966860f21f0c3 --out-dir /firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps --target aarch64-unknown-linux-gnu -C incremental=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/incremental -L dependency=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps -L dependency=/firecracker/build/cargo_target/debug/deps --extern dumbo=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libdumbo-ea66873127194d79.rmeta --extern lazy_static=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/liblazy_static-0925de59eeb12781.rmeta --extern logger=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/liblogger-bddc603ca5b94da7.rmeta --extern micro_http=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libmicro_http-7d059f1cd22f8e20.rmeta --extern serde_json=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libserde_json-ce294f118ad2c211.rmeta --extern snapshot=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libsnapshot-74eb108aec90b439.rmeta --extern utils=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libutils-745e285d9d355f39.rmeta --extern versionize=/firecracker/build/cargo_target/aarch64-unknown-linux-gnu/debug/deps/libversionize-ef8bd22d261d724f.rmeta --extern versionize_derive=/firecracker/build/cargo_target/debug/deps/libversionize_derive-997f2c850e518d2b.so -C link-arg=-lgcc -C link-arg=-lfdt` (signal: 11, SIGSEGV: invalid memory reference)
GDB Backtrace 

Thread 11 "rustc" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xffffe058fc30 (LWP 26541)]
0x0000fffff1ad1488 in llvm::MVT::getSizeInBits() const () from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
(gdb) bt
#0  0x0000fffff1ad1488 in llvm::MVT::getSizeInBits() const () from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#1  0x0000fffff28f2418 in llvm::CallLowering::handleAssignments(llvm::CCState&, llvm::SmallVectorImpl<llvm::CCValAssign>&, llvm::MachineIRBuilder&, llvm::SmallVectorImpl<llvm::CallLowering::ArgInfo>&, llvm::CallLowering::ValueHandler&) const () from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#2  0x0000fffff28f3a80 in llvm::CallLowering::handleAssignments(llvm::MachineIRBuilder&, llvm::SmallVectorImpl<llvm::CallLowering::ArgInfo>&, llvm::CallLowering::ValueHandler&) const () from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#3  0x0000fffff27b6024 in llvm::AArch64CallLowering::lowerFormalArguments(llvm::MachineIRBuilder&, llvm::Function const&, llvm::ArrayRef<llvm::ArrayRef<llvm::Register> >) const
    () from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#4  0x0000fffff290e140 in llvm::IRTranslator::runOnMachineFunction(llvm::MachineFunction&) ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#5  0x0000fffff2d5d280 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) [clone .part.40] [clone .constprop.41] ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#6  0x0000fffff37ecc10 in llvm::FPPassManager::runOnFunction(llvm::Function&) ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#7  0x0000fffff37ed260 in llvm::FPPassManager::runOnModule(llvm::Module&) ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#8  0x0000fffff37ed5a8 in llvm::legacy::PassManagerImpl::run(llvm::Module&) ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#9  0x0000fffff1aa0e10 in LLVMRustWriteOutputFile () from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#10 0x0000fffff1a1fb2c in rustc_codegen_llvm::back::write::write_output_file ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#11 0x0000fffff1a22f1c in rustc_codegen_llvm::back::write::codegen ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#12 0x0000fffff1a0a3f8 in rustc_codegen_ssa::back::write::finish_intra_module_work ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#13 0x0000fffff1a059e8 in rustc_codegen_ssa::back::write::execute_work_item ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#14 0x0000fffff19219b8 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#15 0x0000fffff198439c in core::ops::function::FnOnce::call_once{{vtable-shim}} ()
   from /usr/local/rust/toolchains/1.48.0-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-cdc4f9eebd3191e2.so
#16 0x0000fffff0c02c84 in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once ()
    at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/boxed.rs:1042
#17 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/boxed.rs:1042
#18 std::sys::unix::thread::Thread::new::thread_start () at library/std/src/sys/unix/thread.rs:87
#19 0x0000fffff08f4088 in start_thread (arg=0xffffe1160e8f) at pthread_create.c:463
#20 0x0000fffff0adaffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

My system info:

$ uname -a
Linux cd2a1aab8486 4.15.0-1088-aws #93~16.04.1-Ubuntu SMP Wed Nov 18 01:34:22 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
@acatangiu acatangiu added C-bug Category: This is a bug. I-ICE Issue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️ T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Dec 7, 2020
@jonas-schievink jonas-schievink added I-crash Issue: The compiler crashes (SIGSEGV, SIGABRT, etc). Use I-ICE instead when the compiler panics. and removed I-ICE Issue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️ labels Dec 7, 2020
@LeSeulArtichaut LeSeulArtichaut added the regression-from-stable-to-stable Performance or correctness regression from one stable version to another. label Dec 7, 2020
@rustbot rustbot added the I-prioritize Issue: Indicates that prioritization has been requested for this issue. label Dec 7, 2020
@LeSeulArtichaut LeSeulArtichaut added the O-Arm Target: 32-bit Arm processors (armv6, armv7, thumb...), including 64-bit Arm in AArch32 state label Dec 7, 2020
@acatangiu acatangiu mentioned this issue Dec 7, 2020
Merged
8 tasks
@wesleywiser wesleywiser added the A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. label Dec 7, 2020
@dillona
Copy link
Contributor

dillona commented Dec 8, 2020

cargo-bisect-rustc shows the regression was in:

commit 7ce71c362be9a89e7897ac066aba6e3e6f747800
Merge: e482c86b9de b450c0c86cc
Author: bors <[email protected]>
Date:   Sun Aug 23 04:02:41 2020 +0000

    Auto merge of #73526 - cuviper:rust-llvm11, r=nikic

    Upgrade to LLVM 11 (rc2)

    This builds on #73525 to try actually moving rust-lang/llvm-project to LLVM 11.

@apiraino apiraino added P-high High priority and removed I-prioritize Issue: Indicates that prioritization has been requested for this issue. labels Dec 9, 2020
@apiraino
Copy link
Contributor

apiraino commented Dec 9, 2020

Assigning P-high as discussed as part of the Prioritization Working Group procedure and removing I-prioritize.

@nagisa
Copy link
Member

nagisa commented Dec 9, 2020

SIGSEGV in LLVM is usually an indication of an LLVM assert firing. MVCE is one way to narrow this down, using LLVM with assertions enabled to obtain the assertion message may be another way to sufficiently narrow this down.

@dillona
Copy link
Contributor

dillona commented Dec 9, 2020

596b0d5 seems to have resolved the issue, so presumably it's a bug present in LLVM 11 RC2 and fixed in final.

@JohnTitor JohnTitor added the E-needs-mcve Call for participation: This issue has a repro, but needs a Minimal Complete and Verifiable Example label Jan 8, 2021
@nikic
Copy link
Contributor

nikic commented Jan 20, 2022

Closing this per the preceding comment.

@nikic nikic closed this as completed Jan 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. C-bug Category: This is a bug. E-needs-mcve Call for participation: This issue has a repro, but needs a Minimal Complete and Verifiable Example I-crash Issue: The compiler crashes (SIGSEGV, SIGABRT, etc). Use I-ICE instead when the compiler panics. O-Arm Target: 32-bit Arm processors (armv6, armv7, thumb...), including 64-bit Arm in AArch32 state P-high High priority regression-from-stable-to-stable Performance or correctness regression from one stable version to another. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@dillona @nikic @nagisa @wesleywiser @jonas-schievink @apiraino @JohnTitor @acatangiu @LeSeulArtichaut @rustbot