cargo crashes with illegal instruction under .OPENSSL_cpuid_setup on big-endian ppc64

[hsivonen]

Problem

Running cargo on Linux on big-endian ppc64 does not work at all, because cargo immediately crashes with illegal instruction. The crashing stack frame does not show a readable name but the next one says .OPENSSL_cpuid_setup ().

Steps

1. Install cargo on Linux on big-endian ppc64 (I used Fedora 25 in qemu)
2. Run e.g. cargo --version

Possible Solution(s)

Is there a way to get cargo to use rustls instead of OpenSSL?

Notes

Output of cargo version:

Can't run it, but rustc --version says:
rustc 1.32.0-nightly (6f93e93af 2018-11-14)

[alexcrichton]

In the past this commonly has to do with codegen where the C compiler isn't configured right and is emitting instructions that are "too new". Can you confirm though that the fault happens because of an instruction that shouldn't be generated? If so, do you know the flag we need to pass to C to not generate it?

[pusateri]

This is also happening when installing rustup.rs on powerpc64 gentoo linux.
Next I downloaded
https://static.rust-lang.org/dist/2018-10-30/cargo-beta-powerpc-unknown-linux-gnu.tar.gz
and verified the problem exists here as well. Be glad to help troubleshoot. Not sure what to do next.

[pusateri]

Here's a stack trace:

$ gdb rustup-init-1.14.0 
GNU gdb (Gentoo 8.1 p1) 8.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from rustup-init-1.14.0...done.
(gdb) run -v
Starting program: /home/pusateri/rustup-init-1.14.0 -v
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Welcome to Rust!

This will download and install the official compiler for the Rust programming 
language, and its package manager, Cargo.

It will add the cargo, rustc, rustup and other commands to Cargo's bin 
directory, located at:

  /home/pusateri/.cargo/bin

This path will then be added to your PATH environment variable by modifying the
profile files located at:

  /home/pusateri/.profile
  /home/pusateri/.bash_profile

You can uninstall at any time with rustup self uninstall and these changes will
be reverted.

Current installation options:

   default host triple: powerpc64-unknown-linux-gnu
     default toolchain: stable
  modify PATH variable: yes

1) Proceed with installation (default)
2) Customize installation
3) Cancel installation
>1

verbose: installing toolchain 'stable-powerpc64-unknown-linux-gnu'
verbose: toolchain directory: '/home/pusateri/.rustup/toolchains/stable-powerpc64-unknown-linux-gnu'
info: syncing channel updates for 'stable-powerpc64-unknown-linux-gnu'
verbose: creating temp file: /home/pusateri/.rustup/tmp/rd78ogaxdf9w21_s_file
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256'
verbose: downloading with curl

Program received signal SIGILL, Illegal instruction.
0x0000000100569590 in .OPENSSL_crypto207_probe ()
(gdb) where
#0  0x0000000100569590 in .OPENSSL_crypto207_probe ()
#1  0x0000000100621c9c in .OPENSSL_cpuid_setup ()
#2  0x00000001005b0af0 in .OPENSSL_add_all_algorithms_noconf ()
#3  0x000000010061a834 in .std::sync::once::Once::call_once::{{closure}} ()
#4  0x0000000100841334 in std::sync::once::Once::call_inner ()
    at libstd/sync/once.rs:349
#5  0x000000010052132c in .openssl_sys::init ()
#6  0x0000000100412fb8 in .std::sync::once::Once::call_once::{{closure}} ()
#7  0x0000000100841334 in std::sync::once::Once::call_inner ()
    at libstd/sync/once.rs:349
#8  0x000000010041711c in .<curl::easy::handler::Easy2<H>>::new ()
#9  0x00000001004149d8 in .curl::easy::handle::Easy::new ()
#10 0x00000001003fa1d4 in .download::curl::download::EASY::__init ()
#11 0x00000001003fa35c in .<std::thread::local::LocalKey<T>>::with ()
#12 0x000000010040248c in .download::download_to_path_with_backend ()
#13 0x00000001003c6af4 in .rustup_utils::utils::download_file_with_resume ()
#14 0x00000001003c682c in .rustup_utils::utils::download_file ()
#15 0x00000001003400a0 in .rustup_dist::download::DownloadCfg::download_and_check ()
#16 0x0000000100327ec0 in .rustup_dist::dist::update_from_dist_ ()
#17 0x0000000100327690 in .rustup_dist::dist::update_from_dist ()
#18 0x000000010020e95c in .rustup::install::InstallMethod::run ()
#19 0x0000000100202be8 in .rustup::toolchain::Toolchain::install ()
---Type <return> to continue, or q <return> to quit---
#20 0x0000000100203398 in .rustup::toolchain::Toolchain::install_from_dist_inner ()
#21 0x000000010020308c in .rustup::toolchain::Toolchain::install_from_dist ()
#22 0x0000000100193600 in .rustup_init::self_update::install ()
#23 0x00000001001c8190 in .rustup_init::setup_mode::main ()
#24 0x0000000100181a10 in .rustup_init::main ()
#25 0x0000000100178ec0 in .std::rt::lang_start::{{closure}} ()
#26 0x0000000100825df0 in std::rt::lang_start_internal::{{closure}}::{{closure}} () at libstd/rt.rs:59
#27 std::sys_common::backtrace::__rust_begin_short_backtrace ()
    at libstd/sys_common/backtrace.rs:136
#28 0x00000001008218cc in std::rt::lang_start_internal::{{closure}} ()
    at libstd/rt.rs:59
#29 std::panicking::try::do_call () at libstd/panicking.rs:310
#30 0x0000000100850e48 in __rust_maybe_catch_panic ()
    at libpanic_unwind/lib.rs:105
#31 0x000000010081b7b0 in std::panicking::try () at libstd/panicking.rs:289
#32 std::panic::catch_unwind () at libstd/panic.rs:392
#33 std::rt::lang_start_internal () at libstd/rt.rs:58
#34 0x0000000100181e18 in .main ()
(gdb) 

[pusateri]

According to this closed debian bug report, this is working as designed:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=788720

Apparently, a SIGILL is normal for this cpu feature detection routine. Therefore, the calling program should be prepared to ignore this signal.

[pusateri]

There's supposedly a workaround that skips the call to OPENSSL_cpuid_setup() by presetting the cpu options. I tried 1 which just means ALTIVEC support:
https://github.com/openssl/openssl/blob/9a3b5b766421b7ed9df859956d44fcbe52c2dd82/crypto/ppccap.c#L209

OPENSSL_ppccap=1 ./rustup-init-1.14.0 -v

but had the same SIGILL problem.

$ openssl version
OpenSSL 1.0.2p 14 Aug 2018

Maybe this feature isn't in this version of OpenSSL that rustup/cargo was built with?

[pusateri]

Went back through older versions of rustup-init and 1.0.0 is the only one that works without an illegal instruction on powerpc64.
The first version of the compiler that installs and works is 1.17.0 (2017-04-24).
The last version of the compiler that installs and works is 1.19.0 (2017-07-13).
After that 1.20.0 won't install because the package format must have changed and rustup 1.0.0 doesn't know about the change. It gives an error: missing key: 'url'

[pusateri]

The 2018-06-30 tools seem to run ok but there's no way to install them except manually because rustup 1.0.0 can't understand the new package format and newer versions of rustup won't run. I really think this is a rustup problem at this point, not a cargo problem.

[q66]

Also crashes on powerpc64le, with slightly different stack trace, but also coming from openssl https://gist.github.com/q66/7c7dcde1a7ea057be706c51c4700a373

the first version to do so is 1.31 (persists in nightly), it worked fine on 1.30.1

[pusateri]

I downloaded 1.24.1, 1.30.0, and 1.31.0 and found that 1.24.1 and 1.30.0 run fine and 1.31.0 crashes in OPENSSL_cpuid_setup(). So I think cargo was being built correctly until 1.31.0. Rustup on the other hand has been built incorrectly since 1.0.0 on powerpc64.

pusateri@powermac ~/rust-1.31.0-powerpc64-unknown-linux-gnu/cargo/bin $ gdb cargo
GNU gdb (Gentoo 8.1 p1) 8.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from cargo...done.
(gdb) run -v
Starting program: /home/pusateri/rust-1.31.0-powerpc64-unknown-linux-gnu/cargo/bin/cargo -v
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
0x0000000100000000 in ?? ()
(gdb) where
#0  0x0000000100000000 in ?? ()
#1  0x0000000100197dd4 in .OPENSSL_cpuid_setup ()
#2  0x0000000100ad263c in .__do_global_ctors_aux ()
#3  0x000000010019441c in ._init ()
#4  0x0000000100ad2440 in .__libc_csu_init ()
#5  0x00003ffff7d38088 in ?? () from /lib64/libc.so.6
#6  0x00003ffff7d38350 in .__libc_start_main () from /lib64/libc.so.6
#7  0x0000000000000000 in ?? ()
(gdb) 

[q66]

yeah, that is pretty consistent with LE, just different place in openssl. I can also confirm that self-built cargo 0.32 runs fine.

[q66]

also, it's definitely not that I'm missing some instruction, this is a modern POWER9 box and that's as new as it gets.

[nagisa]

The linked issue #6472 has some insight on what exactly is a problem. Namely:

(gdb) disassemble $pc-8,+16
Dump of assembler code from 0x55480dcc to 0x55480ddc:
   0x0000000055480dcc <.OPENSSL_cpuid_setup+156>:	li      r3,16
   0x0000000055480dd0 <.OPENSSL_cpuid_setup+160>:	bl      0x552f0000
=> 0x0000000055480dd4 <.OPENSSL_cpuid_setup+164>:	nop
   0x0000000055480dd8 <.OPENSSL_cpuid_setup+168>:	rldicl. r9,r3,37,63
End of assembler dump.
(gdb) x/20x 0x552f0000
0x552f0000:	Cannot access memory at address 0x552f0000