Rust-1.54.0-GCCcore-11.2.0.eb does not use OpenSSL from deps and fails.

[surak]

Rust depends on OpenSSL:

easybuild-easyconfigs/easybuild/easyconfigs/r/Rust/Rust-1.54.0-GCCcore-11.2.0.eb

Line 28 in 74f7c40

('OpenSSL', '1.1', '', True),

I would expect it to not need openssl from the system when depending on OpenSSL, but this is what I get on a system without libssl-dev (Ubuntu):

   Compiling openssl-sys v0.9.61
   Compiling libnghttp2-sys v0.1.4+1.41.0
   Compiling libssh2-sys v0.2.19
   Compiling curl-sys v0.4.42+curl-7.76.0
   Compiling libgit2-sys v0.12.18+1.1.0
   Compiling tempfile v3.2.0
   Compiling flate2 v1.0.16
error: failed to run custom build command for `openssl-sys v0.9.61`

Caused by:
  process didn't exit successfully: `/easybuild/2020/build/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/build/x86_64-unknown-linux-gnu/stage2-tools/release/build/openssl-sys-4fbe7f0b41917899/build-script-main` (exit status: 101)
  --- stdout
  cargo:rustc-cfg=const_fn
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
  OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
  OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_DIR
  OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=OPENSSL_STATIC
  cargo:rerun-if-env-changed=OPENSSL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  run pkg_config fail: "`\"pkg-config\" \"--libs\" \"--cflags\" \"openssl\"` did not exit successfully: exit status: 1\n--- stderr\nPackage openssl was not found in the pkg-config search path.\nPerhaps you should add the directory containing `openssl.pc'\nto the PKG_CONFIG_PATH environment variable\nNo package 'openssl' found\n"

  --- stderr
  thread 'main' panicked at '

  Could not find directory of OpenSSL installation, and this `-sys` crate cannot
  proceed without this knowledge. If OpenSSL is installed and this crate had
  trouble finding it,  you can set the `OPENSSL_DIR` environment variable for the
  compilation process.

  Make sure you also have the development packages of openssl installed.
  For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.

  If you're in a situation where you think the directory *should* be found
  automatically, please open a bug at https://github.com/sfackler/rust-openssl
  and include information about your system as well as this message.

  $HOST = x86_64-unknown-linux-gnu
  $TARGET = x86_64-unknown-linux-gnu
  openssl-sys = 0.9.61

  ', /easybuild/2020/build/Rust/1.54.0/GCCcore-11.2.0/cargo/registry/src/github.1485827954.workers.dev-1ecc6299db9ec823/openssl-sys-0.9.61/build/find_normal.rs:174:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed
command did not execute successfully: "/easybuild/2020/build/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "48" "--release" "--manifest-path" "/easybuild/2020/build/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/src/tools/cargo/Cargo.toml" "--message-format" "json-render-diagnostics"
expected success, got: exit status: 101
failed to run: /easybuild/2020/build/Rust/1.54.0/GCCcore-11.2.0/rustc-1.54.0-src/build/bootstrap/debug/bootstrap build --stage 2
Build completed unsuccessfully in 0:11:46
make: *** [Makefile:12: all] Error 1
 (at easybuild/2020/software/EasyBuild/4.5.0/lib/python3.8/site-packages/easybuild/tools/run.py:618 in parse_cmd_output)
== 2022-01-03 18:42:31,668 build_log.py:265 INFO ... (took 11 mins 46 secs)
== ... (took 11 mins 46 secs)
== 2022-01-03 18:42:31,668 filetools.py:1971 INFO Removing lock /easybuild/2020/software/.locks/_easybuild_2020_software_Rust_1.54.0-GCCcore-11.2.0.lock...
== 2022-01-03 18:42:31,668 filetools.py:380 INFO Path /easybuild/2020/software/.locks/_easybuild_2020_software_Rust_1.54.0-GCCcore-11.2.0.lock successfully removed.
== 2022-01-03 18:42:31,668 filetools.py:1975 INFO Lock removed: /easybuild/2020/software/.locks/_easybuild_2020_software_Rust_1.54.0-GCCcore-11.2.0.lock
== 2022-01-03 18:42:31,668 easyblock.py:3915 WARNING build failed (first 300 chars): cmd "export CARGO_HOME=/easybuild/2020/build/Rust/1.54.0/GCCcore-11.2.0/cargo &&  make  -j 48 " exited with exit code 2 and output:
downloading https://static.rust-lang.org/dist/2021-06-17/rust-std-1.53.0-x86_64-unknown-linux-gnu.tar.xz
######################################                        
== 2022-01-03 18:42:31,677 easyblock.py:307 INFO Closing log for application name Rust version 1.54.0
== 2022-01-03 18:42:31,677 build_log.py:265 INFO FAILED: Installation ended unsuccessfully (build directory: /easybuild/2020/build/Rust/1.54.0/GCCcore-11.2.0): build failed (first 300 chars): cmd "export CARGO_HOME=/easybuild/2020/build/Rust/1.54.0/GCCcore-11.2.0/cargo &&  make  -j 48 " exited with exit code 2 and output:
downloading https://static.rust-lang.org/dist/2021-06-17/rust-std-1.53.0-x86_64-unknown-linux-gnu.tar.xz
######################################                         (took 11 mins 55 secs)
== FAILED: Installation ended unsuccessfully (build directory: /easybuild/2020/build/Rust/1.54.0/GCCcore-11.2.0): build failed (first 300 chars): cmd "export CARGO_HOME=/easybuild/2020/build/Rust/1.54.0/GCCcore-11.2.0/cargo &&  make  -j 48 " exited with exit code 2 and output:
downloading https://static.rust-lang.org/dist/2021-06-17/rust-std-1.53.0-x86_64-unknown-linux-gnu.tar.xz
######################################                         (took 11 mins 55 secs)
== 2022-01-03 18:42:31,677 build_log.py:265 INFO Results of the build can be found in the log file(s) /tmp/eb-ipew6lae/easybuild-Rust-1.54.0-20220103.183035.LumvV.log
== Results of the build can be found in the log file(s) /tmp/eb-ipew6lae/easybuild-Rust-1.54.0-20220103.183035.LumvV.log
== 2022-01-03 18:42:31,678 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/2020/software/EasyBuild/4.5.0/lib/python3.8/site-packages/easybuild/base/exceptions.py:124 in __init__): build failed (first 300 chars): cmd "export CARGO_HOME=/easybuild/2020/build/Rust/1.54.0/GCCcore-11.2.0/cargo &&  make  -j 48 " exited with exit code 2 and output:
downloading https://static.rust-lang.org/dist/2021-06-17/rust-std-1.53.0-x86_64-unknown-linux-gnu.tar.xz
######################################                         (at easybuild/2020/software/EasyBuild/4.5.0/lib/python3.8/site-packages/easybuild/main.py:131 in build_and_install_software)
== 2022-01-03 18:42:31,679 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/2020/software/EasyBuild/4.5.0/lib/python3.8/site-packages/easybuild/base/exceptions.py:124 in __init__): Build of /easybuild/2020/software/EasyBuild/4.5.0/easybuild/easyconfigs/r/Rust/Rust-1.54.0-GCCcore-11.2.0.eb failed (err: 'build failed (first 300 chars): cmd "export CARGO_HOME=/easybuild/2020/build/Rust/1.54.0/GCCcore-11.2.0/cargo &&  make  -j 48 " exited with exit code 2 and output:\ndownloading https://static.rust-lang.org/dist/2021-06-17/rust-std-1.53.0-x86_64-unknown-linux-gnu.tar.xz\n\r######################################                        ') (at easybuild/2020/software/EasyBuild/4.5.0/lib/python3.8/site-packages/easybuild/main.py:169 in build_and_install_software)
ERROR: Build of /easybuild/2020/software/EasyBuild/4.5.0/easybuild/easyconfigs/r/Rust/Rust-1.54.0-GCCcore-11.2.0.eb failed (err: 'build failed (first 300 chars): cmd "export CARGO_HOME=/easybuild/2020/build/Rust/1.54.0/GCCcore-11.2.0/cargo &&  make  -j 48 " exited with exit code 2 and output:\ndownloading https://static.rust-lang.org/dist/2021-06-17/rust-std-1.53.0-x86_64-unknown-linux-gnu.tar.xz\n\r######################################                        ')

[branfosj]

('OpenSSL', '1.1', '', True), is either a wrapper to your system OpenSSL or an EB built OpenSSL. Which it is depends on if your system provides a suitable OpenSSL and, in this case, it would be looking for a suitable OpenSSL 1.1.

How recent was your OpenSSL EB package built? Does it have a lib/pkgconfig directory in it? If so, what is the contents of that?

For reference I have:

$ ls -l software/OpenSSL/1.1/lib/pkgconfig/
total 2
-rw-r--r-- 1 branfosj-admin build 350 Aug 19 11:50 libcrypto.pc
-rw-r--r-- 1 branfosj-admin build 367 Aug 19 11:50 libssl.pc
-rw-r--r-- 1 branfosj-admin build 282 Aug 19 11:50 openssl.pc

I know we fixed our install to include this in easybuilders/easybuild-easyblocks#2549 - Can you test by loading the OpenSSL modules and running pkg-config --libs --cflags openssl

[surak]

Turns out it's something else - I installed libssl-dev and got at the same place.

When running your command, I get this, both with and without our OpenSSL-1.1 loaded:

$ pkg-config --libs --cflags openssl
-lssl -lcrypto
$ ml OpenSSL
$ pkg-config --libs --cflags openssl
-lssl -lcrypto

[surak]

When was this integrated into eb? I'm using 4.5.0

[branfosj]

The fix I link to above was in 4.4.2, but if you had built OpenSSL before that then you would have to rebuild it to get that fix. easybuilders/easybuild-easyblocks#2597 may also be relevant on Ubuntu and that was in EB 4.5.0

[surak]

I had to build Rust with this: export OPENSSL_DIR=/easybuild/2020/software/OpenSSL/1.1, otherwise I stumbled on the same error. But I didn't rebuild openssl 1.1, which was from last june. I'm not sure this would've been fixed by those, but you probably know better. If so, this can be closed.

[branfosj]

$ pkg-config --libs --cflags openssl
-lssl -lcrypto
$ module load OpenSSL/1.1
$ pkg-config --libs --cflags openssl
-L/rds/bear-apps/2021a/EL8-has/software/OpenSSL/1.1/lib64 -lssl -lcrypto

The -L... bit should do the equivalent of the export that you did.

[surak]

Sure, but that doesn't make it an automated install, does it? My question is that if this will happen with other people.

[surak]

I tried again, on the same machines, forcing and install of OpenSSL and then that of Rust, without setting the OPENSSL_DIR, and it seems to have fixed the problem.

I would argue that we shouldn't rely on the system library when we can provide our own.

Worth noting that Ubuntu has libssl in /usr/lib/x86_64-linux-gnu/, and Rust hardcodes a /lib at the end of it, so I'm not even sure we can use libssl-dev as it is.