Oscar / Polymake is looking for mpfr in wrong spot on arm64 mac

[tom111]

Hi there,

when precompiling Oscar on a freshly installed arm64 Mac I run into this:

ERROR: LoadError: InitError: "/Users/tom/.julia/artifacts/e75f8bd18c56df0983c9054335c32294bb1eebac/share/polymake/perllib/Polymake/Core/CPlusPlus.pm", line 1785: Can't load shared module /Users/tom/.julia/artifacts/e75f8bd18c56df0983c9054335c32294bb1eebac/lib/polymake/lib/common.bundle: dlopen(/Users/tom/.julia/artifacts/e75f8bd18c56df0983c9054335c32294bb1eebac/lib/polymake/lib/common.bundle, 0x0009): Library not loaded: @rpath/libmpfr.6.dylib
  Referenced from: <08F73057-714A-3F9D-A853-9DE02FE45C14> /Users/tom/.julia/artifacts/e75f8bd18c56df0983c9054335c32294bb1eebac/lib/polymake/lib/common.bundle
  Reason: tried: '/Users/tom/temp/prefix/lib/libmpfr.6.dylib' (no such file)
'/Users/tom/.julia/artifacts/e75f8bd18c56df0983c9054335c32294bb1eebac/lib/polymake/lib/../../libmpfr.6.dylib' (no such file), 
[...]
'/opt/homebrew/Cellar/julia/1.8.5/lib/libmpfr.6.dylib' (no such file), '/usr/local/lib/libmpfr.6.dylib' (no such file), '/usr/lib/libmpfr.6.dylib' (no such file, not in dyld cache)

Possible location for mpfr on this machine are:

~ locate libmpfr
/Applications/Mathematica.app/Contents/SystemFiles/Libraries/MacOSX-ARM64/libmpfr.dylib
/Applications/Mathematica.app/Contents/SystemFiles/Libraries/MacOSX-x86-64/libmpfr.dylib
/opt/homebrew/Caskroom/miniconda/base/pkgs/mpfr-4.2.0-he09a6ba_0/lib/libmpfr.6.dylib
/opt/homebrew/Caskroom/miniconda/base/pkgs/mpfr-4.2.0-he09a6ba_0/lib/libmpfr.dylib
/opt/homebrew/Cellar/julia/1.8.5/lib/julia/libmpfr.dylib
/opt/homebrew/Cellar/mpfr/4.2.0/lib/libmpfr.6.dylib
/opt/homebrew/Cellar/mpfr/4.2.0/lib/libmpfr.a
/opt/homebrew/Cellar/mpfr/4.2.0/lib/libmpfr.dylib
/opt/homebrew/lib/libmpfr.6.dylib
/opt/homebrew/lib/libmpfr.a
/opt/homebrew/lib/libmpfr.dylib

I would suggest looking for it in /opt/homebrew/lib/. Note that the one inside the Julia install is called libmpfr.dylib while the code above is looking for libmpfr6.dylib.

[benlorenz]

The official julia dmg contains both, a symlink and the versioned library:

lorenz@munk ~ % ll /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/*mpfr*
-rwxr-xr-x@ 1 support  539648 Jan  8 09:45 /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libmpfr.6.dylib*
lrwxr-xr-x@ 1 support      15 Jan 10 13:42 /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libmpfr.dylib@ -> libmpfr.6.dylib

We don't really support anything other than the official julia packages, as unfortunately the BinaryBuilder system does not play well with other installations.
To some extent it is possible to override the locations for the jll packages, some details are available here: https://docs.binarybuilder.org/stable/jll/#Non-dev'ed-JLL-packages
With this it might be possible to specify that MPFR is installed at /opt/homebrew, though I am not totally sure if this works for so-called stdlib JLL packages that are shipped directly with julia.

I would say it is a bug in the homebrew package for julia if it ships a libmpfr.dylib (which might be a symlink) but no versioned library (edit: or at least a symlink for it). When building the polymake binaries we just pass -lmpfr, that needed entry is then generated by the linker with the soname of the library (which contains the version).
Note that there were some changes for julia 1.9 and newer that it will also internally use a versioned library instead of the unversioned one, see JuliaLang/julia#47676 (the ticket als applies to MPFR even though it is not mentioned in the title).

[tom111]

You are right that Oscar's Polymake expects libraries that are independent of Julia somehwere inside Julia (is that the right way to go?). But I think it is not about versioned vs. not versioned. I put a symlink to libmpfr.6.dylib in one of the locations it mentions and then it does not find gmp...

ERROR: LoadError: InitError: "/Users/tom/.julia/artifacts/e75f8bd18c56df0983c9054335c32294bb1eebac/share/polymake/perllib/Polymake/Core/CPlusPlus.pm", line 1785: Can't load shared module /Users/tom/.julia/artifacts/e75f8bd18c56df0983c9054335c32294bb1eebac/lib/polymake/lib/common.bundle: dlopen(/Users/tom/.julia/artifacts/e75f8bd18c56df0983c9054335c32294bb1eebac/lib/polymake/lib/common.bundle, 0x0009): Library not loaded: @rpath/libgmp.10.dylib
  Referenced from: <08F73057-714A-3F9D-A853-9DE02FE45C14> /Users/tom/.julia/artifacts/e75f8bd18c56df0983c9054335c32294bb1eebac/lib/polymake/lib/common.bundle
  Reason: tried: '/Users/tom/temp/prefix/lib/libgmp.10.dylib' (no such file), '/libgmp.10.dylib' (no such file), '/workspace/destdir/lib/libgmp.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/workspace/destdir/lib/libgmp.10.dylib' (no such file), '/workspace/destdir/deps/Perl_jll/lib/libgmp.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/workspace/destdir/deps/Perl_jll/lib/libgmp.10.dylib' (no such file), '/workspace/destdir/deps/GMP_jll/lib/libgmp.10.dylib' (no such file)

• I can check on an Intel Mac tomorrow, but I believe it was working just fine there with the homebrew'ed Julia-1.8.5
• Why does it search all those funny "cryptexes" directories on my machine? Something is wrong there. This seem like leftovers from the build machine...?

[benlorenz]

Regarding these paths:

  Reason: tried: '/Users/tom/temp/prefix/lib/libgmp.10.dylib' (no such file), '/libgmp.10.dylib' (no such file), '/workspace/destdir/lib/libgmp.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/workspace/destdir/lib/libgmp.10.dylib' (no such file), '/workspace/destdir/deps/Perl_jll/lib/libgmp.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/workspace/destdir/deps/Perl_jll/lib/libgmp.10.dylib' (no such file), '/workspace/destdir/deps/GMP_jll/lib/libgmp.10.dylib' (no such file)

The first path /Users/tom/temp/prefix is confusing as it does not seem to be related to homebrew and it also doesn't look like any path julia would be using (during build or at runtime). You don't have any DYLD_LIBRARY_PATH or DYLD_FALLBACK_LIBRARY_PATH set?

The Cryptexes stuff seems to be another new security feature in macOS 13. (Such things also tend to break other software from time to time ...)

You are right that Oscar's Polymake expects libraries that are independent of Julia somehwere inside Julia (is that the right way to go?).

I would say that this is not really the way to go (I would prefer to install / manage the dependencies myself; at least on linux), but this is the way the julia developers have written their packaging system. Julia ships various libraries and packages that need those will use exactly those versions, both during build in their sandbox and then during runtime.

But I think it is not about versioned vs. not versioned. I put a symlink to libmpfr.6.dylib in one of the locations it mentions and then it does not find gmp...

ERROR: LoadError: InitError: "/Users/tom/.julia/artifacts/e75f8bd18c56df0983c9054335c32294bb1eebac/share/polymake/perllib/Polymake/Core/CPlusPlus.pm", line 1785: Can't load shared module /Users/tom/.julia/artifacts/e75f8bd18c56df0983c9054335c32294bb1eebac/lib/polymake/lib/common.bundle: dlopen(/Users/tom/.julia/artifacts/e75f8bd18c56df0983c9054335c32294bb1eebac/lib/polymake/lib/common.bundle, 0x0009): Library not loaded: @rpath/libgmp.10.dylib
  Referenced from: <08F73057-714A-3F9D-A853-9DE02FE45C14> /Users/tom/.julia/artifacts/e75f8bd18c56df0983c9054335c32294bb1eebac/lib/polymake/lib/common.bundle
  Reason: tried: '/Users/tom/temp/prefix/lib/libgmp.10.dylib' (no such file), '/libgmp.10.dylib' (no such file), '/workspace/destdir/lib/libgmp.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/workspace/destdir/lib/libgmp.10.dylib' (no such file), '/workspace/destdir/deps/Perl_jll/lib/libgmp.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/workspace/destdir/deps/Perl_jll/lib/libgmp.10.dylib' (no such file), '/workspace/destdir/deps/GMP_jll/lib/libgmp.10.dylib' (no such file)

The place where this library should come from is ${julia_prefix}/lib/julia but this path is not embedded anywhere in the polymake binaries.
We encode only the name (so-name on linux, install-name or dylib-id on macos) of the library, which usually is the versioned name and that is then used for runtime lookup.
Non-versioned libraries (libname.so or libname.dylib) are used when linking to point the linker to the correct library. Then the linker will use the name embedded in the library to embed it in the new binary.

In this case the name of the library is @rpath/libmpfr.6.dylib:

% ll /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libmpfr.dylib
lrwxr-xr-x@ 1 support  15 Jan 10 13:42 /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libmpfr.dylib@ -> libmpfr.6.dylib

% otool -D /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libmpfr.dylib
/Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libmpfr.dylib:
@rpath/libmpfr.6.dylib

Polymake is not really looking anywhere and not expecting any specific paths for these libraries:

$ llvm-objdump --dylibs-used --rpaths --macho common.bundle 
common.bundle:
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 904.4.0)
	@rpath/libflint-17.dylib (compatibility version 17.0.0, current version 17.0.0)
	@rpath/libmpfr.6.dylib (compatibility version 8.0.0, current version 8.1.0)
	@rpath/libgmp.10.dylib (compatibility version 15.0.0, current version 15.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
	@rpath/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
/workspace/destdir/lib
/workspace/destdir/deps/Perl_jll/lib
/workspace/destdir/deps/GMP_jll/lib
/workspace/destdir/deps/MPFR_jll/lib
/workspace/destdir/deps/FLINT_jll/lib
@loader_path/../..

This is from the aarch64-apple-darwin tarball. The /workspace/destdir rpaths are indeed leftovers from the build machine but that should not really hurt.
The important part are the library names like @rpath/libmpfr.6.dylib which matches the name of corresponding library in the julia package.

The way the jlls work is that all dependencies for a library are explicitly loaded (via dlopen) beforehand using the dependency information from the packages.
In this case polymake_jll depends on MPFR_jll and GMP_jll. Loading these should make sure that the correct libmpfr and libgmp libraries are already in the process.
If this is the case then the dlopen step for e.g. common.bundle should not need to find any libraries in the filesystem.

Since GMP and MPFR are stdlibs (and dependencies of julia itself) the libraries should even be in the process as soon as julia is started:

% julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.5 (2023-01-08)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Libdl

julia> filter(contains("gmp"),dllist())
1-element Vector{String}:
 "/Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libgmp.10.dylib"

julia> filter(contains("mpfr"),dllist())
1-element Vector{String}:
 "/Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libmpfr.6.dylib"

[tom111]

OK. Thanks for the explanations. I now agree that there is something wrong with the julia homebrew package. That path @rpath does not resolve correctly, and many libraries are not found. For example, I tried just adding /opt/homebrew/bin to the DYLD_LIBRARY_PATH. The result is this:

julia> using Oscar
[ Info: Precompiling Oscar [f1435218-dba5-11e9-1e4d-f1a5fab5fc13]
┌ Warning: CHOLMOD version incompatibility
│
│ Julia was compiled with CHOLMOD version 3.0.14. It is
│ currently linked with version 4.0.3.
│ This might cause Julia to terminate when working with
│ sparse matrix factorizations, e.g. solving systems of
│ equations with \.
│
│ It is recommended that you use Julia with the same major
│ version of CHOLMOD as the one used during the build, or
│ download the generic binaries from www.julialang.org,
│ which ship with the correct versions of all dependencies.
└ @ SuiteSparse.CHOLMOD /private/tmp/julia-20230204-9518-1tfk4ma/julia-1.8.5/usr/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:186
┌ Error: Error during initialization of module CHOLMOD
│   exception =
│    could not load symbol "SuiteSparse_config":
│    dlsym(0x94e3caa0, SuiteSparse_config): symbol not found
│    Stacktrace:
│     [1] __init__()
│       @ SuiteSparse.CHOLMOD /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:237
└ @ SuiteSparse.CHOLMOD /private/tmp/julia-20230204-9518-1tfk4ma/julia-1.8.5/usr/share/julia/stdlib/v1.8/SuiteSparse/src/cholmod.jl:245
ERROR: LoadError: InitError: could not load library "/Users/tom/.julia/artifacts/808bfb97cb3cba30559230a33f6cc4438bc8598d/lib/libcalcium.dylib"
dlopen(/Users/tom/.julia/artifacts/808bfb97cb3cba30559230a33f6cc4438bc8598d/lib/libcalcium.dylib, 0x0001): Library not loaded: @rpath/libarb-2.14.0.dylib
  Referenced from: <9B579675-75F1-32C7-8A44-0A424ECE0D9F> /Users/tom/.julia/artifacts/808bfb97cb3cba30559230a33f6cc4438bc8598d/lib/libcalcium-0.4.1.dylib
  Reason: tried: '/opt/homebrew/lib/libarb-2.14.0.dylib' (no such file), '/libarb-2.14.0.dylib' (no such file), '/Users/tom/.julia/artifacts/808bfb97cb3cba30559230a33f6cc4438bc8598d/lib/./libarb-2.14.0.dylib' (no such file), '/Users/tom/.julia/artifacts/808bfb97cb3cba30559230a33f6cc4438bc8598d/lib/./libarb-2.14.0.dylib' (no such file), '/opt/homebrew/Cellar/julia/1.8.5/lib/julia/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/Cellar/julia/1.8.5/lib/julia/libarb-2.14.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/Cellar/julia/1.8.5/lib/julia/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/opt/curl/lib/libarb-2.14.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/curl/lib/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/opt/mbedtls@2/lib/libarb-2.14.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/mbedtls@2/lib/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/opt/openblas/lib/libarb-2.14.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/openblas/lib/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/opt/gcc/lib/gcc/current/libarb-2.14.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/gcc/lib/gcc/current/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/lib/libarb-2.14.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/lib/libarb-2.14.0.dylib' (no such file), '/usr/lib/libarb-2.14.0.dylib' (no such file, not in dyld cache), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/Cellar/julia/1.8.5/lib/libarb-2.14.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS@rpath/libarb-2.14.0.dylib' (no such file), '/Users/tom/.julia/artifacts/808bfb97cb3cba30559230a33f6cc4438bc8598d/lib/./libarb-2.14.0.dylib' (no such file), '/Users/tom/.julia/artifacts/808bfb97cb3cba30559230a33f6cc4438bc8598d/lib/./libarb-2.14.0.dylib' (no such file), '/opt/homebrew/Cellar/julia/1.8.5/lib/julia/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/Cellar/julia/1.8.5/lib/julia/libarb-2.14.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/Cellar/julia/1.8.5/lib/julia/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/opt/curl/lib/libarb-2.14.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/curl/lib/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/opt/mbedtls@2/lib/libarb-2.14.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/mbedtls@2/lib/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/opt/openblas/lib/libarb-2.14.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/openblas/lib/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/opt/gcc/lib/gcc/current/libarb-2.14.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/gcc/lib/gcc/current/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/lib/libarb-2.14.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/lib/libarb-2.14.0.dylib' (no such file), '/usr/lib/libarb-2.14.0.dylib' (no such file, not in dyld cache), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/libarb-2.14.0.dylib' (no such file), '/opt/homebrew/Cellar/julia/1.8.5/lib/libarb-2.14.0.dylib' (no such file)

It is also possible to install the dmg version of Julia with homebrew with brew install --cask julia. Using this it works now and then I checked on my Intel Macs and I have been using the cask there too. So maybe it's not about the architecture after all.

[fingolfin]

There is a reason the OSCAR install instructions say this:

Linux and macOS users should generally not install the Julia version provided by their package manager (e.g., apt, pac, dnf, homebrew, ...), as in many cases, these Julia version are either outdated, or crippled, or both.

Indeed, the "crippled" bit holds for (at least) the Julia versions in Conda, Debian, and Homebrew; it also used to be crippled in Arch (and still might be, I didn't check recently).

These days I recommend installing Julia via juliaup as that also makes it quite convenient to update it and to install multiple versions in parallel.

[kalmarek]

@fingolfin It is still unsupported on Arch, linking to system blas, llvm and whatnot;
There's a note on Arch wiki about this