`MPIPreferences` persists through pre-compilation cache.

[t-bltg]

As discussed with @giordano on slack, and I scratched my head a long time on this one:

julia> using MPIPreferences
julia> MPIPreferences.use_jll_binary("MPICH_jll")  # persistent `.toml` written
[...]

# restart julia
julia> using MPICH_jll, OpenMPI_jll  # precompiles
julia> MPICH_jll.is_available()
true
julia> MPIPreferences.use_jll_binary("OpenMPI_jll")  # I now decide to change the MPI implementation used
[...]

# restart julia
julia> using OpenMPI_jll  # does NOT precompile
julia> OpenMPI_jll.is_available()  # confusing since ~/.julia/environments/v1.8/LocalPreferences.toml states `binary = "OpenMPI_jll"`
false

# delete ~/.julia/compiled/OpenMPI_jll, restart julia
julia> using OpenMPI_jll  # precompiles
julia> OpenMPI_jll.is_available()
true

I believe MPIPreferences should invalidate the pre-compilation cache, or we should do something about host_platform being persistent in the precompilation cache.
I shouldn't have to delete ~/.julia/compiled/OpenMPI_jll manually.

This leads to weird mix-ups such as:

julia> using MPICH_jll
julia> MPICH_jll.is_available()
false
julia> MPICH_jll.host_platform  # notice the mpi=openmpi :s
Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.8.1, libc=glibc, libgfortran_version=5.0.0, libstdcxx_version=3.4.30, mpi=openmpi}
julia> using OpenMPI_jll
julia> OpenMPI_jll.is_available()
false
julia> OpenMPI_jll.host_platform  # notice the mpi=mpich :/
Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.8.1, libc=glibc, libgfortran_version=5.0.0, libstdcxx_version=3.4.30, mpi=mpich}

Encountered in #639 (comment).

[simonbyrne]

Is it because MPIPreferences isn't and explicit dependency of OpenMPI_jll?

CC @vchuravy

[giordano]

Is it because MPIPreferences isn't and explicit dependency of OpenMPI_jll?

It actually is: https://github.com/JuliaBinaryWrappers/OpenMPI_jll.jl/blob/a4b7bf25afd5e5c134e8e4f9117d47c2cdf12ac2/Project.toml#L6

[simonbyrne]

Is it because it isn't loaded in https://github.com/JuliaBinaryWrappers/OpenMPI_jll.jl/blob/68a500cbd8a20f66b65acf0471c69bde0796b05e/.pkg/platform_augmentation.jl?

[simonbyrne]

Yes, changing that seems to fix it.

[simonbyrne]

@giordano @vchuravy can we just load MPIPreferences directly from the platform augmentation code?

[simonbyrne]

We can keep the discussion here, but the actual code that would need to be changed is here: https://github.com/JuliaPackaging/Yggdrasil/blob/cf634f846db9a448b747a7baaeafc6da5a874cf2/platforms/mpi.jl

[giordano]

Yes, changing that seems to fix it.

How did you change that? The code comes from https://github.com/JuliaPackaging/Yggdrasil/blob/d247ec95b1e3b5d903a133fe185eb29485a1608b/platforms/mpi.jl (I also just noticed indentation is off)

[simonbyrne]

I dev-ed OpenMPI_jll and modified it manually.

[simonbyrne]

What does

# Can't use Preferences since we might be running this very early with a non-existing Manifest

refer to? When would this occur?

[giordano]

I dev-ed OpenMPI_jll and modified it manually.

Yes, but what's the change you did? https://github.com/JuliaPackaging/Yggdrasil/blob/d247ec95b1e3b5d903a133fe185eb29485a1608b/platforms/mpi.jl#L6 says we may be running without a Manifest

[simonbyrne]

--- a/.pkg/platform_augmentation.jl
+++ b/.pkg/platform_augmentation.jl
@@ -1,4 +1,5 @@
     using Base.BinaryPlatforms
+    using MPIPreferences
         # Can't use Preferences since we might be running this very early with a non-existing Manifest
     MPIPreferences_UUID = Base.UUID("3da0fdf6-3ccc-4f1b-acd9-58baa6c99267")
     const preferences = Base.get_preferences(MPIPreferences_UUID)

[simonbyrne]

I don't know what "running without a Manifest" means in this case. I can't test that as I need a Manifest to use a deved package.

[t-bltg]

Maybe the following might cover # Can't use Preferences since we might be running this very early with a non-existing Manifest:

try
    using MPIPreferences
catch e
    @debug e
end

[simonbyrne]

Not sure? I don't know the situation in which the code would be run without a Manifest.

[vchuravy]

cc: @staticfloat

The reason why we can't load it in the augmentation, is that it is run during Pkg.add and there the Manifes.toml is only partially installed.