Package extensions not loaded when loading dependency from main environment?

[ranocha]

The new feature of package extensions is really nice. Thanks a lot to everybody involved!

The documentation describes package extensions basically as Requires.jl with additional benefits. However, it looks like package extensions are not loaded when loading their dependency from the main environment. I noticed this in SciML/RootedTrees.jl#111
There, I have the following code in the main package

if !isdefined(Base, :get_extension)
    using Requires: @require
end

# more code...

function _distinguishable_colors end

# more code...

function __init__()
    # more code...

    @static if !isdefined(Base, :get_extension)
        @require Plots="91a5bcdd-55d7-5caf-9e0b-520d859cae80" begin include("../ext/PlotsExt.jl") end
    end

    return nothing
end

and the following package extension code

module PlotsExt

if isdefined(Base, :get_extension)
    using Plots: Plots
else
    import ..Plots: Plots
end

using RootedTrees: RootedTrees

RootedTrees._distinguishable_colors(n) = Plots.distinguishable_colors(n)

end # module

The extension is loaded correctly when I set up an environment (Project.toml, Manifest.toml) including Plots.jl and RootedTrees.jl. However, when I activate the environment of RootedTrees.jl itself (julia --project=. in the development directory of this package) and load Plots.jl from my default environment, the package extension is not loaded properly. In this case, I get

julia> using Plots

julia> using RootedTrees

julia> RootedTrees._distinguishable_colors
_distinguishable_colors (generic function with 0 methods)

(RootedTrees) pkg> status --extensions
Project RootedTrees v2.18.1
Status `~/.julia/dev/RootedTrees/Project.toml`
  [23fbe1c1] Latexify v0.16.0
              ├─ DataFramesExt [DataFrames]
              └─ SymEngineExt [SymEngine]

using Julia v1.9.0-rc1. I get the expected result

julia> RootedTrees._distinguishable_colors
_distinguishable_colors (generic function with 1 method)

using Julia v1.8.5 or Julia v1.9.0-rc1 with a project environment including both RootedTrees and Plots.

[KristofferC]

As it is right now, this is intended. The reason for this is that there is no way to ensure that the Plots you are loading from the main environment is compatible with the compatibility constraints defined in your package. We try to make the active environment "sacred" in the sense that you will never load things from "below" that environment.

To give another example, let's say you have an extension that adds an optimization to a method (for example it adds a new method to StaticArrays). Now, we load (maybe an old version of) StaticArrays from the main environment and the extension activates. At this point, you can basically get arbitrary wrong results in your code because the version of StaticArrays is incompatible with the code in the extension. So the important code you have in your active environment (which might be for a scientific paper) gets corrupted by packages installed in the main environment. Bad.

[ranocha]

Okay, I see the reason for the current behavior. But I understood the documentation differently. In particular, it says

An extension will only be loaded if the extension dependencies are loaded from the same environment or environments higher in the environment stack than the package itself.

The part "environments higher in the environment stack" sounded to me like it supported loading something from the default environment. Could you please clarify this?

[KristofferC]

What is meant is that if you do the opposite thing, that is, load your package in Main, and load Plots in the active env, the extension (that is defined in the package loaded in Main) will be loaded.

[ranocha]

Thanks! I made a PR to the docs including your clarifications.