Revise does not apply functions in the two-argument form of `include`

[ranocha]

The two-argument form include(mapexpr::Function, path::AbstractString) allows transforming the code in the file at path. However, Revise does not seem to apply the same transformation mapexpr when re-loading code that is included using this form. For example

julia> using Pkg; Pkg.activate(temp=true); Pkg.add("Revise")
  Activating new environment at `/tmp/jl_D95G2q/Project.toml`
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General`
   Resolving package versions...
    Updating `/tmp/jl_D95G2q/Project.toml`
  [295af30f] + Revise v3.1.17
[...]

julia> "BrokenModule" |> mkdir |> cd

julia> "src" |> mkdir |> cd

julia> open("BrokenModule.jl", "w") do io
           println(io, """
               module BrokenModule

               include(expr -> quote @fastmath begin \$expr end end, "broken_a.jl")

               end
           """)
       end

julia> open("broken_a.jl", "w") do io
           println(io, """
               foo(a, b, c) = a * b + c
           """)
       end

julia> cd(dirname(pwd()))

julia> open("Project.toml", "w") do io
           println(io, """
               name = "BrokenModule"
               uuid = "79e551b4-d023-11eb-261f-b9feb8f119ff"
               authors = ["Someone"]
               version = "0.0.1"

               [compat]
               julia = "1.6"
           """)
       end

julia> run(`git init`)
Initialized empty Git repository in ~/.julia/dev/BrokenModule/.git/
Process(`git init`, ProcessExited(0))

julia> run(`git add .`)
Process(`git add .`, ProcessExited(0))

julia> run(`git commit -m 'initial commit'`)
[master (root-commit) 78b2121] initial commit
 3 files changed, 16 insertions(+)
 create mode 100644 Project.toml
 create mode 100644 src/BrokenModule.jl
 create mode 100644 src/broken_a.jl
Process(`git commit -m 'initial commit'`, ProcessExited(0))

julia> Pkg.add(url=pwd())
    Updating git-repo `~/.julia/dev/BrokenModule`
   Resolving package versions...
    Updating `/tmp/jl_D95G2q/Project.toml`
  [79e551b4] + BrokenModule v0.0.1 `~/.julia/dev/BrokenModule#master`
    Updating `/tmp/jl_D95G2q/Manifest.toml`
  [79e551b4] + BrokenModule v0.0.1 `~/.julia/dev/BrokenModule#master`

julia> Pkg.develop("BrokenModule")
Path `~/.julia/dev/BrokenModule` exists and looks like the correct package. Using existing path.
   Resolving package versions...
    Updating `/tmp/jl_D95G2q/Project.toml`
  [79e551b4] ~ BrokenModule v0.0.1 `~/.julia/dev/BrokenModule#master` ⇒ v0.0.1 `~/.julia/dev/BrokenModule`
    Updating `/tmp/jl_D95G2q/Manifest.toml`
  [79e551b4] ~ BrokenModule v0.0.1 `~/.julia/dev/BrokenModule#master` ⇒ v0.0.1 `~/.julia/dev/BrokenModule`

julia> using Revise, BrokenModule
[ Info: Precompiling BrokenModule [79e551b4-d023-11eb-261f-b9feb8f119ff]

julia> BrokenModule.foo(1.0, 2.0, 3.0) # This used FMA, as desired when using `@fastmath`
5.0

julia> @code_native debuginfo=:none syntax=:intel BrokenModule.foo(1.0, 2.0, 3.0)
        .text
        vfmadd213sd     xmm0, xmm1, xmm2        # xmm0 = (xmm1 * xmm0) + xmm2
        ret
        nop     word ptr cs:[rax + rax]

julia> open(joinpath(dirname(pathof(BrokenModule)), "broken_a.jl"), "w") do io
           println(io, """
               foo(a, b, c) = a + b * c
           """)
       end

julia> BrokenModule.foo(1.0, 2.0, 3.0) # This doesn't use FMA although we have `@fastmath`
7.0

julia> @code_native debuginfo=:none syntax=:intel BrokenModule.foo(1.0, 2.0, 3.0)
        .text
        vmulsd  xmm1, xmm1, xmm2
        vaddsd  xmm0, xmm1, xmm0
        ret
        nop     dword ptr [rax]

The expected output of the last line would be something like

julia> @fastmath foo(a, b, c) = a + b * c
foo (generic function with 1 method)

julia> @code_native debuginfo=:none syntax=:intel foo(1.0, 2.0, 3.0)
        .text
        vfmadd231sd     xmm0, xmm2, xmm1        # xmm0 = (xmm2 * xmm1) + xmm0
        ret
        nop     word ptr cs:[rax + rax]

Hence, this demonstrates that Revise did not apply the transformation expr -> quote @fastmath begin $expr end end passed to include.

[sloede]

Is this an issue that seems easily fixable or is this a tougher nut to crack? It would be really nice to be able to use the 2-argument include with Revise :-)

[timholy]

It's not trivial, but not impossible either. Someone would need to modify Revise's internal data structures to keep track of a mapexpr for each source file. I see some links to Trixi above, is that the only package that uses this feature?

[ranocha]

We wanted to use this feature of include but discovered the issue described above while experimenting with it. Our current workaround is to basically wrap each file in a macro to get the same results while Revise works fine. It seems to be more clean if we could use the two-argument form of include instead.

[MilesCranmer]

Friendly ping on this; would you be able to add this @timholy? Seems like it's the 2nd-most requested feature (and of those top two, perhaps requires the least changes?)

It seems like this might be the reason that Revise.jl doesn't work with DispatchDoctor.jl in #820?

[MilesCranmer]

I tried to get started on this by modifying

Revise.jl/src/packagedef.jl

Line 79 in f4edc99

const watched_files = Dict{String,WatchList}()

to be Dict{String,Tuple{WatchList,Union{Function,Nothing}}}() and then

Revise.jl/src/packagedef.jl

Line 889 in f4edc99

function track(mod::Module, file; mode=:sigs, kwargs...)

to take a required mapexpr::Union{Function,Nothing}=nothing argument. And likewise for includet

However I quickly got lost. @timholy I think you might need to help with this one... Sorry I couldn't be more useful! Would be really nice to have this for compatibility with DispatchDoctor.jl as apparently the precompile times become slower with it.

[timholy]

I don't have time for general Julia development very often these days. This is a bigger project than I can tackle currently.

[MilesCranmer]

@timholy perhaps you could simply point out specifically what needs to be changed for this to work?

[timholy]

It's not so obvious that I can confidently predict it. You want to grab the expression that gets produced by mapexpr and add that to Revise's internal cache. While complex, the internals are described pretty thoroughly: https://timholy.github.io/Revise.jl/stable/internals/ You have likely already read that, but if not, start there.

[disberd]

@MilesCranmer i would be willing to help if possible in trying to achieve this, I have been trying to understand better revise internals for using it in a package of mine and I would love to have this functionality as well.

That being said, I believe an additional challenge is that the specific mapexpr to use is not easy to extract as it's not known by the include_callback called during loading in Julia and it's not tracked in the cache of the precompiled module, so it would have to be extracted after loading from the module itself.

In any case we could have a chat if you are interested in trying to collaborate on this (I'll also be at juliacon in Eindhoven to meet in person if you'll attend)

[timholy]

Yeah, you'll need to store the mapexpr for each included file in some (new) cache in Revise. The handling of include directives is in src/lowered.jl, so that's the place to grab it from.