`nim doc --project` broken with imports below main project file or duplicate names

[timotheecour]

nim doc --project doesn't work when:

• there are relative imports eg import ".." / utils
• there are multiple modules in the project with the same basename

Example

nim doc --index:on --project --outdir:htmldocs /pathto/msub/main.nim

# /pathto/msub/main.nim:
from ".."/utils as m1 import nil 
from "."/utils as m2 import nil # note: would still be buggy if that clashing module is imported from somewhere else

# /pathto/msub/utils.nim
proc fun1*() = discard
# /pathto/utils.nim
proc fun2*() = discard

Current Output

• depending on which import comes first, (from "."/utils as m2 import nil or from ".."/utils as m1 import nil), you'll get one module generated or the other (clashing location)
• you'll get a 404 error when clicking on the import ../utils as the link will be wrong

Possible Solution

the most robust solution is to flatten the hierarchy in exactly the same way as done for nim c

find /tmp/d28/*.c
/tmp/d28/@m..@sthello2.nim.c
/tmp/d28/@mthello2.nim.c
/tmp/d28/@mthello3.nim.c
/tmp/d28/stdlib_io.nim.c
/tmp/d28/stdlib_system.nim.c

perhaps the particular mangling scheme can be adapted for urls but idea is identical.

note

One additional advantage of a flat hierarchy is that we'd be able to use relative instead of absolute href:

 href <link rel="stylesheet" type="text/css" href="nimdoc.css"> # instead of `href="/nimdoc.css"`

and it would work with both httpserver and open pathto/main.html
currently both absolute and relative hrefs would not work depending on situations

Additional Information

• nim devel 9fc04a5
• fix #12998 nim doc regression #13117 does not fix this, as this is a long pre-existing issue

[Araq]

Huh? It was designed not to flatten it.

[timotheecour]

I've suggested a way that should handle any valid nim project (including with duplicate module names and .. in import paths), reusing/adapting existing fakePackageName / demanglePackageName from compiler/packagehandling.nim ; what do you suggest instead? right now docgen can't handle above case

[Araq]

Oh I'm sure it's currently broken but I've regretted the flattening we do for nimcache so it would be nice if the docgen could simply replicate the hierarchy -- maybe introduce helper names for file name clashes.

[timotheecour]

I don't know how you can reliably handle valid nim projects (which allow .., absolute paths, and duplicate module names) with replicated hierarchy, unless you're reproducing the absolute path rooted under nimcache, eg:

# in mypkg/sub/main.nim:
import ".." / utils as util1
import "." / utils as util2
import pkg/utils as util3
import "/some/absolute/path/utils"

with

--nimcache:/pathto/nimcache --path:/Users/timothee/git_clone/mypkg/src/ --path:/Users/timothee/.nimble/pkgs/utils-1.9.0/ 

producing:

/pathto/nimcache/Users/timothee/git_clone/mypkg/src/mypkg/sub/main.c
/pathto/nimcache/Users/timothee/git_clone/mypkg/src/mypkg/sub/utils.c
/pathto/nimcache/Users/timothee/git_clone/mypkg/src/mypkg/utils.c
/pathto/nimcache/some/absolute/path/utils.c

this is much worse IMO that the simple mangling scheme we currently have; for many reasons (eg rm -rf now needed instead of rm *.c; moving a project causes contents of nimcache to change etc)

[Araq]

Ok, go for it, better ugly and correct than beautiful and wrong.

[timotheecour]

after discussions with @Clyybber etc here's another idea:

• NEW --project gets an optional --project:projectbase dir name argument indicating the base with respect to which relative paths are computed
• a naked --project means: the project base is assumed to be the parentDir of project file being compiled (like before)
• NEW any relative path starting with .. will be skipped, and maybe generates a warning; (before: a file would be generated OUTSIDE of specified outdir, which is bad/dangerous)
• as before, paths in other nimble packages than the one compiled are skipped (and no warning shown)

example

suppose you have a nimble package mypkg (but doesn't have to be a nimble package, just an example):

# in mypkg/src/mypkg/sub/main.nim
import ../utils # or: import mypkg/utils which should be the same with `--path:mypkg/src/`
import ../../outside_of_package
import pkg/regex

nim doc —project:mypkg/src/mypkg --outdir:docs mypkg/src/mypkg/sub/main.nim
generates:

• docs/sub/main.html
• docs/utils.html
• it does not generate docs/../outside_of_package.html and instead shows a warning
• it does not generated any file for pkg/regex since it's in a separate nimble package

if —project is passed instead of —project:mypkg/src, the project base is assumed to be the parentDir of project file, ie: mypkg/src/mypkg/sub/

implementation difficulty

easy

note

perhaps we could also support —project:$projectbase, which would auto-compute the project base, maybe using #10530 or similar

[Araq]

Really good, I like it.