Generating haddocks sometimes produces "doc-index.json does not exist"

[Mikolaj]

Edit: this seems fixed in the current 3.8 branch (commit 575b216). To rule out that the way cabal is built affects it (the buggy version was built by gitlab CI with GHC 8.10.7, the new good version is built by hand using the default cabal.project with GHC 9.2.3), I've rebuilt 3.8.RC1 in the same way as the good version and it's still failing.

Edit2: The remaining problems are: we have no test that catches that and we have no clue how it got broken and if the fix is robust enough. Frankly, the only commit I can see that could fix that is the one that bumps process package version. Could a bug in process cause the haddock breakage? I guess it could, given that the haddock exe may be invoked using that machinery.

Describe the bug

See #hackage for an extensive testing session by multiple volunteers. It works with cabal 3.6.2, fails with cabal 3.8.0.RC1. Failures have been reproduced with both GHC 9.2.3 and GHC 9.4.1-alpha3. A log from a failed attempt with GHC 9.4.1-alpha3 and head.hackage:

cabal v2-haddock --builddir=dir --haddock-for-hackage --enable-doc --haddock-options=--quickjump --allow-newer cabal-install-solver
...
haddock: internal error: /home/mikolaj/.cabal/store/ghc-9.4.0.20220623/directory-1.3.7.1-1b6710e8174fc4ee7baecc57c46ce782434603c3b3b58295a4fcbdc139bbd395/share/doc/html/doc-index.json: openBinaryFile: does not exist (No such file or directory)

but at some point (without head.hackage) there was also

haddock: internal error: /home/mikolaj/.cabal/store/ghc-9.4.0.20220623/edit-distance-0.2.2.1-a7ea0a2f3ddd1328ec73dacef20a73fff1fc5fe66f9f3edd881ded197bf6a796/share/doc/html/doc-index.json: openBinaryFile: does not exist (No such file or directory)

so the problem is not limited to packages shipped with GHC that we override and generate new haddocks for.

We should probably rip out all PRs that touch haddock merged after 3.6.2, in turn, and check if this fixes itself. We should also add a test that catches this problem.

Edit: for the bug to manifest with the process package, it needs to be run on the current 3.8 branch (and probably master, too), not on RC1 tag of 3.8 branch.

System information
linux

[Mikolaj]

Edit3: this seems fixed in the current commit of 3.8 branch, so none of the suspects are to blame. Apologies. :)

So far I've found two PRs in that period that obviously touch haddock support: #7827 and #8162 in that chronological order.

Edit: #8162 is innocent, because not included in 3.8 release, neither 3.8.1.0 nor RC1.

Edit2: #7550 could be one more suspect.

[andreasabel]

This is the upstream issue (or rather PR):

• Check if doc-index.json exists before reading it haddock#1488

It seems that the problem was aggravated by including --enable-doc (the cause for haddock failure) in cabal haddock.
Personally, I have to revert to GHC 8.10.7 when using cabal haddock nowadays, because the haddock shipped with GHC 9.0/2 has this bug.

[ulysses4ever]

Is it a dup of #8104?

[Mikolaj]

I'm a bit lost, but probably not?