Fix haddock via Setup.hs with internal libraries #7827

brandonchinn178 · 2021-11-19T05:10:05Z

Fixes #1919

Patches conform to the coding conventions.
Any changes that could be relevant to users have been recorded in the changelog.
The documentation has been updated, if necessary.

brandonchinn178 · 2021-11-19T06:05:02Z

I wrote a test that failed and then made a fix that made it pass. I'm not sure if this is the correct fix, though. Would love guidance on an alternate fix

Mikolaj · 2021-11-19T09:44:39Z

Hi @brandonchinn178! Thank you for the test that shows there is a problem and for the fix. Does anybody have any comments?

The CI failure seems to be not a random breakage, but related to haddocks, so probably there is some more work to do.

brandonchinn178 · 2021-11-20T02:07:50Z

Sorry forgot to commit the output for cabal-testsuite with --with-cabal specified. Updated.

Mikolaj

LGTM. Let's hope this fixes the stack use case.

brandonchinn178 · 2021-11-22T18:00:19Z

@Mikolaj Thanks! Is there anything else I need to do to get this merged?

Mikolaj · 2021-11-22T18:14:07Z

@brandonchinn178: lure one more reviewer with cookies.

brandonchinn178 · 2021-11-22T19:05:02Z

@hsenag @int-e @mietek it seems like you three are the last contributors to comment on the issue. Any of you able/willing to review this?

hsenag · 2021-11-22T22:35:15Z

@brandonchinn178 I tried your branch and I still get the error, but I'm not sure if I did the right things to actually use the dev cabal-install/Cabal. I pointed at eb2dd41 in a source-repository-package stanza in cabal.project, as well as building cabal-install from your branch and running that binary.

Regardless, I'm not really able to review your code itself, but once I can verify it fixes my problem I'd be happy to give a thumbs-up on that basis.

Mikolaj · 2021-11-22T22:48:16Z

@hsenag: I'm afraid you might have used Cabal library from hackage. That's why README says to run cabal install --project-file=cabal.project.release cabal-install and the project file mentions packages: Cabal/ (which needs to be from the same branch/commit). I think the easiest way is to clone Brandon's fork and branch, e.g., with

git checkout -b brandonchinn178-chinn/haddock master
git pull https://github.com/brandonchinn178/cabal.git chinn/haddock

and run the cabal command above in there.

Mikolaj · 2021-11-29T10:02:19Z

@hsenag: ping?

hsenag · 2021-11-29T18:44:55Z

@Mikolaj sorry for the delay, I've kicked off going through the steps now. Feel free to ping me again in a day or two if I don't report back.

hsenag · 2021-11-29T21:41:17Z

I'm still seeing the failure. What I've done this time:

Checked out eb2dd41 into ~/temp/cabal.
In ~/temp/cabal, run cabal install --project-file=cabal.project.release cabal-install --installdir=../bin
In my project folder (a recent version of darcs), added ~/temp/cabal/Cabal to packages
In my project folder, blew away dist-newstyle then ran ~/temp/bin/cabal v2-haddock test:darcs-test --enable-tests --enable-documentation --haddock-tests

and I still get

Running Haddock on test suite 'darcs-test' for darcs-2.17.1..
setup: internal error when calculating transitive package dependencies.
Debug info: []

Mikolaj · 2021-11-29T21:49:09Z

Oh.

So we have a repro. Thank you @hsenag.

@brandonchinn178: are you able to reproduce the failure with darcs-2.17.1? Would you like me to try?

brandonchinn178 · 2021-11-29T21:54:20Z

I use Stack primarily, so I'm not sure how much help I'll be in debugging Cabal invocations. If you could take a look, that'd be great. Thanks!

hsenag · 2021-11-30T12:08:33Z

FYI I've now verified that this also happens with darcs-2.16.4 from hackage. Steps were to cabal unpack it, then add a cabal.project containing packages: . ~/temp/cabal/Cabal, then run the v2-haddock command.

Mikolaj · 2021-11-30T17:39:41Z

@hsenag: I confirm your results, both with and without the cabal.project file.

However, I see Build-Type: Custom and I know custom setup files are usually a different can of worms. BTW, is the custom setup still necessary in the darcs package? Is there a ticket somewhere specifically about similar problems with Custom setups?

@brandonchinn178: did you intend to fix this for build-type: Simple or Build-Type: Custom or both? Can the fix be extended to Custom?

Here's the -v2 log of the run without cabal.project:

~/ttt/darcs-2.16.4$ ~/temp/bin/cabal -v2 v2-haddock test:darcs-test --enable-tests --enable-documentation --haddock-tests
this build was affected by the following (project) config files:
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built:
 - darcs-2.16.4 *test (first run)
creating /home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build
creating /home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/tmp
creating
/home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4
creating
/home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/cache
whenReRegister: nothing to register
creating
/home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/setup
/home/mikolaj/.ghcup/bin/ghc --make -fbuilding-cabal-package -odir /home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/setup -hidir /home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/setup -i -i/home/mikolaj/ttt/darcs-2.16.4/. -optP-include -optP/home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/setup/setup_macros.h -hide-all-packages -no-user-package-db -package-db /home/mikolaj/.cabal/store/ghc-8.10.7/package.db -package-db /home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/packagedb/ghc-8.10.7 -package-id Cabal-3.2.1.0 -package-id base-4.14.3.0 -package-id directory-1.3.6.0 -package-id filepath-1.4.2.1 -package-id process-1.6.13.2 /home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/setup/setup.hs -o /home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/setup/setup -threaded
/home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/setup/setup
haddock --verbose=2
--builddir=/home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4
--tests test:darcs-test
darcs log -a --from-tag 2.16.4 --count
creating
/home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/build/global-autogen
/home/mikolaj/.ghcup/bin/haddock --ghc-version
/home/mikolaj/.ghcup/bin/ghc --print-libdir -hide-all-packages
/home/mikolaj/.ghcup/ghc/8.10.7/bin/ghc-pkg-8.10.7 init /home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/package.conf.inplace
creating
/home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/build/darcs-test
creating
/home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/build/darcs-test/autogen
creating
/home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/build/darcs-test/autogen
Preprocessing test suite 'darcs-test' for darcs-2.16.4..
Running Haddock on test suite 'darcs-test' for darcs-2.16.4..
CallStack (from HasCallStack):
  $, called at libraries/Cabal/Cabal/Distribution/Simple/Haddock.hs:255:13 in Cabal-3.2.1.0:Distribution.Simple.Haddock
  haddock, called at libraries/Cabal/Cabal/Distribution/Simple.hs:636:35 in Cabal-3.2.1.0:Distribution.Simple
  haddockHook, called at libraries/Cabal/Cabal/Distribution/Simple/UserHooks.hs:139:5 in Cabal-3.2.1.0:Distribution.Simple.UserHooks
  haddockHook, called at libraries/Cabal/Cabal/Distribution/Simple/UserHooks.hs:139:5 in Cabal-3.2.1.0:Distribution.Simple.UserHooks
  cmd_hook, called at libraries/Cabal/Cabal/Distribution/Simple.hs:497:9 in Cabal-3.2.1.0:Distribution.Simple
  cmd_hook, called at libraries/Cabal/Cabal/Distribution/Simple.hs:516:4 in Cabal-3.2.1.0:Distribution.Simple
  hookedActionWithArgs, called at libraries/Cabal/Cabal/Distribution/Simple.hs:495:5 in Cabal-3.2.1.0:Distribution.Simple
  hookedAction, called at libraries/Cabal/Cabal/Distribution/Simple.hs:360:3 in Cabal-3.2.1.0:Distribution.Simple
  haddockAction, called at libraries/Cabal/Cabal/Distribution/Simple.hs:187:50 in Cabal-3.2.1.0:Distribution.Simple
  defaultMainHelper, called at libraries/Cabal/Cabal/Distribution/Simple.hs:125:42 in Cabal-3.2.1.0:Distribution.Simple
  $, called at /home/mikolaj/ttt/darcs-2.16.4/dist-newstyle/build/x86_64-linux/ghc-8.10.7/darcs-2.16.4/setup/setup.hs:36:8 in main:Main
setup: internal error when calculating transitive package dependencies.
Debug info: []

CallStack (from HasCallStack):
  withMetadata, called at src/Distribution/Simple/Utils.hs:379:14 in Cabal-3.7.0.0-543aaa840f170abad978ace330d2fed5d9915f52f7fc73a0f5f6df83df6a5532:Distribution.Simple.Utils
Error: cabal: Failed to build documentation for darcs-2.16.4-inplace.

brandonchinn178 · 2021-11-30T17:42:46Z

The function I edited should be the single entrypoint for everything. Short of a custom Setup.hs script that overrides the haddock hook with some completely adhoc function, my fix should still apply. I'll try to download the darcs code today and take a look at the Setup.hs script

Mikolaj · 2021-11-30T17:49:39Z

I'm afraid it does override a whole lot:

  haddockHook = \ pkg lbi hooks flags ->
                let verb = fromFlag $ haddockVerbosity flags
                 in commonBuildHook haddockHook pkg lbi hooks verb >>= ($ flags) ,

-- | For @./Setup build@ and @./Setup haddock@, do some unusual
-- things, then invoke the base behaviour ("simple hook").
commonBuildHook :: (UserHooks -> PackageDescription -> LocalBuildInfo -> t -> a)
                -> PackageDescription -> LocalBuildInfo -> t -> Verbosity -> IO a
commonBuildHook runHook pkg lbi hooks verbosity = do
  versionInfoExists <-
    (&&) <$> doesFileExist "release/distributed-context"
         <*> doesFileExist "release/distributed-version"
  unless versionInfoExists $
    callProcess "runghc" ["./release/gen-version-info.hs"]
  (version, state) <- determineVersion verbosity pkg
  generateVersionModule verbosity lbi version state
  return $ runHook simpleUserHooks pkg lbi hooks

hsenag · 2021-11-30T17:57:27Z

It might just be on us to update the hook then, if so apologies for the noise. We have been trying to get rid of our custom setup but there are still a few pieces left.

brandonchinn178 · 2021-12-17T03:43:29Z

@hsenag do your comments mean that you're unable to verify that this PR is correct, since the darcs repo does its own stuff with Cabal hooks?

@int-e @mietek are either of you able to review this change?

hsenag · 2021-12-17T10:31:27Z

@hsenag do your comments mean that you're unable to verify that this PR is correct, since the darcs repo does its own stuff with Cabal hooks?

Based on my tests on the darcs repo, the problem still happens with your changes. But hopefully it should be possible to update our hook to take account of your changes? I didn't have a chance to take a deeper look to figure that out yet, though it's on my TODO list. If you can point me in the right direction it would probably speed that up :-)

Mikolaj · 2022-01-29T11:10:02Z

@brandonchinn178, @hsenag: how about, to avoid mutual blocking, we merge this PR and test, fix, amend, etc., as needed, later on?

hsenag · 2022-01-29T11:26:55Z

@brandonchinn178, @hsenag: how about, to avoid mutual blocking, we merge this PR and test, fix, amend, etc., as needed, later on?

I'd be very happy with that, and I never intended to block this PR.

If I understood the start of the discussion about whether it fixed the darcs case, it was because @brandonchinn178 needed a second reviewer and that I could review from the point of view of a user whose problem was fixed. Without that, and without knowing the cabal code, I'm not sure my review would be worth much. But given this has been languishing for a while, I could provide that approval anyway?

Mikolaj · 2022-01-29T11:29:21Z

But given this has been languishing for a while, I could provide that approval anyway?

That would be very much appreciated indeed. :)

hsenag · 2022-01-29T11:53:48Z

But given this has been languishing for a while, I could provide that approval anyway?

Turns out it doesn't help because I'm not a reviewer with write access :-(

Mikolaj · 2022-01-29T12:09:30Z

Turns out it doesn't help because I'm not a reviewer with write access :-(

That's fine --- I can force-merge regardless. :)

Kleidukos

LGTM

Mikolaj · 2022-01-29T12:15:02Z

@hsenag, @Kleidukos: thank you so much.

@brandonchinn178: once you rebase and fix the conflicts, let's merge!

brandonchinn178 · 2022-01-29T23:14:20Z

Done!

brandonchinn178 changed the title ~~[WIP] Fix haddock via Setup.hs with internal libraries~~ Fix haddock via Setup.hs with internal libraries Nov 19, 2021

brandonchinn178 mentioned this pull request Nov 19, 2021

stack haddock: internal error when calculating transitive package dependencies. commercialhaskell/stack#4428

Closed

Mikolaj added the Cabal: cmd/haddock label Nov 19, 2021

Mikolaj added the attention: needs-review label Nov 20, 2021

Mikolaj approved these changes Nov 20, 2021

View reviewed changes

lehins mentioned this pull request Dec 28, 2021

hw-ip Haddock failures commercialhaskell/stackage#5014

Closed

hsenag approved these changes Jan 29, 2022

View reviewed changes

Kleidukos approved these changes Jan 29, 2022

View reviewed changes

Mikolaj added merge me Tell Mergify Bot to merge and removed attention: needs-review labels Jan 29, 2022

brandonchinn178 added 6 commits January 30, 2022 02:16

Fix stack.yaml

4b35a59

Fix references for Cabal tests

88c84cb

Add test for haddock with internal libraries

4595a79

Always update index when traversing haddock targets

7ffcd51

Commit test output

dc15dcc

Add changelog entry

bd47cd7

mergify bot merged commit 4879ae1 into haskell:master Jan 30, 2022

jkachmar mentioned this pull request Mar 22, 2022

d10 haddock failure commercialhaskell/stackage#6499

Closed

Mikolaj mentioned this pull request Jul 27, 2022

Generating haddocks sometimes produces "doc-index.json does not exist" #8326

Open

Mikolaj mentioned this pull request Aug 6, 2022

haddock test related failures with GHC 9.0.2 and 9.2.{1,2,3} #7987

Open

RyanGlScott mentioned this pull request Aug 22, 2022

cabal new-haddock regression (setup: internal error when calculating transitive package dependencies.) #5423

Open

Fix haddock via Setup.hs with internal libraries #7827

Fix haddock via Setup.hs with internal libraries #7827

Uh oh!

Conversation

brandonchinn178 commented Nov 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brandonchinn178 commented Nov 19, 2021

Uh oh!

Mikolaj commented Nov 19, 2021

Uh oh!

brandonchinn178 commented Nov 20, 2021

Uh oh!

Mikolaj left a comment

Choose a reason for hiding this comment

Uh oh!

brandonchinn178 commented Nov 22, 2021

Uh oh!

Mikolaj commented Nov 22, 2021

Uh oh!

brandonchinn178 commented Nov 22, 2021

Uh oh!

hsenag commented Nov 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mikolaj commented Nov 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mikolaj commented Nov 29, 2021

Uh oh!

hsenag commented Nov 29, 2021

Uh oh!

hsenag commented Nov 29, 2021

Uh oh!

Mikolaj commented Nov 29, 2021

Uh oh!

brandonchinn178 commented Nov 29, 2021

Uh oh!

hsenag commented Nov 30, 2021

Uh oh!

Mikolaj commented Nov 30, 2021

Uh oh!

brandonchinn178 commented Nov 30, 2021

Uh oh!

Mikolaj commented Nov 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hsenag commented Nov 30, 2021

Uh oh!

brandonchinn178 commented Dec 17, 2021

Uh oh!

hsenag commented Dec 17, 2021

Uh oh!

Mikolaj commented Jan 29, 2022

Uh oh!

hsenag commented Jan 29, 2022

Uh oh!

Mikolaj commented Jan 29, 2022

Uh oh!

hsenag commented Jan 29, 2022

Uh oh!

Mikolaj commented Jan 29, 2022

Uh oh!

Kleidukos left a comment

Choose a reason for hiding this comment

Uh oh!

Mikolaj commented Jan 29, 2022

Uh oh!

brandonchinn178 commented Jan 29, 2022

Uh oh!

Uh oh!

brandonchinn178 commented Nov 19, 2021 •

edited

Loading

hsenag commented Nov 22, 2021 •

edited

Loading

Mikolaj commented Nov 22, 2021 •

edited

Loading

Mikolaj commented Nov 30, 2021 •

edited

Loading