Doctest is broken

[nc6]

When trying to build IntersectMBO/ouroboros-network@eb61e30, the doctest fails with

Error:
    The following packages are broken because other packages they depend on are missing. These broken packages must be rebuilt before they can be used.
installed package ghc-8.6.5 is broken due to missing package binary-0.8.6.0, bytestring-0.10.8.2, containers-0.6.0.1, directory-1.3.3.0, filepath-1.4.2.1, ghc-
boot-8.6.5, ghci-8.6.5, hpc-0.6.0.3, process-1.6.5.0, terminfo-0.4.1.2, time-1.8.0.2, transformers-0.5.6.2, unix-2.7.2.2

builder for '/nix/store/q7c3vcmbxylikpmn33b3jb0xiyl6l2lv-doctest-0.16.0.1-lib-doctest.drv' failed with exit code 1
cannot build derivation '/nix/store/5dynklpbqx3kqd0nh3r193mri4yy3gl1-doctest-0.16.0.1-exe-doctest-config.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/v684c7aa4lprxy0p0mwfk96gr9g0zwjj-ouroboros-consensus-0.1.0.0-test-doctests-config.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/z0y64kmyrmvd68lp9bxnf1zvif3n33wv-ouroboros-consensus-0.1.0.0-test-doctests.drv': 1 dependencies couldn't be built
error: build of '/nix/store/z0y64kmyrmvd68lp9bxnf1zvif3n33wv-ouroboros-consensus-0.1.0.0-test-doctests.drv' failed

Using this version of iohk-nix:

{
  "url": "https://github.com/input-output-hk/iohk-nix",
  "rev": "28149991be2f363eb028fa7ea77198419401cbbc",
  "date": "2019-08-08T04:16:01+00:00",
  "sha256": "1v707gwcl2lpl2rjhczd5m7walbgyg5c4v5f4bp67j74808sa080",
  "fetchSubmodules": false
}

This appears to be the same problem as #66, which was marked as fixed.

[nc6]

Can also see an example of it at https://hydra.iohk.io/build/1024554/nixlog/8, where it's failing in the ledger specs

[chameco]

I'm encountering the same issue when building sbv, which also depends on ghc.

[purefn]

I am hitting this too

builder for '/nix/store/rhw1a3zsr1w43d1vfa4z2rwawisnippy-doctest-0.16.2-lib-doctest.drv' failed with exit code 1; last 10 log lines:
  configuring
  Configure flags:
  --prefix=/nix/store/a8q1yq0qkgklv55gwj4ashqnmgabszfy-doctest-0.16.2-lib-doctest lib:doctest --package-db=clear --package-db=/nix/store/654k37yasbmhxqj2ckvjnh1z718wyl2v-doctest-0.16.2-lib-doctest-config/package.conf.d --exact-configuration --dependency=base-compat=base-compat-0.10.5-FRXoAxOVtbG2qLNIZm1tTr --dependency=code-page=code-page-0.2-3TjNtc9lu3RJN6poyFdQXs --dependency=ghc-paths=ghc-paths-0.1.0.12-8MhNqRZlSJXAzmVO0F6ybT --dependency=syb=syb-0.7.1-1gO4OjQOULM8usUjWwwEyW --dependency=rts=rts --dependency=ghc-heap=ghc-heap-8.6.5 --dependency=ghc-prim=ghc-prim-0.5.3 --dependency=integer-gmp=integer-gmp-1.0.2.0 --dependency=base=base-4.12.0.0 --dependency=deepseq=deepseq-1.4.4.0 --dependency=array=array-0.5.3.0 --dependency=ghc-boot-th=ghc-boot-th-8.6.5 --dependency=pretty=pretty-1.1.3.6 --dependency=template-haskell=template-haskell-2.14.0.0 --dependency=ghc-boot=ghc-boot-8.6.5 --dependency=ghc=ghc-8.6.5 --dependency=Cabal=Cabal-2.4.0.1 --dependency=array=array-0.5.3.0 --dependency=binary=binary-0.8.6.0 --dependency=bytestring=bytestring-0.10.8.2 --dependency=containers=containers-0.6.0.1 --dependency=directory=directory-1.3.3.0 --dependency=filepath=filepath-1.4.2.1 --dependency=ghc-boot=ghc-boot-8.6.5 --dependency=ghc-compact=ghc-compact-0.1.0.0 --dependency=ghc-prim=ghc-prim-0.5.3 --dependency=hpc=hpc-0.6.0.3 --dependency=mtl=mtl-2.2.2 --dependency=parsec=parsec-3.1.13.0 --dependency=process=process-1.6.5.0 --dependency=text=text-1.2.3.1 --dependency=time=time-1.8.0.2 --dependency=transformers=transformers-0.5.6.2 --dependency=unix=unix-2.7.2.2 --dependency=xhtml=xhtml-3000.2.2.1 --with-ghc=ghc --with-ghc-pkg=ghc-pkg --with-hsc2hs=hsc2hs --with-gcc=cc --with-ld=ld --with-ar=ar --with-strip=strip --disable-executable-stripping --disable-library-stripping --disable-library-profiling --disable-executable-profiling --docdir=/nix/store/lbmcpj2wm0cdrvfxj7m3nf7hzqzw3l5j-doctest-0.16.2-lib-doctest-doc/share/doc/doctest --enable-split-sections 
  Configuring library for doctest-0.16.2..
  Warning: The flag --disable-executable-profiling is deprecated. Please use
  --disable-profiling instead.
  Error:
      The following packages are broken because other packages they depend on are missing. These broken packages must be rebuilt before they can be used.
  installed package ghc-8.6.5 is broken due to missing package ghci-8.6.5, terminfo-0.4.1.2

I tried adding

     pkg-def-extras = [                                                                                                                                                                                                                                                                                                     
       # for doctest                                                                                                                                                                                                                                                                                                        
       (hackage: { ghci = hackage.ghci."8.6.5".revisions.default; })                                                                                                                                                                                                                                                        
       (hackage: { terminfo = hackage.terminfo."0.4.1.2".revisions.default; })                                                                                                                                                                                                                                              
     ];                         

but that didn't help at all.

Anyone have a workaround for this?

/cc @angerman @hamishmack

[purefn]

I noticed that terminfo, ghxi, and few others are commented out here https://github.com/input-output-hk/haskell.nix/blob/master/modules/component-driver.nix.

Could that be the cause of these issues?

[purefn]

After uncommenting those and rebuilding ghc, doctest now compiles. Is there any reason not to have these uncommented?

[angerman]

@purefn the idea is to only have minimal set of packages that are non-reinstallable, thus that set of packages needs to be closed and we try to keep it as minimal as possible.

[purefn]

@angerman Ok, is there another way to get doctest building without uncommenting those? If not, it seems like they should be part of the minimal set.

[purefn]

I did eventually find that

{
  reinstallableLibGhc = true;
}

fixes the problem and doctest will then build. Sadly, this means we aren't able to make use of the cachix cache. Is there anyway of making doctest buildable without having to do break our ability to use the cache? Seems like it's a library that is pretty commonly used and it's not a great first user experience to discover it doesn't work by default and then to discover that in order to get it working you have to compile ghc and every other package.

[angerman]

@purefn caching with haskell.nix is by nature substantially worse than with nixpkgs as there is no single currated package set. Setting reinstallableLibGhc = true will increase the set of free packages and thus those that will be built. My recommondation would be to setup caching for projects (you could use cachix --watch-store & and add the cachix key as a substituter to get something like a shared nix-cache for yourself of multiple developers).

Even if we added test-cases with resintalalbleLibGhc = true and build doctest in CI (which is in the end what end up in the provided cachix cache), this would vary with every change in LTS, or cabal index-state :-/

[hamishmack]

I think cabal-doctest support works reasonably well now. See https://github.com/input-output-hk/haskell.nix/tree/master/test/cabal-doctests for an example.