stack test --coverage stopped working

[rubik]

When running tests no code is analyzed, but previously it was working:

$ stack clean; stack build; stack test :argon-test --coverage
[...]
Error: The coverage report for argon's test-suite argon-test did not consider any code. One possible cause of this is if your test-suite builds the library code (see stack issue #1008). It may also indicate a bug in stack or the hpc program. Please report this issue if you think your coverage report should have meaningful results.
Completed all 2 actions.
Generating unified report
Error: The unified report did not consider any code. One possible cause of this is if your test-suite builds the library code (see stack issue #1008). It may also indicate a bug in stack or the hpc program. Please report this issue if you think your coverage report should have meaningful results.

The warning does not apply, because tests do not build the library, instead depend on it. Relevant section of my Cabal file:

library
  hs-source-dirs:      src
  exposed-modules:     Argon
  other-modules:       Argon.Parser
                       Argon.Visitor
                       Argon.Results
                       Argon.Formatters
                       Argon.Types
                       Argon.Preprocess
                       Argon.Loc
                       Argon.Cabal
                       Argon.SYB.Utils
                       Argon.Walker
  build-depends:       base             >=4.7    && <5
                     , ansi-terminal    >=0.6
                     , aeson            >=0.8
                     , bytestring       >=0.10
                     , pipes            >=4.1
                     , pipes-group      >=1.0
                     , pipes-files      >=0.1
                     , pipes-safe       >=2.2
                     , pipes-bytestring >=2.1
                     , lens-simple      >=0.1
                     , ghc              >=7.8    && <8
                     , ghc-paths        >=0.1
                     , ghc-syb-utils    >=0.2
                     , syb              >=0.4
                     , Cabal            >=1.18
                     , containers       >=0.5
                     , directory        >=1.2
  default-language:    Haskell2010
  ghc-options:         -Wall
  if impl(ghc < 7.8)
    buildable: False

test-suite argon-test
  type:                exitcode-stdio-1.0
  hs-source-dirs:      test
  main-is:             Spec.hs
  other-modules:       ArgonSpec
  build-depends:       base             >=4.7    && <5
                     , argon            -any
                     , ghc              >=7.8    && <8
                     , aeson            >=0.8
                     , hspec            >=2.1
                     , QuickCheck       -any
                     , filepath         >=1.3
                     , pipes            >=4.1
                     , pipes-safe       >=2.2
  ghc-options:         -Wall -threaded -rtsopts -with-rtsopts=-N
  default-language:    Haskell2010
  if impl(ghc < 7.8)
    buildable: False

I'm using Stack from the master branch.

[mgsloan]

Hmm, interesting! Looks like what's happening is that the library isn't getting rebuilt, despite reconfiguring with --ghc-options -fhpc and building the package. As a result, the .tix file generated by the test only includes coverage info for the test itself (which gets excluded from the report - #634).

In other words, if you instead do:

stack clean; stack build; stack test :argon-test --coverage --ghc-options "--fforce-recomp"

So, I think this is a ghc or cabal bug.. I've tried to search for an issue in the cabal tracker about it, but haven't found one. I guess the steps are:

• Make sure that this is a bug with ghc or Cabal-the-library, and open an issue about it.
• Implement a workaround. Maybe we need to pass in -fforce-recomp whenever ghc-options change.

[rubik]

Thanks, it works! But I don't think the solution is ideal. This doubles my compilation time, is it strictly necessary?

[mgsloan]

Oh, agreed, that isn't my suggested solution. Everything should be fine if you run stack clean every time you switch from running build to running tests with coverage.

Another solution is to always pass --coverage to build.

I'm leaving this open, though, as it'd be really unfortunate to foist that on people. This could also be a regression, perhaps a difference in how ghc-options are used. I don't think I encountered this circumstance when working on this feature in the past.

[rubik]

If I run stack clean; stack test :argon-test --coverage I get a weird error about build/argon/argon not existing:

argon-0.4.0.0: unregistering (local file changes: app/Main.hs argon.cabal src/Argon.hs src/Argon/Cabal.hs src/Argon/Formatters.hs src/Argon/Loc.hs ...)
argon-0.4.0.0: configure
Configuring argon-0.4.0.0...
argon-0.4.0.0: build
Preprocessing library argon-0.4.0.0...
[ 1 of 11] Compiling Argon.SYB.Utils  ( src/Argon/SYB/Utils.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Argon/SYB/Utils.o )
[ 2 of 11] Compiling Argon.Preprocess ( src/Argon/Preprocess.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Argon/Preprocess.o )
[ 3 of 11] Compiling Argon.Walker     ( src/Argon/Walker.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Argon/Walker.o )
[ 4 of 11] Compiling Argon.Loc        ( src/Argon/Loc.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Argon/Loc.o )
[ 5 of 11] Compiling Argon.Types      ( src/Argon/Types.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Argon/Types.o )
[ 6 of 11] Compiling Argon.Visitor    ( src/Argon/Visitor.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Argon/Visitor.o )
[ 7 of 11] Compiling Argon.Formatters ( src/Argon/Formatters.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Argon/Formatters.o )
[ 8 of 11] Compiling Argon.Cabal      ( src/Argon/Cabal.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Argon/Cabal.o )
[ 9 of 11] Compiling Argon.Results    ( src/Argon/Results.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Argon/Results.o )
[10 of 11] Compiling Argon.Parser     ( src/Argon/Parser.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Argon/Parser.o )
[11 of 11] Compiling Argon            ( src/Argon.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Argon.o )
In-place registering argon-0.4.0.0...
argon-0.4.0.0: copy/register
Installing library in
/home/miki/exp/argon/.stack-work/install/x86_64-linux/lts-3.11/7.10.2/lib/x86_64-linux-ghc-7.10.2/argon-0.4.0.0-EPnFAvdLsdbDHKR24nicp7
Installing executable(s) in
/home/miki/exp/argon/.stack-work/install/x86_64-linux/lts-3.11/7.10.2/bin
setup-Simple-Cabal-1.22.4.0-x86_64-linux-ghc-7.10.2:
.stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/argon/argon: does not exist
No tix files found in /home/miki/exp/argon/.stack-work/install/x86_64-linux/lts-3.11/7.10.2/hpc/, so not generating a unified coverage report.
Progress: 1/2
--  While building package argon-0.4.0.0 using:
      /home/miki/.stack/setup-exe-cache/setup-Simple-Cabal-1.22.4.0-x86_64-linux-ghc-7.10.2 --builddir=.stack-work/dist/x86_64-linux/Cabal-1.22.4.0/ copy
    Process exited with code: ExitFailure 1

This is why I was running stack build before.

[mgsloan]

Ah, interesting. I actually fixed that in this PR, which I've just merged. And here I thought it was introduced by the changes in that PR!

So, updating to the very latest development version should fix that.

[rubik]

I built stack at commit ebd9d1b. Now it always produces reports, but they are for the test suite only.

To reproduce

$ git clone https://github.com/rubik/argon && cd argon
$ stack test :argon-test --coverage

[mgsloan]

What do mean by the reports only being for the test suite? I get the results I'd expect, which is coverage info for the library code, based on its usage by the test-suite.

[rubik]

I'm sorry, disregard everything I said. It was an error on my part and it absolutely works as expected.
I guess this issue can be closed, now that your PR is merged?

EDIT: I was wondering, does your PR lift the restriction that the library code shouldn't be built with the test suite? For example, I would like to test some non-public API, but since the test suite must depend on the library to obtain meaningful coverage (issue #1008) I'm forced to make it public.

[mgsloan]

Cool! My PR doesn't didn't affect coverage stuff much. I've opened this issue to track coverage for non-library modules: #1359