Cabal fails to "properly" fetch and build dependencies from github when mentioned as source-repository-package in cabal.project

[twitu]

Describe the bug
A cabal project that fetches certain dependencies from github fails to compile with errors of the following kind.

To Reproduce

composewell/streamly-process depends on composewell/streamly. Here's streamly-process cabal.project. Building it with v2-build gives the error given below.

packages: streamly-process.cabal

source-repository-package
   type: git
   location: https://github.com/composewell/streamly.git
   tag: master

source-repository-package
  type: git
  location: https://github.com/composewell/streamly.git
  tag: master
  subdir: core

jobs: 1

$ cabal v2-build ...

Cloning into '/home/twitu/Code/streamly-process/dist-newstyle/src/streamly-ec6df83dbd01b094'...
-- snip messages
In order, the following will be built (use -v for more details):
 - streamly-core-0.1.0 (lib:streamly-core) (requires build)
 - streamly-0.9.0 (lib:streamly) (requires build)
 - streamly-process-0.2.0.1 (lib) (first run)
src/Streamly/Internal/Data/Parser/ParserD/Type.hs:211:2: error:
     fatal error: assert.hs: No such file or directory
      211 |
          |  ^
    |
211 |
    |  ^
compilation terminated.

src/Streamly/Internal/Data/Array/Unboxed/Mut/Type.hs:214:2: error:
     fatal error: assert.hs: No such file or directory
      214 | #include "inline.hs"
          |  ^~~~~~~~~~~
    |
214 | #include "inline.hs"
    |  ^
compilation terminated.

src/Streamly/Internal/Data/Parser/ParserD.hs:200:2: error:
     fatal error: assert.hs: No such file or directory
      200 |
          |  ^
    |
200 |
    |  ^
compilation terminated.

src/Streamly/Internal/Data/Array/Unboxed/Mut/Type.hs:1:1: error:
    `gcc' failed in phase `C pre-processor'. (Exit code: 1)
  |
1 | {-# LANGUAGE UnboxedTuples #-}
  | ^

src/Streamly/Internal/Data/Parser/ParserD.hs:1:1: error:
    `gcc' failed in phase `C pre-processor'. (Exit code: 1)
  |
1 | #include "inline.hs"
  | ^

src/Streamly/Internal/Data/Parser/ParserD/Type.hs:1:1: error:
    `gcc' failed in phase `C pre-processor'. (Exit code: 1)
  |
1 | {-# LANGUAGE UndecidableInstances #-}
  | ^

Expected behavior
The build should succeed.

System information
Ubuntu 20.04

haskell-language-server version: 1.7.0.0 (GHC: 9.2.2) (PATH: /home/twitu/.ghcup/hls/1.7.0.0/lib/haskell-language-server-1.7.0.0/bin/haskell-language-server-wrapper)
Tool versions found on the $PATH
cabal:          3.8.1.0
stack:          2.7.5
ghc:            8.10.7

Additional context
The build succeeds when I clone the dependencies and use the local paths as given below.

packages: streamly-process.cabal, ../streamly, ../streamly/core

[Mikolaj]

Is the problem that both deps use the same repo? What happens in you manually clone of of them, but keep the other as source-repository-package?

[philderbeast]

Another thing to try is listing both . and core as subdirectories in a list, #5472.

source-repository-package
  type: git
  location: https://github.com/composewell/streamly.git
  tag: master
  subdir: . core

[harendra-kumar]

Is the problem that both deps use the same repo? What happens in you manually clone of of them, but keep the other as source-repository-package?

It used to work earlier with both packages using the same repo, it started failing only recently. I did not try it but my guess is it has something to do with this commit composewell/streamly@cc3bbd7 in streamly. It probably has something to do with using correct path for finding the pre-processor macro include files.

[twitu]

I tried the above methods.

source-repository-package
  type: git
  location: https://github.com/composewell/streamly.git
  tag: cc3bbd7
  subdir: . core

This doesn't work

packages: streamly-process.cabal, ../streamly

source-repository-package
  type: git
  location: https://github.com/composewell/streamly.git
  tag: cc3bbd7
  subdir: core

This doesn't work.

packages: streamly-process.cabal, ../streamly/core

source-repository-package
  type: git
  location: https://github.com/composewell/streamly.git
  tag: cc3bbd7
  subdir: .

This works!!

The difference between this and the above is that in this case the nested package is fetched locally, while the above case (that doesn't work) the nested package is fetched from github. Weird 🤔.

[Mikolaj]

Thank you for the investigation. It would be wonderful to get to the bottom of this and, at least, document in the manual. Ideally, suggest where the fix needs to go or what a warning to display if the fix is hard.

[fgaz]

assert.hs is missing from extra-source-files, so it doesn't get included in the remote package, but local packages are more impure so that way it's able to pick it up anyway. this should be fixed in the streamly-core cabal file.

cabal doesn't warn about it because during the sdist phase it doesn't know that the file will be required, and during the build phase it doesn't know that the file was there before sdisting

edit: composewell/streamly#1807

[harendra-kumar]

Thanks @fgaz ! I appreciate the PR. I did not know that cabal uses sdist after fetching with git. I thought it uses the package directly as it would use local packages. streamly-core is a new package, not yet on hackage, we have not started doing sdist CI on it yet, so the problem of not including assert.hs in extra-source-files escaped.

The question is whether cabal should use sdist on git repos? If so should it use sdist on local repos as well then? So that both cases show similar behavior?