WIP: Better packages detection

[srid]

Make packages option traverse sub-dirs recursively as well as recognize package.yaml

• Hacky prototype
• Benchmark to validate this approach, and/or parse from cabal.project
• Remove hacks, do proper implementation
  • Optimize traversal

[roberth]

Recursing into arbitrary subdirectories hurts performance. This will be exacerbated by projects that rely on lazy fetching of git submodules.

make haskellPathsInDir perfect

Not sure if perfect exists. I'd prefer a cabal.project or stack.yaml parser.

[srid]

Recursing into arbitrary subdirectories hurts performance.

Doesn't nix copy the source tree to the store anyway? I ran this on a large internal mono repo with 31 packages using package.yaml which all lie as far as 5 sub-directories deep, and didn't notice a perceptible difference (to be fair, this was on my M1 Max). But yes performance degradation is a valid concern. I'll benchmark it properly before we stamp this PR for approval.

This will be exacerbated by projects that rely on lazy fetching of git submodules.

Lazy fetching of git submodules?

I'd prefer a cabal.project or stack.yaml parser.

I don't know about stack (why would someone use stack if there is Nix?) - but cabal.project parsing is indeed better (ie. #76); here's our cabal.project

packages:
  lib/*/*.cabal
  app/*/*/*.cabal
  app/*/*/*/*.cabal
  test/*.cabal

(cabal.project doesn't support hpack, so we would also have to look for package.yaml)

I suppose we can just write a Haskell executable (put it in pkgs.haskellPackages.<whatever>) and then use that to get a JSON back. This Haskell executable can also expose the hpack YAML data as JSON to Nix. In a way, it would be like cabal2nix. Maybe I should just go this route instead of this PR ...

[srid]

Haskell executable

Like https://github.com/NorfairKing/cabal2json#readme

[srid]

This Haskell executable can even do the traversal and return packages information.

[roberth]

Lazy fetching of git submodules?

Sorry, I should have explained myself. I'm considering unimplemented lazy-trees features.

why would someone use stack if there is Nix?

The stack repl multi-module hack is rather useful. You can load a repl where all local packages are quick-reloadable.

There's also horizon-haskell, which is like Nixpkgs Haskell, but more stackage based.

I suppose we can just write a Haskell executable (put it in pkgs.haskellPackages.<whatever>) and then use that to get a JSON back.

It would be really nice to avoid IFD for this purpose. Ideally, the set of flake attribute names can be determined without doing any IFD. This keeps evaluation quick, especially on monorepos, and especially when you don't always need a haskell package in an evaluation.

[roberth]

• See Support nix flake show --option allow-import-from-derivation false #95

[srid]

Closing in favour of #110