Can't download uncached dependencies

[balsoft]

While trying to build https://github.com/serokell/crossref-verifier/tree/balsoft/haskell.nix I face a strange error.

$ nix build -f . --substituters https://nix-tools.cachix.org --no-sandbox --no-restrict-eval
trace: Using index-state: 2019-10-20T00:00:00Z for alex
trace: Using index-state: 2019-10-20T00:00:00Z for happy
trace: Using index-state: 2019-10-20T00:00:00Z for hscolour
builder for '/nix/store/14gr8n5zd05kwiv179i34xdp6hkzr89c-stack-to-nix-pkgs.drv' failed with exit code 1; last 10 log lines:
  Receiving objects: 100% (176/176), 30.10 KiB | 604.00 KiB/s, done.
  Resolving deltas: 100% (68/68), done.
  From https://github.com/int-index/caps
   * [new branch]      master          -> origin/master
   * [new branch]      ref-impl        -> origin/ref-impl
   * [new branch]      to-capabilities -> origin/to-capabilities
  Switched to a new branch 'fetchgit'
  removing `.git'...
  error: cannot open connection to remote store 'daemon': reading from file: Connection reset by peer
  stack-to-nix: nix-prefetch-hg: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
cannot build derivation '/nix/store/l84h1d9793197mlax9pnphhjs80afibj-cache-file.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/51qm98s3q248scwvkf3qkdfaaqjzmz2l-stack-to-nix-pkgs.drv': 1 dependencies couldn't be built
[0 built (1 failed)]
error: build of '/nix/store/51qm98s3q248scwvkf3qkdfaaqjzmz2l-stack-to-nix-pkgs.drv' failed
(use '--show-trace' to show detailed location information)

Just because I feel like this might be something related to nix itself,

$ nix --version                                                                    
nix (Nix) 2.3

Full log of the failed derivation is here: https://0x0.st/zIgk.txt

[kirelagin]

I’m stuck at the same error trying to build https://github.com/serokell/lootbox/tree/kirelagin/haskell.nix 🙁.

(I even tried to commit .stack-to-nix.cache there hoping that stack-to-nix will pick it up, but no luck.)

[angerman]

  error: cannot open connection to remote store 'daemon': reading from file: Connection reset by peer
  stack-to-nix: nix-prefetch-hg: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)

I believe the .stack-to-nix.cache is just not found in the correct folder? There is some logic to construct a cache file from the cache argument, which generates a file; I guess we could extend that to copy over a cache file if one is found?

haskell.nix/lib/call-stack-to-nix.nix

Line 26 in 5884ab0

cp ${mkCacheFile cache} $out/.stack-to-nix.cache

[balsoft]

@angerman I think the problem is deeper. It'd be really nice to not add any extra auto-generated files to the repo, including .stack-to-nix.cache. What we need to fix is to not run nix-prefetch-* scripts from inside the nix build (or use patched scripts that don't use the nix-* commands).

Or, to be fair, we can wait for recursive nix to land in release and then just slightly change stuff so that it all magically works.

[angerman]

@balsoft ok, so I think I have an idea...

First off, we have the .stack-to-nix.cache file, so that re-runing stack-to-nix locally does not repeatedly fetch repositories, but returns quickly if there is nothing or only local changes to be updated.

Then we have the pure/restricted eval issue where we can only use fixed-output-derivations. This in turn requiers all fetchgit invocations to have a sha256 value. So we re-use the cache logic for that (the cache argument to callStackToNix will generate the necessary .stack-to-nix.cache file).

However, and here it gets interesting, you are running this in non-restricted mode. I believe we are missing nix as a nativeBuildInput to the stack-to-nix call? E.g. nix-prefetch-git is not available in the same environment that stack-to-nix is run. This didn't bite us so far as we've never done that, and had to deal with restricted/pure eval already; never reaching the code path you are in now.

This is just a hunch, but it at least explains (to me) the issue you are seeing. I'll have to check if I can find some time to investirage this later. If you feel like investigating, please do!

[balsoft]

I believe we are missing nix as a nativeBuildInput to the stack-to-nix call?

I believe that currently (before recursive nix rolls out) running nix-* or nix * commands is not possible from within the builds, hence the error (the nix executable is available, but it's not working).

I do feel like investigating!

[balsoft]

This is resolved.