Determine if the tag is a ref or a rev

[parsonsmatt]

Test the tag attribute in the cabal.project file. If it looks like a git hash, then assign it to rev. Otherwise, assign it to ref.

This should fix the problem where source-repo-override cannot be used to set a rev or ref, which blocks #1663 from using source-repo-override.

[parsonsmatt]

Some notes while I was digging around

Goal: Provide the branch name in a ref when doing fetchgit

In stack-cache-generator.nix, the code sets ref = branch when doing fetchGit:

            pkgsrc =
              if !is-private && sha256 != null
                then evalPackages.fetchgit {
                  inherit (dep) url rev;
                  inherit sha256;
                }
                else builtins.fetchGit ({
                  inherit (dep) url rev;
                } // evalPackages.lib.optionalAttrs (branch != null) { ref = branch; });

branch is defined here:

            branch = lookupBranch {
              location = dep.url;
              tag = dep.rev;
            };

with lookupBranch being a function argument to the file:

, branchMap    ? null
                     # A way to specify in which branch a git commit can
                     # be found
, lookupBranch ?
  if branchMap != null
    then { location, tag, ...}: branchMap."${location}"."${tag}" or null
    else _: null

How does the call-cabal-project-to-nix.nix file work?

The fetchgit call is here:

              -- line 243
              then fetchgit { inherit (repoData) url sha256; rev = repoData.ref; }

For some reason, we're swapping rev and ref.
repoData is a function argument to fetchPackageRepo, called like this:

      sourceReposEval = builtins.map (fetchPackageRepo evalPackages.fetchgit) sourceRepoPackageResult.sourceRepos;
      sourceReposBuild = builtins.map (x: (fetchPackageRepo pkgs.fetchgit x).fetched) sourceRepoPackageResult.sourceRepos;

And sourceRepoPackageResult is defined as the result of parseSourceRepositoryPackages

      sourceRepoPackageResult = pkgs.haskell-nix.haskellLib.parseSourceRepositoryPackages
        cabalProjectFileName sha256map source-repo-override projectFile;

That function lives in lib/cabal-project-parser.nix.

  parseSourceRepositoryPackages = cabalProjectFileName: sha256map: source-repo-override: projectFile:
    let
      blocks = pkgs.lib.splitString "\nsource-repository-package\n" ("\n" + projectFile);
      initialText = pkgs.lib.lists.take 1 blocks;
      repoBlocks = builtins.map (parseSourceRepositoryPackageBlock cabalProjectFileName sha256map) (pkgs.lib.lists.drop 1 blocks);
      overrideSourceRepo = sourceRepo: (source-repo-override.${sourceRepo.url} or (pkgs.lib.id)) sourceRepo;
    in {
      sourceRepos = pkgs.lib.lists.map (block: overrideSourceRepo block.sourceRepo) repoBlocks;
      otherText = pkgs.lib.strings.concatStringsSep "\n" (
        initialText
        ++ (builtins.map (x: x.otherText) repoBlocks));
    };

Notice that we're passing downt he sha256map, which is analgous to the branchMap that we want to be passing down.
But we also have a source-repo-override thing, which is overriding the URL for the repo.
We could conceivably do either.

source-repo-override

This is an argument to the file call-cabal-project-to-nix:

, source-repo-override ? {} # Cabal seems to behave incoherently when
                            # two source-repository-package entries
                            # provide the same packages, making it
                            # impossible to override cabal.project
                            # with e.g. a cabal.project.local. In CI,
                            # we want to be able to test against the
                            # latest versions of various dependencies.
                            #
                            # This argument is a map from url to
                            # a function taking the existing repoData
                            # and returning the new repoData in its
                            # place. E.g.
                            #
                            # { "https://github.com/input-output-hk/plutus-apps" = orig: orig // { subdirs = (orig.subdirs or [ "." ]) ++ [ "foo" ]; }; }
                            #
                            # would result in the "foo" subdirectory of
                            # any plutus-apps input being used for a
                            # package.

So we could conceivably use this instead - suppose we need to set ref = "ghc-9.4".
Then we can write something like:

    source-repo-override = {
      "https://github.com/TeofilC/servant.git" =
        orig: orig // {
          ref = "ghc-9.4";
          rev = orig.rev;
        };
    };

Unfortunately, this doesn't work, because the downstream code changes ref to rev, so we try to checkout refs/tags/ghc-9.4.
If we want to preserve the original , we'd need to disambiguate a

Getting the branchMap stuff out of the file:

, branchMap     ? null
                     # A way to specify in which branch a git commit can
                     # be found. Set this option if you see an error like:
                     #
                     #     error: Cannot find Git revision 'qewrtyasdf1234' in ref 'refs/heads/master' of repository 'https://github.com/foo/bar.git'! Please make sure that the rev exists on the ref you've specified or add allRefs = true; to fetchGit.
                     #
                     # The syntax is similar to `sha256map`:
                     #
                     #   branchMap =
                     #     { "https://github.com/foo/bar"."qwerasdf1234" =
                     #         = "ghc-9.4";
                     #     }

[hamishmack]

What happens here if we have a repoData.ref but no repoData.rev?

[parsonsmatt]

Good question!

Looks like fetchgit's rev attribute can either be a commit or a ref-like attribute, so I should do rev = repoData.rev or repoData.ref instead of inheriting.

[hamishmack]

This __trace that uses ${repoData.ref}, is that still safe? Should it use ${repoData.rev or repoData.ref}?

[parsonsmatt]

Good catch. I think a complication is that we may have a ref and a rev (which is part of what this is intending to allow). ie, I was trying to fetch a repo with no master, so it didn't fetch the right branch because ref was specified to be something else.

I'll update the trace message to include both.

[hamishmack]

bors try

[iohk-bors]

try

Build succeeded:

• buildkite/haskell-dot-nix
• ci/hydra:Cardano:haskell-nix:required