Write a new stackage-curator tool

[snoyberg]

Some basic ideas: https://github.com/snoyberg/pantry/blob/master/PLANS.md#stackage

[snoyberg]

Basic work is now down, next steps:

• Nice command line interface
• Support adding packages outside of Hackage (to allow for patching)

These changes will likely come after pantry is merged to master.

[dbaynard]

Interesting. I'd been working on https://github.com/dbaynard/stackshot/tree/develop before I heard of pantry, and came back to it a few days ago — it's currently able to update a snapshot's packages to the most recent hackage versions (with a few bugs for good measure). I haven't documented all the recent changes, yet; but I've got some code that pulls cabal/hpack files from github (see the Stackshot.Snapshot and Stackshot.Git modules) which may be helpful.

It was also a bit of an experiment with exception handling, so please forgive the odd IO (Either _ _)…

[snoyberg]

There's more information on the current status and workflow in the README.md for the package: https://github.com/commercialhaskell/stack/tree/master/subs/curator

[snoyberg]

Was discussing with @nh2, some notes on next steps:

• New stackage-curator in nix
  • Testing stack2nix all the way
  • Building with stack inside nix, in addition Setup.hs
  • haddocks: Currently manually calling Setup.hs haddock for the entire stackage contents
• Still need to get nicer output for version bounds issues, like we report currently on the commercialhaskell/stackage repo

[snoyberg]

I just discussed with @qrilka, and we came up with this set of next actions:

---

First step on the stackage-curator work. Write some script/program etc which:

• Uses the curator tool inside the Stack repo today
  • Note: this tool is not 100% complete
• Generates a build plan file (YAML file which matches the contents in https://github.com/commercialhaskell/stackage-snapshots)
  • Note: for testing, you may want to actually use a real LTS snapshot file instead of the generated one so you don't end up trying to debug builds which aren't going to work
• Uses Nix to build and test the full snapshot
• Collects together all of the generated Haddocks into a single directory

Probably it makes the most sense to include this code in the curator tool itself, under a new command.

---

Other steps (not yet detailed):

• Improve the error messages from curator when bounds issues exist to match the current output provided by the stackage-curator tool (see, e.g., base-unicode-symbols 0.2.3 stackage#4156)
• Upload of Haddocks to S3 (or somewhere else)
• Signed Git push of the new YAML file to Github
• Ideally, builds will occur in a Docker container to isolate away Template Haskell and Setup.hs code in untrusted code from Hackage and prevent it from snooping on private files (like the GPG signing key)
• Ability to include Git repositories in the build-constraints.yaml file
  • Need to figure out where the code will live and how we'll deal with hpack's package.yaml files. Possibility: run a Pantry server that includes generated cabal files. May be the best bet for Nix, because we can provide a tarball option. See: Add Github package scraping logic to a Pantry server #4267

[qrilka]

As new Stackage snapshots don't include curator-specific information (e.g. skipped tests) it will be stored in separate files (and the current curator already produces constraints.yaml) in a separate repository. This information is planned to be used in stack2nix so its Nix derivations will be the same as the ones used by Stackage curators.

[qrilka]

It looks like doing Nix builds with tests, benchmarks and haddocks will require substantial effort on supporting Nix-related code so currently we chose to do those builds using Stack itself (this will work as extra test cases for Stack) and we'll do Nix builds in addition to that.