Unify Cabal/cabal-install package/integration Cabal samples

[ezyang]

I recently realized that there is a substantial amount of duplication between Cabal's package-tests and cabal-install's integration-tests. Take any particular Cabal test: it consists of some Cabal files, some source files, and some instructions how to build it (in Haskell code). Create a cabal.project or sandbox, and now those exact same Cabal files can be run using cabal-install. Since there is some amount of duplication between the functionality that Cabal and cabal-install provide, it would be beneficial if we specified the test files only once, and then specified for Cabal and cabal-install separately how to actually run them. It also helps with the problem where we have two substantially similar drivers for both types.

I've been pondering how to do this. I think the answer is that we should create a separate Cabal project (assuming the driver is written in Haskell; it doesn't have to be) just for tests. The driver tests either a Setup executable, or a cabal executable, and each test is associated with all of the various ways you can build the project in question (e.g., a hopefully short cabal new-build invocation, or a longer sequence of Setup invocations). You can write your test script either as a shell script or a self-contained Haskell program (the driver is responsible for runghc'ing these test scripts; decoupling in this way will help alleviate some of the complaints with an entirely Haskell driven interface).

What do people think?

[23Skidoo]

+1 from me, I think that merging the two test suites makes a lot of sense. Would be also nice to move unit tests to the same top-level location, but this will have to wait until cabal-install code is exposed as a library.

[ezyang]

I'd like to avoid doing this before per-component and Backpack branches merge.

[ezyang]

integration-tests2 poses a problem, because it needs to build against private cabal-install modules. Probably can't separate it out.

[ezyang]

Since the package tests rewrite landed, this is a lot closer. We just have to port the cabal-install tests to the new test runner. I have some patches on the way.

[ezyang]

This is fixed!