Make more efficient use of space in cabal-binaries

[ezyang]

Today, I got a message from GitHub saying that cabal-binaries had gotten too big. (For reference, we inaugurated this repo on February 2, so that's about two months of binaries.) To fix the immediate problem, I ran this script to delete every branch besides master from cabal-binaries:

git ls-remote origin | cut -f 2 | cut -d "/" -f 3,4 | grep -v master | grep -v HEAD | xargs -n 1 git push --delete origin -v

In the future, it would be wise to have cabal-binaries clean up after itself. Probably a good way to do this is to have travis-test.sh DELETE its own branch when it finishes successfully. Failed branches will eventually creep us back up to the limit but then we can just delete everything again.

[ezyang]

I received an update that even with nightly old branch pruning, cabal-binaries was taking up too much space. I want to write down some of the updates in developments here.

Findings of fact.

• The size of a single build (remember that every PR/branch spins up ~10 builds for various configurations) previously took about 1.2G of space. This is substantially reduced when put into Git; when I pull a single branch from cabal-binaries today, .git takes only ~160M of space. When uncompressed, the bulk of space usage is taken up by dist-newstyle, which takes a whopping 650M; this is basically all due to object and interface files. We also send over .cabal, which accounts for 200M uncompressed.

• The reason why we send all of dist-newstyle and .cabal is because cabal-testsuite needs to be able to run the GHC interpreter on test files which themselves import the Cabal library. Thus, there needs to be interface files for Cabal and all of its deps available for this to work. (Additionally, some Custom build-type tests link against the built version of Cabla and also need the interface files available.) None of the other test suites in Cabal have this requirement.

• The reason why we transmit our build products to another Travis instance is because otherwise, our builds take too long on Mac OS X. It was decided to perform all tests in the second round, because this was the "logical" way to go about doing things.

• GitHub would prefer our repository to stay under 200M. In contrast, GitHub LFS offers 1G storage free, with 1G of bandwidth a month. Amazon AWS offers 5G free storage and 15G free bandwidth a month. So clearly sending 1G, or even 200M of data is completely untenable.

The current plan.

• I did some experiments with existing Travis to see how much leeway we have in OS X build times. Currently, the slowest OS X build (ghc-8.0) took 20min (https://travis-ci.org/haskell/cabal/builds/229519931). When I started running our "primary" test suite, it now took 35min (https://travis-ci.org/haskell/cabal/builds/229514499). The Travis time limit for builds is 50min, and we set a project-specific limit of 40min to ensure that we can upload intermediate build products to the cache. 35min is not great, but it is low enough to ensure that our slowest OS X build can consistently complete. If our build times increase further or as we add more tests to Cabal, this will be a problem again, but for now it is basically acceptable. (As a stopgap, we can buy ourselves more time at the cost of spending more time CI by running two jobs: one to run the Cabal tests, and one to run the cabal-install tests.)

• If we do not have to run cabal-tests in the second build run, our space requirements drop a lot, since .cabal and dist-newstyle no longer can be sent. We only need to send the (statically) linked binaries we want to test. Binaries in Cabal are 20M, while binaries in cabal-install are 40-50M.

• By default, cabal-install builds 5 binaries. However, a lot of the space is wasted, because they are relinking the same static binary. To solve this problem, I add a new compilation mode to cabal-install, "monolithic", which combines all of the test suites into a single binary that contains all of them.

• Combining all of these improvements + compression, I project that we can now reduce the space needs of each build to 10-15M. If we promptly remove branches after we are done testing them, we can keep the live data usage of our repo under 200M. Right now I am in talks with GitHub as for whether or not we can continue using a repo in this regime. If not, I will have to investigate uploading to AWS (this is problematic, because we will need to figure out how to prevent random people from uploading to our account.)

[23Skidoo]

@ezyang With #4494 merged, can this be closed?

[ezyang]

Yep.