travis: Travis-CI Roadmap

[rbasso]

Right now, we have 4 Issues ( #119 , #124 , #129 and #130 ) and 2 "Pull Requests" ( #120 , #128 ) related to Travis pending. Except for 129, that seems to require substantial changes, all the others appear to be simple and will probably be merged soon.

However, before solving 129, it would be useful to at least have an idea about what we plan to achieve, so we can focus our work in a more productive way.

I think that, ideally, those things would be desirable in Travis:

• Test with multiple versions of GHC.
• Isolated, reproducible test environments.
• Automatic handling of dependencies ( Depends on discussion: Tracking dependencies (update README.md to include package "lens"). #151 )
• Minimal maintenance on software updates.

The only idea I had until now was to use a list of dependencies to generate stack projects on-the-fly to pass to Travis test, using cache to reduce subsequent build times. It works surprisingly well, but only if GHC version >= 7.8.2.

It would still be possible to treat GHC 7.6.x as a special case and use cabal. But that's is an ugly solution and adds a lot of complexity.

Alternatively, we can keep using a global GHC installation with cabal, which is simple and easy, but is also less reproducible and doesn't allow us to enforce correct declaration of exercises' dependencies.

I would really love some feedback about this from as many people as possible.

So... anyone?

[petertseng]

The desirables seem to capture my original pain point that I experienced, mainly "Isolated, reproducible test environments" and "Minimal maintenance on software updates".

• An upstream change broke our builds
• I spent significant effort figuring out why that was even happening before realizing I needed to clear our cache
• Then I had to use creative measures to trick the cache into clearing itself
• Even after my PR got merged I couldn't clear master's cache, so Katrina had to do that to get the build to work on master (well, now you can too!)

I would like to spare future potential contributors from going through that.

---

I personally have no horse in the race on whether to keep supporting 7.6.x, since I'm not using it. To me it seems like that decision is driven by the questions "How many students are still using 7.6.x?" and "Should we continue supporting them instead of asking them to upgrade?" It's also something I was hoping to answer in #124

I'm not sure if there's a good way to get answers to the first question there.

We may be able to infer some things from the supported versions of other tracks. I do know that at least in the Go track we expect something fairly recent: the last two versions, which has historically meant versions released in the last year (maybe a few months more). The Rust track uses the current stable version, which changes every six weeks! In contrast, 7.6.3 was released more than three years ago, so xhaskell is quite generous in its support length. Then again, different support lengths may make sense for languages with different circumstances, communities, and environments.

[rbasso]

I don't think we should just drop support for anything until it gets in the way of a feature most of the people want, and even then, only when it's ready to deploy.

The upkeep of additional versions is minimal. Of course, it gets increasingly hard to keep source code compatible with too distant versions of base, and the probability of things like #122 happening increases.

As Stack gets more popular among beginners - especially now that it's included in the Haskell Platform - it starts to make sense to use it's features to save the users the trouble of dealing with package dependencies and compiler versions.

It would be great to just write exercism fetch; cd leap; stack test to see the tests running in the same environment the contributors use and that was tested by Travis. I believe this convenience would more than pay for the trouble of installing Stack or the Haskell Platform.

But that is just a dream! 😄

Back to Earth... what I have been experimenting for a while is using Stack's snapshots in Travis to test the exercises in complete isolation. If that proves useful and popular here, maybe... just maybe, we'll have a reason to stop the tests with ghc-7.6.x; but even in that case, it would probably still compile everything normally.

[rbasso]

I know only two solutions to avoiding upstream change breaking the builds, and essentially they are the same:

• Pinning a specific stack snapshot in stack.yaml or with --resolver lts-X.Y.Z
• Pinning cabal dependencies directly.

Even without cache, sometimes cabal will decide to use a new version of a library and break the build.

To avoid unneeded maintenance, one solution would be to target the last version of all major LTS releases. Additionally, a nightly snapshot could be marked as allow-failure, so we could see the warnings and errors to prepare for the next release.

Does this makes sense?

[rbasso]

Update: #159 allowed the first successful, isolated build with strict control of dependencies - per source file - on Travis. Rebuild times mostly positive, between 1m26s and 2m32s.

https://travis-ci.org/rbasso/xhaskell/builds/139647895