discussion: Haskell exercises as complete projects

[rbasso]

Haskell exercises as complete projects

What we have now

Currently we are providing most of the exercises as single files containing tests, written with HUnit or QuickCheck.
This is the way it is for historic reasons, but we should discuss if it is the best approach to follow.

Advantages

• As simple as it gets.
• It's very flexible, if you know what you are doing.

Disadvantages

• Needs GHC and all the dependencies previously installed.
• Leaves the problem of dealing with cabal hell to the users.
• Demands exercises compatible with multiple GHC versions and libraries.
• Automatic tests are complex and do not necessarily reflect the user's environment.
• Forces us to keep an updated list of used packages in INSTALLATION.md and README.md.
• It makes harder to write beautiful tests and sample solutions, because they usually depend on multiple packages, newer compiler features and modern libraries.

What we should have

I don't think the state of things is desirable. Ideally, I would love to:

• Make users' lives easier, leaving them just solving the exercises, without caring about compiler versions, libraries or anything else.
• Allow contributors to create beautifully written tests and sample solutions, using a lot of modern libraries and compiler extensions.
• Optionally provide benchmarks for some exercises, so that advanced users could easily check performance.
• Minimize maintenance and simplify automated tests, checking only for the current and next version.

Sounds good? Read on!

Alternative solution

Let's use leap as an example. It could have those files:

stack.yaml
packages.yaml
src/LeapYear.hs
test/LeapTest.hs
bench/LeapBench.hs

The benchmark is here just to illustrate that it's possible to have one. I would not recommend using this functionality so soon because of commercialhaskell/stack#1870

stack.yaml

resolver: lts-6.3

stack.yaml has just the resolver that we would be using for all exercises. This means ghc-7.10.3 and a frozen, curated set of packages that work well with it.

packages.yaml

name: leap

dependencies:
  - base

library:
  exposed-modules: LeapYear
  source-dirs: src
  dependencies:

tests:
  test:
    main: LeapTest.hs
    source-dirs: test
    dependencies:
      - leap
      - tasty
      - tasty-hunit

benchmarks:
  bench:
    main: LeapBench.hs
    source-dirs: bench
    dependencies:
      - leap
      - criterion

package.yaml lists all the dependencies used by the solution, tests and benchmarks. It's a slimmed down and much more readable version of a .cabal file. This format is used by hpack and stack.

test/LeapTest.hs

import Test.Tasty       (defaultMain, testGroup)
import Test.Tasty.HUnit (testCase, (@?=) )

import LeapYear (isLeapYear)

main :: IO ()
main = defaultMain tests
  where
    tests = testGroup "isLeapYear tests" $ test <$> cases
    test (label, year, expected) = testCase label (isLeapYear year @?= expected)
    cases = [ ("leap year"                  , 1996, True )
            , ("standard and odd year"      , 1997, False)
            , ("standard even year"         , 1998, False)
            , ("standard nineteenth century", 1900, False)
            , ("standard eighteenth century", 1800, False)
            , ("leap twenty fourth century" , 2400, True )
            , ("leap y2k"                   , 2000, True ) ]

test/LeapTest.hs contains the tests. Using tasty with tasty-hunit allow us to write test that are at the same time shorter and more readable. Note that we used <$>without importing Control.Applicative, because we are sure that in lts-6.3 it will be already imported in Prelude. We don't need to care about compatibility with older GHCs.

bench/LeapBench.hs

import LeapYear (isLeapYear)

import Criterion.Main (bench, bgroup, defaultMain, whnf)

main = defaultMain [ bgroup "isLeapYear" $ makeBench <$> years]
  where
    makeBench x = bench (show x) $ whnf isLeapYear x
    years = [1900, 2000, 2001, 2004]

bench/LeapBench.hs can implement a benchmark, so if an advanced user cares about the performance of his/her implementation, it can be easily run with stack bench. Here we used criterion, and we don't need to worry about users having to install it, because stack will take care of that if the user decides to run it.

src/LeapYear.hs

module LeapYear (isLeapYear) where

isLeapYear :: Integer -> Bool
isLeapYear year = undefined

src/LeapYear.hs is just a stub solution, so that the user knows exactly what needs to be implemented. See #181 for a discussion about why.

Running the tests

This is where this solution shines. The user just needs to run:

$ stack test

Stack will download and install all the dependencies needed, including the compiler, and run the tests.

Maintenance

• There would be no need to keep updated package lists in INSTALLATION.md and README.md.
• .travis.yml could be greatly simplified. We would probably only test against our current resolver and the latest lts or nightly, to know earlier about any changes needed to upgrade to the newest snapshots.
• We could simply remove all those files:

_test/
_test/bootstrap.sh
_test/check-exercises.hs
_test/dependencies
_test/dependencies.txt
_test/stackalize

Disadvantages

• Needs stack previously installed.
• Demands the user to list additional packages it wants to use in an exercise in packages.yaml (stack handles downloading and installation).

Migration

I think that everything we would need is already in place:

• We have all the dependencies in dependencies.txt
• Stack is already being used in Travis-CI
• Some tracks (C, rust, scala) already use sub-directories, so it should work.
• Stack is currently the recommended way to install Haskell, and it also comes installed in the newer versions of the Haskell Platform.

So... Why not?

Is there any solid reason for us to keep things the way they are?

Are we following this path because it is the best or just because we are comfortable with it?

If you read everything without sleeping, please leave a comment or at least add a reaction.
👍 👎 😄 🎉 😕 ❤️ ❓

[petertseng]

You've mentioned that some tracks also use sub-directories, and indeed some tracks have also converted to a per-exercise project structure with the appropriate tool for their respective language. They originally were not, and maybe we can see what happen with them:

• Rust uses Cargo and has a separate Cargo project for each exercise as of Use Cargo for builds rust#27. Cargo manages the dependencies and each exercise can have its own separate set. In fact we had to have a discussion in Implement Gigasecond exercise rust#79 on whether it was appropriate to add a dependency but ultimately we decided in favor since:
  • it would be contained to the exercise
  • the dependency is easily discoverable - each exercise lists its dependencies (if any) in a section in the Cargo.toml file present in every exercise directory.
  • Cargo is a large part of the Rust ecosystem; it's useful to give students exposure to how it works and how it handles dependencies
  • From the point of the view of the student, it's no different. You just run cargo test and the dependencies (if any) are installed and your tests get run.
• Far as I can tell, Clojure also did the same in Restructure clojure#88 with the result that in each exercise directory lein test is sufficient to run the test for that exercise. The reasoning is that it more closely matches what you would use in the real world if you were writing Clojure, and I imagine the uniform lein test command is helpful as well.

OK so now you tell me "OK great, you have given some information about other tracks, and how does it relate to the Haskell track? Do the same principles for those tracks necessarily apply to this one?"

I think a lot of the goals are similar:

• give students a taste of what it's like to manage typical Haskell projects with a standard tool
• maintainers can better understand dependencies and they'll be contained
• the uniform interface of just stack test benefits everyone

It sounds like the setup for students will be easier: "Install Stack", rather than "Install GHC, install HUnit, install whatever else the exercise depends on". So that is good.

OK, so why are things the way they are now? We may not get a definitive answer as past maintainers have left. It could be possible that when the track only had 10 exercises this approach made sense and everyone just stuck with it. For example, maybe nobody wanted to put in the work to make this happen. We may not be sure of the past motivations but we can definitely reason through the advantages and disadvantages of a new approach and make the best decision for the future.

If going this route, of course, it will be beneficial to update some docs, at least https://github.com/exercism/xhaskell/blob/master/docs/TESTS.md.

A question I wonder - if after making all exercises Stack projects a student has a desire to do the exercises without Stack, how easy is that? Looking at a quick glance at the proposed structure, it looks like it's not too hard. This means that there will not be the complaint of "but this forces a particular tool on the student". I can investigate real quick to confirm that. This may also mean that if a student wishes to use GHC 7.6.x or older, things will probably work for them as well (even if we stop testing it in our automated tests)

To me, it seems like:

• It's a benefit for students who just want to get things working
• It's not too restrictive on students who wish to go their own way
• It helps maintainers and contributors to the repo

If we don't come up with any fatal disadvantages to this, it seems good.

[rbasso]

Thank you for you detailed answer, @petertseng ! 😄

A question I wonder - if after making all exercises Stack projects a student has a desire to do the exercises without Stack, how easy is that?

It's certainly easier than doing the opposite. 😬

... Looking at a quick glance at the proposed structure, it looks like it's not too hard. This means that there will not be the complaint of "but this forces a particular tool on the student".

I believe we should favor what's best for the majority of the least prepared users. The more advanced ones usually have very diverse demands, but they are probably capable of taking care of themselves just having access to the tests.

Also, it would be easier for users using cabal sandboxes. They could easily convert packages.yaml into a .cabal file with hpack.

Any alternative setup would benefit from having the list of dependencies in packages.yaml

This may also mean that if a student wishes to use GHC 7.6.x or older, things will probably work for them as well (even if we stop testing it in our automated tests)

At first, everything would work the same as it works now. As collaborators start to use new compiler features and libraries not compatible with ghc-7.6.3, compatibility will decrease as we improve the sources.

It's a premise that beginners will use stack, so compiler versions would not be a problem for them. We can, of course, keep enforcing compatibility with ghc-7.8.4 for the people not using stack, if we think that would be desirable.

Do you think this would be helpful for a reasonable number of users ❓
Or would be better to expose users to current coding practices ❓

The use of <$> is a good example of how the compiler version can affect coding style:

• Most users of ghc-7.10 use it instead of map and fmap.
• Some users of older versions try to avoid it, so they don't have to import Control.Applicative.

If going this route, of course, it will be beneficial to update some docs

I'll try to make a checklist of all the things that would be changed. With that, we can refine the proposal and start discussing a possible migration path.

If we don't come up with any fatal disadvantages to this, it seems good.

I'm glad you liked! 😄

[petertseng]

We can, of course, keep enforcing compatibility with ghc-7.8.4 for the people not using stack, if we think that would be desirable.
Do you think this would be helpful for a reasonable number of users ❓
Or would be better to expose users to current coding practices ❓

Oof, I can only speak confidently for myself, but I left 7.8 behind a long time ago. If only we had user surveys of Exercism users or something like that.

I kind of want to say "well if it's possible to keep the compatibility without too much detriment to the code, why not?", but that's a bit of a non-answer you already gave the example of <$>. Could keep a Stack lts 2.x in the CI as an optional build, though that might mean nobody will check so it. But maybe that's OK.

I'm biased toward keeping things up to date and being OK with leaving the old ways behind, but that's just me and maybe others think differently.

[rbasso]

I'm biased toward keeping things up to date and being OK with leaving the old ways behind, but that's just me and maybe others think differently.

I would love to try to track the latest lts as a general rule. This way users and collaborators could have access to new features without having to wait too much. Nightly snapshots could also be great (they are essentially the same as LTS), but sometimes there are fewer packages available, so a few exercises could prevent the upgrade.

Could keep a Stack lts 2.x in the CI as an optional build, though that might mean nobody will check so it. But maybe that's OK.

I completely agree! If anyone makes a PR that would break it,then we can discuss if we keep it or drop it.

Seems reasonable for you?

[rbasso]

Expected changes

What would be added

*/stack.yaml    - Specifies the current resolver
*/packages.yaml - Tracks exercise dependencies

What would be renamed and moved

*/name_test.hs -> test/Tests.hs  - A fixed name simplifies things for us.
*/example.hs   -> src/Example.hs - Tile case is commonly used in Haskell
*/Module.hs    -> src/Module.hs  - No need to rename

What would be modified

.gitignore      - Stop ignoring `stack.yaml` and `packages.yaml`
.travis.yml     - Major simplification.
config.json     - Remove references to `_test`
INSTALLATION.md - Complete rewrite
README.md       - Complete rewrite
SETUP.md        - Complete rewrite
TESTS.md        - Complete rewrite

What would be removed

_test/
_test/bootstrap.sh
_test/check-exercises.hs
_test/dependencies
_test/dependencies.txt
_test/stackalize

Additional cleanup

Vagrantfile - Not used by the collaborators.

[rbasso]

Migration

Considering that this would be a huge change, we need a migration plan. This is a possible migration path that allows some time for people to adapt.

Steps

• Change .gitignore.
• Change .travis.yml to also support stack projects.
• Migrate the last exercise to check if everything works as expected.
• Rewrite the documentation recommending stack and explaining that we have two types of exercises.
• Migrate all exercises in phases, starting with the later ones.
• Clean-up the documentation.
• Clean-up .travis.yml.
• Remove _test and update config.json.
• Remove Vagrantfile.

An alternative would be to follow the first three steps and then just migrate everything. It's significantly easier, but may surprise some people.

What do you think @petertseng?
Anyone else have any comment on this?

[kytrinyx]

I can't weigh in on the details, but I really appreciate the detailed and thorough discussion!

[rbasso]

I'll start to set things in motion creating a new issue to track the migration, so we can at least have an online proof-of-concept with the last exercise.

If someone opposes, please tell me ASAP, so I don't waste too much work on that.

[rbasso]

We still need to discuss if we should add stub solutions to the exercises while we convert then to Stack projects.

For the exercise pov, I'll have to add the stub for testing purposes, but I can remove it later.

At least for the first exercises, I think it's really important to have it. If we add stubs to all of the exercises, maybe this could add convenience and lower the bar a little so that people would not give up so fast on the track, which seems to be a problem we have.

[rbasso]

Hi, @kytrinyx !

There were a lot of commits (31) in the last 10 days, and some of them I would like to check online to see if everything is working as expected.

Could you please updated the Haskell sub-module? 😁

Edit: Is there any other way to do this without bothering you ❓

[kytrinyx]

@rbasso So sorry! I'm deploying now.

At the moment I don't have anyone else who deploys. I really should get that!