Skip all but the first test.

[dkinzer]

This change adds a skip to all but the first test for every problem set in
order to avoid overwhelming the users with a bunch of failed test.

This approach is the same as how the Ruby tests have been implemented, which
encourages a pseudo TDD style to solving the exercises.

[sjakobi]

The Python track used to have skipped tests in a few exercises. See #105 for a short discussion leading to the removal of the skips. There are a few issues referencing exercism/exercism#856 that lead to "unskipping" on other language tracks.

It seems plausible that newbies will find the problems more accessible with this change. But is there any data or anecdotal evidence to back this up? User stories maybe?

For everyone else this change introduces a bit more complexity and/or hassle.

In any case, before this change is merged (if it is merged) we'll need to make sure that the CI doesn't skip the tests. The former solution to this can be seen in #105.

[dkinzer]

I don't know... just speaking from self experience (though I'm not a newbie) seeing all those errors always throws me off and when I'm working on one problem I want to see if the change I made fixes that specific problem not get a bunch of errors that have nothing to do with my current iteration.

But I agree that of course the tests must be run in CI. In the Ruby track I think they can still run the tests but I don't know if this is possible in python.

[kytrinyx]

In the ruby track we're not actually running the tests. I'm thinking of doing a thing where I copy all the tests to a tmp dir and edit the files with sed to remove the skips.

[dkinzer]

I have verified that it is possible to still run all the tests if you first "prepare" them by removing the one line from each file.

For example this works:

cat bob_test.py  | sed /@unittest.skip/d | python

A similar solution can be applied that recurses over all the tests and just removes the skip items.

[kytrinyx]

@dkinzer we can't do it in place, since people run the build locally, and if they had changes and were just verifying before committing it could screw things up pretty badly.

[dkinzer]

@kytrinyx :) very good point... we could still do it as long as the orig files aren't touched. Piping the test to python works without any need to change the files in place.

[kytrinyx]

Piping the test to python sounds like a very nice solution!

[sjakobi]

I may have found a simpler solution to the failing test information overload problem:

With the -f or --failfast flag, a test run stops right after the first failure or error:

> python3 -m unittest --failfast pascals_triangle_test.py
F
======================================================================
FAIL: test_is_triangle_false (pascals_triangle_test.PascalsTriangleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/simon/Dropbox/prog/exercism-exercises/python/pascals-triangle/pascals_triangle_test.py", line 22, in test_is_triangle_false
    self.assertEqual(False, is_triangle(inp))
AssertionError: False != None

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

The disadvantage is that the order of the test cases is always alphanumeric by the name of the test method unless we add a custom ordering function for the test loader.

[dkinzer]

For what it's worth I've add a bash script that will pipe these to python.

[Dog]

I did notice a new user in the xphp track that didn't know why tests were being skipped and was looking for help. I think if we are going to skip tests to encourage TDD, we need to find a way to introduce users to TDD at the beginning.

[kytrinyx]

I completely agree. I don't really know what the best place is for this. We even have an issue open for it, but I'm completely stuck as to how to address it: #1869 - just writing documentation doesn't seem to be the thing to do, because how would you know to read it?

[Dog]

What if the first problem in every track is Hello World. We walk them through the submission process, explain TDD in the ReadMe, and can put any track specific information in this problem.

Its very easy for people familiar with the process to finish, and would not be submitted for nitpicking.

Proof of Concept ReadMe

Welcome to the Python track at Exercism!

Every track at Exercism starts with Hello World. Hello World is the traditional first program for beginning programmers. As programmers mature, they eventually want to test their code.

Here at Exercism we encourage Test Driven Development (TDD). Test Driven Development is where you write your tests before writing any functionality. This allows you to consider the requirements of the code, and any edge cases that you may need to support.

Once you finish your tests, you write the functionality. When all the functionality is finished, you can write tests for your next feature. One of the best features of tests is that you can very easily check for regression.

We encourage users to use TDD when solving problems to help form good coding habits,
and help simulate working on Real World™ projects.

Problem

1. Run Tests
   All tests are failing
   • Documentation Link
2. Pass First Test
   All tests are passing
3. Unskip Second Test
   Test Failing
4. Pass Second Test
   All tests are passing
5. Submit Exercise
   You can submit as many iterations you want
   • Documentation LInk

Python Specific Information

Today both Python 2 and Python 3 are widely used. Please keep in mind while nitpicking that
some syntax may vary depending on the version.

Helpful Links:

Documentation
Official Python Tutorial
"Hello, World!" Wikipedia

Note: Exercism does not encourage the use of Fear Driven Development (FDD)

[dkinzer]

Although I hadn't thought about this process as TDD, I really like @Dog's proposal and I think it might work.

[kytrinyx]

@Dog This is really, really good. We had a somewhat vague suggestion for a hello world tutorial (#2021) which was interesting but seemed like a ton of work in the browser to make it feel interactive.

This lets us use all the affordances that are already in place.

I've copied your text into that old issue.

[kytrinyx]

I had a bunch of help from several people to implement a first version of the "hello world" initial exercise.

The hello world solutions are excluded from the nitpicking lists on the site, and rikki provides a short, automated bit of encouragement (not really encouragement, more of a "OK, now go do it f'real").

Over time I think the hello world problems can be massaged to be more helpful, but I think this is going to be a great start.

Only a handful of languages have a hello world problem at the moment, but if this works out it should be fairly straight-forward to get it implemented across the board.

[Dog]

@kytrinyx @dkinzer Am I correct in thinking this issue can be closed now that hello world is implemented?

[dkinzer]

@Dog, I think those two issues are separate. I'm still strongly in favor of only running tests one at a time as is done in the ruby and php tracks. But, as I've stated earlier I can't speak to how this will affect people completely new to programming.

[kytrinyx]

I agree that the issues are separate, but I think that the conclusion (at least for now) is that in python we made the decision to not skip tests.

Here's what I'm seeing:

The argument for skipping tests: With beginners, they often don't have any mental model of how to take small steps, how to break big problems down into smaller problems. It helps to know what you're trying to achieve, but they often panic at the sight of a wall of errors. Having to remove skips encourages you to interact with the test file a bit more, which can be seen as a plus. There's a caveat: we may not actually have any evidence one way or another that it helps the beginner, we've just made the assumption. That said, in the discussion in #856 one person said that it did help working through the list with a bunch of beginners.

The argument against skipping tests: it means you have to edit the test suite. That's a pretty big hassle, especially if seeing errors doesn't make you panic. For beginners, they might not even realize that they have to edit the test suite, and they submit the solution with just a stub of behavior (I've seen this a lot in the Ruby track).

Have I missed any arguments?

[Dog]

I think those are all the arguments. I'm more on the side of not skipping tests personally, but that may be because I'm more experienced than the average exercism user. Seeing test failures while developing doesn't bother me, and I would use a testing suite to run a specific test if needed.

I think its more important to be consistent between tracks.

[kytrinyx]

I think its more important to be consistent between tracks.

I don't think that's achievable, frankly. There are some tracks that will absolutely skip tests, and others that absolutely will not.

I would use a testing suite to run a specific test if needed.

Is there a flag you can pass to run only one test? If so, maybe it would be enough to document that in the SETUP.md (which gets sent along as part of each README.md).

[Dog]

I'm pretty sure it's possible to run an individual test, but I'd have to look up the syntax and try it when I get to a computer. I'm more familiar with nosetests from work, but I remember @sjakobi answering a previous question I had with pytest. I'm also not sure if it'd require extra setup outside of a pip install command. I'll give it a shot when I get home.

If we ask them to install nosetests, then I'm pretty sure they can just run

nosetests -x --failed file_test.py

to run all tests and stop on the first failure. Running the command again reruns the last failed test. Once that test passes, it runs through all the tests again until it sees another failure or completes.

They might even be able to run it in the Python root to test all exercises as well

[Dog]

It appears that all the functionality described is possible in both pytest and nosetest.

I'd have to do more experimenting with pytest, but it appears the equivalent to nosetests -x --failed file_test.py requires a plugin called pytest-cache. Once pytest and pytest-cache are installed, the command is pytest -x --ff file_test.py. The pytest version runs the failed test first, then automatically starts running the rest of the tests from the beginning.

On the other hand, nosetests would require the python track become a package to run all tests.

Installation for users is very easy with both packages:

Pytest

• pip install pytest
• pip install pytest-cache

Nosetests

• pip install nose

At the moment I'm leaning more towards suggesting users run pytest, but I wanted to get some feedback before diving too deep. We also already suggest in the track help page that people install pytest to run all tests in the track at once.

[Dog]

I'm going to close this as I've updated the documentation to recommend pytest.

977e7db

I feel this closes out the issue. Feel free to open if you'd like to discuss this further.