Coloured Output

[JohnPaton]

Currently I write my tests in unittest, but use pytest as a test runner because the coloured output makes it easier to see what's going on at a glance (plus it just looks better). This could be added to marbles as well, since nicer output is a major feature. Examples I'm thinking of:

• Green . / red F as tests run:

..........................................F......

• Green ok / red FAIL in verbose mode:

test_not_found (tests.test_users.TestUpdatePassword) ... ok
test_wrong_password (tests.test_users.TestUpdatePassword) ... ok
test_empty_lists (tests.test_utils.TestDropNones) ... ok
test_empty_strings (tests.test_utils.TestDropNones) ... ok
test_no_nones (tests.test_utils.TestDropNones) ... ok
test_nones (tests.test_utils.TestDropNones) ... ok
test_0_to_plus31 (tests.test_utils.TestFormatPhoneNumber) ... FAIL
test_already_formatted (tests.test_utils.TestFormatPhoneNumber) ... ok
test_bad_input (tests.test_utils.TestFormatPhoneNumber) ... ok
test_dashes (tests.test_utils.TestFormatPhoneNumber) ... ok
test_foreign (tests.test_utils.TestFormatPhoneNumber) ... ok
test_plus (tests.test_utils.TestFormatPhoneNumber) ... ok
test_spaces (tests.test_utils.TestFormatPhoneNumber) ... ok

• Highlight important bits of failure messages
  • Red FAIL in title to make quick scrolling easier (more visible breaks between mesages)
  • Maybe some color for the actual assertion message ( '+31612345678' != 'this will fail')
  • Syntax highlighting in the source lines or traceback (I might be getting carried away here)

======================================================================
FAIL: test_0_to_plus31 (tests.test_utils.TestFormatPhoneNumber)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/johnpaton/some/path/tests/test_utils.py", line 64, in test_0_to_plus31
    self.assertEqual(output, 'this will fail')
  File "/Users/johnpaton/some/path/venv/subsidy/lib/python3.6/site-packages/marbles/core/marbles.py", line 532, in wrapper
    return attr(*args, msg=annotation, **kwargs)
marbles.core.marbles.ContextualAssertionError: '+31612345678' != 'this will fail'
- +31612345678
+ this will fail


Source (/Users/johnpaton/some/path/tests/test_utils.py):
     63 output = utils.format_phone_number(input)
 >   64 self.assertEqual(output, 'this will fail')
     65 
Locals:
	output=+31612345678
	expected=+31612345678
	input=0612345678


----------------------------------------------------------------------

This might be a good use case for colorama and/or pygments at the expense of another dependency.

[leifwalsh]

Probably termcolor rather than pygments, but yes, this seems like a good idea. Pygments could be used on the source, but we'd have to figure out what to do with the line numbers.

This would probably require implementing our own more customized unittest.TestRunner that does more than just customize the unittest.TestResult to remove the traceback, as we do now: https://github.com/twosigma/marbles/blob/master/marbles/core/marbles/core/main.py

FWIW, pytest does this to customize its runner: https://github.com/pytest-dev/pytest/blob/master/src/_pytest/runner.py. Probably more than we need.

Might also be worth setting up Appveyor to test the colored output on windows. We should probably have windows tests anyway, and I'm not sure how we'd test this in general other than checking for explicit ANSI color sequences, which seems gross, but might be acceptable.

Thanks!

[leifwalsh]

@JohnPaton one thing you can do for now is use an alternative test runner like green. I tried it on some marbles tests we have and it does work well, you still get the marbles failure message, but within green's colored output. The stacktrace still shows up though, I wonder how hard it would be to get green to drop that.

[CleanCut]

Not hard at all. I added -e / --no-tracebacks to green 2.13.0 just for you. Enjoy! 😉

Sorry about the delay, by the way. I totally missed the notification for CleanCut/green#191