Improve TAP Output

[jamestalmage]

There are lots of open issues regarding TAP output:

• TAP report prints test titles above not ok message #679 - TAP report prints tests titles above not ok message
• power-assert graph missing from TAP output #399 - power-assert graph not shown in TAP output.
• Include assertions in TAP output #324 - Include individual assertions in the TAP output.
• [Exploratory] Capture good assertion messages. #340 - Capture good assertion messages.
• Returning the line number of the test in the TAP reporter #648 - include line number of passing tests in TAP output

Remaining todo's from #340 (comment)

• Bring our tap output more inline with node-tap. Specifically, the yaml output for our errors. This should make the output from all the tap-mocha-reporters much nicer. For example, our errors currently have expected and actual properties, but the tap reporters prefer found and wanted. Also, we should use stack-utils to attach the at property (file name and line number of failure).
• Finish up work on tap-emitter and integrate into AVA.

Notes:

We should be using the TAP v14 draft specification.

Note that the spec does not address what properties should be included in the yaml blocks. In that case we should just emulate node-tap (see list item above for examples of properties we need to add).

[xjamundx]

Probably related to this is the fact that ava tests/ --tap | tap-xunit > ava-results.xml produces some invalid XML characters that makes jenkins fail.

<testsuite name="calculator ESC[90mESC[2m›ESC[22mESC[39m actions ESC[90mESC[2m›ESC[22mESC[39m __tests__ ESC[90mESC[2m›ESC[22mESC[39m Actions.fetchOriginAmount ESC[90mESC[2m›ESC[22mESC[39m with no changes a noop function should be returned" tests="1" failures="0" errors="0">

Jenkins output looks like this:

Failed to read test report file /jobvolume/test-test-test/workspace/metrics-ci/ava-results.xml
org.dom4j.DocumentException: Error on line 3 of document file:///jobvolume/test-test-test/workspace/metrics-ci/ava-results.xml : An invalid XML character (Unicode: 0x{2}) was found in the value of attribute "{1}" and element is "1b". Nested exception: An invalid XML character (Unicode: 0x{2}) was found in the value of attribute "{1}" and element is "1b".

[sotojuan]

@xjamundx Can you post the actual tests that are giving you the strange XML output? I just tried tap-xunit with some simple tests and it worked fine.

[xjamundx]

@sotojuan it needs to be spread across multiple files:

test/test.js
test/test2.js

running this command

./node_modules/.bin/ava test/test*.js  --tap | ./node_modules/.bin/tap-xunit

The issue (from what I can tell is the little › arrows)

<?xml version="1.0"?>
<testsuites>
  <testsuite name="test › this is ta thing" tests="1" failures="0" errors="0">
    <testcase name="#1 test › this is ta thing"/>
  </testsuite>
  <testsuite name="test2 › this is ta thing" tests="1" failures="0" errors="0">
    <testcase name="#2 test2 › this is ta thing"/>
  </testsuite>
</testsuites>

The test I'm using is:

var test = require('ava')
test('this is ta thing', t => t.pass())

[sotojuan]

Thanks for the feedback. I'll look into it later today.

@sindresorhus would this be something we can fix (the arrows are not valid XML characters) or is that the responsibility of tap reporters?

[sindresorhus]

That's something the reporter should fix. We can't possible correctly escape our titles for every possible scenario.

[xjamundx]

Thanks for the tip @sindresorhus

[sindresorhus]

It seems we also incorrectly include some ansi escape codes in the title ESC[90mESC[2m. We should use strip-ansi on the title in the tap reporter. @sotojuan Wanna do a PR for that?

(The actual issue is still that the reporter doesn't escape the title though)

[xjamundx]

Thanks for following up. I'll get on the tap-xunit folks about fixing it on their end.

[sotojuan]

Opened #792 to help with the ANSI codes. Once it gets merged please test again with the latest master @xjamundx.

[xjamundx]

Can anyone confirm if the todo output is correct? The tap-xunit reporter is listing any ava todos as failures.

Edit: ava seems to be correct, filed an issue here:
aghassemi/tap-xunit#8