Creating a custom error report/traceback?

[michaelboulton]

I've got a pytest plugin that loads files from YAML and generates tests from it (https://github.com/taverntesting/tavern/blob/master/tavern/testutils/pytesthook.py), inspired by the example in the pytest documentation. It loads the tests into a subclass of pytest.Item and overrides runtest to run the tests, and reportinfo to give a hint as to which test it is in the pytest output.

A test might look like this:

test_name: Test approximate numbers

stages:
  - name: Match pi approximately
    request:
      url: "{host}/pi"
      method: GET
    response:
      status_code: 200
      body:
        pi: 3.2

but currently a test failure will look like this (using --tb=short, otherwise it's several hundred lines long):

______ /home/michael/code/tavern/tavern/tests/integration/test_typetokens.tavern.yaml::Test approximate numbers _______
../../.tox/py36-generic/lib/python3.6/site-packages/_pytest/runner.py:198: in __init__
    self.result = func()
../../.tox/py36-generic/lib/python3.6/site-packages/_pytest/runner.py:181: in <lambda>
    lambda: ihook(item=item, **kwds),
../../.tox/py36-generic/lib/python3.6/site-packages/pluggy/__init__.py:617: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
../../.tox/py36-generic/lib/python3.6/site-packages/pluggy/__init__.py:222: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../../.tox/py36-generic/lib/python3.6/site-packages/pluggy/__init__.py:216: in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
../../.tox/py36-generic/lib/python3.6/site-packages/pluggy/callers.py:201: in _multicall
    return outcome.get_result()
../../.tox/py36-generic/lib/python3.6/site-packages/pluggy/callers.py:76: in get_result
    raise ex[1].with_traceback(ex[2])
../../.tox/py36-generic/lib/python3.6/site-packages/pluggy/callers.py:180: in _multicall
    res = hook_impl.function(*args)
../../.tox/py36-generic/lib/python3.6/site-packages/_pytest/runner.py:109: in pytest_runtest_call
    item.runtest()
../../.tox/py36-generic/lib/python3.6/site-packages/tavern/testutils/pytesthook.py:199: in runtest
    run_test(self.path, self.spec, global_cfg)
../../.tox/py36-generic/lib/python3.6/site-packages/tavern/core.py:110: in run_test
    run_stage(sessions, stage, tavern_box, test_block_config)
../../.tox/py36-generic/lib/python3.6/site-packages/tavern/core.py:145: in run_stage
    saved = v.verify(response)
../../.tox/py36-generic/lib/python3.6/site-packages/tavern/_plugins/rest/response.py:206: in verify
    raise TestFailError("Test '{:s}' failed:\n{:s}".format(self.name, self._str_errors()))
E   tavern.util.exceptions.TestFailError: Test 'Match pi approximately' failed:
E   - Value mismatch in body: Key mismatch: (expected["pi"] = '3.2', actual["pi"] = '3.141592653589793')

I want to make this better so it can print out the input YAML and which line caused the failure etc, rather than printing out the traceback from the pytest run. Perhaps so that it looks more like this (just an example of the kind of thing I'm trying to do, there could actually be multiple lines which caused an error):

______ /home/michael/code/tavern/tavern/tests/integration/test_typetokens.tavern.yaml::Test approximate numbers _______
Contents:
---
test_name: Test approximate numbers

stages:
  - name: Match pi approximately
    request:
      url: "{host}/pi"
      method: GET
    response:
      status_code: 200
      body:
        pi: 3.2
        ^^^^^^^ 
E   tavern.util.exceptions.TestFailError: Test 'Match pi approximately' failed:
E   - Value mismatch in body: Key mismatch: (expected["pi"] = '3.2', actual["pi"] = '3.141592653589793')

I'm not really sure where to even start doing this however, I've gone through the pytest source code a bit and there's quite a few places where it handles outputting the errors:

• TerminalReporter
• repr_failure on the Item
• The pytest_runtest_logreport hook

To be clear, I don't want to mess around with the assertion rewriting, I just want to get the final output traceback in a format that's easier to understand for an end user by showing where in the input caused the failure. I also don't want to do something like pytest-html that just generates an external error report based on the result of the test runs.

[pytestbot]

GitMate.io thinks possibly related issues are #756 (internal error building traceback), #642 (misleading error report), #1785 (Extraneous Traceback from a custom non-Python test), #1194 (Add custom reporting output), and #76 (report characters don't support displaying of errors).

[nicoddemus]

Hi @michaelboulton,

I would start with Item.repr_failure. I'm on the run so I can't provide an example at the moment, sorry.

Hope this helps!

[nicoddemus]

Item.repr_failure is how pytest-cpp customizes the exceptions: https://github.com/pytest-dev/pytest-cpp/blob/master/pytest_cpp/plugin.py#L61

You can use that as an example! 👍

[michaelboulton]

For anyone else who comes along this, we've used the above as inspiration along with Pyyaml to get some metadata about the input data for the output. For example showing a missing input variable:

implementation: https://github.com/taverntesting/tavern/blob/efe1c5f3d7c46bd64a0c8ee79308ed31cb291f6c/tavern/testutils/pytesthook.py#L368-L479

Thanks for the help!

[nicoddemus]

Thanks for the follow up! 👍