Flowchart of pytest test session states and hooks

[Sup3rGeo]

Hi,

This is a documentation request to have a flowchart of all the pytest test session states (setup, conftest, collecting, run setup, run test, run teardown, make report, etc..) together with all the applicable hooks.

Something similar to the one on logging module documentation (I suppose that for pytest this would be way more complex):
https://docs.python.org/2/howto/logging.html#logging-flow

I surely read the page on the hooks but not knowing all the steps that are taken by pytest it becomes a bit hard to guess which one we should be using. This would really improve the understanding on how pytest works, especially for plugin developers.

[nicoddemus]

Hi @Sup3rGeo, this would definitely be useful.

I think we can use sphinx.ext.graphviz to generate the flowcharts.

[nicoddemus]

Just played around with sphinx.ext.graphviz and this code:

.. digraph:: foo

    "pytest_cmdline_main" -> "pytest_cmdline_preparse" -> "pytest_load_initial_conftests";

    "pytest_cmdline_main" -> "pytest_cmdline_parse";

Produces this image:

So this is definitely doable. 👍

[Sup3rGeo]

Nice!

I would be willing to help with documentation (with graphviz) but I would just need someone who knows pytest execution flow to define in any way (text or in a tool like draw.io for instance)

[nicoddemus]

@Sup3rGeo thanks for the offer, we appreciate it!

The hook order can be obtained by passing --debug, which will make pytest write a pytestdebug.log file with contents like this:

versions pytest-3.4.1.dev39+ga1c3c3d.d20180226, py-1.5.2, python-3.6.3.final.0
cwd=C:\pytest
args=['.tmp\\test_fail.py', '--debug', '-v']

  pytest_cmdline_main [hook]
      config: <_pytest.config.Config object at 0x0000016BDEBF8BE0>
    pytest_plugin_registered [hook]
        plugin: <Session '.tmp'>
        manager: <_pytest.config.PytestPluginManager object at 0x0000016BDDD63F60>
    finish pytest_plugin_registered --> [] [hook]
    pytest_configure [hook]
        config: <_pytest.config.Config object at 0x0000016BDEBF8BE0>
      pytest_plugin_registered [hook]
          plugin: <_pytest.cacheprovider.LFPlugin object at 0x0000016BDEFF4358>
          manager: <_pytest.config.PytestPluginManager object at 0x0000016BDDD63F60>
      finish pytest_plugin_registered --> [] [hook]
      pytest_plugin_registered [hook]
          plugin: <_pytest.logging.LoggingPlugin object at 0x0000016BDEFF4518>
          manager: <_pytest.config.PytestPluginManager object at 0x0000016BDDD63F60>
      finish pytest_plugin_registered --> [] [hook]
    find_module called for: py._io [assertion]
    find_module called for: py._io.terminalwriter [assertion]
    find_module called for: termios [assertion]
    find_module called for: termios [assertion]
      pytest_plugin_registered [hook]
...

Which can be used as a starting point, and in any case if there are any questions we would gladly answer them over a WIP PR. 😁

[Sup3rGeo]

That is really great, can't believe all this time I didn't know it!

I will try to come up with something then and create PR so you can review it.

[Sup3rGeo]

I was thinking about doing something like this:
https://github.com/Sup3rGeo/pytest/blob/ace24468d336fdd4a8c2eaa3c3e9a32da961f98b/doc/en/img/Pytest%20hooks.png

However I am not sure how to do this with graphviz. Mainly to represent the fact that some hooks are executed inside the context of other hooks.

What do you think?

[nicoddemus]

I was thinking about doing something like this
However I am not sure how to do this with graphviz.

I never used graphviz myself 🤕

What do you think?

I think it looks great! Also, perhaps you can write a script to generate the graphviz output from a pytestdebug.log file? Then we can automate this graph generation for each release.

[Sup3rGeo]

I think it looks great! Also, perhaps you can write a script to generate the graphviz output from a pytestdebug.log file? Then we can automate this graph generation for each release.

That is a great idea. From what I could see, not all the hooks are shown in the log file. Is that dependent on what pytest features the test is actually using?
If that's the case, is there any test that actually outputs all the hooks, based on which we can derive the graph?

[nicoddemus]

If that's the case, is there any test that actually outputs all the hooks, based on which we can derive the graph?

Not offhand, but IMHO don't worry about it too much: having the script in place generating a graph with the most common hooks and integrated with our release process is already a huge addition. We can always improve that bit later. 😁

[Sup3rGeo]

So basically I wrote something to parse the log file and create a tree using anytree module:
https://github.com/Sup3rGeo/pytest/blob/master/doc/pytesthooks.py

The result:

root
└── pytest_cmdline_main
    ├── pytest_plugin_registered
    ├── pytest_configure
    │   └── pytest_plugin_registered
    ├── pytest_sessionstart
    │   ├── pytest_plugin_registered
    │   └── pytest_report_header
    ├── pytest_collection
    │   ├── pytest_collectstart
    │   ├── pytest_make_collect_report
    │   │   ├── pytest_collect_file
    │   │   │   └── pytest_pycollect_makemodule
    │   │   └── pytest_pycollect_makeitem
    │   │       └── pytest_generate_tests
    │   │           └── pytest_make_parametrize_id
    │   ├── pytest_collectreport
    │   ├── pytest_itemcollected
    │   ├── pytest_collection_modifyitems
    │   └── pytest_collection_finish
    │       └── pytest_report_collectionfinish
    ├── pytest_runtestloop
    │   └── pytest_runtest_protocol
    │       ├── pytest_runtest_logstart
    │       ├── pytest_runtest_setup
    │       │   └── pytest_fixture_setup
    │       ├── pytest_runtest_makereport
    │       ├── pytest_runtest_logreport
    │       │   └── pytest_report_teststatus
    │       ├── pytest_runtest_call
    │       │   └── pytest_pyfunc_call
    │       ├── pytest_runtest_teardown
    │       │   └── pytest_fixture_post_finalizer
    │       └── pytest_runtest_logfinish
    ├── pytest_sessionfinish
    │   └── pytest_terminal_summary
    └── pytest_unconfigure

The challenge was to not repeat hook calls and merge internal hooks from some calls that repeat but not exactly one after the other. But it seems to be working fine.

I faced just one problem though - the last pytest_runtest_setup [hook] line did not have a corresponding finish pytest_runtest_setup. But I am going to run the test again to see.

It also has the ability to generate dot graphs out of it, although the tree text representation is already very good and this graph is just unreadable:

https://github.com/Sup3rGeo/pytest/blob/master/doc/pytesthooks.png

[Sup3rGeo]

So I ran a test again and same thing happened: One of the pytest_runtest_setup hooks did not have a corresponding finish pytest_runtest_setup line. This will confuse the script I wrote.

One possible solution is to consider the leading whitespaces to do the nesting.

[Sup3rGeo]

This might be also useful for #3113 and #2670

[RonnyPfannschmidt]

looks fabulous, can we add the scripts somewhere to the pytest repo so we can update in future (maybe it makes sense to put this into pluggy)

[Sup3rGeo]

@RonnyPfannschmidt nice to hear that!

Merging repeated calls inside a parent is problematic because it does not capture the fact that, for instance, pytest__runtest_makereport runs three times, after setup, call, and teardown respectively.

On the other hand, if we leave everything as is then typically you can have much useless repetition, for example many many pytest_plugin_registered and then the hooks from pytest_logstart to pytest_logfinish for every test case if you have many.

So we might have to craft a test that will call just the right amount of hooks, so it looks good on the script, or we should perhaps think of other way.

If we do it manually then it is easier to indicate that a certain hook sequence is repeated for every item collected, etc.