Flowchart of pytest test session states and hooks

[Sup3rGeo]

Hi,

This is a documentation request to have a flowchart of all the pytest test session states (setup, conftest, collecting, run setup, run test, run teardown, make report, etc..) together with all the applicable hooks.

Something similar to the one on logging module documentation (I suppose that for pytest this would be way more complex):
https://docs.python.org/2/howto/logging.html#logging-flow

I surely read the page on the hooks but not knowing all the steps that are taken by pytest it becomes a bit hard to guess which one we should be using. This would really improve the understanding on how pytest works, especially for plugin developers.

[nicoddemus]

Hi @Sup3rGeo, this would definitely be useful.

I think we can use sphinx.ext.graphviz to generate the flowcharts.

[nicoddemus]

Just played around with sphinx.ext.graphviz and this code:

.. digraph:: foo

    "pytest_cmdline_main" -> "pytest_cmdline_preparse" -> "pytest_load_initial_conftests";

    "pytest_cmdline_main" -> "pytest_cmdline_parse";

Produces this image:

So this is definitely doable. 👍

[Sup3rGeo]

Nice!

I would be willing to help with documentation (with graphviz) but I would just need someone who knows pytest execution flow to define in any way (text or in a tool like draw.io for instance)

[nicoddemus]

@Sup3rGeo thanks for the offer, we appreciate it!

The hook order can be obtained by passing --debug, which will make pytest write a pytestdebug.log file with contents like this:

versions pytest-3.4.1.dev39+ga1c3c3d.d20180226, py-1.5.2, python-3.6.3.final.0
cwd=C:\pytest
args=['.tmp\\test_fail.py', '--debug', '-v']

  pytest_cmdline_main [hook]
      config: <_pytest.config.Config object at 0x0000016BDEBF8BE0>
    pytest_plugin_registered [hook]
        plugin: <Session '.tmp'>
        manager: <_pytest.config.PytestPluginManager object at 0x0000016BDDD63F60>
    finish pytest_plugin_registered --> [] [hook]
    pytest_configure [hook]
        config: <_pytest.config.Config object at 0x0000016BDEBF8BE0>
      pytest_plugin_registered [hook]
          plugin: <_pytest.cacheprovider.LFPlugin object at 0x0000016BDEFF4358>
          manager: <_pytest.config.PytestPluginManager object at 0x0000016BDDD63F60>
      finish pytest_plugin_registered --> [] [hook]
      pytest_plugin_registered [hook]
          plugin: <_pytest.logging.LoggingPlugin object at 0x0000016BDEFF4518>
          manager: <_pytest.config.PytestPluginManager object at 0x0000016BDDD63F60>
      finish pytest_plugin_registered --> [] [hook]
    find_module called for: py._io [assertion]
    find_module called for: py._io.terminalwriter [assertion]
    find_module called for: termios [assertion]
    find_module called for: termios [assertion]
      pytest_plugin_registered [hook]
...

Which can be used as a starting point, and in any case if there are any questions we would gladly answer them over a WIP PR. 😁

[Sup3rGeo]

That is really great, can't believe all this time I didn't know it!

I will try to come up with something then and create PR so you can review it.

[Sup3rGeo]

I was thinking about doing something like this:
https://github.com/Sup3rGeo/pytest/blob/ace24468d336fdd4a8c2eaa3c3e9a32da961f98b/doc/en/img/Pytest%20hooks.png

However I am not sure how to do this with graphviz. Mainly to represent the fact that some hooks are executed inside the context of other hooks.

What do you think?

[nicoddemus]

I was thinking about doing something like this
However I am not sure how to do this with graphviz.

I never used graphviz myself 🤕

What do you think?

I think it looks great! Also, perhaps you can write a script to generate the graphviz output from a pytestdebug.log file? Then we can automate this graph generation for each release.

[Sup3rGeo]

I think it looks great! Also, perhaps you can write a script to generate the graphviz output from a pytestdebug.log file? Then we can automate this graph generation for each release.

That is a great idea. From what I could see, not all the hooks are shown in the log file. Is that dependent on what pytest features the test is actually using?
If that's the case, is there any test that actually outputs all the hooks, based on which we can derive the graph?

[nicoddemus]

If that's the case, is there any test that actually outputs all the hooks, based on which we can derive the graph?

Not offhand, but IMHO don't worry about it too much: having the script in place generating a graph with the most common hooks and integrated with our release process is already a huge addition. We can always improve that bit later. 😁

[Sup3rGeo]

So basically I wrote something to parse the log file and create a tree using anytree module:
https://github.com/Sup3rGeo/pytest/blob/master/doc/pytesthooks.py

The result:

root
└── pytest_cmdline_main
    ├── pytest_plugin_registered
    ├── pytest_configure
    │   └── pytest_plugin_registered
    ├── pytest_sessionstart
    │   ├── pytest_plugin_registered
    │   └── pytest_report_header
    ├── pytest_collection
    │   ├── pytest_collectstart
    │   ├── pytest_make_collect_report
    │   │   ├── pytest_collect_file
    │   │   │   └── pytest_pycollect_makemodule
    │   │   └── pytest_pycollect_makeitem
    │   │       └── pytest_generate_tests
    │   │           └── pytest_make_parametrize_id
    │   ├── pytest_collectreport
    │   ├── pytest_itemcollected
    │   ├── pytest_collection_modifyitems
    │   └── pytest_collection_finish
    │       └── pytest_report_collectionfinish
    ├── pytest_runtestloop
    │   └── pytest_runtest_protocol
    │       ├── pytest_runtest_logstart
    │       ├── pytest_runtest_setup
    │       │   └── pytest_fixture_setup
    │       ├── pytest_runtest_makereport
    │       ├── pytest_runtest_logreport
    │       │   └── pytest_report_teststatus
    │       ├── pytest_runtest_call
    │       │   └── pytest_pyfunc_call
    │       ├── pytest_runtest_teardown
    │       │   └── pytest_fixture_post_finalizer
    │       └── pytest_runtest_logfinish
    ├── pytest_sessionfinish
    │   └── pytest_terminal_summary
    └── pytest_unconfigure

The challenge was to not repeat hook calls and merge internal hooks from some calls that repeat but not exactly one after the other. But it seems to be working fine.

I faced just one problem though - the last pytest_runtest_setup [hook] line did not have a corresponding finish pytest_runtest_setup. But I am going to run the test again to see.

It also has the ability to generate dot graphs out of it, although the tree text representation is already very good and this graph is just unreadable:

https://github.com/Sup3rGeo/pytest/blob/master/doc/pytesthooks.png

[Sup3rGeo]

So I ran a test again and same thing happened: One of the pytest_runtest_setup hooks did not have a corresponding finish pytest_runtest_setup line. This will confuse the script I wrote.

One possible solution is to consider the leading whitespaces to do the nesting.

[Sup3rGeo]

This might be also useful for #3113 and #2670

[RonnyPfannschmidt]

looks fabulous, can we add the scripts somewhere to the pytest repo so we can update in future (maybe it makes sense to put this into pluggy)

[Sup3rGeo]

@RonnyPfannschmidt nice to hear that!

Merging repeated calls inside a parent is problematic because it does not capture the fact that, for instance, pytest__runtest_makereport runs three times, after setup, call, and teardown respectively.

On the other hand, if we leave everything as is then typically you can have much useless repetition, for example many many pytest_plugin_registered and then the hooks from pytest_logstart to pytest_logfinish for every test case if you have many.

So we might have to craft a test that will call just the right amount of hooks, so it looks good on the script, or we should perhaps think of other way.

If we do it manually then it is easier to indicate that a certain hook sequence is repeated for every item collected, etc.

[RonnyPfannschmidt]

we have a set of tools that use pluggy - all of those could benefit from a automated way to deal with hooks and their nesting ^^

this is a topic where playing and experimenting with collecting and displaying the data might give insights

[nicoddemus]

If we do it manually then it is easier to indicate that a certain hook sequence is repeated for every item collected, etc.

One idea is to manually indicate to the script a set of hooks that should appear only once, because this will vary from project to project. 😁

[Sup3rGeo]

Just stumbled upon this presentation which has a similar tree view of the hooks (including annotations where there is repetitions):

http://devork.be/talks/pluggy/pluggy.html

Do you guys know him? Should we contact him and see if he might have a script for that already?

[nicoddemus]

Sure we know @flub! 😁

@flub did you use a script to generate that?

[nicoddemus]

Here's the video of the presentation for those interested: https://www.youtube.com/watch?v=zZsNPDfOoHU

[flub]

Hey, I'm afraid I just made that up from what I knew and eyeballing the code somewhat. I didn't attempt to make it complete or overly accurate as that was not the point of that slide.

So you're miles ahead with the work done here! The script with some hints about which hook calls to collapse like @nicoddemus suggests seems like it should be pretty good.

[Sup3rGeo]

I updated the script to use a list of hooks to merge or to exclude from merging, but other than that I am out of ideas on the best way to do things like, for instance, indicate that pytest_runtest_makereport should come after setup,call and teardown.

We could define it to appear a number of times (in this case 3), or indicate that it should have one after each one of them explicitly.

But because this is really code dependent, I cannot really think of a nice general way for solving this.

[nicoddemus]

We could define it to appear a number of times (in this case 3), or indicate that it should have one after each one of them explicitly.

Explicitly defining seems like a good solution to me. 👍

[tadeoos]

Hi @Sup3rGeo, any updates on this? The discussion gives an impression of a great journey towards a fine goal which suddenly stopped and went quiet ;)

[Sup3rGeo]

Hi! So this stopped so far because I couldn't figure out:

• best way to treat repeated hook calls. Some of them we would like to keep (makereport after each state - setup, call, teardown) while some are redundant (plugin_registered)
• how to have a comprehensive image of all hooks without manual intervention. Depending on the settings used to run the pytest section from which we process the debug file, some hooks may be called while others not
• It is better to have an option for the debug file to have a simpler view, with only hook calls, then to process in the script.

I tend to think the best approach would be to implement the last point mentioned as a first step (very clear) and then the resulting tree could be processed the way the user wants as a second step (not clear at all, so maybe should be out of scope of this).

[hectorv]

I got here because I was just thinking how cool this would be 😄
By the way, do you guys know why the pytest_addoption hook doesn't appear in pytestdebug.log? I'm definitely using it in my conftest.py.

[orihabAnyvision]

This is amazing, even unfinished. Doesnt have to be perfect/include EVERYTHING to be a big help. 10x!

[MukunthanML]

So basically I wrote something to parse the log file and create a tree using anytree module:
https://github.com/Sup3rGeo/pytest/blob/master/doc/pytesthooks.py

The result:

root
└── pytest_cmdline_main
    ├── pytest_plugin_registered
    ├── pytest_configure
    │   └── pytest_plugin_registered
    ├── pytest_sessionstart
    │   ├── pytest_plugin_registered
    │   └── pytest_report_header
    ├── pytest_collection
    │   ├── pytest_collectstart
    │   ├── pytest_make_collect_report
    │   │   ├── pytest_collect_file
    │   │   │   └── pytest_pycollect_makemodule
    │   │   └── pytest_pycollect_makeitem
    │   │       └── pytest_generate_tests
    │   │           └── pytest_make_parametrize_id
    │   ├── pytest_collectreport
    │   ├── pytest_itemcollected
    │   ├── pytest_collection_modifyitems
    │   └── pytest_collection_finish
    │       └── pytest_report_collectionfinish
    ├── pytest_runtestloop
    │   └── pytest_runtest_protocol
    │       ├── pytest_runtest_logstart
    │       ├── pytest_runtest_setup
    │       │   └── pytest_fixture_setup
    │       ├── pytest_runtest_makereport
    │       ├── pytest_runtest_logreport
    │       │   └── pytest_report_teststatus
    │       ├── pytest_runtest_call
    │       │   └── pytest_pyfunc_call
    │       ├── pytest_runtest_teardown
    │       │   └── pytest_fixture_post_finalizer
    │       └── pytest_runtest_logfinish
    ├── pytest_sessionfinish
    │   └── pytest_terminal_summary
    └── pytest_unconfigure

The challenge was to not repeat hook calls and merge internal hooks from some calls that repeat but not exactly one after the other. But it seems to be working fine.

I faced just one problem though - the last pytest_runtest_setup [hook] line did not have a corresponding finish pytest_runtest_setup. But I am going to run the test again to see.

It also has the ability to generate dot graphs out of it, although the tree text representation is already very good and this graph is just unreadable:

https://github.com/Sup3rGeo/pytest/blob/master/doc/pytesthooks.png

Could you please tell why pytest_plugin_registered is repeated under pytest_sessionstart. I checked pytestdebug.log and repeats same plugins' registeration. Please clarify.

root
└── pytest_cmdline_main
├── pytest_plugin_registered
├── pytest_configure
│ └── pytest_plugin_registered <-------- registers all _pytest plugins, conftest, setup tools plugins
├── pytest_sessionstart
│ ├── pytest_plugin_registered <-------- again registers all same above _pytest plugins, conftest, setup tools plugins
│ └── pytest_report_header

[Stefanhg]

@RonnyPfannschmidt I keep coming back to this task as it is the only documentation showing the hook call order or in general the hooks available.. Is there a reason why the hooks flow still isn't added to the documentation? I think that the script Sup3rGeo made is fine as-is.

Another thing I am confused about. Why is pytest_runtest_makereport called before pytest_runtest_call? I would assume the report is made after the test is executed.

[The-Compiler]

Is there a reason why the hooks flow still isn't added to the documentation?

Probably because nobody took the time to clean things up, perhaps annotate them a bit, and add them to the docs (and keep them updated once there are new hooks). PRs welcome!

As another starting point, here is what I have in my company training slides:

Another thing I am confused about. Why is pytest_runtest_makereport called before pytest_runtest_call? I would assume the report is made after the test is executed.

It's called once in the setup phase and once in the call phase (and again in the teardown phase). The script probably falsely hides those as duplicates. Another example why this probably needs some more work with someone actually going through the hook output manually and making it human-readable without leaving crucial details out.

[Stefanhg]

As another starting point, here is what I have in my company training slides:

That is surely MUCH better than what I had access to before. Trying to learn all these hooks and how it works is pretty tough when you don't have much to go with.

Another example why this probably needs some more work with someone actually going through the hook output manually

That's a good point. Hopefully somebody will find time to do it at some point.

[storenth]

Looks like there are no actions we need to do, I suggest to close the issue. Pretty doc already exist.

[nhairs-lumin]

Looks like there are no actions we need to do, I suggest to close the issue. Pretty doc already exist.

Specifically I think @storenth is reffering to the Hooks section located on that page: https://docs.pytest.org/en/8.0.x/reference/reference.html#hooks

[Stefanhg]

@storenth I disagree.
The task is not completed and a flow chart is still needed. The amazing pictures that @The-Compiler shared is something I use constantly while developing because the docs are not sufficient.

I think the actual output that the pytesthooks.py file gives is pretty strong, if any improvements should be done it should be either linking the output from pytesthooks.py to the actual hooks in docs so it is easier to see the arguments available for that hook but that is only nice-to-have.

I am not so strong in Sphinx yet but it should be possible to:

1. Generate a tmp path
2. Generate a simple passing testcase def test_pass: pass
3. Run pytest --debug
4. Output pytesthooks.py to a file
5. Attach the contents of the file to Sphinx

And then the task could be completed ..Right?

[The-Compiler]

As I said above, the raw output generated by that script has various issues which will probably confuse people reading the docs rather than help.