Custom markers don't show custom color in 'short test summary info' section

[jeffwright13]

Discussed in #9958

^{Originally posted by jeffwright13 May 14, 2022}
At the end of a pytest run, you have a section called "short test summary info," which terminates in a single line that shows final metrics, all color-coded according to outcome. It doesn't appear as if this final line uses the custom color to color-code tests that are marked with custom markers. I do see that the initial "test session starts" section does color-code those tests, however.

Am I missing something or is this a deficiency in the final section processing code? I'd like to see that final line show my 'snowflake' tests in cyan!

Output:

pytest.ini:

[pytest]
markers =
    snowflake: mark a test as flaky (may pass, may fail)

plugin.py:

def pytest_report_teststatus(report: TestReport, config: Config):
    """Put snowflake tests into a separate group from other failures."""
    if report.when == "call" and 'snowflake' in report.keywords.keys() and report.keywords["snowflake"]:
        return 'snowflake', '❄', ('SNOWFLAKE', {"cyan": True})

test1.py:

def test_aa_ok():
    print("This test doesn't have much to say, but it passes - so OK!!")
    pass

test2.py:

def test_aa_ok():
    print("This test doesn't have much to say, and it fails - not OK!!")
    assert 0

test_snowflake.py:

@pytest.mark.snowflake
def test_snowflake_1():
    assert True


@pytest.mark.snowflake
def test_snowflake_2():
    assert False

[jeffwright13]

I hacked together a solution to this problem by slightly modifying _build_normal_summary_stats_line() in src/_pytest/terminal.py, and adding a custom attribute to the config object. I was hoping to leverage some of the built-in hooks and not change any source code, but I couldn't find one that did the job. Maybe someone else has a better idea or a cleaner implementation?

In plugin.py or conftest.py:

import pytest
from _pytest.config import Config
from _pytest.reports import TestReport


def pytest_report_teststatus(report: TestReport, config: Config):
    """Put snowflake tests into a separate group from other failures."""
    if (
        report.when == "call"
        and "snowflake" in report.keywords.keys()
        and report.keywords["snowflake"]
    ):
        c = config._color_for_special_type = {"type": "snowflake", "symbol": "❄", "color": "cyan"}
        return c["type"], c["symbol"], (c["type"].upper(), {c["color"]: True})

In terminal.py:

    def _build_normal_summary_stats_line(
        self,
    ) -> Tuple[List[Tuple[str, Dict[str, bool]]], str]:
        main_color, known_types = self._get_main_color()
        parts = []

        for key in known_types:
            reports = self._get_reports_to_display(key)
            if reports:
                count = len(reports)
                color = _color_for_type.get(key, _color_for_type_default)
→               # ADDED CODE ↓↓↓
→               if key not in KNOWN_TYPES and self.config._color_for_special_type:
→                   color = self.config._color_for_special_type["color"]
→               # ADDED CODE ↑↑↑
                markup = {color: True, "bold": color == main_color}
                parts.append(("%d %s" % pluralize(count, key), markup))

        if not parts:
            parts = [("no tests ran", {_color_for_type_default: True})]

        return parts, main_color

This results in the short test summary info line correctly showing the custom result type in cyan:

[nicoddemus]

Yeah that color coding of the status is hard coded in terminal.py, without a chance for hooks to customize it.

One solution would be to extend pytest_report_teststatus so it can return either a 3-tuple (the current value) or a 4-tuple, with the last tuple item being the color. I don't like this because returning a 3 or 4 tuple feels hackish, and doesn't leave much room for further customization in the future (if we need yet another information later, do we then can return 3, 4, or 5 item tuples? Yikes).

Possibly a better solution is to return Union of 3-tuple (for backward compatibility) and a new small attrs class with the same information, additionally with color support. This makes it easier to understand and to extend in the future.

[jeffwright13]

Hey, thanks for the reply. Is this something we should leave to the devs, or would it be something I could try myself and submit a PR for? I haven't contributed to a public repo yet so I don't know how these things work. :-)

[nicoddemus]

I'm OK with implementing the change I outlined above (the Union with 3-tuple and a new small attrs class), but let's ask other maintainers if they find it a good idea or if they don't like it (to save you from working on a PR which will then be rejected).

cc @The-Compiler @RonnyPfannschmidt @asottile @bluetech

[asottile]

seems fine to me! it could even be made backward compatible (with deprecation) by making __iter__ act like the 3-tuple

[RonnyPfannschmidt]

as far as im understanding, the desire here is to either take the color of the verbose word, or the symbol and apply it to the status line

i would like to propose that we jsut take the first color given to category as well, and warn if mutliple values are given

[RonnyPfannschmidt]

i want to weight in, that we ideally avoid propagating the terminal-writer styling apis any more than they are already in by accident,
i really would like to see something like Rich styled text instead

[asottile]

is the eventual proposal to replace coloring with rich? as much as rich seems really cool I think it's a little unstable at the moment and a fairly heft dependency to pull in

[RonnyPfannschmidt]

im not proposing to replace coloring with rich anytime soon (in fact - if ever i would like to see a official pytest-rich/pytest-textual plugin in future that enables more fancy output without imposing the details on normal testruns

but i want to avoid adding places of the terminalwriter styling api which is quite dated and locks us into 16 color terminals + no theming forever

[nicoddemus]

as much as rich seems really cool I think it's a little unstable at the moment and a fairly heft dependency to pull in

Same thoughts here: really cool but a hefty dependency to bring in to a popular test runner.

i want to weight in, that we ideally avoid propagating the terminal-writer styling apis any more than they are already in by accident,

I had exactly the same nagging thought, but I think in this case we can be conservative and only allow the normal color names that we support, as strings: Literal["red", "blue", "cyan"] (and so on).

This seems safe enough because it is generic and can be easily ported to something else in the future.

in fact - if ever i would like to see a official pytest-rich/pytest-textual plugin in future that enables more fancy output without imposing the details on normal testruns

I have created https://github.com/nicoddemus/pytest-rich with exactly that purpose in mind. 😁

[RonnyPfannschmidt]

@nicoddemus what i want to propose is that we take the color, that we support for the verbose name/name anyway - and also use it for the unknown statuses

currently:

any unknown status -> yellow

desired -> unknown status but with verbose color -> green + color that extra status using its given color

for that we would have to pass markup into the status counting, and perhaps warn if a different value is passed + choose if we want the first or the last

[nicoddemus]

Oh I'm sorry, I misunderstood the issue: pytest_report_teststatus already supports returning an optional styling:

@hookspec(firstresult=True)
def pytest_report_teststatus(
    report: Union["CollectReport", "TestReport"], config: "Config"
) -> Tuple[str, str, Union[str, Mapping[str, bool]]]:
    ...

We are talking obviously about the counts: 1 failed, 1 passed, 2 snowflake.

Can you clarify what you mean here?

desired -> unknown status but with verbose color -> green + color that extra status using its given color

The -> and + notation you used are not clear to me.

[RonnyPfannschmidt]

@nicoddemus bascially i want to pass the markup that was added in that hook into the status collection as well,

that way we can use the color markup we already use in multiple places (the character and the word) in the summary as well

[jeffwright13]

Would that approach also address the lack of coloring in the === short test summary info === section?

[nicoddemus]

Would that approach also address the lack of coloring in the === short test summary info === section?

This has actually been fixed by #9875 and will be available in pytest 7.2.

[lordwelch]

Until this gets implemented, for non-production code it's possible to import the internal python module and add to the internal _color_for_type dict directly

import _pytest.terminal

_pytest.terminal._color_for_type['incompatible'] = 'red'