Add option to split report

[audricschiltknecht]

Add a new "--split-by" option that allows to split report per user-specified keys.
This is close to what was described in #33 (I think), but gives the user the flexibility to choose how to split the reports.

To do this, one must call pytest passing along one or more "--split-by" values. These are ordered values that will be used as keys to split and group reports in a hierarchy. These keys MUST be present in the report instance. They can be set in eg. the pytest_runtest_makereport hook.

For example, by defining:

@pytest.mark.hookwrapper
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    <...>
    report.language = f'{lang.title()}'
    report.lib = lib

then it can be called with pytest --split-by language --split-by lib and will generate reports grouped by language/lib.

If not option is passed in the pytest invocation, then the current, unified report is generated.

[davehunt]

Thank you for the patch @audricschiltknecht - this looks like a powerful feature indeed! Unfortunately I don't feel like I can give this the time it deserves for the next couple of weeks due to some upcoming travel. I did notice that the tests are failing, so perhaps that's something you could look into? I would also appreciate if you could add new tests for this feature.

Are there default categories that can be split, or is it necessary for categories to be added to the report? Perhaps some additional documentation would make this clearer. Whilst I'm away, if there are any other contributors that would like to provide feedback on this patch, that would really me help on my return. ❤️

[audricschiltknecht]

Hi!

This patch includes quite some changes, and would indeed deserve a bit of documentation. I'll also have a look at the tests, they passed fine on my local machine.

For now, there are no default categories defined. So if you do not pass the --split-by option, you get the current report. If you pass them, then you need to define them in the report instance (eg. using the hook). Maybe we could make some implicit values available? I'm open to suggestions/comments from contributors here.

Enjoy your travel!

[davehunt]

Just to add that I'm now back from my travels (for now). Please add a comment when you've addressed the tests and documentation, and are ready for me to take a look over this.

[audricschiltknecht]

Hello!
Sorry, I had and will have very busy days coming up for a while, not much time to work on this, but will try to do my best!

I took a look at the failing tests: node and python3.6.

I am not really sure what is wrong for the node tests since it looks like an issue with the headless chrome:

>> There was an error with headless chrome
Fatal error: Failed to launch chrome!
[0911/160432.361515:FATAL:zygote_host_impl_linux.cc(116)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/master/docs/linux_suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.
#0 0x55decbe78dfc base::debug::StackTrace::StackTrace()
#1 0x55decbdf8150 logging::LogMessage::~LogMessage()
#2 0x55decd2120c0 service_manager::ZygoteHostImpl::Init()
#3 0x55decbae3c1e content::ContentMainRunnerImpl::Initialize()
#4 0x55decbb18128 service_manager::Main()
#5 0x55decbae21a1 content::ContentMain()
#6 0x55decff1de9d headless::(anonymous namespace)::RunContentMain()
#7 0x55decff1df28 headless::HeadlessBrowserMain()
#8 0x55decbb157fa headless::HeadlessShellMain()
#9 0x55dec9f621ac ChromeMain
#10 0x7fdd0c789f45 __libc_start_main
#11 0x55dec9f6202a _start
Received signal 6
#0 0x55decbe78dfc base::debug::StackTrace::StackTrace()
#1 0x55decbe78961 base::debug::(anonymous namespace)::StackDumpSignalHandler()
#2 0x7fdd12414330 <unknown>
#3 0x7fdd0c79ec37 gsignal
#4 0x7fdd0c7a2028 abort
#5 0x55decbe777b5 base::debug::BreakDebugger()
#6 0x55decbdf85b9 logging::LogMessage::~LogMessage()
#7 0x55decd2120c0 service_manager::ZygoteHostImpl::Init()
#8 0x55decbae3c1e content::ContentMainRunnerImpl::Initialize()
#9 0x55decbb18128 service_manager::Main()
#10 0x55decbae21a1 content::ContentMain()
#11 0x55decff1de9d headless::(anonymous namespace)::RunContentMain()
#12 0x55decff1df28 headless::HeadlessBrowserMain()
#13 0x55decbb157fa headless::HeadlessShellMain()
#14 0x55dec9f621ac ChromeMain
#15 0x7fdd0c789f45 __libc_start_main
#16 0x55dec9f6202a _start
  r8: 00007fdd12a11a40  r9: 000019bc96780800 r10: 0000000000000008 r11: 0000000000000206
 r12: 00007ffeb55fd638 r13: 0000000000000161 r14: 00007ffeb55fd640 r15: 00007ffeb55fd648
  di: 0000000000000b8d  si: 0000000000000b8d  bp: 00007ffeb55fd180  bx: 00007ffeb55fd1f0
  dx: 0000000000000006  ax: 0000000000000000  cx: 00007fdd0c79ec37  sp: 00007ffeb55fd048
  ip: 00007fdd0c79ec37 efl: 0000000000000206 cgf: 002b000000000033 erf: 0000000000000000
 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.
TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md

I can reproduce the same issue on my local computer. Would appreciate any help/hint for that.

For python3.6, not sure what is wrong as it seems the test is stuck. I will try to reproduce it locally.

I will also try to update the documentation in my spare time :)

[khornlund]

Hi guys,

I've been looking for something like this so tried out your code - it splits on the given attribute correctly (very nice!) but looks like it has some bugs. For example:

1. The tick boxes to filter the tests are broken for me. You can remove the tick but they not longer filter.
2. If you click a header in a table to sort on that column, the table width shrinks.

Here is the report I generated from my mock test project:
report.zip

The mock test project I used to generate it is here:
RaceTest

You can use __runtests__.py to run or from command line:
pytest tests --html=report.html --split-by race

:)

[audricschiltknecht]

Hello,

I apologize, I haven't found time to work on this yet...

@khornlund, thanks for the test case. It is interesting because tests seem to include "#" in their name, thus ending up generating some href="#Race #0-title" which is invalid id. I am wondering if that's the reason why you experience some bugs.

I will need to look into that, thanks for the input!

[BeyondEvil]

@audricschiltknecht Any updates on this? Please let us know if you need any assistance! :)

[audricschiltknecht]

Hello,

Sorry for not finding the time to work on this before.
I've updated the PR, including:

• fixes to the issues found by @khornlund.
• some documentation.
• renamed the split-by to group-by, as this is more a grouping than a splitting.

I still need to add some tests.

[ssbarnea]

I am looking forward to see this refreshed and also a link to sample report generated with it, it will make much easier to evaluate and review it.

[AmanKAggarwal]

Hey! I was wondering what is up with this feature? I want to generate pytest report grouped by each test file, but I did not find any documentation on group-by in the latest pytest-html version! Please let me know as soon as possible. If this feature is not supported by pytest-html, I would have to look for some other HTML report generation to cover my requirements

[BeyondEvil]

Hey! I was wondering what is up with this feature? I want to generate pytest report grouped by each test file, but I did not find any documentation on group-by in the latest pytest-html version! Please let me know as soon as possible. If this feature is not supported by pytest-html, I would have to look for some other HTML report generation to cover my requirements

This feature is not available yet. It will be in a not so distant future, however I can't really give an exact time frame. Sorry.

[BeyondEvil]

Would @audricschiltknecht be interested in the reviving this for v4?

[audricschiltknecht]

Hello,
I completely forgot about that PR, thanks for reminding me.
I had a quick look at the changes for V4, and from my point of view, it's close to a full rewrite :)

I'll see what I can do, but I think I'll need some help as it has been some times since I've played with pytest-html.

[BeyondEvil]

Hello, I completely forgot about that PR, thanks for reminding me. I had a quick look at the changes for V4, and from my point of view, it's close to a full rewrite :)

I'll see what I can do, but I think I'll need some help as it has been some times since I've played with pytest-html.

Yes, it's basically a full rewrite.

Happy to help!

[vikshaw-Nokia]

Hi All,
@audricschiltknecht @BeyondEvil
I desperately need this Feature
As My html files are in GBs as of now.

[RonnyPfannschmidt]

What creates that size?

That particular scale may need a backing database

[vikshaw-Nokia]

@RonnyPfannschmidt
If you observe the Below Sample output

The Assertion failure and it trace is enough for me to debug,
But the stdcall is taking the a lot of memory
And when the TC runs for hours, It becomes quite heavy
a 50 Lines before and after the assertion would be enough for me to debug, but at the current point, I can't even open the HTML

[audricschiltknecht]

Hi All, @audricschiltknecht @BeyondEvil I desperately need this Feature As My html files are in GBs as of now.

I had a look at it in the beginning of this year, but unfortunately, the code has changed too much and I basically would need to rewrite from scratch. As I don't work with python much these days, it has dropped out of my TODO-list.

I'm going to close that PR as I don't want to keep people's hope up by making promise that I cannot fulfill, but anyone is welcomed to use the code as a base if they need to.