Junitxml plugin indicates a xfailed test as a skipped test

[harmin-parra]

Environment
platform linux -- Python 3.6.9, pytest-5.4.0, py-1.8.1, pluggy-0.13.1

My test has this fixture
@pytest.mark.xfail(reason="my reason")

Execution command
pytest --junixml=report.xml

Result
The xfailed test is indicated as an skipped test in the junit XML report.

It should be reported as a failed test.

I want my Jenkins test job status to be failed instead of passed.

[The-Compiler]

Could you explain a bit more about your usecase? The whole point about marking a test as xfailed is that it doesn't count as failed, no?

[harmin-parra]

From pytest point of view, I can understand that you want the xfail test to be considered as not failed.

But it bothers me that in the junit XML report it is considered as skipped and that the overall jenkins job status is considered as passed.

When people see the jenkins job status as passed, they don't bother to see the detailed report in order to find out that there is in fact an xfail test.

[anonyknight]

As far as I know, junit report did not include xfail and xpass definition. That's why we can see Jenkinc junit-plugin has no xfail and xpass counter.

https://github.com/jenkinsci/junit-plugin/blob/master/src/main/java/hudson/tasks/junit/TestResultSummary.java

http://doc.pytest.org/en/latest/skipping.html
This test will run but no traceback will be reported when it fails. Instead, terminal reporting will list it in the “expected to fail” (XFAIL) or “unexpectedly passing” (XPASS) sections.

If the test passed, pytest reports xPass in this case, but Pass in junit.xml. If the test failed, then pytest reports xfail but junit.xml reports skipped.

[webknjaz]

@anonyknight it looks like this is the symptom of the fact that pytest treats a test report as xfail/xpass only if it has outcome='skipped' and wasxfail='...' attrs. And it processes them when outputting terminal report: #7327. Other plugins would need to repeat the same logic.

[nicoddemus]

Unfortunately the JUnit XML format doesn't have the notion of xfail/xpass, so there isn't much that pytest can do here.

[Zac-HD]

Unfortunately https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd doesn't have a notion of expected failure 😕