Feature proposal: Capability to mark checks as xfail

[Alex-Izquierdo]

I had to write multiple assertions in the same test and this plugin is awesome. As the same way that you can capture one error and continue executing assertions It would be great if the assertion fails be able to mark it as xfail as the same way that you can do for the whole test or param.

[okken]

I'm glad you are finding the plugin useful.
This is an interesting request.
I'll think about it.
Are you using the check assert helper functions such as check.equal(), check.less(), etc?

With the functions, I could imagine an extra xfail parameter for the reason, for example:

def test_foo(check):
    check.equal(1, 2, xfail="they're not equal")

or more explicitly, perhaps:

def test_foo(check):
    check.equal(1, 2, xfail_reason="they're not equal")

For the context manager case, I'm not sure how that could be expressed. Perhaps also a parameter:

def test_foo(check):
    with check(xfail_reason="they're not equal"):
        assert 1 == 2

Something like this might work.
Although I don't think I could support XPASS with this.

[Alex-Izquierdo]

I'm using only the context manager, but I didn't knew that I can define a xfail_reason with the helpers, so maybe I can use it for these cases. I previously discarded the helpers because I lose the introspection of pytest assertions.

Thanks for your answer.

[okken]

Oh. You can't use xfail_reason, yet.
I'm just kicking around an API design idea.

[harmin-parra]

I'll prefer xfail="they're not equal" over xfail_reason="they're not equal"

[harmin-parra]

But actually, wouldn't it better to have the possibility to choose between failed and xfailed for the outcome status for failed checks ?

By default, the plugin sets the failed status to failed checks.

Perhaps we can have the possibility to choose between failed and xfailed, globally and /or per test ?

Something like this ?

1. Global configuration: with pytest.ini option:

check_outcome_status=failed or xfailed

2. Per-test configuration with decorators:

• If we want fail (current behavior)

@check.mark.fail
def test_foo(check):
    check.equal(1, 2)

• If we want xfail (requested feature)

@check.mark.xfail(reason="my reason")
def test_foo(check):
    check.equal(1, 2)

[jessie-github]

For me I am using pytest-check especially because I have a lot of "expected fails" and I don't want them to stop the test.
So I will have little use for a general @check.mark.xfail associated to the test but I will be happy to have an extra parameter xfail for check.
(and status of the test should be xpass if at least one of the expected fails is passed, but I suppose others may disagree)

[skrawcz]

Would this proposal be useful in the following context? I'm trying to understand it more (and whether this library is something I want to invest in).

1. For a test, I want to run all asserts, i.e. checks.
2. but I only want to mark a few asserts as having the ability to fail a test. The ones marked with xfail, could fail or pass without impacting the test overall.
3. I'd still like to keep track of the results of all asserts, so that I can compare things over different runs.

Cheers!

[okken]

@skrawcz Extremely late reply, but here goes

Would this proposal be useful in the following context?

1. For a test, I want to run all asserts, i.e. checks.
2. but I only want to mark a few asserts as having the ability to fail a test. The ones marked with xfail, could fail or pass without impacting the test overall.
3. I'd still like to keep track of the results of all asserts, so that I can compare things over different runs.

Ummmm. I don't know how to answer.

1. yes
2. "without impacting the overall test", I don't see how that would work
3. yes.

But again, if an check is expected to fail, then what would the result of the test be?

Pass doesn't seem like the right thing. But some people don't want it to be xfail either. and how is xpass acceptable?

[okken]

This has been open for 3 years now.
I think I need to make a decision.
And right now that decision is leaning heavily to NOT support this, and just close it as "won't implement".

But I think I'll add a "someday maybe" tag or something, and think about it.

Reasoning

• pytest now supports subtests, but doesn't support a way to xfail a subtest
  • see Please make it possible to "mark" subtests as xfail #14101
• I don't even really know what that would mean behavior wise

Questions

• Should an xfailed check cause the test as a whole to be treated like xfail?
  • If so, how is that different than the user just marking the test as xfail?
  • If not, then what total result should the test proper have if some of the xfailed checks fail?

I think I need a matrix of:

• test is marked xfail or not
• a check is marked xfail or not
• all checks pass vs at least one check fails
• what the overall test result should be

Also, what should the report look like for checks that expectedly fail?

[okken]

Working on an implementation.

Currently I have something that’s roughly:

• check func xfail with failure -> xfail
• check func xfail with pass -> pass

I’m worried about supporting xpass slowing the normal case behavior.

I’ll run some speed tests.

But also rolling this out in phases might be ok.