[FR] Integrate xfail

[webknjaz]

I have a test that SEGFAULTs under random envs (usually only in one or two jobs in the matrix). So I decided to use forked to isolate it and marked it with xfail as follows:

@pytest.mark.xfail(
    reason='This test causes SEGFAULT, flakily.',
    strict=False,
)
@pytest.mark.forked
def test_exec_command(ssh_channel):
    """Test getting the output of a remotely executed command."""
    ...

But, despite my expectations, when it crashes, pytest shows it as a failure and not as XFAIL.

Could you please clarify whether this behavior is intended or is it a bug? (looks like a bug to me)

Am I doing anything wrong? Am I missing something here?

(Ref: https://github.com/ansible/pylibssh/runs/728770734?check_suite_focus=true#step:9:120)

[RonnyPfannschmidt]

Segfaults are currently not integrated

[webknjaz]

Is it hard to integrate them?

[RonnyPfannschmidt]

i have no idea, i haven't worked on it and i don't plan to maintain/work on pytest-forked i only took it out of xdist to enable dropping the boxed feature from xdist itself

[webknjaz]

@RonnyPfannschmidt where can I see how xfail is processed? I'm trying to figure out how things work in src/_pytest/mark/structures.py but it seems like it's probably not the place where I can hack into...

I want to contribute but struggling to find a starting point.

[RonnyPfannschmidt]

I vaguely recall that xfail is processed in the skipping plugin

[webknjaz]

So it turned out that for xfail to work, I need to set outcome = 'skipped' and wasxfail = 'the reason message' attrs on the report object.

[webknjaz]

So it turned out that for xfail to work, I need to set outcome = 'skipped' and wasxfail = 'the reason message' attrs on the report object.

Opened an issue about this pytest-dev/pytest#7327