xfail

[bertfrees]

Besides focus and pending, an attribute to mark known failures, similar to Automake's xfail (https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html).

I currently mark known failures with pending (which means "Completely skip this scenario"), but xfail would be more appropriate. It means "This scenario MUST fail".

[josteinaj]

You can already assert for errors. Does that cover your usecase?
13. feb. 2014 18:46 skrev "Bert Frees" notifications@github.com følgende:

Besides focus and pending, an attribute to mark known failures, similar to
Automake's xfail (
https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html
).

I currently mark known failures with pending (which means "Completely skip
this test"), but xfail would be more appropriate. It means "This scenario
MUST fail".

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7
.

[bertfrees]

Assert for errors would indeed be the solution if a script would throw an error. But in some of my scenarios, a script returns a result document which is not equal to the expected result (in other words: a failure not an error). Can you assert for this too? (I guess one could make a custom assertion which does exactly the opposite as the normal p:compare, but that's a bit farfetched).

[josteinaj]

Aha, good usecase. I'll put it on my TODO.

Do you want the attribute to be called simply "xfail" or do you want
something more descriptive like "expect-failure"? Or should it be reflected
in the element name instead, like <x:expect-failure ...> ?
13. feb. 2014 19:39 skrev "Bert Frees" notifications@github.com følgende:

Assert for errors would indeed be the solution if a script would throw an
error. But in some of my scenarios, a script returns a result document
which is not equal to the expected result (in other words: a failure not an
error). Can you assert for this too? (I guess one could make a custom
assertion which does exactly the opposite as the normal p:compare, but
that's a bit farfetched).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7#issuecomment-35010489
.

[bertfrees]

xfail appears to be used not only in Automake, so it's a relatively standard term. But nevertheless a more descriptive term is probably better.

I first imagined it as a flag on x:scenario, just like pending and focus. But now that you propose the x:expect-failure element, that might even make more sense. You could then in theory have a mixture of x:expect and x:expect-failure. IDHASO™, it's up to you.

[josteinaj]

@bertfrees: PR #16 adds a xfail attribute. It can be applied to both the x:expect element (for a single assertion) and the x:scenario element (for a set of assertions). The effect of setting xfail="true" is that the result of an assertion is inverted. Does this work for you? Shall I merge or would you like me to tweak it in some way first?

[josteinaj]

closed by #16

@bertfrees: let me know if you'd like any changes made to this.

[bertfrees]

Maybe xfail should resemble pending more, i.e. the value can be a comment?