Skip to content

xfail attribute - invert the result of assertions #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
josteinaj wants to merge 1 commit into from
Closed

xfail attribute - invert the result of assertions #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions www/documentation/generated.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ <h2>The <code>import</code> element</h2><dl class="element"><dt>Content model:</
</section>

<section id="the-scenario-element">
<h2>The <code>scenario</code> element</h2><dl class="element"><dt>Content model:</dt><dd><a href="#the-call-element"><code>call</code></a><small title="optional">?</small></dd><dd><a href="#the-scenario-element"><code>scenario</code></a><small title="zero or more">*</small></dd><dd><a href="#the-pending-element"><code>pending</code></a><small title="zero or more">*</small></dd><dd><a href="#the-context-element"><code>context</code></a><small title="zero or more">*</small></dd><dd><a href="#the-expect-element"><code>expect</code></a><small title="zero or more">*</small></dd><dt>Content attributes:</dt><dd><a href="#attr-scenario-focus"><code>focus</code></a><small title="optional">?</small></dd><dd><a href="#attr-scenario-label"><code>label</code></a></dd><dd>Zero or more attributes from any namespace.</dd></dl>
<h2>The <code>scenario</code> element</h2><dl class="element"><dt>Content model:</dt><dd><a href="#the-call-element"><code>call</code></a><small title="optional">?</small></dd><dd><a href="#the-scenario-element"><code>scenario</code></a><small title="zero or more">*</small></dd><dd><a href="#the-pending-element"><code>pending</code></a><small title="zero or more">*</small></dd><dd><a href="#the-context-element"><code>context</code></a><small title="zero or more">*</small></dd><dd><a href="#the-expect-element"><code>expect</code></a><small title="zero or more">*</small></dd><dt>Content attributes:</dt><dd><a href="#attr-scenario-focus"><code>focus</code></a><small title="optional">?</small></dd><dd><a href="#attr-scenario-label"><code>label</code></a></dd><dd><a href="#attr-scenario-xfail"><code>xfail</code></a><small title="optional">?</small></dd><dd>Zero or more attributes from any namespace.</dd></dl>
<p>
A scenario groups together the definition of the script environment
(the <a href="#the-call-element"><code>call</code></a> element) and the script assertions (the <a href="#the-context-element"><code>context</code></a> and
Expand All @@ -125,6 +125,11 @@ <h2>The <code>scenario</code> element</h2><dl class="element"><dt>Content model:
reported. When present, only that assertion or scenario will be evaluated.
Its content describes why you are focusing on this particular assertion or scenario.</p>
<p id="attr-scenario-label">The <a href="#attr-scenario-label"><code>label</code></a> attribute is used to describe the current element in human-readable words.</p>
<p id="attr-scenario-xfail">Assertions with the <a href="#attr-scenario-xfail"><code>xfail</code></a> attribute will behave just like other assertions, but
will succeed where other assertions fail, and will fail where other assertions succeed (i.e. the result is inverted).
The <a href="#attr-scenario-xfail"><code>xfail</code></a> attribute can also be applied to scenarios, in which case <code>xfail</code> will be implicitly
set to <code>true</code> for all of the scenarios assertions.
The value of the <a href="#attr-scenario-xfail"><code>xfail</code></a> attribute must be <code>true</code>.</p>
</section>

<section id="the-call-element">
Expand Down Expand Up @@ -164,7 +169,7 @@ <h2>The <code>context</code> element</h2><dl class="element"><dt>Content model:<
</section>

<section id="the-expect-element">
<h2>The <code>expect</code> element</h2><dl class="element"><dt>Content model:</dt><dd><a href="#the-document-element"><code>document</code></a><small title="zero or more">*</small></dd><dt>Content attributes:</dt><dd><a href="#attr-expect-equals"><code>equals</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-focus"><code>focus</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-grammar"><code>grammar</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-href"><code>href</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-label"><code>label</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-max"><code>max</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-min"><code>min</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-normalize-space"><code>normalize-space</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-pending"><code>pending</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-step"><code>step</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-test"><code>test</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-type"><code>type</code></a><small title="optional">?</small></dd><dd>Zero or more attributes from any namespace.</dd></dl>
<h2>The <code>expect</code> element</h2><dl class="element"><dt>Content model:</dt><dd><a href="#the-document-element"><code>document</code></a><small title="zero or more">*</small></dd><dt>Content attributes:</dt><dd><a href="#attr-expect-equals"><code>equals</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-focus"><code>focus</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-grammar"><code>grammar</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-href"><code>href</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-label"><code>label</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-max"><code>max</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-min"><code>min</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-normalize-space"><code>normalize-space</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-pending"><code>pending</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-step"><code>step</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-test"><code>test</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-type"><code>type</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-xfail"><code>xfail</code></a><small title="optional">?</small></dd><dd>Zero or more attributes from any namespace.</dd></dl>
<p>
Defines what is expected from the context document(s).
</p>
Expand Down Expand Up @@ -237,6 +242,11 @@ <h2>The <code>expect</code> element</h2><dl class="element"><dt>Content model:</
&lt;x:was xml:space="preserve"&gt;Description of what the value actually was.&lt;/x:was&gt;
&lt;/x:test-result&gt;
</code></pre></li></ul>
<p id="attr-expect-xfail">Assertions with the <a href="#attr-expect-xfail"><code>xfail</code></a> attribute will behave just like other assertions, but
will succeed where other assertions fail, and will fail where other assertions succeed (i.e. the result is inverted).
The <a href="#attr-expect-xfail"><code>xfail</code></a> attribute can also be applied to scenarios, in which case <code>xfail</code> will be implicitly
set to <code>true</code> for all of the scenarios assertions.
The value of the <a href="#attr-expect-xfail"><code>xfail</code></a> attribute must be <code>true</code>.</p>
</section>


Expand Down
16 changes: 13 additions & 3 deletions www/documentation/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ <h1 class="title p-name" id="title" property="dcterms:title">xprocspec</h1>

<h2 property="bibo:subtitle" id="subtitle">A tool for testing XProc scripts</h2>

<h2 property="dcterms:issued" datatype="xsd:dateTime" content="2014-05-09T08:37:41.000Z" id="unofficial-draft-09-may-2014">Unofficial Draft <time class="dt-published" datetime="2014-05-09">09 May 2014</time></h2>
<h2 property="dcterms:issued" datatype="xsd:dateTime" content="2014-05-09T10:35:34.000Z" id="unofficial-draft-09-may-2014">Unofficial Draft <time class="dt-published" datetime="2014-05-09">09 May 2014</time></h2>
<dl>


Expand Down Expand Up @@ -263,7 +263,7 @@ <h3 aria-level="2" role="heading" id="h3_the-import-element"><span class="secno"
</section>

<section id="the-scenario-element" typeof="bibo:Chapter" resource="#the-scenario-element" rel="bibo:Chapter">
<h3 aria-level="2" role="heading" id="h3_the-scenario-element"><span class="secno">1.3 </span>The <code>scenario</code> element</h3><dl class="element"><dt>Content model:</dt><dd><a href="#the-call-element"><code>call</code></a><small title="optional">?</small></dd><dd><a href="#the-scenario-element"><code>scenario</code></a><small title="zero or more">*</small></dd><dd><a href="#the-pending-element"><code>pending</code></a><small title="zero or more">*</small></dd><dd><a href="#the-context-element"><code>context</code></a><small title="zero or more">*</small></dd><dd><a href="#the-expect-element"><code>expect</code></a><small title="zero or more">*</small></dd><dt>Content attributes:</dt><dd><a href="#attr-scenario-focus"><code>focus</code></a><small title="optional">?</small></dd><dd><a href="#attr-scenario-label"><code>label</code></a></dd><dd>Zero or more attributes from any namespace.</dd></dl>
<h3 aria-level="2" role="heading" id="h3_the-scenario-element"><span class="secno">1.3 </span>The <code>scenario</code> element</h3><dl class="element"><dt>Content model:</dt><dd><a href="#the-call-element"><code>call</code></a><small title="optional">?</small></dd><dd><a href="#the-scenario-element"><code>scenario</code></a><small title="zero or more">*</small></dd><dd><a href="#the-pending-element"><code>pending</code></a><small title="zero or more">*</small></dd><dd><a href="#the-context-element"><code>context</code></a><small title="zero or more">*</small></dd><dd><a href="#the-expect-element"><code>expect</code></a><small title="zero or more">*</small></dd><dt>Content attributes:</dt><dd><a href="#attr-scenario-focus"><code>focus</code></a><small title="optional">?</small></dd><dd><a href="#attr-scenario-label"><code>label</code></a></dd><dd><a href="#attr-scenario-xfail"><code>xfail</code></a><small title="optional">?</small></dd><dd>Zero or more attributes from any namespace.</dd></dl>
<p>
A scenario groups together the definition of the script environment
(the <a href="#the-call-element"><code>call</code></a> element) and the script assertions (the <a href="#the-context-element"><code>context</code></a> and
Expand All @@ -279,6 +279,11 @@ <h3 aria-level="2" role="heading" id="h3_the-scenario-element"><span class="secn
reported. When present, only that assertion or scenario will be evaluated.
Its content describes why you are focusing on this particular assertion or scenario.</p>
<p id="attr-scenario-label">The <a href="#attr-scenario-label"><code>label</code></a> attribute is used to describe the current element in human-readable words.</p>
<p id="attr-scenario-xfail">Assertions with the <a href="#attr-scenario-xfail"><code>xfail</code></a> attribute will behave just like other assertions, but
will succeed where other assertions fail, and will fail where other assertions succeed (i.e. the result is inverted).
The <a href="#attr-scenario-xfail"><code>xfail</code></a> attribute can also be applied to scenarios, in which case <code>xfail</code> will be implicitly
set to <code>true</code> for all of the scenarios assertions.
The value of the <a href="#attr-scenario-xfail"><code>xfail</code></a> attribute must be <code>true</code>.</p>
</section>

<section id="the-call-element" typeof="bibo:Chapter" resource="#the-call-element" rel="bibo:Chapter">
Expand Down Expand Up @@ -318,7 +323,7 @@ <h3 aria-level="2" role="heading" id="h3_the-context-element"><span class="secno
</section>

<section id="the-expect-element" typeof="bibo:Chapter" resource="#the-expect-element" rel="bibo:Chapter">
<h3 aria-level="2" role="heading" id="h3_the-expect-element"><span class="secno">1.7 </span>The <code>expect</code> element</h3><dl class="element"><dt>Content model:</dt><dd><a href="#the-document-element"><code>document</code></a><small title="zero or more">*</small></dd><dt>Content attributes:</dt><dd><a href="#attr-expect-equals"><code>equals</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-focus"><code>focus</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-grammar"><code>grammar</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-href"><code>href</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-label"><code>label</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-max"><code>max</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-min"><code>min</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-normalize-space"><code>normalize-space</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-pending"><code>pending</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-step"><code>step</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-test"><code>test</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-type"><code>type</code></a><small title="optional">?</small></dd><dd>Zero or more attributes from any namespace.</dd></dl>
<h3 aria-level="2" role="heading" id="h3_the-expect-element"><span class="secno">1.7 </span>The <code>expect</code> element</h3><dl class="element"><dt>Content model:</dt><dd><a href="#the-document-element"><code>document</code></a><small title="zero or more">*</small></dd><dt>Content attributes:</dt><dd><a href="#attr-expect-equals"><code>equals</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-focus"><code>focus</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-grammar"><code>grammar</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-href"><code>href</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-label"><code>label</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-max"><code>max</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-min"><code>min</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-normalize-space"><code>normalize-space</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-pending"><code>pending</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-step"><code>step</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-test"><code>test</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-type"><code>type</code></a><small title="optional">?</small></dd><dd><a href="#attr-expect-xfail"><code>xfail</code></a><small title="optional">?</small></dd><dd>Zero or more attributes from any namespace.</dd></dl>
<p>
Defines what is expected from the context document(s).
</p>
Expand Down Expand Up @@ -391,6 +396,11 @@ <h3 aria-level="2" role="heading" id="h3_the-expect-element"><span class="secno"
&lt;x:was xml:space="preserve"&gt;Description of what the value actually was.&lt;/x:was&gt;
&lt;/x:test-result&gt;
</code></pre></li></ul>
<p id="attr-expect-xfail">Assertions with the <a href="#attr-expect-xfail"><code>xfail</code></a> attribute will behave just like other assertions, but
will succeed where other assertions fail, and will fail where other assertions succeed (i.e. the result is inverted).
The <a href="#attr-expect-xfail"><code>xfail</code></a> attribute can also be applied to scenarios, in which case <code>xfail</code> will be implicitly
set to <code>true</code> for all of the scenarios assertions.
The value of the <a href="#attr-expect-xfail"><code>xfail</code></a> attribute must be <code>true</code>.</p>
</section>


Expand Down
49 changes: 39 additions & 10 deletions xprocspec/src/it/it-xprocspec/src/test/xprocspec/tests/expect-1.xprocspec
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
<?xml version="1.0" encoding="UTF-8"?>
<?xml-model href="http://www.daisy.org/ns/xprocspec/xprocspec.rng" type="application/xml" schematypens="http://relaxng.org/ns/structure/1.0"?>
<x:description xmlns:c="http://www.w3.org/ns/xproc-step" xmlns:x="http://www.daisy.org/ns/xprocspec" xmlns:e="http://example.net/ns" script="../steps/identity.xpl">

<x:scenario label="identity - base-uri">
<x:call step="e:identity">
<x:option name="option.required" select="'option'"/>
</x:call>

<x:context label="empty sequence"/>
<x:expect type="xpath" label="test='false()' against a empty sequence should succeed" test="false()"/>
<x:expect type="compare" label="compare two empty sequences"/>
<x:expect type="count" label="the empty sequence contains at most 0 documents" max="0"/>

<x:context label="one document">
<x:document type="inline">
<doc1 attr="value1"/>
Expand All @@ -25,7 +25,7 @@
<x:expect type="count" label="a sequence of one document contains between 1 and 10 documents" min="1" max="10"/>
<x:expect type="count" label="a sequence of one document contains between 0 and 1 document" min="0" max="1"/>
<x:expect type="count" label="a sequence of one document contains between 1 and 1 document" min="1" max="1"/>

<x:context label="two documents">
<x:document type="inline">
<doc1 attr="value1"/>
Expand All @@ -47,7 +47,7 @@
<x:expect type="count" label="a sequence of two document contains between 1 and 2 documents" min="1" max="2"/>
<x:expect type="count" label="a sequence of two document contains between 2 and 2 documents" min="2" max="2"/>
<x:expect type="count" label="a sequence of two document contains at least 1 document" min="1"/>

<x:context label="single document with text node">
<x:document type="inline">
<doc><![CDATA[ this string contains several spaces ]]></doc>
Expand All @@ -63,9 +63,9 @@
<doc><![CDATA[ this string contains several spaces ]]></doc>
</x:document>
</x:expect>

<!-- TODO: Validation is not tested. It is not implemented either. -->

<x:context label="a sequence of documents">
<x:document type="inline">
<doc1/>
Expand All @@ -77,8 +77,37 @@
<doc3/>
</x:document>
</x:context>
<x:expect type="custom" label="The custom 'count' step should be successful." xmlns:custom="http://www.example.org/ns/custom" step="custom:count" href="../steps/custom-count.xpl" custom:count="[2,4]"/>

<x:expect type="custom" label="The custom 'count' step should be successful." xmlns:custom="http://www.example.org/ns/custom" step="custom:count" href="../steps/custom-count.xpl"
custom:count="[2,4]"/>

<x:context label="a 'doc1'-document">
<x:document type="inline">
<doc1/>
</x:document>
</x:context>
<x:expect xfail="true" type="compare" label="...should not equal a 'doc2'-document (xfail attribute on `x:expect`-element)">
<x:document type="inline">
<doc2/>
</x:document>
</x:expect>

</x:scenario>

<x:scenario label="identity - xfail attribute on scenario" xfail="true">
<x:call step="e:identity">
<x:option name="option.required" select="'option'"/>
</x:call>

<x:context label="a 'doc1'-document">
<x:document type="inline">
<doc3/>
</x:document>
</x:context>
<x:expect type="compare" label="...should not equal a 'doc2'-document (xfail attribute on `x:expect`-element)">
<x:document type="inline">
<doc4/>
</x:document>
</x:expect>
</x:scenario>

</x:description>
Loading