Skip to content

Commit d03a2d7

Browse files
authored
Consider test level expected results for unexpected pass calculation (#40308)
* Consider test level expected results for unexpected pass calculation Any test level expected result and subtest unexpected pass should be considered as unexpected pass. This is not the case previously. If we expect 'Error" at test level, any unexpectd passed subtest cause the test to fail unexpectedly.
1 parent 08f4559 commit d03a2d7

File tree

1 file changed

+5
-4
lines changed

1 file changed

+5
-4
lines changed

tools/wptrunner/wptrunner/testrunner.py

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -700,16 +700,17 @@ def test_ended(self, test, results):
700700

701701
self.test_count += 1
702702
is_unexpected = expected != status and status not in known_intermittent
703+
is_pass_or_expected = status in ["OK", "PASS"] or (not is_unexpected)
703704

704705
if is_unexpected or subtest_unexpected:
705706
self.unexpected_tests.add(test.id)
706707

707708
# A result is unexpected pass if the test or any subtest run
708-
# unexpectedly, and the overall status is OK (for test harness test), or
709-
# PASS (for reftest), and all unexpected results for subtests (if any) are
710-
# unexpected pass.
709+
# unexpectedly, and the overall status is expected or passing (OK for test
710+
# harness test, or PASS for reftest), and all unexpected results for
711+
# subtests (if any) are unexpected pass.
711712
is_unexpected_pass = ((is_unexpected or subtest_unexpected) and
712-
status in ["OK", "PASS"] and subtest_all_pass_or_expected)
713+
is_pass_or_expected and subtest_all_pass_or_expected)
713714
if is_unexpected_pass:
714715
self.unexpected_pass_tests.add(test.id)
715716

0 commit comments

Comments
 (0)