py.test exits with code 0 on errors

[pixelbombs]

We set up a pytest job in Jenkins to test an automatic bug logging script and we found that when py.test exits with no failures but with errors, we get an exit status of 0.

============================ slowest test durations ============================
84.74s setup    tests/functional/io/dd/test_dd_io_random_data.py::TestDDIORandom::test_4k_block_random_data
0.00s setup    tests/functional/io/dd/test_dd_io_random_data.py::TestDDIORandom::test_8k_block_random_data
0.00s setup    tests/functional/io/dd/test_dd_io_random_data.py::TestDDIORandom::test_32k_block_random_data
0.00s setup    tests/functional/io/dd/test_dd_io_random_data.py::TestDDIORandom::test_64k_block_random_data
0.00s setup    tests/functional/io/dd/test_dd_io_random_data.py::TestDDIORandom::test_256k_block_random_data
0.00s teardown tests/functional/io/dd/test_dd_io_random_data.py::TestDDIORandom::test_4k_block_random_data
0.00s teardown tests/functional/io/dd/test_dd_io_random_data.py::TestDDIORandom::test_256k_block_random_data
0.00s teardown tests/functional/io/dd/test_dd_io_random_data.py::TestDDIORandom::test_32k_block_random_data
0.00s teardown tests/functional/io/dd/test_dd_io_random_data.py::TestDDIORandom::test_8k_block_random_data
0.00s teardown tests/functional/io/dd/test_dd_io_random_data.py::TestDDIORandom::test_64k_block_random_data
=========================== 5 error in 85.21 seconds ===========================
+ RESULT=0

For more insight, here's the part of the bash script that shows the exit code being assigned to RESULT:

py.test -s --durations=0 -v --resultlog $HOME/logs/${logname}.txt --junitxml=$HOME/workspace/$JOB_NAME/${logname}.xml --systems $DEFAULTTARGET --install --clean --branch="B2v0.4" $HOME/test/tests/functional/io/dd/test_dd_io_random_data.py

RESULT=$?

You can imagine this is pretty bad in any instance where success or failure depends on checking the exit code for pytest (like a Jenkins bash script).

[The-Compiler]

How could we reproduce this? Where are the --install, --clean --branch and --systems arguments coming from? Can you show the start of the output which shows version numbers and plugins?

I can't reproduce this with this file:

import pytest

@pytest.fixture
def fix():
    raise Exception

def test_foo(fix):
    pass

$ py.test -s --durations=0 -v --resultlog foo.txt --junitxml foo.xml  blah.py
============================================ test session starts =============================================
platform linux -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /usr/bin/python
rootdir: /home/florian/tmp, inifile: 
plugins: mock
collected 1 items 

blah.py::test_foo ERROR

=================================================== ERRORS ===================================================
_________________________________________ ERROR at setup of test_foo _________________________________________

    @pytest.fixture
    def fix():
>       raise Exception
E       Exception

blah.py:5: Exception
------------------------------- generated xml file: /home/florian/tmp/foo.xml --------------------------------
=========================================== slowest test durations ===========================================
0.00s setup    blah.py::test_foo
0.00s teardown blah.py::test_foo
========================================== 1 error in 0.01 seconds ===========================================

$ echo $?
1

[pixelbombs]

Yeah, we actually just experimented with it a bunch and it turns out it was an error with something in the shell our intern wrote -_-;

======================================== slowest test durations =========================================
66.07s setup    tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_4k_block_known_data
0.00s setup    tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_32k_block_known_data
0.00s setup    tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_4mb_block_known_data
0.00s setup    tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_1mb_block_known_data
0.00s setup    tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_8k_block_known_data
0.00s setup    tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_256k_block_known_data
0.00s setup    tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_64k_block_known_data
0.00s teardown tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_32k_block_known_data
0.00s teardown tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_8k_block_known_data
0.00s teardown tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_256k_block_known_data
0.00s teardown tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_64k_block_known_data
0.00s teardown tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_4mb_block_known_data
0.00s teardown tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_4k_block_known_data
0.00s teardown tests/functional/io/dd/test_dd_io_known_data.py::TestDDIOKnownData::test_1mb_block_known_data
======================================= 7 error in 67.06 seconds ========================================
[csd-user@48-qa-driver9 test]$ echo $?
1

Sorry about that, I'll close the issue.

[pixelbombs]

Fwiw, it was because he added a tee to the shell command and that causes a 0 exit code every time:

py.test -s --durations=0 -v --resultlog /home/csd-user/logs/log.txt --junitxml=/home/csd-user/logs/log.xml --systems qa-test3 --clean --install --branch="B2v0.4" /$HOME/test/tests/functional/io/dd/test_dd_io_random_data.py |& tee ~/log.txt

[pixelbombs]

Oh, and thanks for trying to help @The-Compiler. Sorry for the bother.

[The-Compiler]

Bash is a... fun language. ${PIPESTATUS[0]} to the rescue!

No worries 😉

[dumptyd]

This just happened to me too and I found this issue.
In case someone else runs into this, if you're running pytest from another bash script, make sure to return the exit code of pytest like this

python -m pytest tests/
exit $?

otherwise it will always be 0.