Travis-ci build fails trying to remove .pytest_cache

[SeanXu1984]

• a detailed description of the bug or suggestion

path = '/tmp/pytest-of-travis/garbage-7b2ebcac-468f-4ee5-9433-7967d6217af8'
ignore_errors = False, onerror = <functools.partial object at 0x7f5b3e727788>
    def rmtree(path, ignore_errors=False, onerror=None):
        """Recursively delete a directory tree.
    
        If ignore_errors is set, errors are ignored; otherwise, if onerror
        is set, it is called to handle the error with arguments (func,
        path, exc_info) where func is os.listdir, os.remove, or os.rmdir;
        path is the argument to that function that caused it to fail; and
        exc_info is a tuple returned by sys.exc_info().  If ignore_errors
        is false and onerror is None, an exception is raised.
    
        """
        if ignore_errors:
            def onerror(*args):
                pass
        elif onerror is None:
            def onerror(*args):
                raise
        try:
            if os.path.islink(path):
                # symlinks to directories are forbidden, see bug #1669
                raise OSError("Cannot call rmtree on a symbolic link")
        except OSError:
            onerror(os.path.islink, path, sys.exc_info())
            # can't continue even if onerror hook returns
            return
        names = []
        try:
            names = os.listdir(path)
        except os.error, err:
            onerror(os.listdir, path, sys.exc_info())
        for name in names:
            fullname = os.path.join(path, name)
            try:
                mode = os.lstat(fullname).st_mode
            except os.error:
                mode = 0
            if stat.S_ISDIR(mode):
>               rmtree(fullname, ignore_errors, onerror)
/usr/lib/python2.7/shutil.py:247: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python2.7/shutil.py:247: in rmtree
    rmtree(fullname, ignore_errors, onerror)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
path = '/tmp/pytest-of-travis/garbage-7b2ebcac-468f-4ee5-9433-7967d6217af8/test_cache_writefail_permissions0/.pytest_cache'
ignore_errors = False, onerror = <functools.partial object at 0x7f5b3e727788>
    def rmtree(path, ignore_errors=False, onerror=None):
        """Recursively delete a directory tree.
    
        If ignore_errors is set, errors are ignored; otherwise, if onerror
        is set, it is called to handle the error with arguments (func,
        path, exc_info) where func is os.listdir, os.remove, or os.rmdir;
        path is the argument to that function that caused it to fail; and
        exc_info is a tuple returned by sys.exc_info().  If ignore_errors
        is false and onerror is None, an exception is raised.
    
        """
        if ignore_errors:
            def onerror(*args):
                pass
        elif onerror is None:
            def onerror(*args):
                raise
        try:
            if os.path.islink(path):
                # symlinks to directories are forbidden, see bug #1669
                raise OSError("Cannot call rmtree on a symbolic link")
        except OSError:
            onerror(os.path.islink, path, sys.exc_info())
            # can't continue even if onerror hook returns
            return
        names = []
        try:
            names = os.listdir(path)
        except os.error, err:
>           onerror(os.listdir, path, sys.exc_info())
E           PytestWarning: (rm_rf) unknown function /tmp/pytest-of-travis/garbage-7b2ebcac-468f-4ee5-9433-7967d6217af8/test_cache_writefail_permissions0/.pytest_cache when removing <built-in function listdir>:
E           <type 'exceptions.OSError'>: [Errno 13] Permission denied: '/tmp/pytest-of-travis/garbage-7b2ebcac-468f-4ee5-9433-7967d6217af8/test_cache_writefail_permissions0/.pytest_cache'

• output of pip list from the virtual environment you are using

#6785

https://travis-ci.org/pytest-dev/pytest/builds/655205329#

• pytest and operating system versions
  4.6.10.dev
• minimal example if possible

[Zac-HD]

Travis recently changed their config parser, which could be causing problems? (it broke Hypothesis' builds) https://travis-ci.org/pytest-dev/pytest/jobs/655205330/config

[SeanXu1984]

Can someone take a look at this issue please? Thanks

[blueyed]

@SeanXu1984
Does it have to be for/with 4.6?
IIRC CI is broken there since a while already (check the cron jobs).

[SeanXu1984]

Yes. Currently we are still using Python 2.7 and our Runtime env is not yet migrated to Python 3.x

[blueyed]

@SeanXu1984
It is running too many jobs in the first place it seems (in a single job): https://travis-ci.org/pytest-dev/pytest/jobs/649404964/config (check the validation) - like @Zac-HD mentioned already.
4.6 is not really supported anymore, and you would have to fix it yourself.

This looks pretty bad though, I recommend checking if there are fixes in master in that regard already (at least with regard to the order of arguments for the message I think I've fixed something there):

        except os.error, err:
>           onerror(os.listdir, path, sys.exc_info())
E           PytestWarning: (rm_rf) unknown function /tmp/pytest-of-travis/garbage-7b2ebcac-468f-4ee5-9433-7967d6217af8/test_cache_writefail_permissions0/.pytest_cache when removing <built-in function listdir>:
E           <type 'exceptions.OSError'>: [Errno 13] Permission denied: '/tmp/pytest-of-travis/garbage-7b2ebcac-468f-4ee5-9433-7967d6217af8/test_cache_writefail_permissions0/.pytest_cache'

Also check 262ed56, which this reminded me of.

[blueyed]

Is #6785 relevant for 4.6 only?
Otherwise it should go through master in first anyway.

[SeanXu1984]

Hmmm. Branch 4.6.x is not well maintained?
#6785 applies to 5.x I guess. But if it goes to 5.x first we will still have to merge the change to 4.6.x.

[Zac-HD]

https://docs.pytest.org/en/latest/py27-py34-deprecation.html#maintenance-of-4-6-x-versions

[blueyed]

@SeanXu1984
So I think you should go through master for the original fix.
Then, for the Travis/CI config on the 4.x branch check/sync with https://github.com/blueyed/pytest/blob/my-4.6-maintenance/.travis.yml maybe, where it still works.
Might also make sense to use GitHub actions for the 4.x branch, but that might be more involved.

[SeanXu1984]

Thanks very much for the information guys.

[eb-fmezzabotta]

I'm gonna be trying to migrate the 4.6.x branch to use GitHub actions in here #6884. Travis is completely broken.

[nicoddemus]

Closing this for now as it seems the path to a solution was found.