Test collection to broad, walking sideways, causes PermissionError for tests in /tmp subdirs

[ibressler]

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

Hello, I am a user of pytest and pytest-notebook. I am not sure if this qualifies as a bug here or for pytest-notebook. But since it seems to be very much related to the plugin system, collection and path handling, I post this here. Plus, it seems very similar or related to #11781.

When running a test with a pytester fixture (formerly testdir) a temporary directory is created and used under /tmp. Here, that would be something like /tmp/pytest-of-ingo/pytest-44/test_run_fail0. In the collection phase pytest looks for tests in just /tmp and stumbles on systemd related directories which are inaccessible, causing a PermissionError: [Errno 13] Permission denied: to stop it.

PermissionError details/traceback

-------------------------------------------------------------- Captured stdout call --------------------------------------------------------------
TEST /tmp/pytest-of-ingo/pytest-44/test_run_fail0
============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-8.3.0.dev28+g4080459f0, pluggy-1.5.0 -- /home/ingo/.py11venv/bin/python3
cachedir: .pytest_cache
NB exec dir: /home/ingo/code/3rdp/pytest-notebook/tests/raw_files
rootdir: /home/ingo/code/3rdp/pytest-notebook
configfile: tox.ini
plugins: cov-4.1.0, anyio-4.3.0, dash-2.15.0, pytest_notebook-0.10.0
collecting ... collected 0 items / 1 error

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
/home/ingo/.py11venv/lib/python3.11/site-packages/pluggy/_hooks.py:513: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
/home/ingo/.py11venv/lib/python3.11/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/home/ingo/.py11venv/lib/python3.11/site-packages/pluggy/_manager.py:480: in traced_hookexec
    return outcome.get_result()
/home/ingo/.py11venv/lib/python3.11/site-packages/pluggy/_manager.py:477: in <lambda>
    lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
/home/ingo/.py11venv/lib/python3.11/site-packages/_pytest/python.py:179: in pytest_collect_directory
    if pkginit.is_file():
/usr/lib/python3.11/pathlib.py:1268: in is_file
    return S_ISREG(self.stat().st_mode)
/usr/lib/python3.11/pathlib.py:1014: in stat
    return os.stat(self, follow_symlinks=follow_symlinks)
E   PermissionError: [Errno 13] Permission denied: '/tmp/systemd-private-36e9322d52084bf7bb27cacad8f3ac9e-ModemManager.service-S0fIze/__init__.py'
=========================== short test summary info ============================
ERROR ../../../../home/ingo/code/3rdp/pytest-notebook/::tmp - PermissionError...
=============================== 1 error in 0.12s ===============================
-------------------------------------------------------------- Captured stderr call --------------------------------------------------------------
ERROR: found no collectors for /tmp/pytest-of-ingo/pytest-44/test_run_fail0

But this seems to be the case only for those tests which are run in a different working directory, because the failing tests get the pytest-notebook specific flag --nb-exec-cwd which is forwarded to nbclient and sets the path where the notebook is run.
But this seems to affect the setting of PluginManager._confcutdir which in turn is used to restrict the paths for test collection. The pytest_load_initial_conftests hook seems to be run for the failing tests only and sets the _confcutdir to the cwd folder which might be elsewhere (/home for example) than the tmpdir used by the fixture. In pluginmanager._set_initial_conftests is gets:
invocation_dir=PosixPath('/tmp/pytest-of-ingo/pytest-44/test_run_fail0')
and confcutdir='/home/ingo/code/3rdp/pytest-notebook'
But I can't find out why this happens?

This took me some hours to trace back due to multiple levels of hooks being run (and am I far from familiar with how pytest works internally). And perhaps, someone with more experience with the internals of pytest can tell why _confcutdir is set or not set accordingly.

The test I refer here, is a rather simple example already. It belongs to the pytest-notebook extension. Cloning that and running pytest -v -k test_run_fail should trigger that error while pytest -v -k test_setup_with_skip_meta from the same file works.

I am on Ubuntu 23.10 locally with Python 3.11.6 and the following pip packages in my venv:

Package                   Version
------------------------- ----------------------
anyio                     4.3.0
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
arrow                     1.3.0
attrs                     22.2.0
beautifulsoup4            4.12.3
bleach                    6.1.0
certifi                   2024.2.2
cffi                      1.16.0
charset-normalizer        3.3.2
colorama                  0.4.6
coverage                  7.5.1
defusedxml                0.7.1
fastjsonschema            2.19.1
fqdn                      1.5.1
gitdb                     4.0.11
GitPython                 3.1.43
idna                      3.7
iniconfig                 2.0.0
isoduration               20.11.0
Jinja2                    3.1.4
jsonpointer               2.4
jsonschema                4.22.0
jsonschema-specifications 2023.12.1
jupyter_client            8.6.1
jupyter_core              5.7.2
jupyter-events            0.10.0
jupyter_server            2.14.0
jupyter-server-mathjax    0.2.6
jupyter_server_terminals  0.5.3
jupyterlab_pygments       0.3.0
MarkupSafe                2.1.5
mistune                   3.0.2
nbclient                  0.5.13
nbconvert                 7.16.4
nbdime                    4.0.1
nbformat                  5.10.4
nest-asyncio              1.6.0
overrides                 7.7.0
packaging                 24.0
pandocfilters             1.5.1
pip                       23.2
platformdirs              4.2.1
pluggy                    1.5.0
prometheus_client         0.20.0
ptyprocess                0.7.0
pycparser                 2.22
Pygments                  2.18.0
pytest                    8.3.0.dev28+g4080459f0
pytest_notebook           0.10.0
python-dateutil           2.9.0.post0
python-json-logger        2.0.7
PyYAML                    6.0.1
pyzmq                     26.0.3
referencing               0.35.1
requests                  2.31.0
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rpds-py                   0.18.1
Send2Trash                1.8.3
setuptools                68.1.2
six                       1.16.0
smmap                     5.0.1
sniffio                   1.3.1
soupsieve                 2.5
terminado                 0.18.1
tinycss2                  1.3.0
tornado                   6.4
traitlets                 5.14.3
types-python-dateutil     2.9.0.20240316
uri-template              1.3.0
urllib3                   2.2.1
webcolors                 1.13
webencodings              0.5.1
websocket-client          1.8.0

But this happens with the Github Ubuntu runner as well: https://github.com/chrisjsewell/pytest-notebook/actions/runs/8976635149/job/24653858224#step:5:35

Here, it complains just about a different path being inaccessible (`/tmp/snap-private-tmp/__init__.py`).

Run pytest --durations=10 --cov=pytest_notebook --cov-report=xml --cov-report=term-missing --nb-coverage
============================= test session starts ==============================
platform linux -- Python 3.11.9, pytest-8.2.0, pluggy-1.5.0
rootdir: /home/runner/work/pytest-notebook/pytest-notebook
configfile: tox.ini
plugins: anyio-4.3.0, regressions-2.5.0, datadir-1.5.0, cov-5.0.0, pytest_notebook-0.10.0
collected 73 items

tests/test_cell_diff.py ................                                 [ 21%]
tests/test_coalesce_streams.py ..                                        [ 24%]
tests/test_execution.py ...                                              [ 28%]
tests/test_filter_diff.py .....                                          [ 35%]
tests/test_ipy_magic.py ...........                                      [ 50%]
tests/test_nb_diff.py ...                                                [ 54%]
tests/test_nb_regression.py .....                                        [ 61%]
tests/test_notebook.py ........                                          [ 72%]
tests/test_plugin_collector.py ..FF                                      [ 78%]
tests/test_plugin_fixture.py ..........                                  [ 91%]
tests/test_postprocessors/test_beautifulsoup.py ..                       [ 94%]
tests/test_postprocessors/test_blacken_code.py ..                        [ 97%]
tests/test_postprocessors/test_plugins.py .                              [ 98%]
tests/test_utils.py s                                                    [100%]

=================================== FAILURES ===================================
________________________________ test_run_fail _________________________________

testdir = <Testdir local('/tmp/pytest-of-runner/pytest-0/test_run_fail0')>

    def test_run_fail(testdir):
        copy_nb_to_tempdir("different_outputs_altered.ipynb")
        result = testdir.runpytest(
            "--nb-exec-cwd", os.path.join(PATH, "raw_files"), "--nb-test-files", "-v"
        )
        # fnmatch_lines does an assertion internally
>       result.stdout.fnmatch_lines(
            ["*::nbregression(test_nb) FAILED*", "*CellExecutionError:*"]
        )
E       Failed: nomatch: '*::nbregression(test_nb) FAILED*'
E           and: '============================= test session starts =============================='
E           and: 'platform linux -- Python 3.11.9, pytest-8.2.0, pluggy-1.5.0 -- /opt/hostedtoolcache/Python/3.11.9/x64/bin/python'
E           and: 'cachedir: .pytest_cache'
E           and: 'NB exec dir: /home/runner/work/pytest-notebook/pytest-notebook/tests/raw_files'
E           and: 'rootdir: /home/runner/work/pytest-notebook/pytest-notebook'
E           and: 'configfile: tox.ini'
E           and: 'plugins: anyio-4.3.0, regressions-2.5.0, datadir-1.5.0, cov-5.0.0, pytest_notebook-0.10.0'
E           and: 'collecting ... collected 0 items / 1 error'
E           and: ''
E           and: '==================================== ERRORS ===================================='
E           and: '________________________ ERROR collecting test session _________________________'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/pluggy/_hooks.py:513: in __call__'
E           and: '    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/pluggy/_manager.py:120: in _hookexec'
E           and: '    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/pluggy/_manager.py:480: in traced_hookexec'
E           and: '    return outcome.get_result()'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/pluggy/_manager.py:477: in <lambda>'
E           and: '    lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/_pytest/python.py:179: in pytest_collect_directory'
E           and: '    if pkginit.is_file():'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/pathlib.py:1267: in is_file'
E           and: '    return S_ISREG(self.stat().st_mode)'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/pathlib.py:1013: in stat'
E           and: '    return os.stat(self, follow_symlinks=follow_symlinks)'
E           and: "E   PermissionError: [Errno 13] Permission denied: '/tmp/snap-private-tmp/__init__.py'"
E           and: '=========================== short test summary info ============================'
E           and: "ERROR ../../../../home/runner/work/pytest-notebook/pytest-notebook/::tmp - PermissionError: [Errno 13] Permission denied: '/tmp/snap-private-tmp/__init__.py'"
E           and: '=============================== 1 error in 0.44s ==============================='
E       remains unmatched: '*::nbregression(test_nb) FAILED*'

/home/runner/work/pytest-notebook/pytest-notebook/tests/test_plugin_collector.py:42: Failed
----------------------------- Captured stdout call -----------------------------

Workaround

By specifying a local directory to pytest (for example somewhere in /home) with --basetemp=dir avoids running into root-owned dirs triggering that error.

[bluetech]

Thanks for the report, we have a few issues about this already, so I'll close this one.