cannot extend sys.path in conftest.py

[kalidasya]

Hi,

system and software info:
centos 7.2
python 3.6.3
pytest 3.3.1

I am facing an issue when I try to manipulate sys.path in conftest.py (conftest file in project root). It seems it does not have an effect and related imports will fail afterwards. Moving the conftest into the test folder works the same.

I have the following structure:

├── conftest.py
├── src2                    # source code
          └── util.py
└── scripts2                # tests
          └── test.py

The project itself is an integration test framework so it is not the system under test, the src folder contains util functions.

I run the test with py.test scripts2/test.py -s from root
The content of test.py:

import pytest
import sys
import os
#dir_path = os.path.dirname(os.path.realpath(__file__))
#sys.path.insert(0, dir_path+'/../src2')  # if I do this here and I dont have conftest.py it works
print("PATH0: ", sys.path)
import util  # this is the module from src

def test_print():
    print("PATH: ", sys.path)
    util.myprint()
    assert False

content of conftest.py:

import sys
import os
dir_path = os.path.dirname(os.path.realpath(__file__))
print(dir_path)
sys.path.insert(0, dir_path+'/src2')  # regardless of this the import will fail
print("conftest ", sys.path)

import pytest
from utils import myprint

# not used/interesting
@pytest.fixture
def myprinting():
    print("fixturing")
    myprint()

content of util.py:

import sys

def myprint():
    print(sys.path)

if I have the conftest.py and run the pytest command I get the following error:

Traceback (most recent call last):
  File "/home/centos/xdisttest/v/lib/python3.6/site-packages/_pytest/config.py", line 328, in _getconftestmodules
    return self._path2confmods[path]
KeyError: local('/home/centos/xdisttest/scripts2/test.py')

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/centos/xdisttest/v/lib/python3.6/site-packages/_pytest/config.py", line 328, in _getconftestmodules
    return self._path2confmods[path]
KeyError: local('/home/centos/xdisttest/scripts2')

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/centos/xdisttest/v/lib/python3.6/site-packages/_pytest/config.py", line 359, in _importconftest
    return self._conftestpath2mod[conftestpath]
KeyError: local('/home/centos/xdisttest/conftest.py')

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/centos/xdisttest/v/lib/python3.6/site-packages/_pytest/config.py", line 365, in _importconftest
    mod = conftestpath.pyimport()
  File "/home/centos/xdisttest/v/lib/python3.6/site-packages/py/_path/local.py", line 668, in pyimport
    __import__(modname)
  File "/home/centos/xdisttest/v/lib/python3.6/site-packages/_pytest/assertion/rewrite.py", line 212, in load_module
    py.builtin.exec_(co, mod.__dict__)
  File "/home/centos/xdisttest/conftest.py", line 9, in <module>
    from utils import myprint
ModuleNotFoundError: No module named 'utils'
ERROR: could not load /home/centos/xdisttest/conftest.py

if I remove conftest.py and enable the sys.path manipulation in the test itself, it works. I cannot use this workaround as my real test framework is more complex and there are many tests in it. I also tried a baked dynamic import with importlibs that seems to work to a certain extend but some test files will still fail to import things (I tried only in the real complex project):

For this simple example this solves the problem:

try:
    from utils import myprint
except:
    import importlib
    module_name = 'util'
    filename = dir_path + "/src2/util.py"
    spec = importlib.util.spec_from_file_location(module_name, filename)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    sys.modules[module_name] = module

The bigger picture. I discovered this issue while I was working on an xdist + docker setup where the docker container has the socket server implementation of execnet. I kept getting this error even if the pyexecnetcache had the correct data in the container. I was able to import things via imp or importlib, so I was sure its not related to missing files or file rights. I started to narrow down the problem as I was not sure if execnet or xdist or pytest is the root cause, and this is how I ended up with the smallest/simplest reproduction of the problem.

[nicoddemus]

Hi @kalidasya, sorry for the delay.

I can reproduce the issue but I can't tell right now what might be the problem; I get the error even when I use --assert=plain (which removes the custom import hook).

[kalidasya]

@nicoddemus no worries, thank you for looking into it!

[Reinaard]

Hi @kalidasya I had a similar issue before with exactly that KeyError for importing the conftest.py.

I fixed it with the following command:
find . -name \*.pyc -delete

[yozlet]

Thanks @Reinaard, that fixed it for me too:

docker-compose run web find . -name \*.pyc -delete

(... where web is the docker-compose service alias for my container)

[kalidasya]

I somehow missed the notifications, will try it again

[kalidasya]

still have trouble with it, I will update the ticket when I figured out something new. I delete all pyc file, also changed from socketsever to ssh, also added the test files in the container and not mount them