pytest-django fails pytest (django.core.exceptions.ImproperlyConfigured)

[hkveeranki]

• I installed both pytest and pytest-django libraries in my laptop. Then I wanted to run pytest in a folder in which tests have no dependency of django.
  • Expected: I expected everything to run smoothing
  • What actually happeened? pytest failed with the error

django.core.exceptions.ImproperlyConfigured:` Requested settings, but settings are not configured.
You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

• Pytest version: 3.5.1, Operating System : Ubuntu Linux 18.04 LTS, python version - 3.6.5

I don't know whether this is an intended feature or a bug.

I reported it in pytest repo. They asked me to report it here.

[omegacore]

I struggled with these sorts of problems for a bit. Of course you have followed the steps here?

After I got my settings and django.setup() in the right place, everything worked fine. Figuring out the right place is somewhat challenging.

[hkveeranki]

@omegacore I removed the pytest-django from my machine and ran the tests. My question is that why is pytest-django effecting pytest even when there is no need. I don't have django dependency in the project where I am running pytest.

[omegacore]

Ah, I completely misread that.

[blueyed]

What is the full traceback / error there?
And how did you setup your project?
It clearly looks like Django settings are being accessed for some reason.

Just installing pytest and pytest-django, and having tests that do not use Django/pytest-django does not result in any error for me.

[hkveeranki]

@blueyed You can find the code to our tests here.

[blueyed]

Can you just provide the full traceback/more details yourself first, please?

[olasd]

I have the same issue in my environment: only some of my modules depend on django, and I have pytest-django installed for these (with DJANGO_SETTINGS_MODULE set in pytest.ini); When I reuse the virtualenv to test a module without a django dependency, I get the following traceback on setup for all tests:

============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.6.7rc1, pytest-3.8.2, py-1.6.0, pluggy-0.7.1
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/ndandrim/work/swh-environment/swh-indexer/.hypothesis/examples')
rootdir: /home/ndandrim/work/swh-environment/swh-indexer, inifile: pytest.ini
plugins: requests-mock-1.5.2, litf-0.1.4, django-3.4.3, hypothesis-3.79.0, celery-4.2.1, aiohttp-json-rpc-0.11.2

[... snip ...]
____________________________________________________________________________________ ERROR at setup of IndexerTestStorage.test_revision_metadata_missing _____________________________________________________________________________________

tp = <class 'django.core.exceptions.ImproperlyConfigured'>, value = None, tb = None

    def reraise(tp, value, tb=None):
        try:
            if value is None:
                value = tp()
            if value.__traceback__ is not tb:
>               raise value.with_traceback(tb)

../../../.virtualenvs/swh-test/lib/python3.6/site-packages/six.py:692: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../.virtualenvs/swh-test/lib/python3.6/site-packages/pytest_django/plugin.py:372: in django_test_environment
    dj_settings.DEBUG = False
../../../.virtualenvs/swh-test/lib/python3.6/site-packages/django/conf/__init__.py:70: in __setattr__
    super(LazySettings, self).__setattr__(name, value)
../../../.virtualenvs/swh-test/lib/python3.6/site-packages/django/utils/functional.py:268: in __setattr__
    self._setup()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <LazySettings [Unevaluated]>, name = None

    def _setup(self, name=None):
        """
            Load the settings module pointed to by the environment variable. This
            is used the first time we need any settings at all, if the user has not
            previously configured the settings manually.
            """
        settings_module = os.environ.get(ENVIRONMENT_VARIABLE)
        if not settings_module:
            desc = ("setting %s" % name) if name else "settings"
            raise ImproperlyConfigured(
                "Requested %s, but settings are not configured. "
                "You must either define the environment variable %s "
                "or call settings.configure() before accessing settings."
>               % (desc, ENVIRONMENT_VARIABLE))
E           django.core.exceptions.ImproperlyConfigured: Requested settings, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

../../../.virtualenvs/swh-test/lib/python3.6/site-packages/django/conf/__init__.py:39: ImproperlyConfigured

In that module, nothing explicitly imports django.... but pytest-django thinks that we did.

After instrumenting pytest_django.lazy_django.django_settings_is_configured, I've noticed that DJANGO_SETTINGS_MODULE is indeed unset (duh) but that django.conf got imported somehow. I've therefore added a traceback.print_stack() to django/conf/init.py, and here's the result:

  File "/home/ndandrim/.virtualenvs/swh-test/bin/pytest", line 11, in <module>
    sys.exit(main())
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/_pytest/config/__init__.py", line 58, in main
    config = _prepareconfig(args, plugins)
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/_pytest/config/__init__.py", line 185, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/pluggy/hooks.py", line 258, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/pluggy/manager.py", line 67, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/pluggy/manager.py", line 61, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/pluggy/callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/_pytest/config/__init__.py", line 653, in pytest_cmdline_parse
    self.parse(args)
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/_pytest/config/__init__.py", line 825, in parse
    self._preparse(args, addopts=addopts)
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/_pytest/config/__init__.py", line 777, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/pluggy/manager.py", line 253, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2324, in load
    return self.resolve()
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2330, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/hypothesis/__init__.py", line 26, in <module>
    from hypothesis._settings import settings, Verbosity, Phase, HealthCheck, \
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/_pytest/assertion/rewrite.py", line 290, in load_module
    six.exec_(co, mod.__dict__)
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/hypothesis/_settings.py", line 37, in <module>
    from hypothesis.internal.compat import text_type
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/_pytest/assertion/rewrite.py", line 290, in load_module
    six.exec_(co, mod.__dict__)
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/hypothesis/internal/compat.py", line 552, in <module>
    from django.test import TransactionTestCase
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/django/test/__init__.py", line 5, in <module>
    from django.test.client import Client, RequestFactory
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/django/test/client.py", line 12, in <module>
    from django.conf import settings
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ndandrim/.virtualenvs/swh-test/lib/python3.6/site-packages/django/conf/__init__.py", line 21, in <module>
    traceback.print_stack()

Looks like (in my case) the django.conf module gets imported by the hypothesis pytest plugin, and ends up being unused.

I'd suggest making the pytest_django.lazy_django.django_settings_is_configured check look only for DJANGO_SETTINGS_MODULE, because there's no guarantee that there won't be another plugin spuriously importing a django module in the future.

[blueyed]

@olasd
Thanks for the investigation.
Your suggestion sounds reasonable, please consider creating a PR (with a test).

[olasd]

@blueyed I noticed that pytest-django already almost tests for the same issue but not quite exactly :) I've adapted my propsoed change to stay as close to the original intent as possible.