Incompatible with django-configurations due to the way the settings are imported

[snejus]

Bug report

🐛

What's wrong

Trying to use django-stubs together with django-configurations is a big pain since the latter interferes and controls how django settings are loaded. By default, the following exception gets thrown when trying to run mypy:

(mypackage-vfeg9Sk1-py3.6) ➜  mypackage IN-3486 ✗ python -m mypy -p src.mypackage                                                       
Error constructing plugin instance of NewSemanalDjangoPlugin

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/sarunas/.cache/pypoetry/virtualenvs/mypackage-vfeg9Sk1-py3.6/lib/python3.6/site-packages/mypy/__main__.py", line 12, in <module>
    main(None, sys.stdout, sys.stderr)
  File "mypy/main.py", line 89, in main
  File "mypy/build.py", line 180, in build
  File "mypy/build.py", line 227, in _build
  File "mypy/build.py", line 465, in load_plugins
  File "mypy/build.py", line 443, in load_plugins_from_config
  File "/home/sarunas/.cache/pypoetry/virtualenvs/mypackage-vfeg9Sk1-py3.6/lib/python3.6/site-packages/mypy_django_plugin/main.py", line 80, in __init__
    self.django_context = DjangoContext(django_settings_module)
  File "/home/sarunas/.cache/pypoetry/virtualenvs/mypackage-vfeg9Sk1-py3.6/lib/python3.6/site-packages/mypy_django_plugin/django/context.py", line 88, in __init__
    apps, settings = initialize_django(self.django_settings_module)
  File "/home/sarunas/.cache/pypoetry/virtualenvs/mypackage-vfeg9Sk1-py3.6/lib/python3.6/site-packages/mypy_django_plugin/django/context.py", line 70, in initialize_django
    settings._setup()
  File "/home/sarunas/.cache/pypoetry/virtualenvs/mypackage-vfeg9Sk1-py3.6/lib/python3.6/site-packages/django/conf/__init__.py", line 66, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/sarunas/.cache/pypoetry/virtualenvs/mypackage-vfeg9Sk1-py3.6/lib/python3.6/site-packages/django/conf/__init__.py", line 157, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/sarunas/Documents/projects/mypackage/mypackage/project/settings.py", line 6, in <module>
    from mypackage.settings import Base, IntegratedTests
  File "/home/sarunas/.cache/pypoetry/virtualenvs/mypackage-vfeg9Sk1-py3.6/lib/python3.6/site-packages/mypackage/settings.py", line 26, in <module>
    class Base(Configuration):  # pylint: disable=no-init
  File "/home/sarunas/.cache/pypoetry/virtualenvs/mypackage-vfeg9Sk1-py3.6/lib/python3.6/site-packages/configurations/base.py", line 28, in __new__
    raise ImproperlyConfigured(install_failure)
django.core.exceptions.ImproperlyConfigured: django-configurations settings importer wasn't correctly installed. Please use one of the starter functions to install it as mentioned in the docs: https://django-configurations.readthedocs.io/

That makes sense, since configurations disallows loading the settings manually and requires to run these two calls to configure django, both of them are for example found in manage.py and wsgi.py:

        import configurations
        configurations.setup()

mypy configuration in setup.cfg (it follows [metadata] and [options] package metadata definitions)

[mypy.plugins.django-stubs]
django_settings_module = project.settings

[mypy]
plugins =
    mypy_django_plugin.main

ignore_errors =            false
warn_no_return =           true
strict_equality =          true
strict_optional =          true
warn_return_any =          true
warn_unreachable =         true
implicit_reexport =        false
check_untyped_defs =       true
local_partial_types =      true
warn_unused_ignores =      true
warn_unused_configs =      true
warn_redundant_casts =     true
no_implicit_optional =     true

namespace_packages =       true
fine_grained_incremental = true
show_error_codes =         true

strict =                   true
allow_any_generics =       false
allow_subclassing_any =    true
ignore_missing_imports =   false

[mypy-mypackage.migrations.*]
ignore_errors = true

I tried looking for a workaround which didn't involve changing neither mypy nor django-stubs, nor django-configurations source files and I'm afraid I have failed. Unfortunately this also blocks DRF stubs from being useful.

How is that should be

django-configurations should be supported and allowed to load django settings if the user uses it (unless it is somehow incompatible with the way the stubs are generated?).

System information

• OS: "Ubuntu 18.04.4 LTS"
• python version: 3.6.9
• django version: 2.2.14
• mypy version: 0.770
• django-stubs version: 1.5.0
• django-configurations version: 2.2

Possible solution

One of possible options would be to include those two calls in mypy_django_plugin.django.context initialize_django call, just before the models get imported:

def initialize_django(settings_module: str) -> Tuple['Apps', 'LazySettings']:
    with temp_environ():
        os.environ['DJANGO_SETTINGS_MODULE'] = settings_module

        # add current directory to sys.path
        sys.path.append(os.getcwd())

        def noop_class_getitem(cls, key):
            return cls
        
        try:
             # or use importlib.import_module, or check whether django uses it
            import configurations
            configurations.setup()
        except ModuleNotFoundError:
            pass

        from django.db import models

        models.QuerySet.__class_getitem__ = classmethod(noop_class_getitem)  # type: ignore
        models.Manager.__class_getitem__ = classmethod(noop_class_getitem)  # type: ignore
        ...

I found that this works for me - though I haven't tested it extensively yet. I'd be happy to add a PR for this upon your confirmation.

[sobolevn]

Maybe we can provide a configuration option that leads to a some function hook?

[mypy.plugins.django-stubs]
init-hook = my.custom.function

# my/custom.py

def function():
    import configurations
    configurations.setup()

This will solve your (and possibly all other problems).

[snejus]

Great idea! It's also inline with pylint, so that's not something unseen:

  Master:
    --init-hook=<code>  Python code to execute, usually for sys.path
                        manipulation such as pygtk.require().

I reckon that a config option is enough - no need to support an environmental var, since it's not something that changes between environments or different dev setups considering a single project.

Are you by any chance aware of any side effects that this may cause? The function above calls settings.pre_setup(), setup(), and clears the cache in a suspiciously specific order. Though I don't have enough context to tell whether that's just a coincidence or the intention.

Also, having patched the current master (instead of 1.5.0) I found that it can call either configurations or django pre_setup, like

   try:
        import configurations
        configurations.setup()
    except ModuleNotFoundError:
        settings.pre_setup()

Though there have been some significant changes merged into master so I'm not very surprised. Hope it won't get too involved.

[sobolevn]

Are you by any chance aware of any side effects that this may cause?

The good thing we allow any side-effects during the setup phase 😆

Joking aside, I don't think there will be any. Because we already use django.setup() and it is pretty similar.

[snejus]

It's a lovely one - mypy ends up showing a single error instead of a couple of hundred :D Just released there was a PR intending to solve this issue: #180. Though it looks like it's stale now and wasn't exactly what you'd expect to have here.

Just for clarification, I guess your requirements haven't changed, right?

[sobolevn]

Yes, I still believe that we should not add any support for third-party libraries here.
But, the proposed hook is different: it is just a mechanism that will always work the same way for us.
I don't have any problems with that.

[snejus]

Surely not.

Cool, that's great. I'll get this done in the eve.

[Niedzwiedzw]

I'm getting that error when trying to run mypy in my project...

[Dimitri-WEI-Lingfeng]

any update?

[miaachan]

For anyone who comes here – this non-intrusive workaround #180 (comment) written by @wlf100220 might help you.