Assertion errors when working with existing codebase

[rafales]

I'm having one more problem. Background: I'm adding typing to 2y+ codebase, written in django. I started with those mypy settings:

[mypy]
follow_imports=silent
ignore_missing_imports=true
disallow_untyped_defs=true
plugins=mypy_django_plugin.main

# don't show any errors from migrations
[mypy-*.migrations.*]
ignore_errors = true

I'm running mypy with a set of files and modules which are considered to be properly typed, not all files. The problem is that I often run into this problem:

sms/twilio/api.py:11: error: Function is missing a return type annotation
sms/twilio/api.py:29: error: Function is missing a type annotation
sms/twilio/api.py:34: error: Function is missing a type annotation
sms/twilio/api.py:38: error: Function is missing a type annotation
sms/twilio/api.py:56: error: Function is missing a type annotation
sms/tasks/sms.py:16: error: Function is missing a type annotation
Traceback (most recent call last):
  File "/Users/rafal/Projects/censored/venv/bin/mypy", line 11, in <module>
    sys.exit(console_entry())
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/main.py", line 91, in main
    res = build.build(sources, options, None, flush_errors, fscache)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/build.py", line 162, in build
    result = _build(sources, options, alt_lib_path, flush_errors, fscache)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/build.py", line 217, in _build
    graph = dispatch(sources, manager)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/build.py", line 2360, in dispatch
    process_graph(graph, manager)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/build.py", line 2653, in process_graph
    process_fresh_modules(graph, prev_scc, manager)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/build.py", line 2731, in process_fresh_modules
    graph[id].fix_cross_refs()
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/build.py", line 1714, in fix_cross_refs
    self.options.use_fine_grained_cache)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/fixup.py", line 25, in fixup_module
    node_fixer.visit_symbol_table(tree.names)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/fixup.py", line 88, in visit_symbol_table
    self.visit_type_info(value.node)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/fixup.py", line 45, in visit_type_info
    self.visit_symbol_table(info.names)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/fixup.py", line 90, in visit_symbol_table
    value.node.accept(self)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/nodes.py", line 782, in accept
    return visitor.visit_var(self)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/fixup.py", line 133, in visit_var
    v.type.accept(self.type_fixer)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/types.py", line 620, in accept
    return visitor.visit_instance(self)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/fixup.py", line 157, in visit_instance
    a.accept(self)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/types.py", line 1601, in accept
    return visitor.visit_union_type(self)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/fixup.py", line 229, in visit_union_type
    it.accept(self)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/types.py", line 620, in accept
    return visitor.visit_instance(self)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/fixup.py", line 150, in visit_instance
    inst.type = lookup_qualified_typeinfo(self.modules, type_ref, self.quick_and_dirty)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/fixup.py", line 240, in lookup_qualified_typeinfo
    node = lookup_qualified(modules, name, quick_and_dirty)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/fixup.py", line 254, in lookup_qualified
    stnode = lookup_qualified_stnode(modules, name, quick_and_dirty)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/fixup.py", line 263, in lookup_qualified_stnode
    return lookup_fully_qualified(name, modules, raise_on_missing=not quick_and_dirty)
  File "/Users/rafal/Projects/censored/venv/lib/python3.6/site-packages/mypy/lookup.py", line 47, in lookup_fully_qualified
    assert key in names, "Cannot find %s for %s" % (key, name)
AssertionError: Cannot find models for general.models.CensoredUser

This seems to be happening because of mypy cache. Running rm -rf .mypy_cache always fixes this, but just for the next run.

It usually happens for user model or other "popular" model in the app.

$ python --version
Python 3.6.8
$ mypy --version
mypy 0.670
$ pip freeze | grep django-stubs
django-stubs==0.11.1

it works fine when I remove mypy django plugin.

[mkurnikov]

I had the same problem with the plugin on internal codebase, wasn't able to fix it. incremental=False works fine, and is a nicer alternative to removing cache after each run.

mypy.fixup module, where error happens, is responsible for resolving dependencies after deserialization from mypy cache. So, I think it's just a mypy bug, it stumbles over module relationships in complex Django codebases worsened by artificial dependency of django.conf.global_settings and DJANGO_SETTINGS_MODULE modules.

Probably should be reported there, but I couldn't make a simple reproduce and decided to postpone for better times.

[rafales]

Problem lies in a file where I refer to CensuredUser like this:

class MyModel(models.Model):
    user = models.ForeignKey("general.CensoredUser")

From quickly looking through cache it seems that models.meta.json for a file which refers to general.models.CensoredUser doesn't contain general.models in dependencies which is probably the reason why loading cache fails.

Now need to figure out why the entry is missing from dependencies.

[syastrov]

Maybe it's possible to solve with the new get_additional_deps hook (although I guess to determine the additional dependencies, the AST has to be walked to find the to parameter)?

[mkurnikov]

I've committed some related code in mkurnikov@aeb435c. Could you check out master?