MyPy cannot find module

[DavidCookBrite]

I'm adding MyPy annotations to a single Python 2 file that uses the py2neo package, here's what I have so far (just 2 functions annotated). The file is called gdb.py.

from py2neo import Graph
from contexttimer import Timer
import simpleflake
from datetime import datetime

graph   = None

def init(neo4j_uri):
    # type: (str) -> None
    global graph
    graph = Graph('{}/db/data'.format(neo4j_uri))


def _exec(cypher_text, params):
    # type: (str, Dict[str,Any]) -> Tuple[RecordListList,Timer]
    with Timer() as t:
        result = graph.cypher.execute( cypher_text, params )
    return result, t

I run MyPy from the terminal in the same directory and virtual-environment as normal Python > mypy --py2 gdb.py

I get this error output:

fugu_gdb.py:15: error: Cannot find module named 'py2neo'
fugu_gdb.py:15: note: (Perhaps setting MYPYPATH would help)
fugu_gdb.py:16: error: Cannot find module named 'contexttimer'
fugu_gdb.py:17: error: Cannot find module named 'simpleflake'
fugu_gdb.py: note: In function "_exec":
fugu_gdb.py:34: error: Name 'RecordListList' is not defined

I can't find any documentation on how MYPYPATH is meant to be set.

My __init__.py file doesn't contain anything of significance, just futures imports.

Python itself works fine - python gdb.py runs without any errors when run from the same terminal session as mypy.

[gvanrossum]

MYPYPATH should be set similarly to how you would set PYTHONPATH. In your case I presume you've installed py2neo using pip or another similar way, so Python doesn't need PYTHONPATH; but mypy doesn't know what Python 2's sys.path looks like (nor should it, really) so you need to set MYPYPATH. Try to figure out the directory where py2neo is installed and setting MYPYPATH to that directory.

[DavidCookBrite]

Thanks. When I print sys.path to terminal I get a lot of absolute filesystem paths (e.g. ['','/Users/me/.virtualsenvs/myproject/python2.7/plat-darwin', '/usr/local/Cellary/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/.... What is the best way to set MYPYPATH and what should it contain? Surely it doesn't need a copy of everything in my sys.path? Is there a succinct (and portable) bash command I can execute which will transfer my Python config over to MyPy?

[DavidCookBrite]

FWIW, so I'm not blocked I copied the output from sys.path and converted it to colon-separated paths, so my simple shell-script looks like this for now:

export MYPYPATH=/Users/me/.virtualenvs/myproject/lib/...:/usr/local/Cellar/...
mypy --py2 gdb.py

I get this error output:

Traceback (most recent call last):
  File "/usr/local/bin/mypy", line 6, in <module>
    main(__file__)
  File "/usr/local/lib/python3.5/site-packages/mypy/main.py", line 54, in main
    type_check_only(sources, bin_dir, options)
  File "/usr/local/lib/python3.5/site-packages/mypy/main.py", line 98, in type_check_only
    python_path=options.python_path)
  File "/usr/local/lib/python3.5/site-packages/mypy/build.py", line 206, in build
    result = manager.process(initial_states)
  File "/usr/local/lib/python3.5/site-packages/mypy/build.py", line 403, in process
    next.process()
  File "/usr/local/lib/python3.5/site-packages/mypy/build.py", line 868, in process
    self.semantic_analyzer().visit_file(self.tree, self.tree.path)
  File "/usr/local/lib/python3.5/site-packages/mypy/semanal.py", line 230, in visit_file
    self.accept(d)
  File "/usr/local/lib/python3.5/site-packages/mypy/semanal.py", line 2168, in accept
    node.accept(self)
  File "/usr/local/lib/python3.5/site-packages/mypy/nodes.py", line 673, in accept
    return visitor.visit_if_stmt(self)
  File "/usr/local/lib/python3.5/site-packages/mypy/semanal.py", line 1593, in visit_if_stmt
    self.visit_block(s.body[i])
  File "/usr/local/lib/python3.5/site-packages/mypy/semanal.py", line 969, in visit_block
    self.accept(s)
  File "/usr/local/lib/python3.5/site-packages/mypy/semanal.py", line 2168, in accept
    node.accept(self)
  File "/usr/local/lib/python3.5/site-packages/mypy/nodes.py", line 564, in accept
    return visitor.visit_assignment_stmt(self)
  File "/usr/local/lib/python3.5/site-packages/mypy/semanal.py", line 1027, in visit_assignment_stmt
    self.process_namedtuple_definition(s)
  File "/usr/local/lib/python3.5/site-packages/mypy/semanal.py", line 1326, in process_namedtuple_definition
    named_tuple = self.check_namedtuple(s.rvalue)
  File "/usr/local/lib/python3.5/site-packages/mypy/semanal.py", line 1357, in check_namedtuple
    return self.build_namedtuple_typeinfo('namedtuple', [], [])
  File "/usr/local/lib/python3.5/site-packages/mypy/semanal.py", line 1445, in build_namedtuple_typeinfo
    info.mro = [info] + info.tuple_type.fallback.type.mro
TypeError: can only concatenate list (not "NoneType") to list

*** INTERNAL ERROR ***

/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/collections.py:700: error: Internal error -- please report a bug at https://github.com/JukkaL/mypy/issues

NOTE: you can use "mypy --pdb ..." to drop into the debugger when this happens.

[gvanrossum]

Oh. It looks like setting MYPYPATH like that also finds the stdlib collections.py instead of the stub for it from typeshed. It should be easy enough to find out what the path for py2neo -- use something like `python -c 'import py2neo; print py2neo.__file__'`. I'm no expert in bash so you'll have to figure out for yourself how to turn that output into the directory (maybe `dirname`?)

[JukkaL]

Pointing mypy to the standard library is generally not desirable unless typeshed comes first in the search path as then you'll lose type checking on code that uses the std library (there are no annotations in the std library) and also because mypy will generate spurious errors when running against a bunch of std library code that it can't quite handle yet.

[gvanrossum]

Closing as dupe of #1339.

[timabbott]

I also ran into the traceback above today trying to figure out how to get mypy to work when the Python module dependencies are in a Python 2 virtualenv, using basically env MYPYPATH=/srv/zulip-venv/lib/python2.7/site-packages/ mypy --py2 --check-untyped-defs zerver/ as the command line. I'm not entirely sure what I'm supposed to do in order to use mypy with a venv in general, which is I think what #1339 is about, but perhaps the mypy crash is worth having open as a separate bug from the documentation issue?

[gvanrossum]

The crash is not very interesting. We'd like to do better than crashing, of course, but the real issue here is that (probably through setting MYPYPATH equal to PYTHONPATH) mypy is analyzing the real stdlib instead of the stubs from typeshed. This is just not how it's supposed to be used. What were you hoping to get from setting MYPYPATH to your venv's site-packages? You're really much better off living with errors about missing modules, or using -s/--silent-imports.

[JukkaL]

It's unlikely that mypy will ever be able to make sense of totally arbitrary library modules, because some modules use dynamic magic to define their contents, and mypy would see nothing, and some modules are implemented in C and, again, mypy can't see inside them.

[timabbott]

Ahh, OK, it turns out I had misdiagnosed an issue. When I ran mypy on a machine that only has the dependency libraries installed in a venv, I got a number of errors like this:

zerver/lib/push_notifications.py: note: In function "send_android_push_notification":
zerver/lib/push_notifications.py:121: error: Cannot determine type of 'GCM'
zerver/lib/push_notifications.py:136: error: Cannot determine type of 'GCM'
zerver/lib/push_notifications.py:145: error: Cannot determine type of 'GCM'
zerver/lib/push_notifications.py:150: error: Cannot determine type of 'GCM'
zerver/lib/push_notifications.py:155: error: Cannot determine type of 'GCM'

which I had thought were related to effectively

import gcmclient
gcmclient.GCM

and thinking I needed to have mypy have access to the actual gcmclient to use this

But it turns out the cause is instead:

class PushDeviceToken(models.Model):                                            
    APNS = 1
    GCM = 2

PushDeviceToken.GCM

which I can resolve by adding a # type: int on the GCM = 1 lines. This is probably a mypy bug that I need to add this annotation, since it's clearly an int, but clearly a separate issue (should I open one, or is this known?).

So I guess I don't need to be setting MYPYPATH here; sorry for the confusion!