MYPYPATH not working with local -stubs package

[Dentosal]

This is a bug report. However, if the current behavior is not a bug, then this is request to improve documentation.

Description

Mypy doesn't recognise my local library typing stub.

Reproduction:

Actual Python file

import aiomysql

Directory tree

pythonlibtypes/
└── aiomysql-stubs
    ├── __init__.pyi
    ├── connection.pyi
    ├── cursors.pyi
    ├── pool.pyi
    ├── py.typed
    └── utils.pyi

Mypy command

MYPYPATH="/path/to/project/pythonlibtypes/" pipenv run -- mypy --config mypy.ini module_to_check

mypy.ini:

[mypy]
warn_redundant_casts = True
warn_unused_ignores = True
incremental = True
follow_imports = normal

Actual behavior

path/to/file.py:1: error: Cannot find module named 'aiomysql'

Expected behavior

Import succeeds, Mypy uses the stub

Versions

Python 3.6.5

mypy 0.620

[gvanrossum]

@ethanhs

I wonder if mypy currently only looks for stub packages in the system Python's site-packages directory (and a few related ones, e.g. user-packages) but not along MYPYPATH. But it makes sense to also look there so it's easier to test stub packages. Ethan, what do you think?

[emmatyping]

I think I was initially hesitant to do this as it would not be easy to do with the old search path implementation, and I thought editable installs would handle this use case.

However, with my refactoring of search paths, this is very straightforward to implement.

I still feel like using editable installs is the best way to test things (because that is the closest approximation to a real installation).

To be clear: when you say MYPYPATH, do you mean MYPYPATH (as set by the user) joined with the PYTHONPATH (the files/directories the user points mypy at) or just the former? I think it might be a bit surprising if I check a test from the top level of a source tree to automatically pick up stubs in a subdirectory. In other words, if I checkeded project/test/foo.py and had stubs project/blah-stubs next to project/blah, I would not immediately expect them to be picked up if running mypy in the project directory.

[gvanrossum]

Those are mostly questions for @Dentosal. I'm not sure why you're mentioning PYTHONPATH (which AFAIK is just the environment variable used by Python itself to find modules in addition to its default search path). But your example poses an interesting question. In site packages, of we have both blah and blah-stubs installed, mypy only looks in blah if blah/py.typed exists -- and for blah-stubs the presence of py.typed is irrelevant (right?). But elsewhere (when a package is either passed on the command line, or lives in $MYPYPATH, or elsewhere in mypy's default search path (which starts out with . and also includes __init__-less directories containing files passed on the command line), py.typed is irrelevant.

[Dentosal]

Just adding the folder path pythonlibtypes/aiomysql/ to mypy arguments seems to work, but nothing else does. This doesn't need -stubs suffix, or py.typed file.

pythonlibtypes
└── aiomysql
    ├── __init__.pyi
    ├── connection.pyi
    ├── cursors.pyi
    ├── pool.pyi
    └── utils.pyi

I'm not quite sure what the correct behavior should be. However, Mypy documentation says:

First, mypy has its own search path. This is computed from the following items:

• The MYPYPATH environment variable (a colon-separated list of directories).
• The directories containing the sources given on the command line (see below).
• The installed packages marked as safe for type checking (see PEP 561 support)
• The relevant directories of the typeshed repo.

So I think adding the folder to MYPYPATH should have same effect as specifying it in Mypy arguments, i.e. Mypy should use the local type stubs (in pythonlibtypes/aiomysql/) when importing aiomysql.

[emmatyping]

Okay, I think I've convinced myself that stub packages in the MYPYPATH and pointed to by mypy should be picked up. I will work on an implementation soon.

[emmatyping]

Hmm, I've sat and thought about this, and also implemented it. But I still feel like telling people to either install their stubs or name the package to be compatible with MYPYPATH is a better answer. It would also violate the assertion that packages on the MYPYPATH are valid Python identifiers, which bothers me.

I am open for someone to convince me otherwise, but I'm not too keen to do this (though I am happy to add some documentation about the current state of things).

[gvanrossum]

OK, let's reject the feature request and instead add a bit of documentation.