Navigating to file within site-packages and type checking with mypy causes site-packages is in the PYTHONPATH #8567
Comments
|
Hm, so we add the directory that mypy is run from to the PYTHONPATH like Python would. Could that be the culprit? You can run mypy with -vv to get the python/package paths to debug this further. And yeah ideally we should print to stderr here. |
emmatyping
added
bug
|
Thanks for the quick response, if I run the flycheck mypy checker and add -vv switch, the log contains the following trace of various python_paths - let me know if you need to see the complete trace or if these are sufficient: |
|
Thank you for that! It doesn't look like the trace you gave is in the site packages directory, this is still from an erroring run? It may be we are searching the directory above the file you pass perhaps? I am a bit rusty on our module searching code. |
|
Sorry I may have misunderstood what you were asking for. So if I run mypy with the full path and -vv switch, no trace is written to the console and the error described above is generated: If I then navigate to the package directory where frame.py resides and run again without the full path then we do see the current directory and full path of that current directory in the python_path. Equally in this case a number of mypy errors and warnings are generated as you'd expect for this file. |
|
Sorry, I probably wasn't clear :) Thank you for running those, it seems we exit before the debug information can be printed, which is unfortunate. We probably should print the paths and give more information here. I will try to reproduce this myself. |
|
I'm happy to try and help if I can - I was going to pull the code down over the weekend and have a poke around. If you have higher priority tasks you're working on, point me in the right direction and I can report the more detailed information and work towards a fix. |
|
That would be great! If you want more information, you can set a breakpoint for pdb around here (this is where the error comes from): https://github.com/python/mypy/blob/master/mypy/modulefinder.py#L545 Let me know if you have more questions related to that code (I am also on the gitter chat a few times a day) |
|
The issue relates to modulefinder's check to see if the site packages retrieved from get_site_packages lies in python_path. python_path is constructed from from the sources passed, in the case where a fully qualified path that lies within the site-packages area is passed it checks to see if the source has a base_dir and then adds that base_dir to the python_path: python_path = [] # type: List[str]
if not alt_lib_path:
for source in sources:
# Include directory of the program file in the module search path.
if source.base_dir:
dir = source.base_dir
if dir not in python_path:
python_path.append(dir)In my example the source would be /home/dsyzling/miniconda3/envs/pandastest/lib/python3.7/site-packages/pandas/core/frame.py and the base.dir added to python_path ends up being /home/dsyzling/miniconda3/envs/pandastest/lib/python3.7/site-packages. My site packages contain: '/home/dsyzling/miniconda3/envs/mypy/lib/python3.7/site-packages'
'/home/dsyzling/.local/lib/python3.7/site-packages'So the check then fails because site-packages is within python_path. When running within the directory of pandas and just specifying the name of the file on its own - frame.py - then we don't add the site-packages directory, the current directory /home/dsyzling/miniconda3/envs/pandastest/lib/python3.7/site-packages/pandas/core is added but this won't match against /home/dsyzling/miniconda3/envs/mypy/lib/python3.7/site-packages and so doesn't trigger the error allowing mypy to type check the file. |
|
Ah, ok, that makes total sense. I think this error will have to go away anyway once #5701 is implemented (which I am hoping to work on soon), otherwise I'd change the error message. I'm not entirely sure if there is an action item here if that is the case. |
|
No problem, I can track #5701 and once that has been merged I can retest with flycheck and my specific scenario. Thanks for your help with this issue, much appreciated. |
Essentially mypy seems to have bad behaviour when you try and run on it site-packages. This is a very common paradigm for testing conda packages that is a known isue python/mypy#8567 So I do a crazy hack to work around this and just create a symlink to the created package such that mypy doesn't detect it's in site-packages and refuse to run
Essentially mypy seems to have bad behaviour when you try and run on it site-packages. This is a very common paradigm for testing conda packages that is a known isue python/mypy#8567 So I do a crazy hack to work around this and just create a symlink to the created package such that mypy doesn't detect it's in site-packages and refuse to run
I'm seeking to establish the expected/defined behaviour for mypy given the context of the problem below when working with python lsp servers and editor environments - in particular with Emacs/flycheck and python-mypy. I have reported this issue to the flycheck authors and we are seeking further guidance from the mypy team.
My environment is:
flycheck 20200320.1402
mypy 0.770
python 3.7.6
Emacs 26.2 on Arch Linux using lsp-mode and in this case the Microsoft python language server.
If I navigate to a source file within Emacs (go to definition) - the example below jumps to a source file within pandas, flycheck will run mypy to type check the file. The full path is passed to mypy and here is the result:
mypy --show-column-numbers /home/dsyzling/miniconda3/envs/pandastest/lib/python3.7/site-packages/pandas/core/frame.py
/home/dsyzling/miniconda3/envs/pandastest/lib/python3.7/site-packages is in the
PYTHONPATH. Please change directory so it is not.
However if I navigate to the directory manually and run:
mypy frame.py
The file type checks, . a non zero return code is returned along with a set of errors.
The issue for flycheck is that it will always pass the full path of the file loaded within the editor and in this case mypy returns a non zero return code but I think the author said it doesn't report the PYTHONPATH error on standard error output. At the very least that error could be reported by flycheck.
So I have a number of questions:
In my case PYTHONPATH doesn't contain site-packages, however the full path passed to mypy does - is this expected behaviour? Since running with just the filename works, there is a slight inconsistency here, what is the expected behaviour?
Should the error be reported to standard error as well? I think in this case with flycheck it may confuse the user because they will check their PYTHONPATH which may not contain site-packages. However it's better than the current situation which constantly nags the user that mypy returned a non zero error code without any errors and the user should make sure all of their packages are up to date and raise a bug report.
As a developer I probably don't want these files being type checked - if I'm navigating to a third party source library to check docs or source code I don't necessarily want flake8, mypy checking the files and reporting errors. However at the moment between lsp-mode, flycheck and mypy (as well as other linters) any kind of inclusion/exclusion criteria falls through the cracks - there is no behaviour to specify not applying linters/checkers to a set of directories outside of the project area.
I should say in the case where developers are using pyls (Palentir), the language server performs linting checks with plugins and I'm guessing the current third party plugin for mypy probably ignores the error message (flycheck and python-mypy are not used under this scenario). I ran into the problem trying to work on a solution to incorporate running other linters/checkers with the Microsoft python language server (mypls) which doesn't follow the same approach as pyls.
The text was updated successfully, but these errors were encountered: