Auto-detect src and/or test directories

[brettcannon]

Some people keep their source in a src directory and some people keep their tests in test. This seems to up often enough that we should either detect this and automatically do something or warn people they should create a .env file that manipulates their PYTHONPATH to include the appropriate directories.

[DonJayamanne]

and some people keep their tests in test.

We already have some code that looks for directories named test (or similar) and we display a prompt asking the user whether they'd like to configure VS Code to discover and run the tests.

[DonJayamanne]

We might want to update our documentation too (see #1328), other IDEs do the same.
Besides that's how python works (using PYTHONPATH)

[DonJayamanne]

We already have some code that looks for directories named test (or similar) and we display a prompt asking the user whether they'd like to configure VS Code to discover and run the tests.

I take that back, the auto detection of tests was take out of the extension 99ff84c
I guess this was removed as users found the prompts too annoying.

[DonJayamanne]

If we do go back with the auto-prompt, we might want to make the prompt more subtle. As it is we have a linter prompt. I guess it would be a review of the extension startup (setup) experience.

[brettcannon]

https://twitter.com/r1chardj0n3s/status/1019521759156924416 suggests at least on Twitter people like to use src.

[sloria]

I use src/ directories for my libraries. I think the addressing #4954 would also fix this issue more directly than workarounds like suggesting PYTHONPATH modification.

[jakebailey]

@sloria I'm working on something that will improve import detection when editable installs are used; i.e. if a package has been installed as editable, then the language server will recognize that and add it to its list of user import paths automatically, as opposed to the situation right now where editable installs work but still require extraPaths/PYTHONPATH to be set.

It has the downside of only working if something has been editable installed at all (so not if you just open any project, and is not the same as the original issue filed here, per se), but would that be helpful/in the vein of what you were interested in?

[karrtikr]

Please also include the workspace folder directory itself. (the one referred using PYTHONPATH=.)

[jakebailey]

If you are talking to me, that's actually already done in the LS; we always have the workspace root as an import path (non-optional).

Also, linking this, since it involves auto-configuration for src directories: microsoft/python-language-server#1440

[karrtikr]

I was referring to the person who works on this in general, not just LS. I just noticed that I had to explicitly add PYTHONPATH=. for tests to be discovered, and it was not auto-detecting modules in the root directory.

[luabud]

#6544

[uchuugaka]

Why close #5710 for this one?
Nobody is going to use src and test directories all the time, it forces an unrealistic directory structure on Python projects.
Seems more like we need the app to recognize local modules the same way Python itself does without going through hoops of setting up additional, unnecessary files.

[jakebailey]

Nobody is going to use src and test directories all the time, it forces an unrealistic directory structure on Python projects.

This issue is about auto-detection. If you do not use this structure (most projects do not use src, many use test), you are not affected. The language server has done this for about 10 months to good effect. Jedi's import resolution is looser, and may resolve things more forgivingly. Other tooling (like linters) may or may not be able to do this yet.

Seems more like we need the app to recognize local modules the same way Python itself does without going through hoops of setting up additional, unnecessary files.

This is a pretty difficult thing to get correct in all cases. The biggest offenders are scripts; it's not generically possible to know if any given .py file is intended to be imported from somewhere else or run as a script, nor can we really know from where a user is intending to run their code (where imports are rooted, which we typically assume to be the workspace root). If it's a script, then the interpreter will add the script's directory as a search path. But the editor doesn't know this is going to happen 100% of the time. Tools like pytest may also modify the search path. And if the user has a src directory, or some other format, it can't know for certain if a given import is going to resolve or not.

Anything automatic/default is a heuristic to be improved, but the notion of configuring search paths is not uncommon in editors that need to handle Python, as there is no perfect algorithm that can both provide good feedback about errors and handle the wide range of cases.

[uchuugaka]

When you follow the same import rules as Python, it is really simple. A file or package in the same directory should be importable normally in the current file. It’s really not much to ask to silence an incorrect warning that all other editors seem to get right.

…

On Jun 5, 2020, at 13:35, Jake Bailey ***@***.***> wrote:  Nobody is going to use src and test directories all the time, it forces an unrealistic directory structure on Python projects. This issue is about auto-detection. If you do not use this structure (most projects do not use src, many use test), you are not affected. The language server has done this for about 8 months to good effect. Jedi's import resolution is looser, and may resolve things more forgivingly. Other tooling (like linters) may or may not be able to do this yet. Seems more like we need the app to recognize local modules the same way Python itself does without going through hoops of setting up additional, unnecessary files. This is a pretty difficult thing to get correct in all cases. The biggest offenders are scripts; it's not generically possible to know if any given .py file is intended to be imported from somewhere else or run as a script, nor can we really know from where a user is intending to run their code (where imports are rooted, which we typically assume to be the workspace root). If it's a script, then the interpreter will add the script's directory as a search path. But the editor doesn't know this is going to happen 100% of the time. Tools like pytest may also modify the search path. And if the user has a src directory, or some other format, it can't know for certain if a given import is going to resolve or not. Anything automatic/default is a heuristic to be improved, but the notion of configuring search paths is not uncommon in editors that need to handle Python, as there is no perfect algorithm that can both provide good feedback about errors and handle the wide range of cases. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[brettcannon]

Pylance picks up on src/ directories already, so closing as outdated.