Add `distutils` as a top-level package included with `types-setuptools` #10948

AlexWaygood · 2023-10-29T18:45:40Z

This breaks stubtest_third_party.py when you run it on Python <3.12 (as we do in CI), unfortunately, as it gets very confused at there being both a stdlib/distutils package and a stubs/setuptools/distutils package. Not sure how to fix that...

Closes #10255

AlexWaygood · 2023-10-29T19:02:43Z

And, I see that the test cases I added fail with pyright...

AlexWaygood · 2023-10-30T18:53:41Z

I got stubtest working, with a few hacks. If we're happy with the hacks, I can ask the pyright maintainers to look at why the test cases I added might be failing.

Or we could just delete the test cases for now, and try to figure out later why it isn't working for pyright. I don't think this PR would cause any regressions for pyright users; it just wouldn't offer the same benefits as it's offering for mypy users

Avasam · 2023-11-01T01:21:44Z

stubs/setuptools/@tests/stubtest_allowlist.txt

+setuptools._distutils.versionpredicate
+
+# Reexported from setuptools._distutils; problems should be fixed there
+distutils\..+


Good call. Should make stubtest errors clearer and contributors will be less tempted to try to apply a fix in stdlib/distutils (I know I have in the past)

srittau · 2023-11-01T10:45:40Z

I would prefer if we copy the distutils stubs into setuptools. From what I understand, they are different from stdlib's distutils at this point anyway, and may evolve further. I'd even argue that copied stubs might be easier to maintain: We probably won't need to update stdlib's distutils much, but might need to update setuptools's from time to time.

AlexWaygood · 2023-11-01T10:56:23Z

I would prefer if we copy the distutils stubs into setuptools. From what I understand, they are different from stdlib's distutils at this point anyway, and may evolve further. I'd even argue that copied stubs might be easier to maintain: We probably won't need to update stdlib's distutils much, but might need to update setuptools's from time to time.

We've already done that. We copied them into setuptools/_distutils some time ago. This PR attempts to reflect the fact that setuptools essentially re-exports the entirety of setuptools._distutils as a standalone distutils package.

At runtime, setuptools._distutils becomes distutils via a crazy importlib hack, but we can't do that in the stubs.

srittau · 2023-11-01T11:06:32Z

Does it still use an import hack? At least the freshly installed version of setuptools (68.2.2) vendors distutils and does seem to use it:

Python 3.11.4 (main, Jun  9 2023, 07:59:55) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools._distutils as d
>>> d.__file__
'/home/srittau/foo/lib/python3.11/site-packages/setuptools/_distutils/__init__.py'
>>> import setuptools._distutils.log as l
>>> l.__file__
'/home/srittau/foo/lib/python3.11/site-packages/setuptools/_distutils/log.py'

AlexWaygood · 2023-11-01T11:19:43Z

setuptools._distutils does not use an import hack, and never has done, but the plain distutils package that's installed into your environment as part of pip install setuptools still does.

srittau · 2023-11-01T11:40:46Z

But setuptools.distutils is equivalent to setuptools._distutils, at least in my tests:

System Python:

Python 3.11.4 (main, Jun  9 2023, 07:59:55) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from setuptools import distutils as d
>>> d.__file__
'/usr/lib/python3/dist-packages/setuptools/_distutils/__init__.py'

Virtualenv with locally updated setuptools:

Python 3.11.4 (main, Jun  9 2023, 07:59:55) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from setuptools import distutils as d
>>> d.__file__
'/home/srittau/foo/lib/python3.11/site-packages/setuptools/_distutils/__init__.py'

AlexWaygood · 2023-11-01T11:47:14Z

Not from setuptools import distutils, just plain import distutils.

pip installing setuptools into your environment also installs a distutils package into your environment that is standalone to setuptools.

And yes, it gets you the same thing as setuptools._distutils. That's what this PR is trying to reflect! :)

AlexWaygood · 2023-11-01T14:39:53Z

@srittau, this is the behaviour I'm trying to reflect in this PR:

(fresh-venv) C:\Users\alexw\coding>python -VV
Python 3.12.0 (tags/v3.12.0:0fb18b0, Oct  2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)]

(fresh-venv) C:\Users\alexw\coding>python -m pip install --upgrade pip
Requirement already satisfied: pip in c:\users\alexw\coding\fresh-venv\lib\site-packages (23.2.1)
Collecting pip
  Obtaining dependency information for pip from https://files.pythonhosted.org/packages/47/6a/453160888fab7c6a432a6e25f8afe6256d0d9f2cbd25971021da6491d899/pip-23.3.1-py3-none-any.whl.metadata
  Using cached pip-23.3.1-py3-none-any.whl.metadata (3.5 kB)
Using cached pip-23.3.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.2.1
    Uninstalling pip-23.2.1:
      Successfully uninstalled pip-23.2.1
Successfully installed pip-23.3.1

(fresh-venv) C:\Users\alexw\coding>python -c "import setuptools"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools'

(fresh-venv) C:\Users\alexw\coding>python -c "import distutils"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'distutils'

(fresh-venv) C:\Users\alexw\coding>python -m pip install setuptools
Collecting setuptools
  Downloading setuptools-68.2.2-py3-none-any.whl.metadata (6.3 kB)
Downloading setuptools-68.2.2-py3-none-any.whl (807 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 807.9/807.9 kB 4.3 MB/s eta 0:00:00
Installing collected packages: setuptools
Successfully installed setuptools-68.2.2

(fresh-venv) C:\Users\alexw\coding>python -c "import setuptools, distutils"

(fresh-venv) C:\Users\alexw\coding>python
Python 3.12.0 (tags/v3.12.0:0fb18b0, Oct  2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils
>>> distutils
<module 'distutils' (C:\Users\alexw\coding\fresh-venv\Lib\site-packages\setuptools\_distutils\__init__.py)>

Avasam · 2024-03-16T05:21:16Z

In pywin32 I just hit error: Cannot find implementation or library stub for module named "distutils" [import-not-found] with mypy that only happens in Python 3.12, because it's using setuptools but still needs distutils functionality to build executable files.

https://github.com/mhammond/pywin32/actions/runs/8305626045/job/22732748471?pr=2211#step:5:11

I think this PR would resolve that issue as it would add distutils-stubs when installing types-setuptools ?

Avasam · 2024-03-16T05:48:44Z

@erictraut Is there some special-casing in pyright that prevents detecting a would-be distutils-stubs from the typeshed path when checking against Python 3.12 ?

erictraut · 2024-03-16T05:58:46Z

@Avasam, no, pyright has no special-case knowledge of distutils or its stub file. However, the VERSIONS file in typeshed says that the stub is unsupported on Python 3.12: distutils: 3.0-3.11. That's probably the cause.

AlexWaygood · 2024-03-16T10:19:58Z

@erictraut, the VERSIONS file only gives type checkers information regarding the stdlib stubs (this is why the file is in our stdlib/ directory). I don't think pyright should be inferring from that file that the third-party distutils stubs being added here are not available on py312+.

This situation arises because the distutils package is no longer part of the stdlib on py312+, and is now instead provided by the third-party setuptools distribution.

erictraut · 2024-03-16T14:34:57Z

@AlexWaygood, that's right. On python 3.11 and earlier, pyright is resolving the import to the stdlib stub. On python 3.12 and newer, the VERSIONS file indicates that it should no longer use the stdlib stub for resolution. Since there is no other distutils library or stub package present, pyright reports that it cannot be resolved in this case. I think that's the correct behavior, no?

I'm not sure what you mean by "provided by the third-party setuptools distribution. I've tried installing the latest setuptools from pypi, and I don't see any distutils package installed along with it. I don't see where pyright would find a distutils.pyi or distutils/__init__.pyi file in this case.

AlexWaygood · 2024-03-16T14:39:38Z

I'm not sure what you mean by "provided by the third-party setuptools distribution. I've tried installing the latest setuptools from pypi, and I don't see any distutils package installed along with it. I don't see where pyright would find a distutils.pyi or distutils/__init__.pyi file in this case.

@erictraut, I mean the behaviour I described in this comment here: #10948 (comment). setuptools includes a copy of distutils in this directory here: https://github.com/pypa/setuptools/tree/main/setuptools/_distutils. This is exposed for users (via some importlib hacks that we don't need to worry about here) as a top-level package that is installed and available when you do pip install setuptools. This PR adds an equivalent stubs package to typeshed's stubs/setuptools directory, in an attempt to get type checkers to understand that import distutils still works on Python 3.12+ if you've pip installed setuptools. However, the failing tests in this PR indicate that pyright still does not understand import distutils to work on Python 3.12 if setuptools has been installed, even with the third-party distutils stubs I'm adding as part of this PR. I'm unclear as to why that might be.

Avasam · 2024-03-16T14:41:50Z

I'm not sure what you mean by "provided by the third-party setuptools distribution. I've tried installing the latest setuptools from pypi, and I don't see any distutils package installed along with it. I don't see where pyright would find a distutils.pyi or distutils/__init__.pyi file in this case.

Setuptools adds their own vendored distsutils to the path. Of course this means it can't be analyzed statically and pyright users have to turn off reportMissingModuleSource, but that's where stubs come in to fill in the gap.

AlexWaygood · 2024-03-16T14:54:52Z

Huh, I see that this seems to be fixed on the latest version of pyright; all CI is now passing on this PR. I wonder what changed, and when.

Sorry for the noise, @erictraut -- thanks for your time, as ever!

AlexWaygood · 2024-03-16T14:56:13Z

I think the only thing still blocking this PR is the question of whether we're happy with the hacks I'm introducing to stubtest_third_party.py in order to get stubtest to still pass? Any thoughts on that from any other maintainers?

Avasam · 2024-03-16T15:01:27Z

stubs/setuptools/@tests/test_cases/check_distutils.py

@@ -0,0 +1,5 @@
+import distutils.command.sdist
+import distutils.config
+


Maybe we should also try to use something exclusive to setuptool's version?

For example (the first one I found):

Suggested change

from distutils.util import split_version

s = split_version("")

Whelp, it showed that checkers (at least within the context of test_cases) still prefer the stdlib if it's found. (this now only passes on 3.12).

Since those weren't in stdlib anyway, I can live with not supporting them rather than adding more hacky workarounds to the test suite. But that's up to you.

That fails mypy/pyright on Python <3.12, because both of them choose the stdlib distutils stubs over the third-party distutils stubs. That's the correct behaviour for any normal case where two packages of the same name are simultaneously provided by the stdlib and a third-party distribution; setuptools only does its "override the stdlib distutils if we're installed" magic through some importlib hacks, and unfortunately I think it's unreasonable to expect type checkers to add support for that

You could write this as if sys.version_info >= (3, 12): split_version(""). No opinion on whether it's needed though.

You could write this as if sys.version_info >= (3, 12): split_version("").

Eh, I feel like that could be more confusing than clarifying, since it is actually available on <py312 if you have setuptools installed, there's just no way of reasonably getting type checkers to understand that

Co-authored-by: Avasam <[email protected]>

This reverts commit 5663525.

Avasam

The special-casing in tests/stubtest_third_party.py isn't too wild, I can't think of a better solution, and it should resolve actual issues I've encountered in the wild, so this has my approval. (let's get at least another maintainer's approval though)

JelleZijlstra · 2024-03-16T15:42:14Z

stubs/setuptools/@tests/test_cases/check_distutils.py

@@ -0,0 +1,5 @@
+import distutils.command.sdist
+import distutils.config
+


You could write this as if sys.version_info >= (3, 12): split_version(""). No opinion on whether it's needed though.

tests/stubtest_third_party.py

AlexWaygood · 2024-03-16T15:52:32Z

Huh, seems like the stubtest issue has also been magically fixed by one of our mypy upgrades since I originally filed this PR in October? Or maybe the fact that we now run third-party stubtest on py311 rather than py310? No idea. Anyway, looks like we don't even need the stubtest hacks anymore!

github-actions · 2024-03-16T15:56:18Z

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

AlexWaygood and others added 4 commits October 29, 2023 18:43

Add distutils as a top-level package included with types-setuptools

12b5787

fix pyright

9811aab

Try to make the assorted linters happy

53eb07c

[pre-commit.ci] auto fixes from pre-commit.com hooks

1e5ee15