Missing importlib re-exported top-level modules

[Avasam]

>>> import importlib
>>> dir(importlib)
['_RELOADING', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__import__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_bootstrap', '_bootstrap_external', '_imp', '_pack_uint32', '_unpack_uint32', 'abc', 'find_loader', 'import_module', 'invalidate_caches', 'machinery', 'reload', 'sys', 'types', 'util', 'warnings']
>>> importlib.abc
<module 'importlib.abc' from 'C:\\Program Files\\Python39\\lib\\importlib\\abc.py'>
>>> importlib.machinery
<module 'importlib.machinery' from 'C:\\Program Files\\Python39\\lib\\importlib\\machinery.py'>
>>> importlib.util
<module 'importlib.util' from 'C:\\Program Files\\Python39\\lib\\importlib\\util.py'>

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

kornia (https://github.com/kornia/kornia)
- kornia/testing/__init__.py:20: error: Module has no attribute "util"  [attr-defined]

[hauntsaninja]

This definitely isn't always the case:

λ python3.9
Python 3.9.17 (main, Jun 15 2023, 08:01:12) 
[Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> dir(importlib)
['_RELOADING', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__import__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_bootstrap', '_bootstrap_external', '_imp', '_pack_uint32', '_unpack_uint32', 'find_loader', 'import_module', 'invalidate_caches', 'reload', 'sys', 'types', 'warnings']
>>> importlib.abc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'abc'
>>> importlib.machinery
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'machinery'
>>> importlib.util
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'

Maybe you have something in site or some PEP 660 editable installs or something that are importing stuff?

[AlexWaygood]

I concur with @hauntsaninja, with the caveat that importlib.machinery specifically does always seem to be set as an attribute on importlib on py310+. Not sure that that's something users should be relying on, though.

Python 3.9.12 (main, Apr  4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.machinery
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'machinery'

Python 3.10.8 | packaged by conda-forge | (main, Nov 24 2022, 14:07:00) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.machinery
<module 'importlib.machinery' from 'C:\\Users\\alexw\\.conda\\envs\\py310\\lib\\importlib\\machinery.py'>

Python 3.13.0a0 (heads/main:d4cea794a7, Sep 21 2023, 13:24:35) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.machinery
<module 'importlib.machinery' (frozen)>

[Avasam]

Hum, I came across this in pywin32 https://github.com/mhammond/pywin32/blob/main/win32/Lib/win32serviceutil.py#L20
Seeing that, I tested directly in my console on python 3.9.13 (since that wasn't mentioned). Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32
Guess now I gotta try to figure out why or what made those 3 work for me. (So I can at least fix it there and in libs raised by primer, and explain why it shouldn't be relied on, if that's the case)

Edit: ran pip freeze then uninstalled all editable installs, same results

[JelleZijlstra]

I also see importlib.util available on a "clean" shell. I used the advanced debugging method of making importlib.util raise an exception on import and it seems like it's from .pth files:

Error processing line 1 of /home/jelle/ans/venv3.9/lib/python3.9/site-packages/google_api_core-1.32.0-py3.10-nspkg.pth:

  Traceback (most recent call last):
    File "/home/jelle/ans/venv3.9/lib/python3.9/site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "/home/jelle/ans/venv3.9/lib/python3.9/importlib/util.py", line 13, in <module>
      fail
  NameError: name 'fail' is not defined

Remainder of file ignored

[Avasam]

Smart. So I had protobuff, zope.event, zope.interface, ruamel.yaml and flake8_helper. Once all uninstalled the modules are no longer accessible with only import importlib.
Dunno if leaving a comment in-stubs is worth. If not, you can close this.
Either way I can reference this conversation when making the change in sources using a bare import importlib.

util

Error processing line 1 of C:\Program Files\Python39\lib\site-packages\protobuf-3.20.1-py3.9-nspkg.pth:

  Traceback (most recent call last):
    File "C:\Program Files\Python39\lib\site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "C:\Program Files\Python39\lib\importlib\util.py", line 20, in <module>
      raise "found it"
  TypeError: exceptions must derive from BaseException

Remainder of file ignored
Error processing line 1 of C:\Program Files\Python39\lib\site-packages\ruamel.yaml-0.17.21-py3.9-nspkg.pth:

  Traceback (most recent call last):
    File "C:\Program Files\Python39\lib\site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "C:\Program Files\Python39\lib\importlib\util.py", line 20, in <module>
      raise "found it"
  TypeError: exceptions must derive from BaseException

Remainder of file ignored
Error processing line 1 of C:\Program Files\Python39\lib\site-packages\zope.event-4.5.0-py3.6-nspkg.pth:

  Traceback (most recent call last):
    File "C:\Program Files\Python39\lib\site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "C:\Program Files\Python39\lib\importlib\util.py", line 20, in <module>
      raise "found it"

Remainder of file ignored
Error processing line 1 of C:\Program Files\Python39\lib\site-packages\zope.interface-5.4.0-py3.9-nspkg.pth:

  Traceback (most recent call last):
    File "C:\Program Files\Python39\lib\site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "C:\Program Files\Python39\lib\importlib\util.py", line 20, in <module>
      raise "found it"
  TypeError: exceptions must derive from BaseException

Remainder of file ignored
Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

machinery

``` Error processing line 3 of C:\Program Files\Python39\lib\site-packages\flake8_helper.pth:

Traceback (most recent call last):
File "C:\Program Files\Python39\lib\site.py", line 169, in addpackage
exec(line)
File "", line 1, in
File "C:\Program Files\Python39\lib\doctest.py", line 97, in
import inspect
File "C:\Program Files\Python39\lib\inspect.py", line 39, in
import importlib.machinery
File "C:\Program Files\Python39\lib\importlib\machinery.py", line 2, in
raise "found it"
TypeError: exceptions must derive from BaseException

Remainder of file ignored
Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

</details>

[JelleZijlstra]

Seems like we shouldn't make this change.