Sphinx 7.2.5 breaks with PyO3: "modules may only be initialized once per interpreter process"

[pbybee]

Describe the bug

New error message with this patch release. 7.2.4 succeeds in building the same documents.
PyO3 modules may only be initialized once per interpreter process

Full traceback
Traceback (most recent call last):
  File ".venv/lib/python3.10/site-packages/sphinx/cmd/build.py", line 298, in build_main
    app.build(args.force_all, args.filenames)
  File ".venv/lib/python3.10/site-packages/sphinx/application.py", line 351, in build
    self.builder.build_all()
  File ".venv/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 252, in build_all
    self.build(None, summary=__('all source files'), method='all')
  File ".venv/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 312, in build
    with logging.pending_warnings():
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File ".venv/lib/python3.10/site-packages/sphinx/util/logging.py", line 222, in pending_warnings
    memhandler.flushTo(logger)
  File ".venv/lib/python3.10/site-packages/sphinx/util/logging.py", line 187, in flushTo
    logger.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 964, in handle
    rv = self.filter(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 821, in filter
    result = f.filter(record)
  File ".venv/lib/python3.10/site-packages/sphinx/util/logging.py", line 427, in filter
    raise exc
sphinx.errors.SphinxWarning: autodoc: failed to import module 'test' from module 'test'; the following exception was raised:
PyO3 modules may only be initialized once per interpreter process

How to Reproduce

install latest sphinx and run
sphinx-build -WaE docs/project docs/project/_build/html

Environment Information

Platform:              linux; (Linux-5.4.0-149-generic-x86_64-with-glibc2.31)
Python version:        3.10.12 (main, Jun  7 2023, 12:45:35) [GCC 9.4.0])
Python implementation: CPython
Sphinx version:        7.2.5
Docutils version:      0.18.1
Jinja2 version:        3.1.2
Pygments version:      2.16.1

Sphinx extensions

myst-parser==2.0.0

Additional context

No response

[AA-Turner]

@pbybee Please provide a full reproducer. PyO3 is for rust projects, which you haven't mentioned in your description. Does the error appear with Sphinx 7.2.4?

A

[pbybee]

No 7.2.4 builds correctly. We are not using PyO3 in our project. Perhaps it's a dependancy of sphinx that revved. I'll dig a little more as I don't have good repro steps at the moment without digging through the sphinx build process.

[AA-Turner]

If you're able to bisect, try before/after this commit: 8248be3

[AA-Turner]

Perhaps it's a dependancy of sphinx that revved.

Very unlikely, more likely is that a dependency of the object you're documenting uses PyO3.

[picnixz]

Can you check if the bad module contains a TYPE_CHECKING block and if any of the imports in this block contains some PyO3 object somewhere (or depends on it)? Could you also give us the dependencies of your project?

As Adam pointed out in another issue where the feature of reloading a module with TYPE_CHECKING enabled makes jax fails, can you check the bad module with pyanalyze?

@AA-Turner I am wondering, but for projects that would fail like that, could we perhaps make the TYPE_CHECKING feature optional? (it's probably impossible to make it optional per module due to dependencies that we would then need to build in advance).

[picnixz]

Ok I know what happened.

• Assume that you have a module using PyO3 say yay and a bad module boo
• Assume you have a module a resp. b) importing yay (resp. yay and boo) in a TYPE_CHECKING block.
• Import a: success for yay, failure for boo. We remove yay from sys.modules (the commit Adam referred to).
• Import b: failure for yay even if sys.modules does not contain yay.

Reason: PyO3 seems to import modules but does not seem to "unimport" them if they are deleted from sys.modules. As such, it's probably impossible for us to do anything about that.

EDIT: I concluded as such by creating a module using PyO3, importing it, deleting it from.sys.modules, and re-importing it again and here is my configuration:

Platform:              linux; (Linux-5.10.136-android12-9-26258365-abA336BXXS7CWG1-aarch64-with-libc)
Python version:        3.11.4 (main, Jul  2 2023, 11:17:00) [Clang 14.0.7 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0)
Python implementation: CPython
Sphinx version:        7.2.4
Docutils version:      0.18.1
Jinja2 version:        3.1.2
Pygments version:      2.16.1

[adityagoel4512]

This may not apply exclusively to PyO3. We are seeing a similar issue with importing the onnx package (which is built using Pybind11).

WARNING: autodoc: failed to import module 'onnx.testing' from module <module-name>; the following exception was raised:
Traceback (most recent call last):
  File "/opt/conda/envs/env-name/lib/python3.9/site-packages/sphinx/ext/autodoc/importer.py", line 65, in import_module
    return importlib.import_module(modname)
  File "/opt/conda/envs/env-name/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

This error is not present in 7.2.4.

[picnixz]

Actually, it's probably something related to how extension modules are imported. It appears that deleting them from sys.modules is not sufficient. By the way, what is the full traceback error? you are missing the exception message (or there is none?)

[jaraco]

I've also encountered this issue in jaraco/skeleton#88 and I've produced a minimal reproducer in https://github.com/jaraco/jaraco.mongodb/tree/triage/36-minimal. Run with tox -e docs.

In this case, the offending import is import inflect, which has a dependency on pydantic, which has a rust extension.

[picnixz]

Mmh thank you. As I only have my phone for investing this, it will be hard. In the meantime, maybe the issue could be directly related to PyO3? Or, alternatively, for extension modules, maybe we could skip their removal, though they would still be leftovers modules... So I don't really know what to do (except pinning the version prior to 7.1.x. I wouldn't recommend 7.2.x as this is kind of subject to some unstability).

[jaraco]

If you're able to bisect, try before/after this commit: 8248be3

I installed the commit prior to that one and the issue remains.

Correction - after ensuring that I'm forcing the reinstall, I have confirmed that commit is implicated:

 jaraco.mongodb triage/36-minimal @ .tox/docs/bin/pip install --force --quiet 'git+https://github.com/sphinx-doc/sphinx@8248be3'
  WARNING: Did not find branch or tag 'e494baa', assuming revision or ref.
 jaraco.mongodb triage/36-minimal @ tox -e docs
.pkg: _optional_hooks> python /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_editable> python /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_editable> python /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
docs: install_package> python -I -m pip install --force-reinstall --no-deps /Users/jaraco/code/jaraco/jaraco.mongodb/.tox/.tmp/package/39/jaraco.mongodb-0.0.0-0.editable-py3-none-any.whl
docs: commands[0] /Users/jaraco/code/jaraco/jaraco.mongodb/docs> python -m sphinx -W --keep-going . /Users/jaraco/code/jaraco/jaraco.mongodb/build/html
Running Sphinx v7.2.5+/f90b3be
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
writing output... 
building [html]: targets for 1 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
reading sources... [100%] index
WARNING: autodoc: failed to import module 'oplog' from module 'jaraco.mongodb'; the following exception was raised:
PyO3 modules may only be initialized once per interpreter process
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
copying assets... copying static files... done
copying extra files... done
done
writing output... [100%] index
generating indices... genindex done
writing additional pages... search done
dumping search index in English (code: en)... done
dumping object inventory... done
build finished with problems, 1 warning.
docs: exit 1 (0.63 seconds) /Users/jaraco/code/jaraco/jaraco.mongodb/docs> python -m sphinx -W --keep-going . /Users/jaraco/code/jaraco/jaraco.mongodb/build/html pid=49155
.pkg: _exit> python /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
  docs: FAIL code 1 (1.15=setup[0.52]+cmd[0.63] seconds)
  evaluation failed :( (1.20 seconds)
 jaraco.mongodb triage/36-minimal @ .tox/docs/bin/pip install --quiet --force 'git+https://github.com/sphinx-doc/sphinx@8248be3~1'
  WARNING: Did not find branch or tag '8248be3~1', assuming revision or ref.
 jaraco.mongodb triage/36-minimal @ tox -e docs
.pkg: _optional_hooks> python /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_editable> python /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_editable> python /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
docs: install_package> python -I -m pip install --force-reinstall --no-deps /Users/jaraco/code/jaraco/jaraco.mongodb/.tox/.tmp/package/41/jaraco.mongodb-0.0.0-0.editable-py3-none-any.whl
docs: commands[0] /Users/jaraco/code/jaraco/jaraco.mongodb/docs> python -m sphinx -W --keep-going . /Users/jaraco/code/jaraco/jaraco.mongodb/build/html
Running Sphinx v7.2.5+/f90b3be
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
writing output... 
building [html]: targets for 1 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
reading sources... 
looking for now-outdated files... none found
preparing documents... done
copying assets... copying static files... done
copying extra files... done
done
writing output... [100%] index
generating indices... genindex py-modindex done
writing additional pages... search done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.

The HTML pages are in ../build/html.
.pkg: _exit> python /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
  docs: OK (1.16=setup[0.54]+cmd[0.62] seconds)
  congratulations :) (1.22 seconds)

[AA-Turner]

cc @godlygeek as the author of #11645 if you've any ideas re PyO3/extension module unloading in general.

[AA-Turner]

Another simple reproducer:

import importlib, sys
orig_modules = frozenset(sys.modules)
module = importlib.import_module('pydantic')
for m in [m for m in sys.modules if m not in orig_modules]:
    sys.modules.pop(m)
module = importlib.import_module('pydantic')
# ImportError: PyO3 modules may only be initialized once per interpreter process