gh-93939: Build C extensions without setup.py

[tiran]

Combines GH-93940, GH-94452, and GH-94433

• Issue: Build shared extensions without distutils #93939

Automerge-Triggered-By: GH:tiran

[merwok]

Is there a need for force pushes? Github notifications lead to nothing, which is a little annoying.

[pradyunsg]

FWIW, You can click on "force-pushed" in the event about the force push, to see what changed between the two force-pushes.

Submitting feedback to GitHub to have the notifications for force-pushed go to that, instead of "oh, I can't show you what changed" may help make GitHub prioritze fixing that higher.

[tiran]

I only force push to rebase. It is the easiest and fastest way for me to resolve merge conflicts. The PR has frequent conflicts, even right now.

[tiran]

[tiran]

setup.py's check_extension_import feature is now provided by a script. Here is an example output with tk-dev missing and a modified _sqlite that is missing _sqlite/util.c dependency.

[ERROR] _sqlite3 failed to import: build/lib.linux-x86_64-3.12/_sqlite3.cpython-312-x86_64-linux-gnu.so: undefined symbol: _pysqlite_seterror

The necessary bits to build these optional modules were not found:
_tkinter                                                       
To find the necessary bits, look in configure.ac and config.log.

Following modules built successfully but were removed because they could not be imported:
_sqlite3                                                       

Checked 109 modules (30 built-in, 76 shared, 1 n/a on linux-x86_64, 0 disabled, 1 missing, 1 failed on import)

validate_tzpath is performed by configure.ac

xxsubtype is now an optional extension. It is skipped when Python is built without test C extension modules.

[zware]

One minor naming suggestion, otherwise LGTM.

[nascheme]

We could leave the makefile targets oldsharedmods and oldsharedinstall as aliases but I suspect it's better just to remove them (as is done in the PR). If someone depends on building those targets, they can adapt. The removal of setup.py is probably more disruptive anyhow.

Is it a mistake that Tools/ssl/multissltests.py tries to make oldsharedmods? Should be sharedmods maybe.

I was worried that this would add a lot of complexity to configure.ac. I think that was the main motivation to using setup.py in the first place (or was it better for cross-platform builds too?). This PR doesn't really add any complexity to configure.ac. So, removing setup.py is a good idea I think.

[tiran]

Is it a mistake that Tools/ssl/multissltests.py tries to make oldsharedmods? Should be sharedmods maybe.

Yes, that's a bug. Thanks for spotting it!

I was worried that this would add a lot of complexity to configure.ac. I think that was the main motivation to using setup.py in the first place (or was it better for cross-platform builds too?). This PR doesn't really add any complexity to configure.ac. So, removing setup.py is a good idea I think.

Well, we kind of cheated here. GH-90005 added complexity to configure.ac. @erlend-aasland and I improved configure.ac slowly to detect all C extension dependencies. The approach has benefit that we now use pkg-config whenever possible and cross-compiling works out of the box.

[nascheme]

Kind of cheating but that code is already in configure.ac (i.e. a sunk cost). This change just removes about 1400 lines of code that's not needed anymore. Unless someone can think of good reason that setup.py needs to be kept, we should remove that code, IMHO.

$ git diff 0c66074e9f8c9728e1d920910d35da0c62f30403 --stat
 Lib/_bootsubprocess.py                             |   97 --
 Lib/test/test_descr.py                             |    9 +
 Makefile.pre.in                                    |   69 +-
 .../2022-06-28-09-42-10.gh-issue-93939._VWxKW.rst  |    2 +
 Modules/Setup                                      |    2 +-
 Modules/Setup.stdlib.in                            |    1 +
 Python/stdlib_module_names.h                       |    1 -
 Tools/scripts/check_extension_modules.py           |   11 +-
 Tools/scripts/generate_stdlib_module_names.py      |   11 -
 Tools/ssl/multissltests.py                         |    4 +-
 configure                                          |   69 +-
 configure.ac                                       |   21 +-
 setup.py                                           | 1294 --------------------
 13 files changed, 84 insertions(+), 1507 deletions(-)

[miss-islington]

Sorry, I can't merge this PR. Reason: Base branch was modified. Review and try the merge again..

[tiran]

#94848 addresses the build issue with LTO + pydebug.

[vstinner]

My little precious setup.py... YOU WILL NEVER BE FORGOTTEN 😭😭😭

_bootsubprocess.py (that I wrote)... I hate you, I'm glad that you're dead! If I recall correctly, I had to write _bootsubprocess to use the clean subprocess module, rather than naive fork+exec code, in distutils. distutils... I also hate you by the way.

[ayappanec]

This broke the AIX build with the below error.

./python -E -S -m sysconfig --generate-posix-vars ;
if test $? -ne 0 ; then
echo "generate-posix-vars failed" ;
rm -f ./pybuilddir.txt ;
exit 1 ;
fi

Traceback (most recent call last):
File "/home/buildusr/cpython_master/cpython/Lib/_aix_support.py", line 7, in
import subprocess
File "/home/buildusr/cpython_master/cpython/Lib/subprocess.py", line 104, in
from _posixsubprocess import fork_exec as _fork_exec
ModuleNotFoundError: No module named '_posixsubprocess'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/buildusr/cpython_master/cpython/Lib/sysconfig.py", line 851, in
_main()
File "/home/buildusr/cpython_master/cpython/Lib/sysconfig.py", line 839, in _main
_generate_posix_vars()
File "/home/buildusr/cpython_master/cpython/Lib/sysconfig.py", line 511, in _generate_posix_vars
pybuilddir = f'build/lib.{get_platform()}-{_PY_VERSION_SHORT}'
^^^^^^^^^^^^^^
File "/home/buildusr/cpython_master/cpython/Lib/sysconfig.py", line 781, in get_platform
from _aix_support import aix_platform
File "/home/buildusr/cpython_master/cpython/Lib/_aix_support.py", line 11, in
import _bootsubprocess as subprocess
ModuleNotFoundError: No module named '_bootsubprocess'
generate-posix-vars failed
gmake: *** [Makefile:871: pybuilddir.txt] Error 1
gmake: *** Waiting for unfinished jobs....

[tiran]

Please open a new bug for the problem.

[hroncok]

I found a regression, reported as #98707

[haampie]

Thanks for doing this!