Can't repair wheel archive on manylinux_2_28

[Mizux]

For google/or-tools we are migrating from manylinux2014 to quay.io/pypa/manylinux_2_28_x86_64 to generate our python native library (framework)

Here the wheel generated inside the docker image against the /opt/python/cp312-cp312/bin/python inside a virtualenv.
ortools-9.12.4478-cp312-cp312-linux_x86_64.whl.zip (ed to remove .zip extension needed to upload on github)

General Layout of the wheel package:

* ortools/ # root module containing each component
  * .libs/
     *  libortools.so # our main C++ solver everything in a big fat dynamic library #  RUNPATH $ORIGIN
     * libxxx.so # various third party needed by libortools.so (e.g abseil-cpp, protobuf, etc..)
  * graph/ # module
     * python/ # pybind11 wrapper
        *   /max_flow.cpython-312-x86_64-linux-gnu.so # RUNPATH $ORIGIN:$ORIGIN/../../../ortools/.libs
  * algorithms/ # an other module etc...

note: take a look at the ortools/graph/python/max_flow.cpython-312-x86_64-linux-gnu.so
and objdump -p ...

Protocol

note: I'm using a python 3.12 venv on a x86_64 linux

$ python --version
Python 3.12.8
$ pip show auditwheel
Name: auditwheel
Version: 6.2.0
Summary: Cross-distribution Linux wheels
Home-page: https://github.com/pypa/auditwheel
Author: Robert T. McGibbon
Author-email: rmcgibbo@gmail.com
License: MIT
Location: /tmp/py12/lib/python3.12/site-packages
Requires: packaging, pyelftools

$ auditwheel -v repair --plat "manylinux_2_28_x86_64" ortools-9.12.4478-cp312-cp312-linux_x86_64.whl

Expected

A ortools-9.12.4478-cp312-cp312-manylinux_2_28_x86_64.whl generated file

Observed

auditwheel failed...

...
DEBUG:auditwheel.policy:Package requires GLIBC_2.26, incompatible with policy manylinux_2_24_x86_64 which requires {'GLIBC_2.3.3', 'GLIBC_2.7', 'GLIBC_2.13', 'GLIBC_2.8', 'GLIBC_2.11', 'GLIBC_2.4', 'GLIBC_2.2.6', 'GLIBC_2.24', 'GLIBC_2.3', 'GLIBC_2.15', 'GLIBC_2.9', 'GLIBC_2.2.5', 'GLIBC_2.3.4', 'GLIBC_2.23', 'GLIBC_2.17', 'GLIBC_2.12', 'GLIBC_2.10', 'GLIBC_2.5', 'GLIBC_2.6', 'GLIBC_2.22', 'GLIBC_2.14', 'GLIBC_2.3.2', 'GLIBC_2.16', 'GLIBC_2.18'}
DEBUG:auditwheel.policy:Package requires GLIBC_2.25, incompatible with policy manylinux_2_24_x86_64 which requires {'GLIBC_2.3.3', 'GLIBC_2.7', 'GLIBC_2.13', 'GLIBC_2.8', 'GLIBC_2.11', 'GLIBC_2.4', 'GLIBC_2.2.6', 'GLIBC_2.24', 'GLIBC_2.3', 'GLIBC_2.15', 'GLIBC_2.9', 'GLIBC_2.2.5', 'GLIBC_2.3.4', 'GLIBC_2.23', 'GLIBC_2.17', 'GLIBC_2.12', 'GLIBC_2.10', 'GLIBC_2.5', 'GLIBC_2.6', 'GLIBC_2.22', 'GLIBC_2.14', 'GLIBC_2.3.2', 'GLIBC_2.16', 'GLIBC_2.18'}
DEBUG:auditwheel.policy:Package requires GCC_7.0.0, incompatible with policy manylinux_2_24_x86_64 which requires {'GCC_3.4.2', 'GCC_4.7.0', 'GCC_3.3.1', 'GCC_4.3.0', 'GCC_3.4', 'GCC_4.8.0', 'GCC_3.3', 'GCC_4.0.0', 'GCC_4.2.0', 'GCC_3.4.4', 'GCC_3.0'}
DEBUG:auditwheel.policy:Package requires CXXABI_1.3.11, incompatible with policy manylinux_2_24_x86_64 which requires {'CXXABI_1.3.10', 'CXXABI_1.3.6', 'CXXABI_FLOAT128', 'CXXABI_1.3.8', 'CXXABI_1.3.3', 'CXXABI_1.3.1', 'CXXABI_1.3.4', 'CXXABI_TM_1', 'CXXABI_1.3', 'CXXABI_1.3.9', 'CXXABI_1.3.7', 'CXXABI_1.3.2', 'CXXABI_1.3.5'}
Traceback (most recent call last):
  File "/tmp/py12/bin/auditwheel", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/tmp/py12/lib/python3.12/site-packages/auditwheel/main.py", line 54, in main
    return args.func(args, p)
           ^^^^^^^^^^^^^^^^^^
  File "/tmp/py12/lib/python3.12/site-packages/auditwheel/main_repair.py", line 178, in execute
    out_wheel = repair_wheel(
                ^^^^^^^^^^^^^
  File "/tmp/py12/lib/python3.12/site-packages/auditwheel/repair.py", line 82, in repair_wheel
    raise ValueError(msg)
ValueError: Cannot repair wheel, because required library "libabsl_spinlock_wait.so.2407.0.0" could not be located

note: Using the same "workflow" but with quay.io/pypa/musllinux_1_2_x86_64:latest works as expected i.e.
auditwheel repair it as ortools-9.12.4466-cp39-cp39-musllinux_1_2_x86_64.whl

note: On my way to perform some test, hack my local auditwheel install to add some logs etc...

[Mizux]

dev note:
The "zip" wheel above has been generated using the repo https://github.com/google/or-tools, branch mizux/rc,
patch to disable the auditwheel step and get the wheel:

git diff
diff --git a/tools/docker/python/build-manylinux.sh b/tools/docker/python/build-manylinux.sh
index 6c31c30998..fe804231ee 100755
--- a/tools/docker/python/build-manylinux.sh
+++ b/tools/docker/python/build-manylinux.sh
@@ -149,7 +149,7 @@ function check_wheel() {
     # if no files found do nothing
     [[ -e "$FILE" ]] || continue
     python -m auditwheel show "$FILE" || true
-    python -m auditwheel -v repair --plat "manylinux_2_28_$PLATFORM" "$FILE" -w "$export_root"
+    #python -m auditwheel -v repair --plat "manylinux_2_28_$PLATFORM" "$FILE" -w "$export_root"
     #python -m auditwheel -v repair --plat manylinux_2_28_x86_64 "$FILE" -w "$export_root"
     #python -m auditwheel -v repair --plat manylinux_2_28_aarch64 "$FILE" -w "$export_root"
   done

and command:

git clone -b mizux/rc git@github.com:google/or-tools.git && cd or-tools
cd tools/docker
make python_amd64_manylinux_cp312_export

note: I have a Makefile to orchestrate all the docker commands
note2: to get a container you can use

# devel stage is before the build/auditwheel which fail
make sh_python_amd64_manylinux_cp312_devel
[inside the docker]: ./build-manylinux.sh build

Dockerfile in python/amd64/manylinux.Dockerfile

[Mizux]

More test using Archlinux...

TLDR: archlinux auditwheel pkg is broken, using auditwheel 6.2.0 (venv) works.

1. Current Archlinux extra/python-auditwheel 6.1.0-1 pkg is broken since they try to use the system package extra/python-wheel 0.45.0-3 which doesn't contains the pkginfo contrary to the vendored wheel 36.2
   ref: https://github.com/pypa/auditwheel/tree/main/src/auditwheel/_vendor/wheel

$ auditwheel repair --plat "manylinux_2_27_x86_64" ortools-9.12.4478-cp312-cp312-linux_x86_64.whl
Traceback (most recent call last):
  File "/usr/bin/auditwheel", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/usr/lib/python3.13/site-packages/auditwheel/main.py", line 54, in main
    rval = args.func(args, p)
  File "/usr/lib/python3.13/site-packages/auditwheel/main_repair.py", line 109, in execute
    from .repair import repair_wheel
  File "/usr/lib/python3.13/site-packages/auditwheel/repair.py", line 21, in <module>
    from .wheel_abi import get_wheel_elfdata
  File "/usr/lib/python3.13/site-packages/auditwheel/wheel_abi.py", line 20, in <module>
    from .genericpkgctx import InGenericPkgCtx
  File "/usr/lib/python3.13/site-packages/auditwheel/genericpkgctx.py", line 4, in <module>
    from .wheeltools import InWheelCtx
  File "/usr/lib/python3.13/site-packages/auditwheel/wheeltools.py", line 26, in <module>
    from wheel.pkginfo import read_pkg_info, write_pkg_info
ModuleNotFoundError: No module named 'wheel.pkginfo'

note: should be fixed by https://gitlab.archlinux.org/archlinux/packaging/packages/python-auditwheel/-/merge_requests/1

2. So I try to use a python3.13 venv with a "clean" auditwheel

$ python -m venv /tmp/venv
$ source /tmp/venv/bin/activate
$ pip install auditwheel
$ pip show auditwheel | grep Version
Version: 6.2.0
$ python -m auditwheel show .../ortools-9.12.4478-cp312-cp312-linux_x86_64.whl
$ python -m auditwheel repair --plat "manylinux_2_28_x86_64" .../ortools-9.12.4478-cp312-cp312-linux_x86_64.whl -w .
...
Fixed-up wheel written to ./ortools-9.12.4478-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl

EDIT: on my archlinux I've already installed the libabsl package so auditwheel could found them in /usr/lib/ too...

-> Now need to know why is not working insed the manylinux_2_28 image...

[Mizux]

Found a dirty workaround, since I ALSO have the library in the BUILD_DIR (before they are copied to the wheel by our CMake workflows) I've used:

LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${ROOT_DIR}/${BUILD_DIR}/lib64" python -m auditwheel repair --plat "manylinux_2_28_$PLATFORM" "$FILE" -w .

But still since the libraries are in the wheel and RUN_PATH seems legit, I still think there is a bug in the current auditwheel implem -> Will try to create a MRE to dig this issue...

[mayeut]

Thanks for digging into this.
Indeed if all libraries are in the wheel and RUN_PATH are correct, this seems like a bug in auditwheel.
An MRE would be much appreciated.