`auditwheel` misses hashing name of `/lib64/libexpat.so`

[nilsnolde]

In a fairly big C++ project with Python bindings, which we want to "fix" with auditwheel, the final wheel's _valhalla.xxx.so tries to find libexpat.so.1 instead of the mangled/hashed libexpat.so name I'd expect auditwheel to produce, so that

ldd wheelhouse/pyvalhalla_git-3.5.1.post191-cp313-cp313-manylinux_2_28_x86_64/_valhalla.cpython-313-x86_64-linux-gnu.so | grep expat

produces libexpat.so.1 => /lib64/libexpat.so.1 (0x000075a8523c4000), while a similar dependency libhdf5_hl-0b60eabd.so.100.1.2 produces the patched binary name properly and finds it in pyvalhalla.libs. It's very puzzling to me, nothing seems to be off, /lib64/libexpat.so.1 exists as a symlink and libexpat is a direct dependency of libgdal (which is a direct dependency to our bindings), and so is libhdf5 which is properly resolved and also installed system-wide at /lib64/.

Then I ran auditwheel -vvv to get DEBUG output and to me it seems inconsistent, potentially hinting at a bug? The full log of building, packaging & auditing (in DEBUG stdout) the wheel is attached to this issue. In short:

The dict in DEBUG:auditwheel.wheel_abi:full_elftree: says (in dot json notation)

    _valhalla.cpython-313-x86_64-linux-gnu.so.libraries.libexpat.so.1 = {
                "soname": "libexpat.so.1",
                "path": "/lib64/libexpat.so.1",
                ...
   }

and it's in a bunch of needed section of other libraries keys. What seems odd is that it doesn't show up in _valhalla.cpython-313-x86_64-linux-gnu.manylinux_2_28_x86_64.libs, which is the manylinux variant we're using. However, it does show up in e.g. the manylinux_2_5_x86_64.libs key as "libexpat.so.1": "/lib64/libexpat.so.1.6.7".

As a side note, in that project we also distribute some native C++ CLI executables, which auditwheel thankfully re-writes as well. Though I realized it's actually breaking one executable so that it segfaults when executed. Also ldd doesn't show anything. I didn't dig any further yet, but it must be smth in auditwheel that's breaking the binary (ELF header?). See below if you're interested in the compilation process or want to download the executable for further inspection (located in valhalla/bin/valhalla_service in the PyPI wheel). If I ever dig into it and manage to find smth I'll shout out of course.

EDIT: So my actual question/request is: do you have any idea what might cause that behavior? Or a clue how to debug further myself?

How to reproduce

I realize this is quite an effort to fully reproduce building from source with the whole building chain. But such seems Python packaging for C++-extended projects with lots of dependencies.. I also have no idea if it's even necessary, so if you just would like to look at the latest wheel, which shows all the problems I'm referring to in this issue, can be found on Test PyPI.

I'm currently working on this in a PR on this branch, so the repro can unfortunately only fully be done with that.

It takes me around 3-4 mins of processing to run the below (not having to download the pretty heavy Docker image):

git clone --recurse-submodules https://github.com/valhalla/valhalla && cd valhalla
git checkout nn-python-packaging  # branch which contains the commit

# this downloads and runs the manylinux image where we installed things like libprotobuf etc
# https://github.com/valhalla/manylinux: manylinux fork
# image URL: https://github.com/valhalla/manylinux/pkgs/container/manylinux
docker run -dt -v $PWD:/valhalla-py --name valhalla-py --workdir /valhalla-py ghcr.io/valhalla/manylinux:2_28_valhalla_python

# this script builds the C++ component, libvalhalla, into the manylinux image
# NOTE, first you might want to patch "./src/bindings/python/build_linux.sh" to add "-vvv" to auditwheel command
docker exec -t valhalla-py /valhalla-py/src/bindings/python/build_linux.sh build_manylinux 3.13

# once the wheel is built and audited, install it in a fresh docker container
docker run -dt --name python -v $PWD/wheelhouse:/wheelhouse python:slim-bullseye
docker exec -it python bash

# inside the container, install the local audited wheel
python -m pip install /wheelhouse/*

# execute some commands
# NOTE: currently failing with:
#  File "/usr/local/lib/python3.13/site-packages/valhalla/__init__.py", line 6, in <module>
#     from _valhalla import *
# ImportError: libexpat.so.1: cannot open shared object file: No such file or directory
python -m valhallla --help
python -m valhalla valhalla_build_tiles -h

Attachments

• build_python_log.txt: full log of building, packaging & auditing (in DEBUG stdout) the wheel
• _valhalla.cpython-313-x86_64-linux-gnu.zip: the bindings .so from ./dist, before auditwheel processing

[nilsnolde]

Sorry for this wall of text. I tried to capture all info I found. Now I'm shying away from working myself into auditwheel and hope you have some suggestions on how to debug further:)

[nilsnolde]

And thanks a LOT for auditing every ELF file. I was worried I have to do the dirty work myself when distributing C++ executables on top of the shared library.

[mayeut]

libexpat.so.1 is whitelisted as it is required & thus available globally by some python distribution (debian & ubuntu).
see #152
It could have been either blacklisted or whitelisted, it was whitelisted meaning that the system one should be installed.

[nilsnolde]

Yep, thanks, @xnox already helped me out with that in valhalla/valhalla#5292 (comment).

Eventually I came up with this snippet in the pyproject.toml:

# we prepend a few commands to patch the auditwheel source a bit
# main challenge is to find the same python version auditwheel will use for repairing
[[tool.cibuildwheel.overrides]]
    select = "*-manylinux_*"
    inherit.repair-wheel-command = "prepend"
    repair-wheel-command = [
        'link_target="$(readlink -f $(which "auditwheel"))"',
        'python_bin="$(dirname "$link_target")/python"',
        'echo "[INFO] Using $($python_bin -V) from $(which "$python_bin")"',
        '$python_bin src/bindings/python/scripts/auditwheel_patch.py libexpat.so.1 libz.so.1',
    ]

where auditwheel_patch.py is a small script (source) removing the libraries names from the manylinux-policy.json lib_whitelist list, which I'd like to have vendored.

Is that behavior documented somewhere? I'm baffled that this doesn't happen much more frequently. It's probably pretty common that C++-extended projects are repaired with auditwheel which have somewhere a libexpat dependency. I guess it's just not very common that libexpat is not installed on a Linux system. I used a python:slim-bullseye image with python3.13, which doesn't link lots of common Python-linked libraries, so expat is missing from the system.