site.py removal in 49.0.0 breaks eggs installed outside system prefix

[ystreet]

site.py was removed in #2165 and first released with setuptools 49.0.0

This now causes installed eggs to not be loaded when installed to a prefix and pointed to by PYTHONPATH.

E.g. a concrete example of this use is with GStreamer's cerbero build system which installs the meson build system to a custom prefix to avoid messing up user/system installations. Relevant code for that is: https://gitlab.freedesktop.org/gstreamer/cerbero/-/blob/master/recipes/build-tools/meson.recipe#L30

PYTHONPATH is constructed by cerbero to point to the necessary path for the installed location (e.g. PYTHONPATH=/path/to/cerbero/build/build-tools/lib/python3.8/site-packages/). Without the site.py that is installed, this does not work anymore.

One can reproduce by doing the following:

git clone https://gitlab.freedesktop.org/gstreamer/cerbero.git
cd cerbero
./cerbero-uninstalled -c config/build-tools.cbc build meson
./cerbero-uninstalled shell
python3 -c "import sys; print(sys.path)"
# should output a value containing '/path/to/cerbero/build/build-tools/lib/python3.8/site-packages/meson-0.54.0-py3.8.egg'

Reverting to python-setuptools < 49.0.0 is one workaround.

[jaraco]

Thanks for the report.

The usage of setup.py install (which implicitly invokes easy_install) in that project is deprecated, so it's probably best to start working out a long-term fix that doesn't involve easy_install.

I recommend that the project use pip install --prefix or possibly pip install --target to install packages and supply a sitecustomize.py (example) to ensure that directory ends up as a site dir and gets .pth processing. That approach should be future-proof (at least against the sunset of easy_install).

Please let me know if that approach sounds viable (or if not, why) and how I can help.

[ystreet]

I had a go at implementing the sitecustomize.py modification you have described however it does not seem that the sitecustomize does not have any effect and is not executed at all. WIP is available from https://gitlab.freedesktop.org/ystreet/cerbero/-/commits/sitecustomize

The reproduction instructions for this version require running ./cerbero-uninstalled bootstrap instead of ./cerbero-uninstalled -c config/build-tools.cbc build meson to get the custom sitecustomize.py written to /path/to/cerbero/build/build-tools/lib/python3.8/site-packages/sitecustomize.py

./cerbero-uninstalled bootstrap # at least until things start building.  Can Ctrl-C and run the meson-specific command once things start building
./cerbero-uninstalled shell
python3 -c "import sys; print(sys.path)"
# should output a value containing '/path/to/cerbero/build/build-tools/lib/python3.8/site-packages/meson-0.54.0-py3.8.egg'

To wipe all built sources to start again:

./cerbero-uninstalled wipe --force --build-tools

For completeness, this is currently running on Arch Linux with Python 3.8.5 and python-setuptools 49.6.0

[jaraco]

Could it be a typo in sitecustomize.y (note the missing 'p')?

[ystreet]

Ah, silly me 🤦

Now, the paths are in the wrong order:

python3 -c "import sys; print(sys.path)" produces:
['', '/home/matt/Projects/cerbero/build/dist/linux_x86_64/lib/python3.8/site-packages', '/home/matt/Projects/cerbero/build/build-tools/lib/python3.8/site-packages', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/matt/.local/lib/python3.8/site-packages', '/usr/lib/python3.8/site-packages', '/home/matt/Projects/cerbero/build/build-tools/lib/python3.8/site-packages/meson-0.54.0-py3.8.egg']
with: PYTHONPATH=/home/matt/Projects/cerbero/build/dist/linux_x86_64/lib/python3.8/site-packages/:/home/matt/Projects/cerbero/build/build-tools/lib/python3.8/site-packages/

I need the meson path before all the default system paths otherwise a system install of meson is going to be used instead :)

[ystreet]

The path ordering looks like it was taken care of within the original site.py by inserting .pth entries before the system determined paths which is actually what is needed here.

[ystreet]

For now, we are going to reinstate the removed site.py until this is fixed: https://gitlab.freedesktop.org/gstreamer/cerbero/-/merge_requests/655