[RFC] [python] 'setup.py install' is deprecated

[jameslamb]

Description

Running python setup.py install for the Python package raises the following deprecation warning:

/Users/jlamb/mambaforge/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(

I found a very detailed write-up explaining this issue: "Why you shouldn't invoke setup.py directly" (October 2021).

Here are some of the main things we need to know:

...as of the last few years all direct invocations of setup.py are effectively deprecated in favor of invocations via purpose-built and/or standards-based CLI tools like pip, build and tox.

As mentioned in this issue, direct setup.py invocations have effectively been unmaintained for several years now. Whenever someone raises an issue with a reproducer that involves invoking setup.py, the maintainers ask for a reproducer that doesn't hit this code path, and if one can't be found the issue is closed. Put another way, direct invocations of setup.py that currently work for you do so essentially by chance — if something breaks, you are on your own.

Reproducible example

I observed this warning building from source on macOS 12.2.1, with Python v3.9.7 and setuptools==59.8.0.

git clone --recursive https://github.com/microsoft/LightGBM.git
cd LightGBM/python-package

This warning is raised with python setup.py install.

python setup.py install

It is also raised with pip.

pip install -v .

You can also see this warning on LightGBM's CI builds that use python setup.py {command}. For example, the most recent Linux regular job running on Azure DevOps (build link).

References

This warning was added in setuptools=v58.3.0 (October 22, 2021), pypa/setuptools#2824.

That PR mentions https://www.python.org/dev/peps/pep-0517/ as a description of the standard packages should meet to continue to be compliant with the main Python package management tools.

The warning comes from setuptools.command.install.initialize_options(): https://github.com/pypa/setuptools/blob/391bb5d4d09c9eb8d6b2b98968e623455ae0a384/setuptools/command/install.py#L32-L38.

lightgbm currently sub-classes setuptools.command.install to customize the behavior of setup.py install.

LightGBM/python-package/setup.py

Lines 220 to 225 in f6d654b

	class CustomInstall(install):
	user_options = install.user_options + LIGHTGBM_OPTIONS
	def initialize_options(self) -> None:
	install.initialize_options(self)

It also does that to customize setup.py {install_lib, bdist_wheel, sdist}.

LightGBM/python-package/setup.py

Lines 361 to 366 in f6d654b

	cmdclass={
	'install': CustomInstall,
	'install_lib': CustomInstallLib,
	'bdist_wheel': CustomBdistWheel,
	'sdist': CustomSdist,
	},

• "pyproject.toml quickstart" (setuptools docs)
• "Configuring setuptools using setup.cfg files" (setuptools docs)

Why does this matter?

If nothing is changed, the LightGBM Python package might eventually not be installable with some future versions of Python / setuptools.

What should be done?

I don't have a recommendation yet.

https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html, https://www.python.org/dev/peps/pep-0517/, and the other things they link to are very detailed and I'm not deeply familiar with the details of how setuptools works.

Just opening this up to track discussion about this issue.

[StrikerRUS]

Oh, it's a rabbit hole!
The most suffered projects are ones that use Cython and numpy to build their source code. I learned that during working on scikit-learn-contrib/lightning#188.

Linking some more related discussions:

• how to handle numpy.distutils and setuptools interaction pypa/setuptools#2372
• RFC Migrate away from numpy.distutils before CPython 3.12 (October 2023) scikit-learn/scikit-learn#21499
• RFC: switch to Meson as a build system scipy/scipy#13615
• https://iscinumpy.gitlab.io/post/scikit-build-proposal/

TL;DR: the future of Python building system is extremely foggy.

[jameslamb]

Thanks for those links @StrikerRUS !

I think scikit-build is really compelling, since it supports CMake. I'll add two more relevant projects here:

• https://pybind11.readthedocs.io/en/stable/
• https://cibuildwheel.readthedocs.io/en/stable/

[StrikerRUS]

Recently released scipy 1.9.0 is the first version built with Meson: http://scipy.github.io/devdocs/release.1.9.0.html#scipy-switched-to-meson-as-its-build-system.

[edzul]

Hi,

I tried with pip as the warning suggests as follows:

Instead of

python setup.py install --precompile

I ran, in the LightGBM/python-package folder

python -m pip install --install-option=--precompile --user .

The --user param is probably not necessary in all cases, but on the machine I'm working is required since site-packages folder is at a specific location specified in the USER_SITE variable. You can try without it, but if you run pip list and don't see lightgbm, try adding the --user param.

And with this there is no warning and works fine.

[ifredom]

it's work for me

pip install setuptools==58.2.0

[jameslamb]

Thank you, but "just pin to a 1+-year-old version of setuptools" is not an acceptable constraint for LightGBM to put on its users.

[jameslamb]

python setup.py install now raises a stricter, slightly more worrying deprecation warning.

DEPRECATION: lightgbm is being installed using the legacy 'setup.py install' method, because the '--no-binary' option was enabled for it and this currently disables local wheel building for projects that don't have a 'pyproject.toml' file. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at pypa/pip#11451

pip is currently on 22.3.1 (release history), so I'm not sure when the 23.x series will start.

But I think this adds new urgency to the project to switch lightgbm over to a pyproject.toml-based build.

If no one else picks it up, I'll try working on that after resolving #5514 .