-
-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Add parallel compile from pybind11 #8990
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
setup.py
Outdated
from setuptools import Extension, setup | ||
from setuptools.command.build_ext import build_ext | ||
|
||
ParallelCompile("MAX_CONCURRENCY").install() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ParallelCompile("MAX_CONCURRENCY").install() | |
ParallelCompile("MAX_CONCURRENCY", default=4).install() |
Actually, from our documentation, this should be 4.
https://pillow.readthedocs.io/en/stable/installation/building-from-source.html#build-options
Config setting: -C parallel=n. Can also be given with environment variable: MAX_CONCURRENCY=n. Pillow can use multiprocessing to build the extension. Setting -C parallel=n sets the number of CPUs to use to n, or can disable parallel building by using a setting of 1. By default, it uses 4 CPUs, or if 4 are not available, as many as are present.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've created wiredfool#12 to use -C parallel=n
as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From pybind11 docs:
You can also pass
default=N
to set the default number of threads (0 will take the number of threads available) andmax=N
, the maximum number of threads; if you have a large extension you may want set this to a memory dependent number.
If not set, pybind11 uses 0.
Pillow's limit of 4 was added in 2014 (#1043) but machines have more CPUs today.
Last I looked, the lowest amount on a macOS on the market was 8, and Linux laptops/desktops usually have at least around this many. Servers and shared systems may be lower.
Shall we leave the default to 0 (and update the docs)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I've pushed a commit.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice speedup.
One concern is pybind11 might not support new Python versions until after we want to start testing.
For example, pybind11 added 3.14 support after beta 1 in May (pybind/pybind11#5646), but we'd already started testing after alpha 3 in January (#8690).
But we can address this when it comes up next time: if pybind11 doesn't support 3.15 when we want to test it, we can make it conditional or temporarily remove it.
setup.py
Outdated
from setuptools import Extension, setup | ||
from setuptools.command.build_ext import build_ext | ||
|
||
ParallelCompile("MAX_CONCURRENCY").install() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From pybind11 docs:
You can also pass
default=N
to set the default number of threads (0 will take the number of threads available) andmax=N
, the maximum number of threads; if you have a large extension you may want set this to a memory dependent number.
If not set, pybind11 uses 0.
Pillow's limit of 4 was added in 2014 (#1043) but machines have more CPUs today.
Last I looked, the lowest amount on a macOS on the market was 8, and Linux laptops/desktops usually have at least around this many. Servers and shared systems may be lower.
Shall we leave the default to 0 (and update the docs)?
Fixes #8971 .
Changes proposed in this pull request:
With:
On main: