Skip to content

Conversation

wiredfool
Copy link
Member

@wiredfool wiredfool commented May 31, 2025

Fixes #8971 .

Changes proposed in this pull request:

  • Use the ParallelCompile from pybind11 to compile the c extension in parallel.

With:

$ make clean && time make install
...
real	0m9.226s
user	0m33.522s
sys	0m3.639s

$ make clean && time make debug
...
real	0m4.724s
user	0m15.480s
sys	0m2.941s

On main:

$ make clean && time make install
...
real	0m21.224s
user	0m20.892s
sys	0m2.613s

$ make clean && time make debug
...
real	0m10.243s
user	0m9.450s
sys	0m2.123s

setup.py Outdated
from setuptools import Extension, setup
from setuptools.command.build_ext import build_ext

ParallelCompile("MAX_CONCURRENCY").install()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ParallelCompile("MAX_CONCURRENCY").install()
ParallelCompile("MAX_CONCURRENCY", default=4).install()

Actually, from our documentation, this should be 4.

https://pillow.readthedocs.io/en/stable/installation/building-from-source.html#build-options

Config setting: -C parallel=n. Can also be given with environment variable: MAX_CONCURRENCY=n. Pillow can use multiprocessing to build the extension. Setting -C parallel=n sets the number of CPUs to use to n, or can disable parallel building by using a setting of 1. By default, it uses 4 CPUs, or if 4 are not available, as many as are present.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've created wiredfool#12 to use -C parallel=n as well.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From pybind11 docs:

You can also pass default=N to set the default number of threads (0 will take the number of threads available) and max=N, the maximum number of threads; if you have a large extension you may want set this to a memory dependent number.

If not set, pybind11 uses 0.

Pillow's limit of 4 was added in 2014 (#1043) but machines have more CPUs today.

Last I looked, the lowest amount on a macOS on the market was 8, and Linux laptops/desktops usually have at least around this many. Servers and shared systems may be lower.

Shall we leave the default to 0 (and update the docs)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I've pushed a commit.

@hugovk hugovk added the Build label Jun 25, 2025
Copy link
Member

@hugovk hugovk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice speedup.

One concern is pybind11 might not support new Python versions until after we want to start testing.

For example, pybind11 added 3.14 support after beta 1 in May (pybind/pybind11#5646), but we'd already started testing after alpha 3 in January (#8690).

But we can address this when it comes up next time: if pybind11 doesn't support 3.15 when we want to test it, we can make it conditional or temporarily remove it.

setup.py Outdated
from setuptools import Extension, setup
from setuptools.command.build_ext import build_ext

ParallelCompile("MAX_CONCURRENCY").install()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From pybind11 docs:

You can also pass default=N to set the default number of threads (0 will take the number of threads available) and max=N, the maximum number of threads; if you have a large extension you may want set this to a memory dependent number.

If not set, pybind11 uses 0.

Pillow's limit of 4 was added in 2014 (#1043) but machines have more CPUs today.

Last I looked, the lowest amount on a macOS on the market was 8, and Linux laptops/desktops usually have at least around this many. Servers and shared systems may be lower.

Shall we leave the default to 0 (and update the docs)?

@wiredfool wiredfool merged commit d560320 into python-pillow:main Jul 15, 2025
71 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parallel build is no longer parallel
3 participants