Implement a make jobserver: #9210
Draft
+190
−8
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
Going to work on the failures. |
Edouard-chin
force-pushed
the
ec-jobserver-auth
branch
from
4a495d1 to
be7bafe
Compare
Edouard-chin
added a commit
to Shopify/rubygems
that referenced
this pull request
Dec 19, 2025
- While working on ruby#9210, I found those leaky tests which break the test suite. They remove Bundler constants without setting them back. The reason it's not a problem on master is because there are no tests that install gems in the same process (all tests shell out to `bundle install`).
- Fix ruby#9170 - In ruby#9131, we added running `make` in parallel with the `-j` flag. It's problematic because since Bundler installs gems in parallel, we could end up installing `N gem x M processors` which would exhaust the machine resources and causes issues like ruby#9170. In this patch, I have implemented a make [jobserver](https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html), so that each thread installing a gem can take available job slots from the pool. I also want to highlight that with this patch, we can still end up running 2 more jobs than what's available. The reason being that if a gem is waiting for slots to be available, then the whole `bundle install` takes much longer which defeats the whole purpose of the `make -j` optimization. So if there is no more slots available, we compile the extension without parallelization (we still use one cpu though). Example ---------- If `bundle install -j 6` is running, then we have 6 slots available. Each gem with native extension that gets installed may take up to *3* slots (see my reasoning on this number below). In the event where 3 gems with native extensions get installed, 2 gems will consume all 6 slots, leaving the third gem without any. Ultimately, this means that we could use *at maximum 2 more slots than what's available* (e.g. `bundle install -j 3` for 3 gems will use 5 slots). I chose `3` slots per `make` process because after installing multiple gems, I didn't see any benefit to adding more jobs. It's possible that some gems have many `make` recipes that could run more than 3 in parallel, but I don't think this is very frequent.
Edouard-chin
force-pushed
the
ec-jobserver-auth
branch
from
be7bafe to
e6fb3de
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What was the end-user or developer problem that led to this PR?
Fix #9170
What is your fix for the problem, implemented in this PR?
Problem
In #9131, we added running
makein parallel with the-jflag.It's problematic because since Bundler installs gems in parallel, we could end up installing
N gem x M processorswhich would exhaust the machine resources and causes issues like #9170.Solution
In this patch, I have implemented a make jobserver, so that each thread installing a gem can take available job slots from the pool.
Important
I also want to highlight that with this patch, we can still end up running more jobs than what's available. The reason being that if a gem is waiting for slots to be available, then the whole
bundle installtakes much longer which defeats the whole purpose of themake -joptimization. So if there is no more slots available, we compile the extension without parallelization (we still use one cpu though).Example
If
bundle install -j 6is running, then we have 6 slots available. Each gem with native extension that gets installed may take up to 3 slots (see my reasoning on this number below).In the event where 3 gems with native extensions get installed, 2 gems will consume all 6 slots, leaving the third gem without any.
I chose
3slots permakeprocess because after installing multiple gems, I didn't see any benefit to adding more jobs. It's possible that some gems have manymakerecipes that could run more than 3 in parallel, but I don't think this is very frequent.Make sure the following tasks are checked