Improve inlining of scope latch counters #1057
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `increment` and `set` methods now have `#[inline]` hints, and the `counter` fields in `CountLatch` and `CountLockLatch` are now listed first to increase the chance that layout puts them at the same offset. (That layout is not critical to ensure, but works out nicely.)
|
bors r+ |
|
Build succeeded! The publicly hosted instance of bors-ng is deprecated and will go away soon. If you want to self-host your own instance, instructions are here. If you want to switch to GitHub's built-in merge queue, visit their help page.
|
This was referenced Jun 20, 2023
cuviper
added a commit
to cuviper/rayon
that referenced
this pull request
Jun 21, 2023
The former `enum ScopeLatch` forced a `match` during both `increment` and `set` (decrement), even though both variants only need to update an `AtomicUsize` most of the time. rayon-rs#1057 helped hide that for `increment`, but `set` branching still showed up in perf profiles. Now this is refactored to a unified `CountLatch` that has a direct field for its `counter` used in the frequent case, and then an internal enum for the one-time notification variants. Therefore, most of its updates will have no `match` reached at all. The only other use of the former `CountLatch` was the one-shot termination latch in `WorkerThread`, so that's now renamed to `OnceLatch`.
bors bot
added a commit
that referenced
this pull request
Jun 22, 2023
1059: Refactor scope latches to reduce matching r=cuviper a=cuviper The former `enum ScopeLatch` forced a `match` during both `increment` and `set` (decrement), even though both variants only need to update an `AtomicUsize` most of the time. #1057 helped hide that for `increment`, but `set` branching still showed up in perf profiles. Now this is refactored to a unified `CountLatch` that has a direct field for its `counter` used in the frequent case, and then an internal enum for the one-time notification variants. Therefore, most of its updates will have no `match` reached at all. The only other use of the former `CountLatch` was the one-shot termination latch in `WorkerThread`, so that's now renamed to `OnceLatch`. Co-authored-by: Josh Stone <[email protected]>
cuviper
added a commit
to cuviper/rayon
that referenced
this pull request
Apr 15, 2025
The former `enum ScopeLatch` forced a `match` during both `increment` and `set` (decrement), even though both variants only need to update an `AtomicUsize` most of the time. rayon-rs#1057 helped hide that for `increment`, but `set` branching still showed up in perf profiles. Now this is refactored to a unified `CountLatch` that has a direct field for its `counter` used in the frequent case, and then an internal enum for the one-time notification variants. Therefore, most of its updates will have no `match` reached at all. The only other use of the former `CountLatch` was the one-shot termination latch in `WorkerThread`, so that's now renamed to `OnceLatch`. (cherry picked from commit 32d3774)
cuviper
added a commit
to cuviper/rayon
that referenced
this pull request
Apr 15, 2025
The former `enum ScopeLatch` forced a `match` during both `increment` and `set` (decrement), even though both variants only need to update an `AtomicUsize` most of the time. rayon-rs#1057 helped hide that for `increment`, but `set` branching still showed up in perf profiles. Now this is refactored to a unified `CountLatch` that has a direct field for its `counter` used in the frequent case, and then an internal enum for the one-time notification variants. Therefore, most of its updates will have no `match` reached at all. The only other use of the former `CountLatch` was the one-shot termination latch in `WorkerThread`, so that's now renamed to `OnceLatch`. (cherry picked from commit 32d3774)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
incrementandsetmethods now have#[inline]hints, and thecounterfields inCountLatchandCountLockLatchare now listedfirst to increase the chance that layout puts them at the same offset.
(That layout is not critical to ensure, but works out nicely.)