Repository navigation
Deduplicate binary payloads in the wheel cache - #19694
Closed
charliermarsh wants to merge 17 commits into
Closed
charliermarsh wants to merge 17 commits into
charliermarsh wants to merge 17 commits into
Conversation
charliermarsh
force-pushed
the
charlie/dirhash-binaries
branch
from
June 5, 2026 19:25
f8c324d to
6ee9545
Compare
Merging this PR will not alter performance
Comparing Footnotes
|
charliermarsh
force-pushed
the
charlie/dirhash-binaries
branch
from
June 5, 2026 21:04
6ee9545 to
d35be39
Compare
uv test inventory changesThis PR changes the tests when compared with the
|
charliermarsh
force-pushed
the
charlie/dirhash-binaries
branch
from
June 5, 2026 21:49
d35be39 to
a958aa0
Compare
charliermarsh
force-pushed
the
charlie/dirhash-binaries
branch
from
June 7, 2026 23:32
a958aa0 to
7402c2e
Compare
charliermarsh
force-pushed
the
charlie/dirhash-binaries
branch
from
June 9, 2026 19:41
7402c2e to
f594ab8
Compare
charliermarsh
force-pushed
the
charlie/dirhash-binaries
branch
3 times, most recently
from
June 23, 2026 00:10
f19d76c to
aeb6d29
Compare
charliermarsh
force-pushed
the
charlie/dirhash-binaries
branch
from
August 5, 2026 14:53
aeb6d29 to
b1d6242
Compare
charliermarsh
temporarily deployed
to
automations
August 5, 2026 14:58 — with
GitHub Actions
Inactive
charliermarsh
force-pushed
the
charlie/dirhash-binaries
branch
from
August 5, 2026 15:33
b1d6242 to
420663c
Compare
charliermarsh
temporarily deployed
to
automations
August 5, 2026 15:42 — with
GitHub Actions
Inactive
charliermarsh
temporarily deployed
to
automations
August 25, 2026 22:41 — with
GitHub Actions
Inactive
charliermarsh
temporarily deployed
to
automations
August 25, 2026 22:59 — with
GitHub Actions
Inactive
charliermarsh
temporarily deployed
to
automations
August 25, 2026 23:18 — with
GitHub Actions
Inactive
charliermarsh
force-pushed
the
charlie/dirhash-binaries
branch
from
August 27, 2026 18:33
19b4880 to
a3f977e
Compare
charliermarsh
had a problem deploying
to
automations
August 27, 2026 18:33 — with
GitHub Actions
Failure
charliermarsh
force-pushed
the
charlie/dirhash-binaries
branch
from
August 28, 2026 00:08
a3f977e to
1f28fd5
Compare
charliermarsh
temporarily deployed
to
automations
August 28, 2026 00:09 — with
GitHub Actions
Inactive
charliermarsh
added a commit
that referenced
this pull request
Aug 31, 2026
## Summary On main, we support content-addressed caching, but only at the wheel-level. That is, if you download the same wheel twice from different sources, they share a cache entry. But files _within_ or _across_ wheels are not deduplicated at all. This PR adds deduplication at the file level: every file is now stored under its BLAKE3 hash in a `files-v0` bucket. We hardlink these objects into their original locations in `archive-v0`, so the installation step doesn't change at all -- we're just deduping _within_ the cache (and cache cleanup removes file objects when their hardlink count drops to one). In the prior proposal (#19694), we included the following table: | File selection | Additional savings | Files hardlinked | Distinct `files-v0` objects | | --- | ---: | ---: | ---: | | Executables and native libraries | 275.7 MiB | 3,336 | 3,088 | | Any payload file ≥ 10 MiB | 235.0 MiB | 80 | 78 | | Any payload file ≥ 1 MiB | 279.7 MiB | 373 | 349 | | Any payload file ≥ 100 KiB | 353.4 MiB | 2,830 | 2,458 | | Any payload file ≥ 10 KiB | 475.7 MiB | 23,767 | 18,722 | | Any payload file ≥ 1 KiB | 537.4 MiB | 95,156 | 66,422 | | All payload files | 545.2 MiB | 134,222 | 87,129 | So we're saving 545.2 MiB on my local machine, or about 10% of the cache. In return, the net effect seems to be something like a <4% slowdown for cold installs (and no effect on warm installs), which I think is _probably_ worthwhile here. --------- Co-authored-by: Charlie Marsh <charliemarsh@openai.com>
Member
|
Closing as superseded by #21327, which has been merged. |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On main, we support content-addressed caching, but only at the wheel-level. That is, if you download the same wheel twice from different sources, they share a cache entry. But files within or across wheels are not deduplicated at all.
This PR adds deduplication for select files -- executables and native libraries (
.so, versioned.so.*,.dylib,.dll, and.pyd), which are stored in afiles-v0bucket keyed by BLAKE3 content digest and executable status. We hardlink these objects into their original locations inarchive-v0, so the installation step doesn't change at all -- we're just deduping within the cache (and cache cleanup removes file objects when their hardlink count drops to one).Cache analysis
Looking at the cache on my own machine, I have 5.54 GiB of file contents across 923 wheel archives.
The whole-wheel deduplication removes 30 duplicate archives and saves 121.1 MiB, leaving 5.42 GiB.
This table compares the additional savings from this PR with some alternative deduplication policies:
files-v0objects.dist-infoPerformance comparison
I also compared this PR with a local prototype that deduplicates every file, including
.dist-info. These are median times to prepare and install each workload into a fresh environment with an empty uv cache:So hardlinkg every file from the cache would cause a fairly significant slowdown in cold-install time.