Skip to content

Deduplicate binary payloads in the wheel cache - #19694

Closed
charliermarsh wants to merge 17 commits into
mainfrom
charlie/dirhash-binaries
Closed

charliermarsh wants to merge 17 commits into
mainfrom
charlie/dirhash-binaries

Conversation

@charliermarsh

@charliermarsh charliermarsh commented Jun 5, 2026 •

Copy link
Copy Markdown
Member

Summary

On main, we support content-addressed caching, but only at the wheel-level. That is, if you download the same wheel twice from different sources, they share a cache entry. But files within or across wheels are not deduplicated at all.

This PR adds deduplication for select files -- executables and native libraries (.so, versioned .so.*, .dylib, .dll, and .pyd), which are stored in a files-v0 bucket keyed by BLAKE3 content digest and executable status. We hardlink these objects into their original locations in archive-v0, so the installation step doesn't change at all -- we're just deduping within the cache (and cache cleanup removes file objects when their hardlink count drops to one).

Cache analysis

Looking at the cache on my own machine, I have 5.54 GiB of file contents across 923 wheel archives.

The whole-wheel deduplication removes 30 duplicate archives and saves 121.1 MiB, leaving 5.42 GiB.

This table compares the additional savings from this PR with some alternative deduplication policies:

File selection Additional savings Files hardlinked Distinct files-v0 objects
Executables and native libraries (this PR) 275.7 MiB 3,336 3,088
Any payload file ≥ 10 MiB 235.0 MiB 80 78
Any payload file ≥ 1 MiB 279.7 MiB 373 349
Any payload file ≥ 100 KiB 353.4 MiB 2,830 2,458
Any payload file ≥ 10 KiB 475.7 MiB 23,767 18,722
Any payload file ≥ 1 KiB 537.4 MiB 95,156 66,422
All payload files 545.2 MiB 134,222 87,129
Every file, including .dist-info 548.3 MiB 138,945 90,300

Performance comparison

I also compared this PR with a local prototype that deduplicates every file, including .dist-info. These are median times to prepare and install each workload into a fresh environment with an empty uv cache:

Workload, including dependencies Current PR Every file
SymPy 0.23 s 0.58 s
NumPy/Pandas/SciPy 0.62 s 1.45 s
PyTorch 1.59 s 5.46 s

So hardlinkg every file from the cache would cause a fairly significant slowdown in cold-install time.

@charliermarsh
charliermarsh force-pushed the charlie/dirhash-binaries branch from f8c324d to 6ee9545 Compare June 5, 2026 19:25
@codspeed

codspeed Bot commented Jun 5, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 25 untouched benchmarks
⏩ 12 skipped benchmarks1


Comparing charlie/dirhash-binaries (1f28fd5) with main (3b5853d)

Open in CodSpeed

Footnotes

  1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@charliermarsh
charliermarsh force-pushed the charlie/dirhash-binaries branch from 6ee9545 to d35be39 Compare June 5, 2026 21:04
@astral-sh-bot

astral-sh-bot Bot commented Jun 5, 2026 •

Copy link
Copy Markdown

uv test inventory changes

This PR changes the tests when compared with the main base revision.

  • Added tests: 5
  • Removed tests: 0
  • Changed suites: 2
uv-extract: +1 / -0

Added:

  • uv-extract::dirhash::archive::tests::extracted_file_executable_status

Removed: none

uv::pip_install: +4 / -0

Added:

  • uv::pip_install::pip_install::binary_payload_copy_fallback_uses_archive_file_store
  • uv::pip_install::pip_install::binary_payload_selection
  • uv::pip_install::pip_install::binary_payloads_stay_in_archive_without_preview
  • uv::pip_install::pip_install::binary_payloads_use_archive_file_store

Removed: none

@charliermarsh
charliermarsh force-pushed the charlie/dirhash-binaries branch from d35be39 to a958aa0 Compare June 5, 2026 21:49
@charliermarsh
charliermarsh force-pushed the charlie/dirhash-binaries branch from a958aa0 to 7402c2e Compare June 7, 2026 23:32
@charliermarsh
charliermarsh force-pushed the charlie/dirhash-binaries branch from 7402c2e to f594ab8 Compare June 9, 2026 19:41
@charliermarsh
charliermarsh force-pushed the charlie/dirhash-binaries branch 3 times, most recently from f19d76c to aeb6d29 Compare June 23, 2026 00:10
@charliermarsh
charliermarsh force-pushed the charlie/dirhash-binaries branch from aeb6d29 to b1d6242 Compare August 5, 2026 14:53
@charliermarsh
charliermarsh force-pushed the charlie/dirhash-binaries branch from b1d6242 to 420663c Compare August 5, 2026 15:33
@charliermarsh charliermarsh changed the title Deduplicate binary payloads with archive manifests Deduplicate binary payloads in the wheel cache Aug 27, 2026
@charliermarsh
charliermarsh force-pushed the charlie/dirhash-binaries branch from 19b4880 to a3f977e Compare August 27, 2026 18:33
@charliermarsh
charliermarsh force-pushed the charlie/dirhash-binaries branch from a3f977e to 1f28fd5 Compare August 28, 2026 00:08
charliermarsh added a commit that referenced this pull request Aug 31, 2026
## Summary

On main, we support content-addressed caching, but only at the
wheel-level. That is, if you download the same wheel twice from
different sources, they share a cache entry. But files _within_ or
_across_ wheels are not deduplicated at all.

This PR adds deduplication at the file level: every file is now stored
under its BLAKE3 hash in a `files-v0` bucket. We hardlink these objects
into their original locations in `archive-v0`, so the installation step
doesn't change at all -- we're just deduping _within_ the cache (and
cache cleanup removes file objects when their hardlink count drops to
one).

In the prior proposal (#19694), we
included the following table:

| File selection | Additional savings | Files hardlinked | Distinct
`files-v0` objects |
| --- | ---: | ---: | ---: |
| Executables and native libraries | 275.7 MiB | 3,336 | 3,088 |
| Any payload file ≥ 10 MiB | 235.0 MiB | 80 | 78 |
| Any payload file ≥ 1 MiB | 279.7 MiB | 373 | 349 |
| Any payload file ≥ 100 KiB | 353.4 MiB | 2,830 | 2,458 |
| Any payload file ≥ 10 KiB | 475.7 MiB | 23,767 | 18,722 |
| Any payload file ≥ 1 KiB | 537.4 MiB | 95,156 | 66,422 |
| All payload files | 545.2 MiB | 134,222 | 87,129 |

So we're saving 545.2 MiB on my local machine, or about 10% of the
cache.

In return, the net effect seems to be something like a <4% slowdown for
cold installs (and no effect on warm installs), which I think is
_probably_ worthwhile here.

---------

Co-authored-by: Charlie Marsh <charliemarsh@openai.com>

zsol commented Oct 13, 2026

Copy link
Copy Markdown
Member

Closing as superseded by #21327, which has been merged.

@zsol zsol closed this Oct 13, 2026

This branch was previously deployed

1 inactive deployment
automations — 1f28fd5f Deployed Aug 28, 2026 by charliermarsh via review / security review #45399
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants