Repository navigation
Compress benchmark artifacts with zstd - #22660
charliermarsh merged 1 commit into
Conversation
Merging this PR will improve performance by 9.71%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ |
Simulation | discover_workspace_from_all_members_with_excludes |
155.3 ms | 133.7 ms | +16.19% |
| ⚡ |
Simulation | discover_workspace_from_all_members |
147.5 ms | 137.2 ms | +7.5% |
| ⚡ |
Simulation | run_python_version_synthetic_workspace |
546.3 ms | 516.7 ms | +5.72% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing charlie/benchmark-zstd-artifacts (f87a942) with charlie/benchmark-build-runners (11672ac)2
Footnotes
-
54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
charlie/benchmark-build-runners(41a001d) during the generation of this report, so 4bfa4af was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
a36f776 to
f495c73
Compare
We currently repopulate the warm Git cache each time Criterion invokes the benchmark callback. Populate it once per selected benchmark and reuse it across warmup and samples. | PR | Change | Build + run | vs baseline | vs previous | Elapsed | |---|---|---:|---:|---:|---:| | [#22651](#22651) | Baseline | 35m28s | — | — | 40m53s | | **[#22657](#22657 | **Warm Git cache fix** | **24m48s** | **−10m40s** | **−10m40s** | **25m47s** | | [#22658](#22658) | Marker sampling | 23m38s | −11m50s | −1m10s | 25m07s | | [#22659](#22659) | Larger build runners | 20m01s | −15m27s | −3m37s | 21m14s | | [#22660](#22660) | Zstd artifacts | 17m25s | −18m03s | −2m36s | 18m39s | | [#22656](#22656) | Sharding | 14m58s | −20m30s | −2m27s | 15m51s | Rows are cumulative measurements from the original marker-based stack, including the removed marker-sampling layer, before rebasing onto `main`; this PR was measured at `9f8a992`. Build + run sums the build job and longest wall-time job; elapsed includes queueing. These are single runs, so build and network variation affect the differences ([CI run](https://github.com/astral-sh/uv/actions/runs/38958624816)).
f495c73 to
3456d7b
Compare
3456d7b to
13775ab
Compare
13775ab to
f87a942
Compare
Move the wall-time benchmark build from a 4-core to a 16-core Depot runner so the benchmarks can start sooner. | PR | Change | Build + run | vs baseline | vs previous | Elapsed | |---|---|---:|---:|---:|---:| | [main](c42c495) | Baseline | 20m40s | — | — | 21m48s | | **#22659** | **Larger build runner** | **16m34s** | **−4m06s** | **−4m06s** | **18m20s** | | #22660 | Zstd artifacts | 15m40s | −5m00s | −54s | 17m25s | | #22656 | Sharding | 14m11s | −6m29s | −1m29s | 16m15s | At [Depot's published rates](https://depot.dev/docs/github-actions/runner-types), the estimated cost of the two Depot benchmark jobs rises from $0.31 on `main` to $0.53 per run: about $0.22 more (+72%). In the runs above, build + run fell by 4m06s and total elapsed time fell by 3m28s.
Compress the wall-time benchmark artifact with multithreaded zstd and disable GitHub's additional compression to reduce packaging and upload time.
Measured once per revision against
main(c42c495781), which includes the warm Git fix and one codegen unit. All runs restored matching caches and rebuilt the same 389 profiling dependencies. The PRs use 16 cores for the wall-time build; the baseline uses 4. Simulation uses 4 cores throughout. Setup and transfer times vary.Build + run adds the wall-time build job and longest wall-time execution job. Elapsed includes waiting and ends when all benchmark jobs finish.
Baseline run · Run for this change (
f87a942a58).