Skip to content

Compress benchmark artifacts with zstd - #22660

Merged
charliermarsh merged 1 commit into
charlie/benchmark-build-runnersfrom
charlie/benchmark-zstd-artifacts
Oct 17, 2026
Merged

charliermarsh merged 1 commit into
charlie/benchmark-build-runnersfrom
charlie/benchmark-zstd-artifacts

Conversation

@charliermarsh

@charliermarsh charliermarsh commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Compress the wall-time benchmark artifact with multithreaded zstd and disable GitHub's additional compression to reduce packaging and upload time.

PR Change Build + run vs baseline vs previous Elapsed
main Baseline 20m40s — — 21m48s
#22659 Larger build runner 16m34s −4m06s −4m06s 18m20s
#22660 Zstd artifacts 15m40s −5m00s −54s 17m25s
#22656 Sharding 14m11s −6m29s −1m29s 16m15s

Measured once per revision against main (c42c495781), which includes the warm Git fix and one codegen unit. All runs restored matching caches and rebuilt the same 389 profiling dependencies. The PRs use 16 cores for the wall-time build; the baseline uses 4. Simulation uses 4 cores throughout. Setup and transfer times vary.

Build + run adds the wall-time build job and longest wall-time execution job. Elapsed includes waiting and ends when all benchmark jobs finish.

Baseline run · Run for this change (f87a942a58).

@codspeed

codspeed Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 9.71%

⚠️ 8 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
✅ 66 untouched benchmarks
⏩ 54 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ ⚠️ Simulation discover_workspace_from_all_members_with_excludes 155.3 ms 133.7 ms +16.19%
⚡ ⚠️ Simulation discover_workspace_from_all_members 147.5 ms 137.2 ms +7.5%
⚡ ⚠️ Simulation run_python_version_synthetic_workspace 546.3 ms 516.7 ms +5.72%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing charlie/benchmark-zstd-artifacts (f87a942) with charlie/benchmark-build-runners (11672ac)2

Open in CodSpeed

Footnotes

  1. 54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on charlie/benchmark-build-runners (41a001d) during the generation of this report, so 4bfa4af was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@charliermarsh
charliermarsh removed this pull request from stack #22662 October 16, 2026 20:59
@charliermarsh
charliermarsh force-pushed the charlie/benchmark-zstd-artifacts branch from a36f776 to f495c73 Compare October 16, 2026 21:02
charliermarsh added a commit that referenced this pull request Oct 16, 2026
We currently repopulate the warm Git cache each time Criterion invokes
the benchmark callback. Populate it once per selected benchmark and
reuse it across warmup and samples.

| PR | Change | Build + run | vs baseline | vs previous | Elapsed |
|---|---|---:|---:|---:|---:|
| [#22651](#22651) | Baseline |
35m28s | — | — | 40m53s |
| **[#22657](#22657 | **Warm Git
cache fix** | **24m48s** | **−10m40s** | **−10m40s** | **25m47s** |
| [#22658](#22658) | Marker sampling
| 23m38s | −11m50s | −1m10s | 25m07s |
| [#22659](#22659) | Larger build
runners | 20m01s | −15m27s | −3m37s | 21m14s |
| [#22660](#22660) | Zstd artifacts
| 17m25s | −18m03s | −2m36s | 18m39s |
| [#22656](#22656) | Sharding |
14m58s | −20m30s | −2m27s | 15m51s |

Rows are cumulative measurements from the original marker-based stack,
including the removed marker-sampling layer, before rebasing onto
`main`; this PR was measured at `9f8a992`. Build + run sums the build
job and longest wall-time job; elapsed includes queueing. These are
single runs, so build and network variation affect the differences ([CI
run](https://github.com/astral-sh/uv/actions/runs/38958624816)).
@charliermarsh
charliermarsh force-pushed the charlie/benchmark-zstd-artifacts branch from f495c73 to 3456d7b Compare October 16, 2026 21:23
@charliermarsh
charliermarsh force-pushed the charlie/benchmark-zstd-artifacts branch from 3456d7b to 13775ab Compare October 16, 2026 23:50
@charliermarsh
charliermarsh force-pushed the charlie/benchmark-zstd-artifacts branch from 13775ab to f87a942 Compare October 17, 2026 00:20
@charliermarsh
charliermarsh marked this pull request as ready for review October 17, 2026 14:57
@charliermarsh
charliermarsh merged commit a0a01a4 into main Oct 17, 2026
86 of 89 checks passed
@charliermarsh
charliermarsh deleted the charlie/benchmark-zstd-artifacts branch October 17, 2026 14:57
charliermarsh added a commit that referenced this pull request Oct 17, 2026
Move the wall-time benchmark build from a 4-core to a 16-core Depot
runner so the benchmarks can start sooner.

| PR | Change | Build + run | vs baseline | vs previous | Elapsed |
|---|---|---:|---:|---:|---:|
|
[main](c42c495)
| Baseline | 20m40s | — | — | 21m48s |
| **#22659** | **Larger build runner** | **16m34s** | **−4m06s** |
**−4m06s** | **18m20s** |
| #22660 | Zstd artifacts | 15m40s | −5m00s | −54s | 17m25s |
| #22656 | Sharding | 14m11s | −6m29s | −1m29s | 16m15s |

At [Depot's published
rates](https://depot.dev/docs/github-actions/runner-types), the
estimated cost of the two Depot benchmark jobs rises from $0.31 on
`main` to $0.53 per run: about $0.22 more (+72%). In the runs above,
build + run fell by 4m06s and total elapsed time fell by 3m28s.

This branch was previously deployed

1 inactive deployment
automations — f87a942a Deployed Oct 17, 2026 by charliermarsh via review / security review #51476
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant