Skip to content

Shorten marker benchmark sampling - #22658

Closed
charliermarsh wants to merge 9 commits into
charlie/benchmark-warm-gitfrom
charlie/benchmark-marker-sampling
Closed

charliermarsh wants to merge 9 commits into
charlie/benchmark-warm-gitfrom
charlie/benchmark-marker-sampling

Conversation

@charliermarsh

@charliermarsh charliermarsh commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Reduce marker benchmark warmup from 3s to 1s and measurement time from 5s to 2s. Keep all 25 cases and 100 samples per case.

PR Change Build + run vs baseline vs previous Elapsed
#22651 Baseline 35m28s — — 40m53s
#22657 Warm Git cache fix 24m48s −10m40s −10m40s 25m47s
#22658 Marker sampling 23m38s −11m50s −1m10s 25m07s
#22659 Larger build runners 20m01s −15m27s −3m37s 21m14s
#22660 Zstd artifacts 17m25s −18m03s −2m36s 18m39s
#22656 Sharding 14m58s −20m30s −2m27s 15m51s

Rows are cumulative. Build + run sums the build job and longest wall-time job; elapsed includes queueing. These are single runs, so build and network variation affect the differences (CI run).

zanieb and others added 9 commits October 15, 2026 18:55
Benchmark marker DNF conversion, extra substitution, and conflict simplification directly, with marker construction outside the timed loop.

DNF conversion uses conflict guards with 4, 16, and 32 extras, producing 7, 121, and 497 clauses. Extra substitution uses compact diagrams with 4, 16, and 256 paths that all reduce to the same platform condition. These cases measure both clause simplification and the cost of expanding shared subgraphs.

Conflict simplification measures `UniversalMarker::imbibe` alone and followed by formatting, using 4, 16, and 32 mutually exclusive extras. Environment-only markers cover redundant conflict guards; conflict-dependent markers and a no-conflict case provide controls. Each iteration starts from the original marker, so formatting measures the cost of the guards that remain after simplification.

The benchmarks reuse the global marker interner across iterations and run in both CodSpeed modes. Related to #21954.
@codspeed

codspeed Bot commented Oct 16, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 12.68%

⚠️ 8 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 68 untouched benchmarks
🆕 50 new benchmarks
⏩ 54 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ ⚠️ Simulation discover_workspace_from_all_members 156 ms 138.4 ms +12.68%
🆕 WallTime imbibe_and_format_marker_conflict_dependent[16] N/A 18.5 µs N/A
🆕 WallTime imbibe_and_format_marker_conflict_dependent[32] N/A 64.9 µs N/A
🆕 WallTime imbibe_and_format_marker_conflict_dependent[4] N/A 2.1 µs N/A
🆕 WallTime imbibe_and_format_marker_environment_only[16] N/A 18.2 µs N/A
🆕 WallTime imbibe_and_format_marker_environment_only[32] N/A 64.5 µs N/A
🆕 WallTime imbibe_and_format_marker_environment_only[4] N/A 1.9 µs N/A
🆕 WallTime imbibe_and_format_marker_no_conflicts[0] N/A 167 ns N/A
🆕 WallTime imbibe_marker_conflict_dependent[16] N/A 133 ns N/A
🆕 WallTime imbibe_marker_conflict_dependent[32] N/A 133 ns N/A
🆕 WallTime imbibe_marker_conflict_dependent[4] N/A 130 ns N/A
🆕 WallTime imbibe_marker_environment_only[16] N/A 103 ns N/A
🆕 WallTime imbibe_marker_environment_only[32] N/A 102 ns N/A
🆕 WallTime imbibe_marker_environment_only[4] N/A 101 ns N/A
🆕 WallTime imbibe_marker_no_conflicts[0] N/A 6 ns N/A
🆕 WallTime resolve_activated_extras_independent_pairs[2] N/A 316 ns N/A
🆕 WallTime resolve_activated_extras_independent_pairs[4] N/A 645 ns N/A
🆕 WallTime resolve_activated_extras_independent_pairs[8] N/A 1.2 µs N/A
🆕 WallTime marker_to_dnf_conflicting_extras[16] N/A 10.9 µs N/A
🆕 WallTime marker_to_dnf_conflicting_extras[32] N/A 36.7 µs N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing charlie/benchmark-marker-sampling (1c66d32) with main (e32cc50)2

Open in CodSpeed

Footnotes

  1. 54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on charlie/benchmark-warm-git (9f8a992) during the generation of this report, so main (e32cc50) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@charliermarsh
charliermarsh removed this pull request from stack #22662 October 16, 2026 20:59
@charliermarsh
charliermarsh force-pushed the charlie/benchmark-warm-git branch from 9f8a992 to 26a98ed Compare October 16, 2026 20:59
charliermarsh added a commit that referenced this pull request Oct 16, 2026
We currently repopulate the warm Git cache each time Criterion invokes
the benchmark callback. Populate it once per selected benchmark and
reuse it across warmup and samples.

| PR | Change | Build + run | vs baseline | vs previous | Elapsed |
|---|---|---:|---:|---:|---:|
| [#22651](#22651) | Baseline |
35m28s | — | — | 40m53s |
| **[#22657](#22657 | **Warm Git
cache fix** | **24m48s** | **−10m40s** | **−10m40s** | **25m47s** |
| [#22658](#22658) | Marker sampling
| 23m38s | −11m50s | −1m10s | 25m07s |
| [#22659](#22659) | Larger build
runners | 20m01s | −15m27s | −3m37s | 21m14s |
| [#22660](#22660) | Zstd artifacts
| 17m25s | −18m03s | −2m36s | 18m39s |
| [#22656](#22656) | Sharding |
14m58s | −20m30s | −2m27s | 15m51s |

Rows are cumulative measurements from the original marker-based stack,
including the removed marker-sampling layer, before rebasing onto
`main`; this PR was measured at `9f8a992`. Build + run sums the build
job and longest wall-time job; elapsed includes queueing. These are
single runs, so build and network variation affect the differences ([CI
run](https://github.com/astral-sh/uv/actions/runs/38958624816)).

This branch was previously deployed

1 inactive deployment
automations — 1c66d328 Deployed Oct 16, 2026 by charliermarsh via review / security review #51433
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants