Skip to content

Index large marker DNF expressions - #2107

Draft
zaniebot wants to merge 1 commit into
zb/allocator-api-owned-dnf-scratchfrom
zb/allocator-api-dnf-index
Draft

zaniebot wants to merge 1 commit into
zb/allocator-api-owned-dnf-scratchfrom
zb/allocator-api-dnf-index

Conversation

@zaniebot

@zaniebot zaniebot commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Large conflict markers can make uv lock and uv export spend most of their time repeatedly comparing marker expressions during DNF simplification, as in astral-sh#21954. Intern the expressions and use bit sets for subset checks, with the expression map, clause indexes, and bit sets allocated through the standard Allocator API. Keep the sequential simplification order and use the linear implementation for fewer than eight DNF clauses, terms repeated within a clause, or a dense index larger than 32 MiB. Stacked on #2094.

Hyperfine 2.0.0 measurements of 567ffbbc1 → c0d8a3aaf used release builds of the complete CLI with rustc 1.100.0-nightly (6eeff9a52 2026-09-23) on an AMD EPYC-Genoa Linux VM, pinned to four CPUs. Output went to /dev/null. The supplied interpreter was CPython 3.12.13.

Commands include uv workspace metadata --script SCRIPT --frozen --python PYTHON --preview-features workspace-metadata, uv export --frozen, and uv lock --no-index --find-links WHEELS. Synthetic frozen script locks supply markers with 1, 7, 8, 9, 32, or 128 Boolean OR terms, plus a categorical export control. The input term count does not by itself establish the number of DNF clauses seen by the simplifier. Synthetic local-wheel workspaces provide fresh uv lock cases with 30 or 50 mutually exclusive extras. Root dependencies activate different extras below and from Python 3.13; the extras request alternating versions of one shared package name, alongside one common dependency conditional on Linux. Each command starts a fresh process; OS pages, local wheel metadata, and interpreter-query caches are warm. Fresh-lock cases remove uv.lock outside wall time and RSS. Frozen exports and metadata commands read existing synthetic script locks. No external index, source backend, or package installation is used.

Values are means, parent → head. Changes are (head / parent - 1) with 95% confidence intervals; negative values indicate a reduction.

Workload Wall time Change, 95% CI Peak RSS Change, 95% CI
metadata / 64 edges / Boolean OR terms / 1 per marker / distinct markers 16.1 → 16.1 ms -0.43% [-2.08%, +1.11%] 25.20 → 25.12 MiB -0.30% [-0.79%, +0.27%]
metadata / 64 edges / Boolean OR terms / 7 per marker / distinct markers 20.4 → 20.6 ms +1.19% [+0.31%, +2.12%] 27.46 → 27.46 MiB -0.01% [-0.22%, +0.19%]
metadata / 64 edges / Boolean OR terms / 8 per marker / distinct markers 20.8 → 20.8 ms -0.01% [-1.46%, +1.36%] 27.70 → 27.68 MiB -0.10% [-0.35%, +0.16%]
metadata / 64 edges / Boolean OR terms / 9 per marker / distinct markers 21.0 → 21.1 ms +0.49% [-0.85%, +2.04%] 28.18 → 28.18 MiB +0.01% [-0.44%, +0.43%]
metadata / 64 edges / Boolean OR terms / 32 per marker / distinct markers 109.2 → 87.0 ms -20.35% [-21.02%, -19.65%] 71.82 → 71.61 MiB -0.29% [-0.44%, -0.16%]
metadata / 16 edges / Boolean OR terms / 128 per marker / distinct markers 1,589.3 → 316.9 ms -80.06% [-80.24%, -79.87%] 213.39 → 213.25 MiB -0.07% [-0.11%, -0.03%]
export / 64 edges / categorical values / 32 per marker / distinct markers 53.0 → 50.6 ms -4.49% [-5.56%, -3.19%] 40.83 → 40.90 MiB +0.16% [-0.06%, +0.39%]
lock / 30 extras / 1 shared name / 1 common dependency / conflicts / conditional activation 1,759.0 → 103.2 ms -94.14% [-94.22%, -94.06%] 52.39 → 52.53 MiB +0.28% [+0.01%, +0.56%]
lock / 50 extras / 1 shared name / 1 common dependency / conflicts / conditional activation 31,415.0 → 576.4 ms -98.17% [-98.19%, -98.14%] 96.43 → 96.61 MiB +0.19% [-0.0003%, +0.39%]

Each workload used ten comparison blocks with alternating revision order, one warmup followed by three measured runs per revision in each block (30 measurements per revision). The pointwise 95% percentile-bootstrap intervals use 20,000 resamples of paired block means and retain all samples. They estimate within-run uncertainty, not variation across machines, independent builds, or separate benchmark campaigns.

The 30- and 50-extra locks ran 94.14% and 98.17% faster. The 32- and 128-term Boolean metadata cases ran 20.35% and 80.06% faster, and the categorical export ran 4.49% faster. The 7-term metadata case ran 1.19% slower; the 1-, 8-, and 9-term runtime intervals include zero. Peak RSS fell by 0.29% and 0.07% for the 32- and 128-term cases and rose by 0.28% for the 30-extra lock. The other six RSS intervals include zero, including the 50-extra lock interval with a lower endpoint of -0.0003%. The full source comparison combines indexed simplification with arena storage; these whole-command results do not isolate the allocator contribution. RSS is Hyperfine's Linux wait4 peak, not an allocation count or summed simultaneous process-tree memory.

Earlier measurements

Earlier benchmark study:

On macOS arm64 with Rust 1.100.0-nightly and uv's optimized profiling profile, an offline 50-extra lock reproduction dropped from 5.100 s to 96.1 ms elapsed (98.12%) and from 5.079 s to 83.1 ms CPU (98.36%; paired 95% interval 98.33–98.40%) across 11 alternating pairs against the study's immediate parent. The fixture uses two local versions of a shared wheel and a Python-version split. The complete indexed algorithm accounts for this gain: changing only its storage from &Global to the arena reduced CPU by 0.12% across 51 pairs (95% interval −0.88–1.14%), so an independent allocator speedup is not established.

For the checked-in project and lockfile at MTEB 674c65d, the immediate parent's offline uv export --frozen --no-dev --no-emit-project --no-hashes --no-header exceeded a 60-second limit. The indexed arena version averaged 11.13 s elapsed across 11 paired runs; the same algorithm with &Global averaged 10.91 s. The timeout is a lower bound. These figures are consistent with indexed simplification providing the improvement; the allocator-only comparison does not establish an independent speedup.

@codspeed

codspeed Bot commented Sep 24, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 46 untouched benchmarks


Comparing zb/allocator-api-dnf-index (c0d8a3a) with zb/allocator-api-owned-dnf-scratch (567ffbb)

Open in CodSpeed

@zaniebot zaniebot added risk:3 Credible risk of meaningful incorrect behavior or regression; bounded or recoverable. priority:3 Normal: useful features, documentation, usability, or routine efficiency improvements. size:3 Substantial: interacting behaviors, difficult compatibility questions, or a careful migration. labels Sep 24, 2026
@zaniebot zaniebot added the build:skip-release Disable building release binaries for a pull request label Oct 7, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build:skip-release Disable building release binaries for a pull request performance priority:3 Normal: useful features, documentation, usability, or routine efficiency improvements. risk:3 Credible risk of meaningful incorrect behavior or regression; bounded or recoverable. size:3 Substantial: interacting behaviors, difficult compatibility questions, or a careful migration.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants