Repository navigation
Use bitsets for all DNF simplification - #22613
charliermarsh wants to merge 2 commits into
Conversation
c382f59 to
9061b45
Compare
75e6452 to
4fc1518
Compare
4fc1518 to
bd031cc
Compare
9061b45 to
5e95425
Compare
bd031cc to
6fb72f1
Compare
5e95425 to
647406c
Compare
Merging this PR will regress 8 benchmarks
|
6fb72f1 to
fa4e939
Compare
647406c to
6e34068
Compare
| if clauses.len() >= 256 && clauses.len() > negated.len().saturating_mul(2) { | ||
| return simplify_candidates(dnf, clauses, &negated, words); | ||
| let clause_count = clauses.len(); | ||
| let mut sets: SmallVec<[u64; 8]> = smallvec![0; clause_count.saturating_mul(words)]; |
There was a problem hiding this comment.
[P2] Retain a memory bound for the dense bitset index
A publisher can change the Requires-Dist markers in a previously locked direct-URL wheel. During an online uv sync --locked with missing or stale metadata, Lock::package_metadata fetches and caches that metadata; serialization invokes this simplifier before checking the wheel's locked hash. Removing the previous 32 MiB limit and linear fallback makes this allocation consume 8 × clauses × ceil(distinct_terms / 64) bytes, allowing sparse disjunctions to cause quadratic memory exhaustion. The failure threshold depends on the process memory budget. Retain a bounded dense index with a sparse or linear fallback.
Superseded by #22626.
Summary
We currently choose between direct expression comparisons and indexed comparisons when simplifying DNFs. This PR interns the terms and uses bitsets for every DNF with more than one clause, so we maintain one production implementation of the removal rules.
Small indexes stay inline; larger indexes use heap storage. We still cache clause differences when many clauses share terms, and repeated terms follow the same removal order. The original linear simplifier is compiled only for tests, where it checks the exact ordered output.
Stacked on #22606. On a quiet Linux devbox using uv's allocator, the following medians compare
main(c8e4086), the prior stack at #22606 (6e34068), the shared direct/indexed alternative in #22626 (4d72c96), and this PR. Positive percentages mean time saved relative to the prior stack:to_dnfcasemainEach value is the median of 30 samples, with the implementations in the same optimized binary and their measurement order rotated. The marker diagrams are built before timing. These measure DNF collection, simplification, and sorting; they do not establish a lock or export speedup.
Always building an index adds overhead for small expressions. Sparse expressions also use more memory: the bitsets occupy
8 × clauses × ceil(distinct_terms / 64)bytes, so storage grows quadratically when each clause introduces distinct terms. DNF creation and quadratic clause comparisons remain.CodSpeed comparison against #22606 measures the 25 marker cases in both walltime and simulation.