Skip to content

Use bitsets for all DNF simplification - #22613

Closed
charliermarsh wants to merge 2 commits into
charlie/marker-dnf-candidatesfrom
charlie/marker-dnf-interned
Closed

charliermarsh wants to merge 2 commits into
charlie/marker-dnf-candidatesfrom
charlie/marker-dnf-interned

Conversation

@charliermarsh

@charliermarsh charliermarsh commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Superseded by #22626.

Summary

We currently choose between direct expression comparisons and indexed comparisons when simplifying DNFs. This PR interns the terms and uses bitsets for every DNF with more than one clause, so we maintain one production implementation of the removal rules.

Small indexes stay inline; larger indexes use heap storage. We still cache clause differences when many clauses share terms, and repeated terms follow the same removal order. The original linear simplifier is compiled only for tests, where it checks the exact ordered output.

Stacked on #22606. On a quiet Linux devbox using uv's allocator, the following medians compare main (c8e4086), the prior stack at #22606 (6e34068), the shared direct/indexed alternative in #22626 (4d72c96), and this PR. Positive percentages mean time saved relative to the prior stack:

to_dnf case main Prior stack #22626 This PR Time saved vs. prior stack
Two clauses 0.898 µs 0.885 µs 0.858 µs 1.122 µs −26.8%
Mixed clauses 1.555 µs 1.552 µs 1.600 µs 1.832 µs −18.0%
4 conflicting extras 5.036 µs 5.020 µs 5.120 µs 3.844 µs +23.4%
16 conflicting extras 7.988 ms 0.362 ms 0.283 ms 0.321 ms +11.3%
32 conflicting extras 488.432 ms 4.234 ms 4.023 ms 4.118 ms +2.7%
512 sparse clauses 14.165 ms 8.739 ms 8.659 ms 10.022 ms −14.7%

Each value is the median of 30 samples, with the implementations in the same optimized binary and their measurement order rotated. The marker diagrams are built before timing. These measure DNF collection, simplification, and sorting; they do not establish a lock or export speedup.

Always building an index adds overhead for small expressions. Sparse expressions also use more memory: the bitsets occupy 8 × clauses × ceil(distinct_terms / 64) bytes, so storage grows quadratically when each clause introduces distinct terms. DNF creation and quadratic clause comparisons remain.

CodSpeed comparison against #22606 measures the 25 marker cases in both walltime and simulation.

@codspeed

codspeed Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 8 benchmarks

⚠️ 8 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 10 improved benchmarks
❌ 8 regressed benchmarks
✅ 101 untouched benchmarks
⏩ 104 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ WallTime marker_to_dnf_two_clauses 476 ns 584 ns -18.49%
❌ Simulation marker_to_dnf_two_clauses 34.4 µs 42 µs -18.16%
❌ Simulation marker_to_dnf_platform_specific_python_versions 43.5 µs 51.6 µs -15.63%
❌ Simulation resolve_activated_extras_independent_pairs[2] 23.3 µs 27.4 µs -15.06%
❌ WallTime marker_to_dnf_platform_specific_python_versions 903 ns 1,021 ns -11.56%
❌ WallTime marker_to_dnf_sparse_clauses[512] 6.7 ms 7.2 ms -7.04%
❌ ⚠️ Simulation discover_workspace_from_all_members 132.2 ms 141.9 ms -6.81%
❌ Simulation imbibe_and_format_marker_conflict_dependent[4] 111.8 µs 118.6 µs -5.68%
⚡ Simulation imbibe_and_format_marker_conflict_dependent[16] 11 ms 7.9 ms +38.32%
⚡ WallTime imbibe_and_format_marker_conflict_dependent[16] 590.5 µs 457 µs +29.2%
⚡ Simulation marker_to_dnf_conflicting_extras[16] 3.2 ms 2.5 ms +28.56%
⚡ WallTime marker_to_dnf_conflicting_extras[4] 2.3 µs 1.9 µs +26.57%
⚡ WallTime marker_to_dnf_conflicting_extras[16] 175.9 µs 142.4 µs +23.49%
⚡ Simulation imbibe_marker_no_conflicts[0] 398.5 ns 343 ns +16.2%
⚡ Simulation marker_to_dnf_conflicting_extras[4] 74.3 µs 67.7 µs +9.79%
⚡ WallTime marker_to_dnf_conflicting_extras[32] 1.9 ms 1.8 ms +6.94%
⚡ WallTime imbibe_and_format_marker_conflict_dependent[32] 5.8 ms 5.4 ms +6.1%
⚡ Simulation marker_to_dnf_sparse_clauses[128] 5.8 ms 5.5 ms +5.13%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing charlie/marker-dnf-interned (cc52b9c) with charlie/marker-dnf-candidates (6e34068)

Open in CodSpeed

Footnotes

  1. 104 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@charliermarsh charliermarsh changed the title Use interned terms for all DNF simplification Use bitsets for all DNF simplification Oct 15, 2026
if clauses.len() >= 256 && clauses.len() > negated.len().saturating_mul(2) {
return simplify_candidates(dnf, clauses, &negated, words);
let clause_count = clauses.len();
let mut sets: SmallVec<[u64; 8]> = smallvec![0; clause_count.saturating_mul(words)];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Retain a memory bound for the dense bitset index

A publisher can change the Requires-Dist markers in a previously locked direct-URL wheel. During an online uv sync --locked with missing or stale metadata, Lock::package_metadata fetches and caches that metadata; serialization invokes this simplifier before checking the wheel's locked hash. Removing the previous 32 MiB limit and linear fallback makes this allocation consume 8 × clauses × ceil(distinct_terms / 64) bytes, allowing sparse disjunctions to cause quadratic memory exhaustion. The failure threshold depends on the process memory budget. Retain a bounded dense index with a sparse or linear fallback.

@charliermarsh
charliermarsh removed this pull request from stack #22616 October 16, 2026 01:15

This branch was previously deployed

1 inactive deployment
automations — cc52b9c3 Deployed Oct 15, 2026 by charliermarsh via review / report #51274
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant