Skip to content

Skip marker DNF simplification for Boolean transformations - #22605

Closed
charliermarsh wants to merge 3 commits into
charlie/marker-dnf-bitsetsfrom
charlie/marker-dnf-unsimplified
Closed

charliermarsh wants to merge 3 commits into
charlie/marker-dnf-bitsetsfrom
charlie/marker-dnf-unsimplified

Conversation

@charliermarsh

@charliermarsh charliermarsh commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Superseded by #22608, which resolves extras and checks marker reachability directly on the decision diagram. The raw-DNF API is no longer needed. Lockfile and export regression coverage is retained in #22604 and #22606.

@charliermarsh
charliermarsh marked this pull request as draft October 15, 2026 19:35
@charliermarsh
charliermarsh force-pushed the charlie/marker-dnf-bitsets branch from e18c632 to 8454aaa Compare October 15, 2026 19:37
@charliermarsh
charliermarsh force-pushed the charlie/marker-dnf-unsimplified branch from c90f875 to 5c5ab16 Compare October 15, 2026 19:37
Large groups of mutually exclusive extras can make `uv lock` slow, as in #21954. Add offline lock benchmarks for 4, 16, and 32 extras, using a shared dependency whose releases split resolution across Python versions.

Each iteration starts without a lockfile and reuses a warm metadata cache, measuring a fresh solve through lockfile generation. Run the benchmarks in both CodSpeed modes.
Large conflict sets produce marker DNFs with many clauses. Simplification repeatedly compares their terms, including string and version values. Intern those terms and represent clauses as bit sets so we can compare integer IDs and machine words instead.

Keep direct comparisons for small DNFs and use indexed term lookups for sparse clauses. Cap the dense bit-set index at 32 MiB and fall back to the linear algorithm for larger indexes or repeated terms within a clause. Terms and clauses are eliminated in the same order, so earlier removals affect later comparisons and serialized markers remain consistent.

This reduces the cost of each comparison; DNF materialization and the quadratic number of clause comparisons remain. Related to #21954.
Resolving activated extras and checking marker reachability convert marker trees to DNF, simplify them, then reconstruct a Boolean result. Removing redundant terms and clauses does not change that result, so skip simplification and sorting in these two callers.

Add `MarkerTree::to_unsimplified_dnf` for these transformations. Public marker output and conflict diagnostics continue to use simplified DNF, keeping marker text compact and avoiding repeated warnings from redundant expressions. Cover lockfile generation, locked revalidation, and export for an environment-dependent dependency shared by conflicting extras.
@charliermarsh
charliermarsh force-pushed the charlie/marker-dnf-bitsets branch from 8454aaa to b94d029 Compare October 15, 2026 19:53
@charliermarsh
charliermarsh force-pushed the charlie/marker-dnf-unsimplified branch from 5c5ab16 to b5c3118 Compare October 15, 2026 19:53
@charliermarsh
charliermarsh removed this pull request from stack #22607 October 15, 2026 20:26
@charliermarsh
charliermarsh force-pushed the charlie/marker-dnf-bitsets branch from b94d029 to b5bea7f Compare October 15, 2026 20:26
@codspeed

codspeed Bot commented Oct 15, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 5.07%

⚠️ 11 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

⚡ 1 improved benchmark
✅ 71 untouched benchmarks
🆕 3 new benchmarks
⏩ 54 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ WallTime git_fetch_cold_precise[sampleproject] 37.6 ms 35.8 ms +5.07%
🆕 WallTime uv_lock_no_lockfile_warm_metadata_cache_16_conflicting_extras N/A 2.9 ms N/A
🆕 WallTime uv_lock_no_lockfile_warm_metadata_cache_32_conflicting_extras N/A 14.1 ms N/A
🆕 WallTime uv_lock_no_lockfile_warm_metadata_cache_4_conflicting_extras N/A 1.2 ms N/A

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing charlie/marker-dnf-unsimplified (b5c3118) with charlie/marker-dnf-bitsets (b94d029)

Open in CodSpeed

Footnotes

  1. 54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

This branch was previously deployed

1 inactive deployment
automations — b5c3118c Deployed Oct 15, 2026 by charliermarsh via review / security review #51185
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants