Skip to content

Speed up marker DNF conversion and simplification - #2140

Draft
zaniebot wants to merge 14 commits into
zb/bench-large-marker-dnffrom
zb/index-marker-dnf
Draft

zaniebot wants to merge 14 commits into
zb/bench-large-marker-dnffrom
zb/index-marker-dnf

Conversation

@zaniebot

@zaniebot zaniebot commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Large conflict sets can make uv lock slow, as reported in astral-sh#21954. Mutually exclusive extras can produce large marker expressions: converting them to disjunctive normal form (DNF) can yield many clauses, and simplification repeatedly compares their marker terms. This also adds work to uv export, which transforms marker trees while constructing output.

Intern marker expressions and represent clauses as bit sets to make those comparisons cheaper. For large DNFs whose clauses reuse relatively few terms, count missing terms before enumerating bit-set differences to discard candidates that cannot justify removing a term. Cache the remaining differences for reuse while simplifying the clause.

Candidate caching is limited to expressions where its setup can pay off; smaller or sparser expressions compare directly, and short clauses use indexed term lookups rather than scanning mostly empty bit sets. Cap the dense bit-set index at 32 MiB and fall back to linear comparisons for larger indexes or repeated terms within a clause. Terms and clauses are still eliminated in the same order, so earlier removals affect later comparisons and the simplified marker output remains consistent.

Skip DNF simplification when resolving activated extras and checking marker reachability, where the Boolean result does not depend on removing redundant clauses. Public marker output and conflict diagnostics continue to use simplified DNF, keeping marker text compact and avoiding repeated warnings from redundant expressions.

@codspeed

codspeed Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 30.33%

⚠️ 11 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 68 untouched benchmarks
🆕 6 new benchmarks
⏩ 41 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Simulation load_python_download_catalog 452.7 ms 347.3 ms +30.33%
🆕 WallTime uv_lock_no_lockfile_warm_metadata_cache_16_conflicting_extras N/A 2.8 ms N/A
🆕 WallTime uv_lock_no_lockfile_warm_metadata_cache_32_conflicting_extras N/A 9 ms N/A
🆕 WallTime uv_lock_no_lockfile_warm_metadata_cache_4_conflicting_extras N/A 1.1 ms N/A
🆕 ⚠️ Simulation uv_lock_no_lockfile_warm_metadata_cache_16_conflicting_extras N/A 32.8 ms N/A
🆕 ⚠️ Simulation uv_lock_no_lockfile_warm_metadata_cache_32_conflicting_extras N/A 166.2 ms N/A
🆕 ⚠️ Simulation uv_lock_no_lockfile_warm_metadata_cache_4_conflicting_extras N/A 9.9 ms N/A

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing zb/index-marker-dnf (ec10f8a) with zb/bench-large-marker-dnf (3363300)2

Open in CodSpeed

Footnotes

  1. 41 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on zb/bench-large-marker-dnf (11dd3da) during the generation of this report, so bf47876 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@zaniebot
zaniebot changed the base branch from main to zb/bench-large-marker-dnf September 29, 2026 19:32
@zaniebot
zaniebot added this pull request to stack #2155 September 29, 2026 22:34
@zaniebot zaniebot added risk:3 Credible risk of meaningful incorrect behavior or regression; bounded or recoverable. priority:3 Normal: useful features, documentation, usability, or routine efficiency improvements. size:2 Moderate: one subsystem with meaningful invariants, edge cases, or operational evidence to inspect. labels Sep 30, 2026
@zaniebot zaniebot added the build:skip-release Disable building release binaries for a pull request label Oct 7, 2026
@zaniebot
zaniebot deployed to automations October 7, 2026 23:05 — with GitHub Actions Active
@zaniebot
zaniebot deployed to automations October 7, 2026 23:10 — with GitHub Actions Active
Comment thread crates/uv-pep508/src/marker/simplify.rs Outdated
Comment on lines +269 to +272
let words = expressions.len().div_ceil(64);
// A sparse expression can have far more distinct terms than terms per clause.
// Bound the dense index to 32 MiB and use the linear algorithm beyond that.
if dnf.len().saturating_mul(words) > MAX_INDEX_WORDS {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Fall back to linear comparisons for sparse DNFs

The memory cap still admits sparse inputs that add substantial comparison work. collect_dnf produces singleton clauses for disjunctions such as platform_machine == 'arch-0' or platform_machine == 'arch-1' or .... Both indexed passes scan leading zero words before finding each other clause's term, adding a factor proportional to the number of distinct terms. At 16,384 singleton clauses, the index fits exactly within 32 MiB but requires approximately 69 billion word visits; the linear algorithm performs constant work per clause pair. Include clause density in the fallback decision, or skip unpopulated words, and cover a sparse input below the storage limit.

@zaniebot
zaniebot removed this pull request from stack #2155 October 7, 2026 23:37
@zaniebot
zaniebot added this pull request to stack #2337 October 7, 2026 23:38
@zaniebot
zaniebot removed this pull request from stack #2337 October 8, 2026 00:02
@zaniebot
zaniebot added this pull request to stack #2339 October 8, 2026 00:02
@zaniebot
zaniebot removed this pull request from stack #2339 October 8, 2026 00:19
@zaniebot
zaniebot added this pull request to stack #2340 October 8, 2026 00:19
@zaniebot
zaniebot deployed to automations October 8, 2026 03:51 — with GitHub Actions Active
@zaniebot
zaniebot deployed to automations October 8, 2026 03:55 — with GitHub Actions Active
Comment on lines +739 to +743
for _ in 0..next() % 30 {
let mut clause = Vec::new();
for _ in 0..next() % 12 {
let term = expressions[next() % expressions.len()].clone();
if case % 2 == 0 || !clause.contains(&term) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Exercise negation in the sparse comparison tests

Replaying this fixed generator produces at most 119 distinct terms, so even the many_expressions pass never reaches sparse_comparisons, which requires more than 128. The dedicated sparse fixtures contain no complementary terms, while the multiword fixtures use dense comparisons. Consequently, the new sparse path’s acceptance of a negated term lacks equivalence coverage. Add a deterministic fixture with more than 128 distinct terms and short complementary clauses, such as (A and B) and (not A and B), and compare its ordered result with simplify_linear.

@zaniebot
zaniebot deployed to automations October 8, 2026 04:10 — with GitHub Actions Active
@zaniebot
zaniebot deployed to automations October 8, 2026 06:51 — with GitHub Actions Active
@zaniebot zaniebot changed the title Use bit sets to simplify large marker DNFs Speed up marker DNF conversion and simplification Oct 15, 2026
@zaniebot
zaniebot deployed to automations October 15, 2026 18:25 — with GitHub Actions Active
@zaniebot
zaniebot deployed to automations October 15, 2026 18:31 — with GitHub Actions Active
Comment on lines +1014 to +1017
/// Both [`MarkerTree::TRUE`] and [`MarkerTree::FALSE`] return an empty list. Callers that
/// need to distinguish these constants must handle them before converting to DNF.
pub fn to_unsimplified_dnf(self) -> Vec<Vec<MarkerExpression>> {
simplify::to_unsimplified_dnf(self)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Preserve constant truth values in the new DNF API

Returning [] for both constants makes this Boolean-transformation API lossy: the reconstruction fold in unsimplified_dnf_reconstructs_marker would turn TRUE into FALSE. Both production callers guard constants correctly, but every future consumer must remember this extra precondition. Return one empty conjunction for TRUE and no conjunctions for FALSE in the new method, leaving the existing to_dnf contract unchanged, and include a tautology in the reconstruction test.

This branch was successfully deployed

1 active deployment
automations — ec10f8a7 Deployed Oct 15, 2026 by zaniebot via review-code-quality / code quality review #10429
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build:skip-release Disable building release binaries for a pull request performance priority:3 Normal: useful features, documentation, usability, or routine efficiency improvements. risk:3 Credible risk of meaningful incorrect behavior or regression; bounded or recoverable. size:2 Moderate: one subsystem with meaningful invariants, edge cases, or operational evidence to inspect.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants