Repository navigation
Simplify selection markers under conflict constraints - #22639
charliermarsh wants to merge 32 commits into
Conversation
Merging this PR will degrade performance by 4.06%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Large conflict sets produce marker DNFs with many clauses. Simplification repeatedly compares their terms, including string and version values. Intern those terms and represent clauses as bit sets so we can compare integer IDs and machine words instead. Keep direct comparisons for small DNFs and use indexed term lookups for sparse clauses. Cap the dense bit-set index at 32 MiB and fall back to the linear algorithm for larger indexes or repeated terms within a clause. Terms and clauses are eliminated in the same order, so earlier removals affect later comparisons and serialized markers remain consistent. This reduces the cost of each comparison; DNF materialization and the quadratic number of clause comparisons remain. Related to #21954.
21f6da8 to
10cf428
Compare
d8dd421 to
494c832
Compare
Large marker DNFs often reuse a small set of terms across many clauses. The indexed simplifier recomputes the difference between two clauses for each term it considers removing. Compute viable differences once per clause and reuse them across those term checks. Count missing terms first to discard candidates that cannot justify a removal, then update the cached differences after each removal so later checks use the surviving terms. Enable this path only when enough clauses reuse relatively few terms; smaller and sparser DNFs retain direct comparisons. Differential tests check that both paths produce the same ordered output as the linear simplifier.
Keep an environment marker when it describes the dependency under every valid conflict selection. Retain conflict-dependent and impossible edges so lockfile readers can still discover conflicting transitive extras. Port astral-sh#2149. Related to #21954.
c15e2df to
989c236
Compare
494c832 to
1954319
Compare
Keep an environment marker when it describes the dependency under every valid conflict selection. Retain conflict-dependent and impossible edges so lockfile readers can still discover conflicting transitive extras. Port astral-sh#2149. Related to #21954.
1954319 to
e93cb56
Compare
567cd3a to
2b99716
Compare
We add a temporary, constraint-aware view of dependency markers for installation and export. This lets us simplify markers that still depend on extras or groups, while keeping the original markers for lockfiles and conflict discovery.
For example, suppose
cpuandgpuare mutually exclusive extras:A dependency needed by
cpuon Linux can have a marker like this, written here as Boolean shorthand:The first term keeps the dependency edge reachable when both conflicting extras are active. That matters because dependencies can activate extras transitively: the existing lockfile reader follows those edges to discover the conflict and report an error. We cannot simply delete that term from the stored marker.
For a valid selection, however,
cpu AND gpuis always false. We can evaluate the simpler marker:This still depends on
cpu. #22615 handles the narrower case where every valid selection reduces to the same environment-only marker. This PR also allows simplification when the result still depends on an extra or group.The new
ConflictMarkerContextkeeps the assumptions together with the simplified view. It uses the existingMarkerTree::restrictoperation to simplify the decision diagram, and only uses that view where the assumptions hold. For example, selecting both extras on macOS makes the compact marker false but the original marker true. In that case we retain the original marker, so the existing conflict-discovery and error-reporting behavior keeps its input. A reachable edge does not mean we permit the invalid installation.The context is integrated in two places:
The context caches compact views and shares constraints between markers that reference the same conflict items. Constraint construction lets each item exclude the union of preceding items, avoiding enumeration of every conflicting pair. Collecting referenced extras visits each shared decision node once.
This generalizes in-memory simplification. It preserves the v1 lockfile representation and the existing activation-discovery algorithm, so it does not shrink lockfiles or remove every marker-scaling limit.
Local measurements compare #22615 at
e93cb5696cwith this PR at962d915168, using 31 paired trials on the same Linux devbox:Constraint construction improves for larger sets, but preparing compact views adds work. With a warm BDD interner, preparing a context and evaluating 32 markers takes 309 µs, compared with 14 µs for direct evaluation. The roughly 1% sync increase should be treated as a small regression; these measurements do not establish a general end-to-end speedup.
All 11 lockfiles are byte-for-byte identical, all 88 command-result comparisons match, and all three installed environments match.