Slow solving with `build-tool-depends` on local packages with tests

[michaelpj]

Describe the bug
Suppose I have a local package B which provides a build tool, and which either has tests or depends on a package A that has tests. Finally, I have another local package C which depends on B's build tool.

Finally, I have tests: true in my cabal.project, since I naturally want to test my local packages.

The result of this is that the solver becomes slower, because (I hypothesize) it tries to create a build plan for A without tests to satisfy the build-tool dependency, then finds that it can't do that (due to the project constraint) and has to redo it.

That looks something like this (from cabal configure -v3):

[_13] trying: C:B:exe.A~>A-0.1.0.0 (dependency of C:B:exe.B)
[_14] rejecting: C:B:exe.A:!test (dependencies not linked: stanza "test" incompatible; conflict set: A, B, C:B:exe.A, C:B:exe.B, A:test, C:B:exe.A:test)

This is not a big deal in a small project. However, if we have a long chain of packages up to the build tool dependency like so:

A->B-> ... ->Z->Tool

all of which with tests, then the solver will redo most of the work over and over until it finally realises it has to build all of A to Z with tests. This can cause a very large slowdown in solving time.

cabal freeze does not help, since it only pins the versions and the test/no-test incompatibility still bites.

To Reproduce

I made a repository that exhibits the behaviour I described in the small: https://github.com/michaelpj/cabal-build-tool-depends-solver-repro

Changing tests:true to tests:false in cabal.project stops the solver from failing and having to backtrack.

Expected behavior

I don't understand how cabal's solver works at all, but I would like this to not be slow! Maybe the test/no-tests constraint needs to be pushed down eagerly?

System information

• NixOS
• cabal 3.4.0.0, ghc 8.10.4

[michaelpj]

Quick check for "does this affect me":

cabal configure -v3 > out.txt
cat out.txt | grep 'stanza "test"' | wc -l

[michaelpj]

A few more notes for the interested:

• We observed this in the wild on some Cardano projects, notably https://github.com/input-output-hk/plutus and https://github.com/input-output-hk/cardano-node.
  • It's not appallingly slow, but you can observe the test/non-test confusions in the solver output.
• The problem is especially acute on those repositories with cabal 3.2, since in those days tests: true applied to source-repository-packages, multiplying the number of test/non-test confusions that get hit.
• It's also acute for haskell.nix (Resolving dependencies is much slower for haskell.nix with lots of source-repository-package stanzas input-output-hk/haskell.nix#1156), which replaces source-repository-package stanzas with local packages in order to avoid network access, but thus ends up in the same place as cabal 3.2.

[michaelpj]

For newer versions of cabal which don't run the solver when calling configure, the easiest way to see the issue is probably cabal build --dry -v3 all.

[gbaz]

Here's the result of my investigation thus far. The planPackages call constructs a DepResolverParams (this gets printed out at v3) that in turn includes "preferences" which include, in this example:

  A [BenchStanzas]
  B [BenchStanzas]
  C [BenchStanzas]

Those preferences say try with bench enabled before bench disabled. In this case we should set the preferences to also have TestStanzas.

Since this section also sets up constraints, and tests are definitely included in the constraints, all the data should be there to include them in preferences as well.

If this is done the search tree will be reordered to try packages first with test enabled (the success path!) and only then disabled. Not quite sure how to tweak the code to do this though, if anyone else wants to take a run at it first :-)

[gbaz]

Anyone affected by this issue is welcome to try out the linked PR an see if it improves things. (Assuming it gets a good review, I think it should probably be merged even if there's no noticeable speedup since it seems slightly more correct, and at worst will remove some "line noise" from solver output by eliminating some unnecessary local backjumps.

[grayjay]

I ran the reproduction, but I'm not sure it reproduces the performance issue, even though it contains the log message about incompatible stanzas. I didn't see any backtracking with cabal 3.4.0.0.

Full solver log:

[__0] trying: A-0.1.0.0 (user goal)
[__1] trying: base-4.14.1.0/installed-4.14.1.0 (dependency of A)
[__2] trying: rts-1.0/installedrts (dependency of base)
[__3] trying: integer-gmp-1.0.3.0/installed-1.0.3.0 (dependency of base)
[__4] trying: ghc-prim-0.6.1/installed-0.6.1 (dependency of base)
[__5] rejecting: A:!test (constraint from config file, command line flag, or user target requires opposite flag selection)
[__5] trying: A:*test
[__6] trying: B-0.1.0.0 (user goal)
[__7] trying: C-0.1.0.0 (user goal)
[__8] trying: C:B:exe.B~>B-0.1.0.0 (dependency of C)
[__9] trying: C:B:exe.base~>base-4.14.1.0/installed-4.14.1.0 (dependency of C:B:exe.B)
[_10] trying: C:B:exe.rts~>rts-1.0/installedrts (dependency of C:B:exe.base)
[_11] trying: C:B:exe.integer-gmp~>integer-gmp-1.0.3.0/installed-1.0.3.0 (dependency of C:B:exe.base)
[_12] trying: C:B:exe.ghc-prim~>ghc-prim-0.6.1/installed-0.6.1 (dependency of C:B:exe.base)
[_13] trying: C:B:exe.A~>A-0.1.0.0 (dependency of C:B:exe.B)
[_14] rejecting: C:B:exe.A:!test (dependencies not linked: stanza "test" incompatible; conflict set: A, B, C:B:exe.A, C:B:exe.B, A:test, C:B:exe.A:test)
[_14] trying: C:B:exe.A:*test
[_15] done

The log shows that cabal is trying to use the same instance of A for both the top-level build target and C's build tool dependency (level 13), which is an optimization. Using the same instance of A means that the two uses must make the same choice for enabling tests. The solver first tries disabling tests for the build tool dependency at level 14 but immediately encounters the conflict and enables tests instead. This is how I would expect the solver to behave, even with more packages in the chain of dependencies.

I think I would need to see an example of how increasing the chain of dependencies causes backtracking, especially if it causes the solver to make the same choice multiple times.

I also think that cabal should add the same preferences for all local packages as in #7472 (comment), for consistency.

[michaelpj]

I've attached the actual constraint-solver log from the project in question. It's quite long, but perhaps illuminating.

out.txt

[michaelpj]

In particular it has:

• Lots of steps even in the happy path
• Lots of backracking when it hits the test/non-test conflict

[michaelpj]

(Sorry that the reproduction wasn't obviously slow, I was trying to just show the test/non-test conflict issue, which seemed more self-contained!)