Fix space leaks in dependency solver logging.

[grayjay]

This commit removes references to the solver log that prevented it from being garbage collected. It also forces evaluation of the current level and variable stack in Message.showMessages.

There is still a space leak in backjumping, so I profiled without backjumping enabled by merging in this branch: https://github.com/grayjay/cabal/tree/no-backjumping . I stopped each of these runs after a minute, but the times on my branch and master were similar when they solved for fewer packages.

EDIT: I compiled with GHC 7.10.2 and ran cabal --ignore-sandbox install --dry-run --max-backjumps -1 hackage-server-0.4 aeson yesod --with-compiler C:/ghc-7.6.3_64/bin/ghc +RTS -p -s -h -L50

master (bbc3638):

this branch (90c7772):

this branch without backjumping:

[kosmikus]

Wow. Thank you! This looks extremely promising.

You did not (intend to) change any of the actual behavior, right?

Do you think there's any way we could add some kind of regression test for this? Or even a semi-manual test? But at least something that makes it easy to test after further solver changes that the space behavior has not become worse again?

[grayjay]

@kosmikus Thanks! My goal was to keep the existing behavior but avoid converting the Progress to and from a list of messages paired with a result.

I'm not familiar with regression testing for performance, especially comparing across versions. I mainly tested this change by checking that the heap profile leveled off after cabal finished reading the package index. A test program could at least isolate the solver and use a very small set of unchanging packages. Then a space leak would have a much larger effect on maximum residency, and we might be able to use a fixed cutoff. We could choose a difficult test and run it for a limited number of steps, so that run time isn't affected by other changes to the solver, like goal order. I'm not sure how to test for increases in constant memory use, though.

[grayjay]

I started writing a benchmark, and it seems to detect these space leaks reliably. It runs cabal on a .cabal file with a lot of flags but no actual dependencies. The cabal.config file points to an empty package index to avoid reading the index from Hackage into memory. Then the test checks the memory use after a fixed number of backjumps with +RTS -t --machine-readable. I had to use the average, instead of the maximum, because there was often a spike at the beginning of the run.
https://github.com/grayjay/cabal/tree/solver-benchmarks .

[BardurArantsson]

I looked through this a few days ago and it seemed reasonable to me. That said... a lot of the code complexity/churn here seems to arise from not using a special-purpose monad for the solver (and/or logging its doings) -- of course doing that would be a huge refactoring, but would it make sense?

Regardless, I don't think that this should have to wait for a method-of-doing-automated-performance-regression-testing. You've provided ample evidence that this concrete PR fixes a leak. (Don't get me wrong... if you feel like pursuing the automated-performance-regression-testing thing, then I'd be very excited about that!)

[grayjay]

@BardurArantsson Thanks for the feedback. I'm not sure I understand the monad approach. The solver creates the log when it explores the tree, so it's only logging the last step. Earlier steps just leave information in the tree, such as pruned nodes, so that it can be included in the log. Then there are several log-processing steps. I think it would be nice to move some of that processing into the exploration step, where it might be easier. It would help with #2917 (2nd bullet point in #2917 (comment)). Also, we could remove the variable stack here if we could record the current variable during exploration:

cabal/cabal-install/Distribution/Client/Dependency/Modular/Message.hs

Line 50 in da8caf4

go :: [Var QPN] -> Int -> [Message] -> [String]

.

Even though I tested this PR, I'd really like to add some solver performance tests. It's too easy to accidentally introduce performance problems. I have a test for space leaks, but I'm not sure how it could fit into the project. Testing speed would be even more useful. Do you have any ideas for testing?

[BardurArantsson]

@grayjay It really was just an off-hand "wouldn't this be nice?" comment, so REALLY don't worry about it :).

AFAIU the PR is absolutely fine. (But I stress that my understanding of the solver is limited... very limited. Just FTR)

[grayjay]

I was just curious because I had some ideas for refactoring that area of code, and I used a State monad in #2917. It was mostly for supporting the new feature, though.

[BardurArantsson]

@23Skidoo What do you think about just merging this given the arguments above? I think adding performance regression tests would be a different PR.

[23Skidoo]

@BardurArantsson
Yes, I want to see this and #2916 in 1.24, I just need to do a review first.

I think adding performance regression tests would be a different PR.

This is fine with me.

[23Skidoo]

Since this PR only touches cabal-install code, I'm postponing it for the 1.24.1 release.

[kosmikus]

I'm looking at this (and #2916) right now.

[kosmikus]

Still looking. Have been running a few performance comparisons, and they all take a long time to complete. But in general, it is all looking very good so far.

[kosmikus]

This one definitely looks good to me. Still looking at #2916.

[BardurArantsson]

Excellent to have this merged! I may revisit the solver split this weekend (but in a slightly more step-wise way)... @kosmikus Are you planning any big changes to the solver soon?

My revised approach will be to start by generalizing various types to have the package location be a type parameter. Unless I've missed some drastic recent developments this will allow breaking the dependency from the modular solver to cabal-install... which will be the second step.

[kosmikus]

Yes, @grayjay, let me say again that I am very very sorry for not dealing with these PRs for so long. These PRs are fantastic work. Thank you very much.

@BardurArantsson I am not generally opposed to the solver split (if we can work out the details). It's a bit difficult to say what I'm planning. It's the first time in ages I've managed to set aside a bit of time for work on this, and I will first try to look at outstanding PRs and issues. There are a few things I'd like to do myself, ultimately, but I don't think any of that will happen too soon. In any case, I'm aware of the ideas to separate out the solver, and I won't do anything major that isn't yet on the issue tracker in some form without coordinating with you.

[BardurArantsson]

Ok, thanks. That sounds great.

I'm very happy to discuss the details -- I think the first step should be pretty uncontroversial (and non-disruptive unlike my first attempt), so I'll try to get a PR done sometime this weekend and we can perhaps discuss it then.

EDIT: I'll make sure to /cc you on that.

[kosmikus]

@BardurArantsson Ok, sounds good. Thanks.

[grayjay]

@kosmikus No problem. Thank you for the thorough testing and review!

[23Skidoo]

Thanks, @grayjay!

[BardurArantsson]

@kosmikus First step filed as PR #3210

[grayjay]

I created an issue for the performance tests: #3211