Make dependency resolution plan platform-agnostic

[bennofs]

Problem

The solver currently depends on system information to generate the install plan. The install plan is then also only valid on the given system. For example, the fsnotify package depends on either hfsevents (mac) or inotify (linux).

This makes it hard to have truly deterministic install plans, since you have to generate a different install plan for each platform that you want to support. It also complicates life for automated tools since they have to deal with different resolutions for different platforms.

If this was supported. lockfiles could just specify a version of each package and the solver wouldn't have to run again for a different platform.

Do you think it would be a good idea to work towards eliminating these platform dependencies? Or is this out of scope for cabal?

Possible solution

Of couse, we would still need conditional compilation. Also, having flags influence the dependency graph is not a problem per se. The problem only occurs because flags are allowed to be automatically resolved based on platform properties.

One way to avoid this would be to do something similar to what Rust does: make every package available on all platforms, but make the compile phase be a no-op if it isn't supported. So fsnotify would always depend on both hfsevents and inotify (both would be resolved), but only one of them would actually be compiled.

Thoughts?

[hvr]

The problem only occurs because flags are allowed to be automatically resolved based on platform properties.

Personally I consider that a strength rather than a weakness, as it allows to localize platform-specific solver constraints into the single package which provides the platform specific services; and afaik, Cabal is one of the few ecosystems which have such an advanced feature. And btw, you don't even need to use automatic flags to get into this situation. The problem here is that trying to shoehorn the design to support "lockfiles" is unrealistic IMO, as I've always considered freeze-files to be specific to the specific environment (compiler, compiler features, os, arch, available C libraries etc), and by requiring lockfiles to be globally satisfiable sounds to me like the equivalent of putting the cart before the horse in terms of design.

but make the compile phase be a no-op if it isn't supported. So fsnotify would always depend on both hfsevents and inotify (both would be resolved), but only one of them would actually be compiled.

If I understand you right, this sounds like it would result in more rather than less conditional APIs, as when you track a dependency on a package hfsevents you declare to depend unconditionally on its API being available. And you need to mirror the conditional-logic into the consumers of the package rather than have the cabal solver do that for you (See also FAQ entry on conditional APIs).

TLDR, I'm skeptical :-)

[bennofs]

Thank you for your answer!

as I've always considered freeze-files to be specific to the specific environment (compiler, compiler features, os, arch, available C libraries etc)

Makes sense. However, at some point, there has to be the question where we put the line. For some of these it certainly makes sense, as they are very easy to control / are visible: OS, compiler, compiler features and arch. They also have few possible values each, so that an approach of one lockfile for each combination is quite feasible for these factors.

Making freeze files depend on available C libraries is borderline to me though. The set of possible installed C libraries is quite large so the space of possible configurations is huge. It is also not very visible: I'm probably not going to think about what C libraries I have installed if I build a haskell package. An argument could be made that if it depends on available C libraries, then it may as well depend on available Haskell libraries (which are already installed), nullifying the benefits of a freeze file all together.

If I understand you right, this sounds like it would result in more rather than less conditional APIs, as when you track a dependency on a package hfsevents you declare to depend unconditionally on its API being available. And you need to mirror the conditional-logic into the consumers of the package rather than have the cabal solver do that for you.

That is a good point, I hadn't considered that.

Overall, I see that the current situation has its advantages. It sucks for tools, since dealing with conditional dependencies trees is a little annoying, but it may be necessary complexity (the particular tool in my mind here is things like the autogeneration of the nix package set, where a static dep tree would be much easier to deal with. Also potentially stackage, since you may get a totally different build plan than what stackage tested with (stackage tests run on linux))

[jneira]

This issue has arise recently, around fsnotify again: #8059
Being lock/freeze files os dependent reduce its uselfuness but it seems is an unavoidable thing, given how cabal handle system dependencies by design.
I am gonna close this one as it seems there is nothing actionable here, feel free to reopen in other case