Support for managing sandbox packages

[mightybyte]

I've been using the sandbox functionality from HEAD, and the lack of list/unregister/etc functionality is a big problem. We need the equivalent of ghc-pkg for managing packages in sandboxes, or at least basic list and unregister functionality. Right now the only solution is to delete the whole sandbox, which is a huge waste of time, or manually hunt down directories to delete and hope that this doesn't leave the sandbox in a broken state.

[tibbe]

I would hope that you would never had to manually touch the sandbox. The idea is that it should be enough to just update your project's .cabal file and, in some cases, the (optional) per-project cabal.config file. To better understand what the right fix is, could you describe in which circumstance you want to unregister a package or list the packages in the sandbox.

[mightybyte]

It happens all the time. Before sandboxes, ghc-pkg has been indispensable to me for maintenance of my package repository. Never manually touching the sandbox may sound good in theory, but the real world just doesn't work that nicely. Sometimes multiple versions of a package get installed and sometimes I get linker errors because the wrong version is getting used in the wrong place. I don't want to have to specify constraints from now on, so I unregister the one I don't want. In the absence of fine-grained control over packages, I would resort to blowing away the whole repository, which would be a huge waste of time as I'd often have to rebuild 100+ packages.

The case that just happened to me was that I installed my package foo and it retrieved a dependency bar from hackage. Then I discovered that I in fact needed my locally modified version of bar. So I do cabal sandbox-add-source ../bar. But that doesn't do anything since it already has the version from hackage which satisfies the dependencies. Again, I don't want to delete my whole sandbox because that sets me back another 20 minutes or more. I just want to delete the version of bar that is already installed in the sandbox so that it will use my local one.

[tibbe]

Note that you can still use ghc-pkg on a sandbox directory. It's a valid package DB. Just use the ghc-pkg flag to point ghc-pkg to it. For what it's worth I almost never touch my package DB, except when it has broken due to Cabal dependency hell.

I don't understand how you arrive at linker errors. If you don't want to specify constraints you will have a bad time. The compiler needs them to pick the right versions of things. Perhaps that's why you're getting linker errors?

That a Hackage package is picked over a add-source package sounds like a bug that we ought to fix before the release. Please file a separate ticket for that one.

[mightybyte]

Hmmmm, then it would be nice if ghc-pkg would just work on the sandbox in the current or most recent parent directory if it exists. I don't manually touch my package DB either unless dependency hell comes along. The problem is that it comes along on a very regular basis.

I specify both upper and lower version bounds in my packages religiously. I'm talking about adding manual command-line version constraints.

I'll add another ticket for the priority thing, but that won't eliminate the need to manually modify the repository. I can just as easily imagine a scenario where I'd want the reverse behavior. I agree that add-source should have the highest priority, but that's not always the right thing.

I opened a new issue for the sub-problem at #1197.

[tibbe]

I don't manually touch my package DB either unless dependency hell comes along. The problem is that it comes along on a very regular basis.

Hopefully that should happen much less often with sandboxing. The only shared package DB will be the global one and as long as you don't install anything there (you shouldn't) separate project shouldn't be able to stomp on each other (the infamous --force-reinstall breakage problem).

I'm talking about adding manual command-line version constraints.

You might want to try the new per-project cabal.config feature. You can put a file named cabal.config in the same directory as the project itself and specify a constraints section in there. It acts like the --constraints flag but it's persistent so you only have to specify the constraints once. The intended use is to freeze dependencies in a project that you intend to ship (e.g. an executable to run on a server or to ship to a customer).

[23Skidoo]

Once #1104 is fixed, cabal info and other standard commands will use the sandbox if it is present. With regards to ghc-pkg, we can implement the analog of cabal-dev buildopts so that you could run ghc-pkg cabal sandbox buildopts list.

[mightybyte]

@tibbe

Well, I've been using sandboxing for less than a week and I've already had at least three or four instances where I wanted to unregister packages from my sandbox. I don't think constraints in a per-project config file will solve my problems because I'm often making changes to four or more packages in concert.

@23Skidoo

ghc-pkg cabal sandbox buildopts list is rather cumbersome. I guess I would begrudgingly accept it for my purposes if I combine it with a custom shell alias. But I think my distaste for it is more well-founded than that. It's not very discoverable for the uninitiated. If I use cabal install to install something, then the universal first guess would be that I should use cabal uninstall or perhaps cabal unregister to remove that something. Not some crazy unrelated program called ghc-pkg. And definitely not that program combined with some other bizarre cabal command. If we can implement cabal sandbox buildopts, then why can't we implement cabal unregister that just uses cabal's knowledge of the sandbox location and shells out to ghc-pkg? And if cabal info can get me the information for a single installed package, why can't cabal installed get me a list of the installed packages?

[tibbe]

@mightybyte Then lets figure out why you wanted to unregister packages and make sure that's no longer needed. Building a system where that is common practice is not desirable.

[23Skidoo]

If I use cabal install to install something, then the universal first guess would be that I should use cabal uninstall or perhaps cabal unregister to remove that something.

I agree, but that's a separate (and longstanding) issue (#227).

[tibbe]

Re. installing. We've set a bad precedence by focusing on installing in the past. Installing is the wrong thing to do. Rarely do you need software under development or software you intend to ship somewhere else to be installed on your system. The focus should be on projects. You're working on something and you need to build it (to test it or whatever). cabal build (or cabal test, etc) should do that. What it needs to do in order to do that (build dependencies, etc.) is not important from the user point of view.

Having experience a build system that works like this, is used on tens of millins of lines of code, and by thousands of developers I'm sure that it's both possible and desirable. The chance of a system based on installing things working goes towards 0 as the number of packages goes towards BIG_NUMBER.

cabal build in a project dir should work as-if you wiped your whole system clean, reinstalled GHC and all your dependencies from scratch. If you can tell, from within a project directory, that there are other projects on the same machine, we still have work to do.

[mightybyte]

Then lets figure out why you wanted to unregister packages and make sure that's no longer needed. Building a system where that is common practice is not desirable.

I'm not sure that I agree with this, but that discussion should probably take place somewhere else.

What it needs to do in order to do that (build dependencies, etc.) is not important from the user point of view.

This would be true, except for the fact that Haskell builds are slow, therefore they need to be cached....i.e. installed. If you have a system for caching builds, then you either need a mechanism for removing things from the cache or you need more granular build disambiguation than the current system gives us.

[tibbe]

then you either need a mechanism for removing things from the cache or you need more granular build disambiguation than the current system gives us.

Nix-style builds give us the latter. It makes the cache key be dependent on all inputs and thus the cache is always valid and never needs to be manually pruned.

[mightybyte]

Right, so since we don't have nix-style builds right now, we need a way to manage the package cache.

[tibbe]

@mightybyte Just use ghc-pkg.

[23Skidoo]

@mightybyte

And if cabal info can get me the information for a single installed package, why can't cabal installed get me a list of the installed packages?

There is cabal list --installed.

[23Skidoo]

If/when we'll have an uninstall command, it will support sandboxes. I've opened a separate ticket about sandbox hc-pkg / sandbox buildopts (#1200). Anything else we want to do here?

[mightybyte]

@23Skidoo Ok, that sounds good to me.

[23Skidoo]

Closing in favor of #1200 and #227.