Rebuild source directories added to the sandbox.

[23Skidoo]

Adds an offline mode for the install command and an onlyDepsOf flag that allows to remove a subset of targets from the install plan (see also my message to the mailing list).

Rebuilding of add-source deps themselves is implemented by running

install --offline --only-dependencies-of="." "." /path/to/add-source-dep-1 ... /path/to/add-source-dep-N

before cabal build.

[tibbe]

@dcoutts I'd really like your input on this one.

@23Skidoo To avoid UI discussion issues, couldn't this all be done internally, instead of calling some command with some new command line flags?

Here's how I imagine what needs to be done. Each of these could be implemented as a function or two:

1. Create an install plan for the package (just like cabal install already does).
2. Remove the package itself (just like --only-dependencies).
3. Remove already installed packages (just like cabal install already does).
4. Figure out which add-source dependencies need to be rebuilt. This is currently all dependencies as we don't track enough information to know what to rebuild.
5. Find all reverse dependencies (except the main package) of the add-source dependencies and add them to the plan.
6. Run cabal install --force-reinstalls on this set of packages.

I realize that this is probably what you're already doing (I haven't reviewed the code in depth yet). I think it would be useful to be able to do this internally in cabal, by calling functions such as reverseDepedenciesOf :: Package -> [Package] and addToInstallPlan :: Plan -> Package -> Plan or something along those lines.

[23Skidoo]

To avoid UI discussion issues, couldn't this all be done internally, instead of calling some command with some new command line flags?

This is basically how it is done. In sandboxBuild, I call Distribution.Client.Install.install and set installFlags to

let installFlags = mempty { installOfflineMode       = Flag True,
                            installOnlyDepsOf        = Flag ["."] }

Now, installOfflineMode is exposed as the --offline option for install, but this is not really necessary for my code to work (I just decided that it might be useful). installOnlyDepsOf is not exposed at all since it is so esoteric.

[tibbe]

@23Skidoo What could happen if we didn't use the --offline flag?

[23Skidoo]

What could happen if we didn't use the --offline flag?

The --offline flag prohibits downloading packages from the Internet. Without --offline, the first run of cabal sandbox-build is equivalent to cabal sandbox-install --only-dependencies && cabal build.

[tibbe]

The --offline flag prohibits downloading packages from the Internet. Without --offline, the first run of cabal sandbox-build is equivalent to cabal sandbox-install --only-dependencies && cabal build.

OK. So I guess we want this behavior until/if we decide to have build imply installing dependencies (from the internet).

[23Skidoo]

OK. So I guess we want this behavior until/if we decide to have build imply installing dependencies (from the internet).

Yes, then it won't be needed (though we might allow the users to enable offline mode explicitly).

[tibbe]

Yes, then it won't be needed (though we might allow the users to enable offline mode explicitly).

Yes. Explicitly disabling package downloads using a flag is something that would be nice to support.

[23Skidoo]

Regarding this part of your comment:

1. Figure out which add-source dependencies need to be rebuilt. This is currently all dependencies as we don't track enough information to know what to rebuild.
2. Find all reverse dependencies of the add-source dependencies and add them to the plan.

This is done by the solver itself because the current package and all add-source dependencies are install targets. The only thing that remains is to remove the current package from the install plan (achieved with --only-dependencies-of).

[tibbe]

Sorry for the radio silence on this issue. I'm still trying to wrap my head around it. There's something about the way we're implementing this that doesn't rub me the right way. Let me try to explain my current thinking:

I think (part of) the issue is that cabal install is context sensitive. cabal install --only-dependencies only make sense relative to some current working directory that you're currently in. This gives rise to weird cases. For example, what does

cabal install --only-dependencies foo

mean if the current package doesn't depend on foo? Then we're no longer installing "only" dependencies. In fact, I don't think the --only-dependencies flag has any effect once you start listing explicit package names (this is the weird mode switch from being about the current package to being about other packages cabal does).

I think what we're actually doing when we're rebuilding add-source packages is:

cabal install add-source1 add-source2
cabal install --reverse-dependencies add-source1 add-source2 --exclude=.

I don't have a good understanding if the above --reverse-dependencies flag is hard to implement or not. I think it would be useful even for people who don't use sandboxing (e.g. when they upgrade bytestring) and there's precedence for it in e.g. Gentoo (which has a revdep-rebuild command). There's also an issue of what happend if --exclude excludes a package that's needed by some other reverse dependency (that will then fail to build).

[dcoutts]

Isn't what we want to do, to make an install plan for the whole sandbox, so that we know we're always doing everything consistently. Then we prune that install plan graph back to the dependencies of the target(s).

I don't see why we need an offline mode for that (though I have considered it in the past as an independently useful feature, but I'm pretty sure it ought to be orthogonal here). I also don't think we need to add any command line flags for this. We ought to be able to do it just as manipulations of the install plan.

[dcoutts]

I think it might help if you called the individual parts/phases of install directly, rather than going via the high-level install action.

That is, call the planner directly, get back the InstallPlan, transform the InstallPlan and then use the existing machinery for executing the modified InstallPlan. So we should export more of the things from the Install module, and we'll have to add some queries to the InstallPlan module, so that it becomes possible to implement the transforms in terms of its api. (InstallPlan transforms should still go via the dynamic validity check.)

[23Skidoo]

@tibbe

In fact, I don't think the --only-dependencies flag has any effect once you start listing explicit package names

No, it has the same effect - create an install plan and then prune the install targets:

$ cabal install --only-dependencies --dry-run ghc-core
Resolving dependencies...
In order, the following would be installed (use -v for more details):
ansi-terminal-0.5.5
haskell-lexer-1.0
colorize-haskell-1.0.1
pcre-light-0.4

There's also an issue of what happend if --exclude excludes a package that's needed by some other reverse dependency (that will then fail to build).

This will make the install plan invalid and install will exit with an error.

I don't have a good understanding if the above --reverse-dependencies flag is hard to implement or not.

If someone can point me in the right direction, I can try my hand at it.

@dcoutts

I don't see why we need an offline mode for that

So that to prohibit build to access Internet. It's not strictly necessary.

Isn't what we want to do, to make an install plan for the whole sandbox, so that we know we're always doing everything consistently.

Can you advise me how to implement this without unnecessarily reinstalling everything? By default, we already refuse to proceed if the reinstall is likely to break something.

I also don't think we need to add any command line flags for this. We ought to be able to do it just as manipulations of the install plan.

Just to clarify: the only command-line flag I added is --offline. installOnlyDepsOf is a member of the InstallFlags record that is only accessible internally.

I think it might help if you called the individual parts/phases of install directly, rather than going via the high-level install action.

Maybe. Should I break up install into a set of smaller functions and remove installOnlyDepsOf?

[tibbe]

Maybe. Should I break up install into a set of smaller functions and remove installOnlyDepsOf?

I think so. I think we want to create the functions necessary to be able to create the InstallPlan we need directly, without jumping through hoops such as --force-reinstalls etc. To do that we need some API on top of InstallPlan, such as addReverseDepedenciesOf and removePackage (these are just examples). The implementation of build for the sandbox would call some of these and create an InstallPlan that contains exactly what we want to build and then hand it off to some other part of cabal that builds it.

[23Skidoo]

I think so. I think we want to create the functions necessary to be able to create the InstallPlan we need directly, without jumping through hoops such as --force-reinstalls etc.

OK. It will take me some time.

[tibbe]

OK. It will take me some time.

I appreciate that this is more difficult and perhaps not as easy as you hoped, but I do think it's the right thing to do and it will make cabal better overall.

[23Skidoo]

Should I still leave the --offline install option in?

[23Skidoo]

@dcoutts
I've removed the installOnlyDepsOf flag and made the InstallPlan manipulations more explicit (turned out to be easier than I expected). Please tell me if there's something more to be done.

[tibbe]

debug doesn't add an extra newline. Do we really want to add one here?

[23Skidoo]

debug itself doesn't, but wrapText does.

[tibbe]

Overall the change looks good to me. @dcoutts could you also take a look please? Also @kosmikus as well?

[tibbe]

I'm inclined to merge this. @dcoutts any objections?

[dcoutts]

I don't see why we're moving the 'install' code into Main. It's fine to export the component parts from the Install module, but why move the main part? The general organisation is that the Main module really only deals with command line args, config files etc, and the logic of each command lives in the corresponding module. I think it's sensible to keep that.

[dcoutts]

I still don't think we need an offline mode for this. The way I think it should work is that we construct the install plan when we configure, then we keep that install plan as long as it remains valid (ie until someone changes one of the .cabal files in the source trees). Then build always just uses the same install plan, but we adjust which bits of it we actually want to build, given the current target(s).

That said, an offline mode is independently useful, though I think we would implement it differently. We'd tell the solver to avoid all packages that are not already fetched.

Ok, lets talk interim solutions since perhaps my suggestion to shift the planning into the configure phase and reusing the same plan (without re-running the solver) during build is too big a step to do immediately. Lets commit what we have now with the following changes:

• no offline mode
• keep the existing install code in the Install module (the exporting of more stuff is fine)

[23Skidoo]

@dcoutts I've added offline mode because @kosmikus was against having commands other than cabal install download packages from the Internet. This can happen e.g. if the user edits the .cabal file of an add-source package and adds a new dependency. The next cabal build will fetch and install that dependency into the sandbox. If this behaviour is considered acceptable, the offline mode can be dropped.

[dcoutts]

In a context where we're handling a whole set of local packages, I don't see a sensible alternative to making 'build' build dependencies and that may involve getting them from the network. Perhaps we could move the network part earlier to the configure phase, so it fetches them at that point. That may well be a good idea but that doesn't seem like a very significant difference, since if you do a 'build' and the configuration is out of date (because the user changed a .cabal file) then we're going to want to automatically reconfigure anyway.

If we default to an offline mode by default for that, then it just means we're likely to fail instead. As a longer term thing, I've been thinking of making some of the UI interactive by default, so that it gets confirmation for some things, like confirming when new packages will be installed.

[23Skidoo]

I'll remove the offline mode then.

[23Skidoo]

@dcoutts @tibbe I've updated my patches. Main changes:

• Offline mode removed
• Added an alternative interface for D.C.Install.install that allows to manipulate the install plan

[tibbe]

A TODO for the future: we probably want to make InstallContext and InstallArgs into proper data types with documented fields.

[23Skidoo]

Thought about this. Not that hard to change.

[tibbe]

Overall the change looks good to me.

[dcoutts]

Ok, looks fine to me too.

About those type aliases, they're fine when the thing is basically an internal api, but we could look at them again when we consider making it more of a public api. For now it's fine.