Draft announcement for cabal new-build

[ezyang]

I plan on posting this to my blog. Comments welcome. (There are some minor formatting wibbles because I didn't bother converting my ReST to Markdown.)

CC @hvr @dcoutts @bgamari @23Skidoo

Rendered: https://github.com/ezyang/cabal/blob/pr/nix-announce/nix-announce.rst

[23Skidoo]

Nice write-up. The only issue blocking the release is #3199, I think I'll be able to fix it this week.

[dcoutts]

Probably it's easier to tell people to list the dir rather than the .cabal file. Either are possible of course.

packages: Cabal/
          cabal-install/

[dcoutts]

For example, cabal new-build Cabal:package-tests exe:cabal --enable-tests requests two components be built: the component package-tests from the package Cabal, and the executable named cabal (which happens to be from the package cabal-install).

I don't particularly want to encourage the --enable-tests approach, it's still there if you need it but I think the approach we want to encourage is just asking to build what you want to build, rather than doing it indirectly.

So you can just do cabal new-build Cabal:package-tests exe:cabal or simpler cabal new-build package-tests cabal

The workflow I'm aiming for here is that that's the first thing people will go for, and it'll tell them if 1. the targets were ambiguous and if so what the more qualified syntax for the choices would be (this works already) and 2. if the testsuites were not enabled optimistically that it tells users they can try --enable-tests for that package or the command line or in the cabal.project file (this bit is only partially implemented).

The only time you really need to specify --enable-tests is if the solver was not able to optimistically enable the testsuite. Adding --enable-tests forces the solver to either enable them or to fail. So it's possible it'll either fail or pick a different set of libs than it would otherwise.

Perhaps it's confusing to have --enable-tests do two things: force the solver to solve for them, and also add the tests to the "default" package target. Perhaps it should only do the first thing. I had it do both since that's closer to the old/classic behaviour.

[hvr]

@dcoutts btw, a wildcard syntax cabal new-build test:* or more explicitly, something like cabal new-build mypackage:test:* to denote building all tests contained in mypackage would be convenient...

[lspitzner]

• might be worth mentioning ~/.cabal/store and ghc-pkg list --package-db=$HOME/.cabal/store/ghc-version/package.db/
• might be worth to include git clone https://github.com/haskell/cabal.git --branch="1.24" --depth=1 cabal-new-build. Or without the depth.
• how do i achieve the equivalent of sandbox add-source? just building the dependency, hoping it would end up in the user-global store did not seem to work out. adding the local directory does not either (i'll create an issue for that one, one sec).

I don't like the tl;dr. Is this an announcement of a tech-preview or of an almost-release of a feature truly capable of replacing sandboxes/stack? But then i don't know how much you plan to implement in the time until the announcement, or maybe my first impression of new-build was just bad.

I am grateful for the work you put into this, and in time i am sure it will become a very useful feature.

[kosmikus]

cabal new-build

[Sorry, my very first point is bikeshedding, and not your fault.]
I've said it many times before, I think new-build is an unforgivably bad name for this, even if it's a tech-preview. If we wan't find a short name, then why not nix-build?

(it's missing commands and no user interface for freezing dependency solver results)

I find this difficult to parse.

As of writing, cabal-install 1.24 has not yet been uploaded to Hackage, but you can download a prerelease off of GitHub.

Do you really want to post this prior to the release? I think it's much better to have "lots" of people look at it once 1.24 is released, and not at some random point before.

In addition, I think you may want to highlight a few of the cool implications this has, such as finally being able to enable profiling "later" and just have it do the right thing.

Apart from these points, great post.

[ezyang]

@dcoutts Oh! Then I have some sort of mysterious bug where I can't remove --enable-tests from my invocations of new-build without breaking the build; I get this error:

ezyang@sabre:~/Dev/cabal-shake$ cabal new-build Cabal:package-tests cabal-install:unit-tests --with-ghc ghc-shake   --disable-optimization 
Resolving dependencies...
In order, the following will be built (use -v for more details):
Cabal-1.25.0.0
hackage-security-0.5.0.2
unbounded-delays-0.1.0.10
tasty-0.11.0.2
tasty-hunit-0.9.2
tasty-quickcheck-0.8.4
cabal-install-1.25.0.0
Configuring Cabal-1.25.0.0...
cabal: Encountered missing dependencies:
QuickCheck >=2.7 && <2.9,
exceptions -any,
old-time -any,
regex-posix -any,
tagged -any,
tasty -any,
tasty-hunit -any,
tasty-quickcheck -any,
transformers -any

I don't quite have a reproducible test case yet.

@lspitzner Thanks for the comments. It's very useful to hear non power user impressions of the feature.

how do i achieve the equivalent of sandbox add-source? just building the dependency, hoping it would end up in the user-global store did not seem to work out. adding the local directory does not either (i'll create an issue for that one, one sec).

That would be adding it to the list of packages in cabal.project. I bet this section can be made clearer.

I don't like the tl;dr. Is this an announcement of a tech-preview or of an almost-release of a feature truly capable of replacing sandboxes/stack? But then i don't know how much you plan to implement in the time until the announcement, or maybe my first impression of new-build was just bad.

I think the current code in 1.24 is what is going into the release, unless we decide to blitz a raft of UI improvements in the next few days (maybe the right call!) And is it a tech preview or a feature release? That's a good question.

Here's what I know:

• new-build works well enough for me to do all of my Cabal development in it. So it is good enough for bootstrapping.
• There ARE bugs, and we are not going to be able to fix them all in time for 1.24 (just look at the list of issues I've filed!)
• I (personally) have not used new-build on projects beyond cabal-install. I think it should work OK. Maybe others have used it on different, bigger projects. But it would not be surprising if things broke in fascinating ways.
• I know the UI is not there. There is no documentation. Everything I know about new-build I figured out by reading the source.
• I'm pretty sure the UI is going to change. At the very least, we're not going to keep calling this new-build. Eventually new-configure/new-build should just replace configure/build.

So, is tech preview the right moniker for this? I guess! But from my perspective, for every new Cabal based project I am going to do development on, I am going to use new-build for it. Maybe I'm an outlier, because I know enough about the codebase to know when things are bugs and usefully contribute bugfixes.

[hvr]

@lspitzner

how do i achieve the equivalent of sandbox add-source? just building the dependency, hoping it would end up in the user-global store did not seem to work out.

local packages should never end up in the user-global store (~/.cabal/store), local packages (and non-local packages depending on local packages!) end up in ./dist-newstyle instead

there's also no explicit cabal install-equivalent to manage the global store; the user-global store/cache is only populated as a side-effect, and there's IMO little reason you'd ever need to manage it explicitly (other than maybe to garbage-collect it) -- in fact, the install plan solver should never be affected by the user-global store state. This is a big departure from the cabal install-era where the state of your user package db affected the solution the cabal solver would find.

[lspitzner]

@dcoutts after some more playing around, the main problem seems to have been documentation; i had to guess about what is supposed to work and how i would observe that it worked correctly.

@hvr thanks, that the /store/ only is for non-local dependencies was an important insight. On learning that i could indeed query things with ghc-pkg --package-db …, i had been confused that the db did not contain my foo package after ((seemingly) successfully) running new-build in the directory containing foo.cabal.

Properly setting up a cabal.package, as described in the draft, worked just fine.

All in all my first impression was "just" bad (and a bit unlucky; i think there is a bug to the effect that gtk3 cannot be installed as a dependency, will report separately). Now that i have an idea of how things are supposed to work, i am more optimistic about releasability, given proper documentation in general.

[ezyang]

D'oh! The "Encountered missing dependencies" bug is #3199. The workaround is to pass --enable-tests, which forces cabal-install to configure all components and thus works around the problem.

[dcoutts]

wildcard syntax cabal new-build test:* or more explicitly, something like cabal new-build mypackage:test:* to denote building all tests contained in mypackage would be convenient...

@hvr I'm a bit reluctant to use * since without a blah: qualifier it'll get expanded by the shell. Any other obvious syntax or convention? Or would it always always be used with a qualifier so we don't accidentally guide users into doing build *.

[hvr]

@dcoutts then what about leaving off the *, as in cabal new-build test: / cabal new-build mypackage:test:? or use an _ instead of *

[hvr]

@ezyang btw, you didn't mention cabal new-repl (nor new-configure managing the cabal.project.local file)

incomplete example of what's possible within cabal.project (@dcoutts may want to chime in...)
(empirical way to find out syntax is to pass flags to new-configure and look at resulting cabal.project.local file)

packages: */*.cabal
-- general settings
with-compiler: /opt/ghc/8.0.1/bin/ghc
constraints: cryptohash < 0.11.8
benchmarks: False

-- per package settings
package cryptohash
  profiling: True   

package bytestring
  flags: -integer-simple

[ezyang]

OK, I incorporated all of the comments, and also put it in a PR so that you can view the rendered rst. See here: https://github.com/ezyang/cabal/blob/pr/nix-announce/nix-announce.rst

[hvr]

@ezyang

• re cabal.project: it may also be noteworthy that now you can call cabal build exe:cabal from any subfolder, e.g. in $toplevel/cabal-install/Distribution/Client (afaik with build you had to be in the project-top-level-folder)

• even though there's no new-clean yet, its trivial form can be emulated by rm -rf $toplevel/dist-newstyle (this is needed sometimes if new-build gets confused about its state, not often but still...)

• -- So cryptohash will get its profiling suites built the term seems strange; sounds like bench/test suite, I would have written "...will be built with profiling support" -- but tbh, this is a bad example, as you typically need all your deps built with profiling enabled... I rather meant to use a different example:

  set profiling: True in the general section, and tweak the --profiling-detail=level flag, i.e.

  -- ...
  profiling: true
  
  package cryptohash
      profiling-detail: all-functions
      optimization: False
      -- ghc-options: ...
  

PS: the allow-newer: property is also a neat use-case for cabal.project, here's Cabal's current cabal.project as an example (we also want to allow per-package allow-newer syntax to implicitly set the allow-newer-scope to the current package):

packages: Cabal/ cabal-install/

-- Uncomment to allow picking up extra local unpacked deps:
--optional-packages: */

-- current hackage-security release needs Cabal < 1.25
allow-newer: hackage-security:Cabal

[grayjay]

D'oh! The "Encountered missing dependencies" bug is #3199. The workaround is to pass --enable-tests, which forces cabal-install to configure all components and thus works around the problem.

That might be a different issue, since #3199 should only affect components that use Buildable: False.

[hvr]

@kosmikus

I've said it many times before, I think new-build is an unforgivably bad name for this, even if it's a tech-preview. If we wan't find a short name, then why not nix-build?

As I'm not sure I remember the reasons you put forward for new-build being a bad name, why do you consider new-build a bad name?

(also, what about the dist-newstyle-folder naming?)

Btw, here's some names (modulo suffix/prefix swap)

• new-build (for completeness...)
• novel-build 😆
• nix-build
• ng-build
• tng-build 🌟 🚀 🌟
• nextgen-build
• x-build
• build2
• build-newstyle (in line with dist-newstyle)
• tp-build (tp = tech-preview)
• local-build
• modern-build
• adv(anced)-build

[kosmikus]

I think that naming anything "new" is always strange. I've fallen into this trap myself in the past (I named one of the modes newcode, and it's still there). The word "new" doesn't convey any real meaning. It doesn't even tell you it's better, just that it's different. Also, there's the danger despite good intentions that the name will stick, and then you'll have it around long after it's new. I've made this mistake myself in the past (I named one of the modes of lhs2tex newcode, and it's still there).

For the same reasons, most of your list is also bad, with the exception of local-build.

Then again, if you believe that picking an ugly enough name now is going to absolutely ensure we properly fix the interface later, I'm happy to wait and see if this will indeed happen :)

[kosmikus]

(And yes, I'm not overly happy with dist-newstyle either, even less so because we'll probably want to keep a different name from dist for this for all eternity.)

[hvr]

@ezyang I've adapted the PPA-based .travis.yml generator to use new-build (including travis-caching support): https://github.com/hvr/multi-ghc-travis/blob/master/make_travis_yml_2.hs

[lspitzner]

@ezyang announcement looks good! hope is was not too negative before.

i have made an overview about old-vs-new style, and the new-style semantics (all in the currently observed state). Not meant for the announcement, but perhaps useful as an overview regarding feature-completeness.

[gbaz]

It is hard to look this far ahead, but in my experience working on long-lived software project I have learned "never ever name anything 'new X'" because one day, in the future you don't imagine, there will be a newer X. and then what do you name that? a name descriptive of what it is is much more resilient under future unanticipated changes.

[dcoutts]

The rationale for "new-blah" was the thought that we would not have these in parallel in any official encouraged use case (beyond tech preview testing etc), that once it's ready it would simply replace the normal commands (possibly with some flag for the old impl for a while until we can kill it off).

Calling it something sensible makes it look like we plan to have them in parallel for a while, or worse that it's yet another extension to an already complex CLI, like the sandbox stuff.

I'm not the only decision maker here though, so perhaps we should discuss the transition plan.

[ezyang]

I updated the draft with all comments, in particular incorporating in @lspitzner's notes (and expanding and elaborating them a bit.)

[hvr]

@ezyang I'd change

git clone https://github.com/haskell/cabal.git \
    --branch="1.24" cabal-nix-local-build
cd cabal-nix-local-build

into something like

git clone https://github.com/haskell/cabal.git \
    --branch="1.24" cabal-1.24.x
cd cabal-1.24.x/

to avoid any confusion about where that nix-local-build term comes from

[ezyang]

@dcoutts OK, I think I understand the current algorithm for determining if a test should be enabled, and I think that it is a better experience for users if they pass --enable-tests if they know they want to run tests.

Here's what's going on:

1. It is true that the solver attempts to solve for packages with all stanzas enabled. Empirically, if I solve for Cabal, I will get back a solution which has the test suite enabled.
2. However, suppose I type cabal new-build Cabal with a blank global store. If the planner decides to configure the package with tests, it will have to go off and install of the test-suite dependencies to appease the configuration phase of Cabal. So the project planner has a hack, where it only ACTUALLY enables the tests if doing so would not force us to install any new packages (or a user actually asked for Cabal:package-tests to be built).

OK, so now here is a user interaction which results in bad behavior:

1. A user runs cabal new-build Cabal because they want to just build the library. Due to the sticky heuristic above, Cabal is configured WITHOUT enabling tests.
2. They try to run the tests, but they notice that it didn't actually get built. So they run cabal new-build Cabal:package-tests. Now that tests are explicitly enabled, Cabal must reconfigure with tests.
3. In Cabal 1.24, there is still a bug that version_macros.h record package global information. Thus the reconfiguration will modify this file with extra information about the test suite dependencies. Consequently, most of Cabal needs to be rebuilt upon the reconfiguration.

All-in-all, if you know you are going to want to run the tests, it seems better to just pass --enable-tests from the getgo and eat the cost of installing the test suite dependencies and avoid an expensive rebuild later.

UPDATE: However, it is true that on master, the experience is a lot better, because the reconfigure will probably not force a rebuild.

[cartazio]

Question: is the cabal.project file or 1.24 style cabal install able to express executable dependencies?

[23Skidoo]

@ezyang Just asked @rthomas to cut the first release of Cabal/cabal-install from the 1.24 branch.

[hvr]

@cartazio

Question: is the cabal.project file or 1.24 style cabal install able to express executable dependencies?

Are you asking about something like #2965 (comment) ?

[cartazio]

@hvr yesss

[ezyang]

No, I haven't gotten around to implementing that feature.

[hvr]

Fyi, @rthomas just uploaded Cabal-1.24 & cabal-install-1.24 to Hackage... so I guess we should look into finishing this up

[ezyang]

Out! http://blog.ezyang.com/2016/05/announcing-cabal-new-build-nix-style-local-builds/