Haskell source pre-processing in Haskell no longer works

[ygale]

The standard technique for pre-processing Haskell source code using Haskell has stopped working in recent versions of Cabal. The technique is as follows:

The cabal file is structured like this:

Name: foo
...
Executable foo-meta
  ...
Library
  ...
Executable foo
  ...

Then you pass "-F -pgmF dist/build/foo/foo-meta" to GHC for the source file you want to pre-process, e.g., using an OPTIONS_GHC pragma.

The effect is that first cabal builds foo-meta, then uses it to build the library, then builds the final executable.

But now, Cabal refuses to build the artifacts in the order I specified in the Cabal file. It always tries to build the library first, before the foo-meta executable, and the build fails.

Note: We have only tried this so far inside a sandbox. However, we have verified that the problem is definitely that cabal is not even trying to build foo-meta, not a problem with differences in the path to the foo-meta executable in a sandbox.

[23Skidoo]

Can't you just release foo-meta as a separate package?

[ygale]

That's really ugly, for a few reasons.

1. The preprocessing code and the main code are all one code base. Maintaining them as two separate packages is counterintuitive.
2. You need to coordinate the version numbers of the two packages manually to ensure that the preprocessor gets rebuilt, or not, as needed.
3. There is an awkward extra requirement to set up some external relationship between the two packages. For example, you can create sandbox add-source links manually every time you check out the combined project. Or you can give up on using a single cabal install command and write a build script that builds things in the right order. Or upload (manually) the packages frequently to hackage or a local yackage server so that the preprocessor will build. But then what happens if there are code dependencies and you make a breaking change that affects both the preprocessor and the main code at the same time? You can't compile the main code until you have compiled the preprocessor, but you can't compile the preprocessor until you have uploaded the new version of the main code.

This being Haskell, I agree that it would be nice to move away from the imperative feel of requiring the artifacts to built in the order they physically appear in the cabal file. Perhaps we should add some cabal file syntax for expressing a dependency of one artifact on another defined in the same cabal file. In one direction we already have something primitive: an executable can list its own package name as a build dependency, causing the library to be built before this executable.

But in the meantime, cabal has broken behavior that was previously working.

[bermanjosh]

I may have found a bug which is related to why cabal seems to be ignoring the build order.

in: Cabal/Distribution/Simple/LocalBuildInfo.hs, lines 342 - 357:

{-# DEPRECATED withComponentsLBI "Use withAllComponentsInBuildOrder" #-}
withComponentsLBI :: PackageDescription -> LocalBuildInfo
                  -> (Component -> ComponentLocalBuildInfo -> IO ())
                  -> IO ()
withComponentsLBI = withAllComponentsInBuildOrder

-- | Perform the action on each buildable 'Library' or 'Executable' (Component)
-- in the PackageDescription, subject to the build order specified by the
-- 'compBuildOrder' field of the given 'LocalBuildInfo'
withAllComponentsInBuildOrder :: PackageDescription -> LocalBuildInfo
                              -> (Component -> ComponentLocalBuildInfo -> IO ())
                              -> IO ()
withAllComponentsInBuildOrder pkg lbi f =
    sequence_
      [ f (getComponent pkg cname) clbi
      | (cname, clbi) <- allComponentsInBuildOrder lbi ]

Based on the comments of the withAllComponentsInBuildOrder function, it is looking for a field compBuildOrder of the LocalBuildInfo type, which seems to be defined on lines 116-160. However, I don't see a compBuildOrder field in that definition.

data LocalBuildInfo = LocalBuildInfo {
        configFlags   :: ConfigFlags,
        -- ^ Options passed to the configuration step.
        -- Needed to re-run configuration when .cabal is out of date
        extraConfigArgs     :: [String],
        -- ^ Extra args on the command line for the configuration step.
        -- Needed to re-run configuration when .cabal is out of date
        installDirTemplates :: InstallDirTemplates,
                -- ^ The installation directories for the various differnt
                -- kinds of files
        --TODO: inplaceDirTemplates :: InstallDirs FilePath
        compiler      :: Compiler,
                -- ^ The compiler we're building with
        hostPlatform  :: Platform,
                -- ^ The platform we're building for
        buildDir      :: FilePath,
                -- ^ Where to build the package.
        --TODO: eliminate hugs's scratchDir, use builddir
        scratchDir    :: FilePath,
                -- ^ Where to put the result of the Hugs build.
        componentsConfigs   :: [(ComponentName, ComponentLocalBuildInfo, [ComponentName])],
                -- ^ All the components to build, ordered by topological sort, and with their dependencies
                -- over the intrapackage dependency graph
        installedPkgs :: PackageIndex,
                -- ^ All the info about the installed packages that the
                -- current package depends on (directly or indirectly).
        pkgDescrFile  :: Maybe FilePath,
                -- ^ the filename containing the .cabal file, if available
        localPkgDescr :: PackageDescription,
                -- ^ The resolved package description, that does not contain
                -- any conditionals.
        withPrograms  :: ProgramConfiguration, -- ^Location and args for all programs
        withPackageDB :: PackageDBStack,  -- ^What package database to use, global\/user
        withVanillaLib:: Bool,  -- ^Whether to build normal libs.
        withProfLib   :: Bool,  -- ^Whether to build profiling versions of libs.
        withSharedLib :: Bool,  -- ^Whether to build shared versions of libs.
        withDynExe    :: Bool,  -- ^Whether to link executables dynamically
        withProfExe   :: Bool,  -- ^Whether to build executables for profiling.
        withOptimization :: OptimisationLevel, -- ^Whether to build with optimization (if available).
        withGHCiLib   :: Bool,  -- ^Whether to build libs suitable for use with GHCi.
        splitObjs     :: Bool,  -- ^Use -split-objs with GHC, if available
        stripExes     :: Bool,  -- ^Whether to strip executables during install
        progPrefix    :: PathTemplate, -- ^Prefix to be prepended to installed executables
        progSuffix    :: PathTemplate -- ^Suffix to be appended to installed executables
  } deriving (Read, Show)

Sorry if I missed something; I'm still relatively new to Haskell.
Thanks

[dcoutts]

@ygale we already have a system for this, which is the build-tool field. You can use that to list that your lib or exe depends on the build tool defined within the same package.

Now having said that, while I wrote that code I've never actually tested it (or if I did, I don't remember!) So could you give that a go and tell us if it works, and we can close this ticket, or re-purpose it.

And about having broken existing things, we never promised the components would be built in order, and using -F -pgmF dist/build/foo/foo-meta is not actually supported (because we do not promise that the dist dir is called dist, nor the internal layout of files within that dir). This build-tools thing has been there for a while however, so it's safe to use that (assuming it is actually working). The "right way" to do pre-processors within the same package is to use code in Setup.hs to declare the pre-processor.

[23Skidoo]

@dcoutts Yes, this seems to work: https://gist.github.com/23Skidoo/7930870

Leaving the issue open for now because I want to add a test case for this.

[ezyang]

This is a little known feature of Cabal, not mentioned in the manual, and rarely known by many who have this problem in the wild.

Additionally, it would be quite nice if we could support the preprocessor use-case without needing people to add a custom Setup.hs. Perhaps we can support -pgmF style: what if Cabal adds every mentioned build-tool to the PATH?