Skip to content

Cannot install pango with new-build when using local copy of gtk2hs-buildtools #5164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
RyanGlScott opened this issue Feb 24, 2018 · 19 comments

Comments

@RyanGlScott
Copy link
Member

I'm experiencing this issue with both:

$ cabal --version
cabal-install version 2.0.0.1
compiled using version 2.0.1.1 of the Cabal library 
$ cabal-head --version
cabal-install version 2.1.0.0
compiled using version 2.1.0.0 of the Cabal library

To trigger the issue:

$ cabal get pango-0.13.4.0
Unpacking to pango-0.13.4.0/
$ cd pango-0.13.4.0/
$ cabal get gtk2hs-buildtools-0.13.3.1
Unpacking to gtk2hs-buildtools-0.13.3.1/
$ cat > cabal.project << EOF
> packages: .
>           ./gtk2hs-buildtools-0.13.3.1
> EOF
$ cabal new-build all
Resolving dependencies...
In order, the following will be built (use -v for more details):
 - gtk2hs-buildtools-0.13.3.1 (lib) (first run)
 - gtk2hs-buildtools-0.13.3.1 (exe:gtk2hsTypeGen) (first run)
 - gtk2hs-buildtools-0.13.3.1 (exe:gtk2hsHookGenerator) (first run)
 - gtk2hs-buildtools-0.13.3.1 (exe:gtk2hsC2hs) (first run)
 - glib-0.13.5.0 (lib:glib) (requires build)
 - cairo-0.13.4.2 (lib:cairo) (requires build)
 - pango-0.13.4.0 (lib:pango) (first run)
Configuring library for gtk2hs-buildtools-0.13.3.1..
Preprocessing library for gtk2hs-buildtools-0.13.3.1..
<build output elided>
[1 of 1] Compiling Main             ( /home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/setup.hs, /home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/Main.o )

/home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/setup.hs:51:18: warning: [-Wdeprecations]
    In the use of ‘autogenModulesDir’
    (imported from Distribution.Simple.BuildPaths):
    Deprecated: "If you can, use 'autogenComponentModulesDir' instead, but if you really wanted package-global generated modules, use 'autogenPackageModulesDir'.  In Cabal 2.0, we avoid using autogenerated files which apply to all components, because the information you often want in these files, e.g., dependency information, is best specified per component, so that reconfiguring a different component (e.g., enabling tests) doesn't force the entire to be rebuilt.  'autogenPackageModulesDir' still provides a place to put files you want to apply to the entire package, but most users of 'autogenModulesDir' should seriously consider 'autogenComponentModulesDir' if you really wanted the module to apply to one component."
   |
51 |     targetDir  = autogenModulesDir lbi
   |                  ^^^^^^^^^^^^^^^^^

/home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/setup.hs:55:3: warning: [-Wdeprecations]
    In the use of ‘die’ (imported from Distribution.Simple.Utils):
    Deprecated: "Messages thrown with die can't be controlled with Verbosity; use die' instead, or dieNoVerbosity if Verbosity truly is not available"
   |
55 |   die $ "unexpected pango version number: " ++ display version
   |   ^^^
Linking /home/rgscott/Documents/Hacking/Haskell/pango-0.13.4.0/dist-newstyle/build/x86_64-linux/ghc-8.2.2/pango-0.13.4.0/setup/setup ...
Configuring pango-0.13.4.0...
/u/rgscott/.cabal/share/x86_64-linux-ghc-8.2.2/gtk2hs-buildtools-0.13.3.1/hierarchyGen/Hierarchy.chs.template: openFile: does not exist (No such file or directory)cabal: Failed to build pango-0.13.4.0. The failure occurred during the
configure step.

Note that this bug will not be triggered if you have installed gtk2hs-buildtools-0.13.3.1 previously using cabal install, as that will cause ~/.cabal/share/x86_64-linux-ghc-8.2.2/gtk2hs-buildtools-0.13.3.1/hierarchyGen/Hierarchy.chs.template to have been created.

Moreover, this bug does not occur if gtk2hs-buildtools-0.13.3.1 is not put into cabal.project and is instead downloaded from Hackage.

I suspect the issue may have something to do with this line of code in gtk2hs-buildtools-0.13.3.1:

templateFile <- getDataFileName "hierarchyGen/Hierarchy.chs.template"
@RyanGlScott
Copy link
Member Author

I've created https://github.com/RyanGlScott/cabal-gh5164, which minimizes this bug. To use it:

$ git clone https://github.com/RyanGlScott/cabal-gh5164
$ cd cabal-gh5164/
$ cabal new-build all
Resolving dependencies...
In order, the following will be built (use -v for more details):
 - setup-lib-0.1 (lib) (first run)
 - fails-with-new-build-0.1 (lib:fails-with-new-build) (first run)
Configuring library for setup-lib-0.1..
Preprocessing library for setup-lib-0.1..
Building library for setup-lib-0.1..
[1 of 2] Compiling Paths_setup_lib  ( /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/setup-lib-0.1/build/autogen/Paths_setup_lib.hs, /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/setup-lib-0.1/build/Paths_setup_lib.o )
[2 of 2] Compiling SetupLib         ( src/SetupLib.hs, /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/setup-lib-0.1/build/SetupLib.o )
[1 of 1] Compiling Main             ( /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/fails-with-new-build-0.1/setup/setup.hs, /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/fails-with-new-build-0.1/setup/Main.o )
Linking /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/fails-with-new-build-0.1/setup/setup ...
setup: /u/rgscott/.cabal/share/x86_64-linux-ghc-8.2.2/setup-lib-0.1/example.txt: openFile: does not exist (No such file or directory)

@RyanGlScott
Copy link
Member Author

In contrast, here's what happens with cabal install:

$ cabal install setup-lib/
Resolving dependencies...
Configuring setup-lib-0.1...
Building setup-lib-0.1...
Installed setup-lib-0.1
$ cd fails-with-new-build/
$ cabal build
Resolving dependencies...
[1 of 1] Compiling Main             ( dist/setup/setup.hs, dist/setup/Main.o )
Linking ./dist/setup/setup ...
Example data file

Configuring fails-with-new-build-0.1...
Example data file

Preprocessing library for fails-with-new-build-0.1..
Building library for fails-with-new-build-0.1..
[1 of 1] Compiling FailsWithNewBuild ( src/FailsWithNewBuild.hs, dist/build/FailsWithNewBuild.o )

@hvr
Copy link
Member

hvr commented Feb 24, 2018

I believe this to be a long-standing issue w/ local packages, because we construct a Paths_ that points to paths that make no sense for nix-style local builds... but I can't seem to find the issue which contains the discussion (and fwiw, a related issue exists for local build-tools #5104).

So the solution would be either to

  • have Paths_... contain the actual local filepaths, or
  • for cabal new-build to set the respective environment-variables (like it's done e.g. for cabal new-run/new-test) during the custom setup invocations via new-build so the package-data lookup functions will locate the right folder

@RyanGlScott
Copy link
Member Author

Out of curiosity, is there an environment variable that I can set manually to work around this issue?

@hvr
Copy link
Member

hvr commented Feb 24, 2018

Sure, it's documented here:

http://cabal.readthedocs.io/en/latest/developing-packages.html?highlight=datadir#accessing-data-files-from-package-code

So you need to set an env-var named normalised_pkgname_datadir (you can cheat by looking at the autogenerated Paths_....hs source code)

And for reference, this here is the place were e.g. new-runsets those env-vars automatically:

-- | Construct the environment needed for the data files to work.
-- This consists of a separate @*_datadir@ variable for each
-- inplace package in the plan.
dataDirsEnvironmentForPlan :: ElaboratedInstallPlan
-> [(String, Maybe FilePath)]
dataDirsEnvironmentForPlan = catMaybes
. fmap (foldPlanPackage
(const Nothing)
dataDirEnvVarForPackage)
. toList
-- | Construct an environment variable that points
-- the package's datadir to its correct location.
-- This might be:
-- * 'Just' the package's source directory plus the data subdirectory
-- for inplace packages.
-- * 'Nothing' for packages installed in the store (the path was
-- already included in the package at install/build time).
-- * The other cases are not handled yet. See below.
dataDirEnvVarForPackage :: ElaboratedConfiguredPackage
-> Maybe (String, Maybe FilePath)
dataDirEnvVarForPackage pkg =
case (elabBuildStyle pkg, elabPkgSourceLocation pkg)
of (BuildAndInstall, _) -> Nothing
(BuildInplaceOnly, LocalUnpackedPackage path) -> Just
(pkgPathEnvVar (elabPkgDescription pkg) "datadir",
Just $ path </> dataDir (elabPkgDescription pkg))
-- TODO: handle the other cases for PackageLocation.
-- We will only need this when we add support for
-- remote/local tarballs.
(BuildInplaceOnly, _) -> Nothing

@RyanGlScott
Copy link
Member Author

Thanks, that's very helpful.

A related question: does new-build put the data-files somewhere in dist-newstyle? I just searched for example.txt, but couldn't find it anywhere in my dist-newstyle.

@RyanGlScott
Copy link
Member Author

For the time being, I'm finding myself having to hardcode the directory in which the data-files live to make this work. For instance:

$ setup_lib_datadir="$(pwd)/setup-lib" bash -c 'cabal new-build all'
In order, the following will be built (use -v for more details):
 - fails-with-new-build-0.1 (lib:fails-with-new-build) (first run)
Example data file

Configuring fails-with-new-build-0.1...
Example data file

Preprocessing library for fails-with-new-build-0.1..
Building library for fails-with-new-build-0.1..
[1 of 1] Compiling FailsWithNewBuild ( src/FailsWithNewBuild.hs, /home/rgscott/Documents/Hacking/Haskell/cabal-gh5164/dist-newstyle/build/x86_64-linux/ghc-8.2.2/fails-with-new-build-0.1/build/FailsWithNewBuild.o )
Example data file

Example data file

Obviously, this is quite ugly, and won't scale if your data-files are spread out over multiple directories. But since new-build apparently doesn't put data-files into dist-newstyle, I don't see any other way to make this work at the moment.

@hvr
Copy link
Member

hvr commented Feb 24, 2018

A related question: does new-build put the data-files somewhere in dist-newstyle?

Maybe for (non-local) "inplace" packages (I'd have to check), but certainly not for local "inplace" ones.

@RyanGlScott
Copy link
Member Author

Note that stack does the correct thing here:

$ stack build fails-with-new-build                   
setup-lib-0.1: configure (lib)
setup-lib-0.1: build (lib)
setup-lib-0.1: copy/register
fails-with-new-build-0.1: configure (lib)
[1 of 2] Compiling Main             ( /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/fails-with-new-build/Setup.hs, /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/fails-with-new-build/.stack-work/dist/x86_64-linux/Cabal-2.0.1.0/setup/Main.o )
[2 of 2] Compiling StackSetupShim   ( /home/rgscott/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs, /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/fails-with-new-build/.stack-work/dist/x86_64-linux/Cabal-2.0.1.0/setup/StackSetupShim.o )
Linking /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/fails-with-new-build/.stack-work/dist/x86_64-linux/Cabal-2.0.1.0/setup/setup ...
Example data file
             
Configuring fails-with-new-build-0.1...
fails-with-new-build-0.1: build (lib)
Example data file
             
Preprocessing library for fails-with-new-build-0.1..
Building library for fails-with-new-build-0.1..
[1 of 1] Compiling FailsWithNewBuild ( src/FailsWithNewBuild.hs, .stack-work/dist/x86_64-linux/Cabal-2.0.1.0/build/FailsWithNewBuild.o )
fails-with-new-build-0.1: copy/register
Example data file
             
Installing library in /home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/lib/x86_64-linux-ghc-8.2.2/fails-with-new-build-0.1-BoDEqEiYoIHEwENsULGL10
Example data file
             
Registering library for fails-with-new-build-0.1..
Completed 2 action(s).
$ more ./setup-lib/.stack-work/dist/x86_64-linux/Cabal-2.0.1.0/build/autogen/Paths_setup_lib.hs
{-# LANGUAGE CPP #-}
{-# OPTIONS_GHC -fno-warn-missing-import-lists #-}
{-# OPTIONS_GHC -fno-warn-implicit-prelude #-}
module Paths_setup_lib (
    version,
    getBinDir, getLibDir, getDynLibDir, getDataDir, getLibexecDir,
    getDataFileName, getSysconfDir
  ) where

import qualified Control.Exception as Exception
import Data.Version (Version(..))
import System.Environment (getEnv)
import Prelude

#if defined(VERSION_base)

#if MIN_VERSION_base(4,0,0)
catchIO :: IO a -> (Exception.IOException -> IO a) -> IO a
#else
catchIO :: IO a -> (Exception.Exception -> IO a) -> IO a
#endif

#else
catchIO :: IO a -> (Exception.IOException -> IO a) -> IO a
#endif
catchIO = Exception.catch

version :: Version
version = Version [0,1] []
bindir, libdir, dynlibdir, datadir, libexecdir, sysconfdir :: FilePath

bindir     = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/bin"
libdir     = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/lib/x86_64-linux-ghc-8.2.2/setup-lib-0.1-8hekJdedMBc9fTEHUxKQOX"
dynlibdir  = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/lib/x86_64-linux-ghc-8.2.2"
datadir    = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/share/x86_64-linux-ghc-8.2.2/setup-lib-0.1"
libexecdir = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/libexec/x86_64-linux-ghc-8.2.2/setup-lib-0.1"
sysconfdir = "/home/rgscott/Documents/Hacking/Haskell/sandbox/cabal-gh5164/.stack-work/install/x86_64-linux/lts-10.7/8.2.2/etc"

getBinDir, getLibDir, getDynLibDir, getDataDir, getLibexecDir, getSysconfDir :: IO FilePath
getBinDir = catchIO (getEnv "setup_lib_bindir") (\_ -> return bindir)
getLibDir = catchIO (getEnv "setup_lib_libdir") (\_ -> return libdir)
getDynLibDir = catchIO (getEnv "setup_lib_dynlibdir") (\_ -> return dynlibdir)
getDataDir = catchIO (getEnv "setup_lib_datadir") (\_ -> return datadir)
getLibexecDir = catchIO (getEnv "setup_lib_libexecdir") (\_ -> return libexecdir)
getSysconfDir = catchIO (getEnv "setup_lib_sysconfdir") (\_ -> return sysconfdir)

getDataFileName :: FilePath -> IO FilePath
getDataFileName name = do
  dir <- getDataDir
  return (dir ++ "/" ++ name)

@hvr
Copy link
Member

hvr commented Mar 1, 2018

@RyanGlScott this looks a lot like what the old cabal sandbox did as well...

@RyanGlScott
Copy link
Member Author

Indeed, I can confirm that this works with sandboxes as well.

This is an annoying enough issue (I experience it several times per day) that I'm inclined to figure out how to fix it. stack fixes this issue by wiring in a different datadir. Is there an equivalent code path for new-build that would accomplish the same thing?

@hvr
Copy link
Member

hvr commented Mar 1, 2018

Well, first we'd need to agree which of the two variants mentioned in my comment #5164 (comment) we want to pursue (there's a reason we didn't just go w/ the obsolete cabal sandbox approach, but I don't remember the details); I remember some discussion about this in another ticket, and it would be worth locating that again to learn about the conclusion from there.

PS: related #4120 and #4558 (comment) (but I don't think that's the discussion I was thinking of)

/cc @phadej @23Skidoo

@23Skidoo
Copy link
Member

23Skidoo commented Mar 1, 2018

  • for cabal new-build to set the respective environment-variables (like it's done e.g. for cabal new-run/new-test) during the custom setup invocations via new-build so the package-data lookup functions will locate the right folder

I'm +1 on this idea, what are the arguments against?

@RyanGlScott
Copy link
Member Author

I'd be in favor of that as well.

@hvr
Copy link
Member

hvr commented Mar 4, 2018

@23Skidoo Can't think of any showblocker arguments against either.

@RyanGlScott
Copy link
Member Author

Great! Now the question arises of how to implement this.

I'm a neophyte when it comes to the Cabal codebase—are there any places where we currently set these environment variables? And where in cabal new-build would I need to set them?

@23Skidoo
Copy link
Member

23Skidoo commented Mar 8, 2018

@RyanGlScott

are there any places where we currently set these environment variables?

new-run does it in https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/CmdRun.hs#L217

And where in cabal new-build would I need to set them?

https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/ProjectBuilding.hs#L930 is the place where new-build runs the compiled setup script.

@RyanGlScott
Copy link
Member Author

A question for the Cabal folks. I looked into implementing this recently and was presented with a design concern that scared me. Presumably, we're going to want to set these environment variables with something like withEnv. But as the comment for that notes:

-- Warning: This operation is NOT thread-safe, because current
-- environment is a process-global concept.

This is pretty worrying—if we're going to be building things in parallel (which is often the case), it means that we don't have a particularly strong guarantee that when we build a particular component, <component>_datadir et al. will be set to the right thing.

Note that new-run doesn't suffer from this issue since it runs an executable in a separate process, so it only needs to set environment variables once. In a new-build sort of situation, however, we might need different <component>_datadir values for each thread.

Thoughts?

@23Skidoo
Copy link
Member

Setting the vars should be fine when using the external setup method (running a compiled setup script) or the self-exec setup method (cabal act-as-setup), because in both cases you're running a separate process. IIUC, this bug manifests only when using build-type: Custom, which means the former.

Though you're right about withEnv, you shouldn't use it. Instead, pass the env arg to runProcess in D.C.SetupWrapper when running the setup script. Adding a useExtraEnvVars setting to SetupScriptOptions is IMO the logical place for this, there's already a useExtraPathEnv setting.

23Skidoo added a commit that referenced this issue Mar 27, 2018
Fix #5164 by overriding environment variables when new-building custom Setups
23Skidoo added a commit that referenced this issue Mar 27, 2018
Fix #5164 by overriding environment variables when new-building custom Setups

(cherry picked from commit 6ca34b0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants