Check for required C libraries during configure

[bos]

(Imported from Trac #262, reported by guest on 2008-03-25)

Cabal should check for required C libraries and other non-Haskell dependencies at the configure stage,
rather than just waiting for the linker to fail with an obscure error message during the build stage.

Error messages should be helpful for the typical Cabal user, who should not be assumed
to know anything about C libraries or linking.

That is not so simple. For example:

The haddock package current requires the C libraries for readline, due to its dependency
on the GHC API. If that library is missing, the user should have some way of figuring out how
do the right thing on the given platform - for most Linux distros, install the readline-dev
package (as opposed to just readline), for Mac OS X download the GNUreadline framework
from the GHC website and drag it into the appropriate Frameworks folder depending on whether
this is Tiger and below or Leopard and above, etc.

[bos]

(Imported comment by guest on 2008-03-25)

See also:

• fix unix dependency for ghc 6.2 #15 and sdist puts Happy-generated .hs files in the wrong place #159 - implement dependency analysis
• allow per-package configuration options in config file #216 - specify more kinds of dependencies in .cabal files
• cabal-install should report build results to hackage server #177 - check for required directories

[bos]

(Imported comment by @dcoutts on 2008-03-25)

I don't think the dependency analysis tickets are relevant here. Just knowing the required C headers and C libs and the include and library search paths is enough information to do the checks. It's just the same check that autoconf does and we can do it in a similar way.

I'm not sure that being able to specify the foreign package dependencies would help here as we don't have a platform independent way of checking for them anyway. Those are more hints for package managers I'd say.

So we should look up exactly how autoconf does these checks. My understanding is:

• for headers it makes a small .c file that #includes the header and calls gcc/cpp to process the file. It passes the extra include search dirs with the -I flag.
• for libs it makes a small .o file and calls gcc/ld with the -l and -L flags.

If they fail then it means we cannot find the header/lib in question.

[bos]

(Imported comment by @dcoutts on 2008-03-25)

I've been thinking about this and I think that with packages that use confiure scripts there is actually no point at which we can do this check. The reason is that we need any extra settings from the .buildinfo file, however that file is generated by ./configure and the Setup.hs script never sees it.

In fact the first point we could see the full information is during preBuild, we could not do the check during configure at all.

We could try and hack it by reading the .buildinfo file directly during postConf but that's pretty hairy since the user can modify the hooks however they like and there is no requirement that they use any particular file.

So we may have to only do this check for build-type: Simple.

[bos]

(Imported comment by alexeevg on 2009-01-16)

we should look up exactly how autoconf does these checks

I started looking into it and stumbled upon a little problem. Maybe it's a silly question, but I'm going to ask it anyway. Autoconf uses AC_CHECK_LIB macro to check if the library is present. AC_CHECK_LIB takes the library name and a function(symbol) name that is expected to be exported from the library, generates a C program depending on this symbol and tries compiling it. This is a bit of info we miss in BuildInfo? - we have extra-libs, but no function names.

Error messages should be helpful for the typical Cabal user, who should not be assumed to know > anything about C libraries or linking.

This is a bit unclear to me - we have no info on required packages, we have only includes/extra-libs, how do we make error messages user-friendly? Autoconf doesn't do it.

[bos]

(Imported comment by alexeevg on 2009-01-22)

Answering my own question:

This is a bit of info we miss in `BuildInfo?`? - we have extra-libs, but no function names.

Hmm, it seems that Autoconf is a little paranoid here, ld will fail if library is not found regardless of whether or not the program depends on it.

[bos]

(Imported comment by @dcoutts on 2009-01-22)

Replying to alexeevg:

Autoconf uses AC_CHECK_LIB macro to check if the library is present. AC_CHECK_LIB takes the library name and a function(symbol) name that is expected to be exported from the library, generates a C program depending on this symbol and tries compiling it. This is a bit of info we miss in `BuildInfo?` - we have extra-libs, but no function names.

Right, we do not have this additional information. As you say, we will have to rely on ld being paranoid and failing when it cannot find a library, even if the object files do not depend on any function exported by that library.

Actually, I think ld pretty much has to have this behavior since it does not know what symbols the missing library might export that would shadow symbols exported by other libraries. So it cannot actually determine if the library is not needed, at least for ELF linkers.

Error messages should be helpful for the typical Cabal user, who should not be assumed to know > anything about C libraries or linking.

This is a bit unclear to me - we have no info on required packages, we have only includes/extra-libs, how do we make error messages user-friendly? Autoconf doesn't do it.

You're right that we cannot easily or accurately map missing C libs back to which native OS packages that provide them. I think we can still produce reasonable error messages. We can at least explain which C libraries are missing, we must make it clear that these are C libraries (not Haskell packages) that we expected to find on the system. Then there are two possible causes with corresponding solutions:

• The most likely is that the C package is just not installed, or that the development flavour of the package is not installed (binary Linux distros often split packages into runtime and dev variants). We can recommend users look for a native package with a similar name to the C lib, and if there is a "-dev" version then install that. We could provide more detail and say the native package should provide a file libfoo.so or libfoo.a or whatever.

• Another possibility is that the C lib is installed but in some non-standard location. In that case we should tell users about the --extra-include-dirs and/or --extra-lib-dirs flags.

Just for my own future reference:

• [AC_CHECK_LIB](http://www.gnu.org/software/hello/manual/autoconf/Libraries.html)
• [AC_CHECK_HEADER](http://www.gnu.org/software/hello/manual/autoconf/Generic-Headers.html)

[bos]

(Imported comment by @dcoutts on 2009-01-22)

Now partially done!

[bos]

(Imported comment by @dcoutts on 2009-01-30)

Thanks! Works great.

For example:

$ setup configure -O0
Configuring foo-1.0...
Warning: Required C header not found: foo.h
Warning: Required C library not found: bar
Warning: Required C library not found: baz

And adding --extra-include-dirs=./test/ --extra-lib-dirs=./test/ makes it work fine (with the test headers and lib files in ./test/ obviously).

Mon Jan 26 18:58:32 GMT 2009  [email protected]
- Warn if C dependencies not found (kind of fixes #262)
  This is just a basic check - generate a sample program and check if it compiles and links with relevant flags. Error messages (warning messages,
  actually) could use some improvement.
  Fri Jan 30 12:02:28 GMT 2009  Gleb Alexeyev <[email protected]>
- #262 iterative tests for foreign dependencies
  Optimize for succesful case. First try all libs and includes in one command,
  proceed with further tests only if the first test fails. The same goes for lib
  s
  and headers: look for an offending one only when overall test fails.
  

The last thing I need to fix before we can make these errors rather than just warnings is doing the check in the right place for packages that use configure scripts. We can also make some improvements to the error messages but it's already a lot more informative than it used to be when it'd just fail half way through building.

[bos]

(Imported comment by @dcoutts on 2009-01-30)

The check is now done after running ./configure so we can pick up any settings it provides. Those settings (cpp-options, cc-options and ld-options) are also now taken into account. This means the check should now be accurate so we can make it fatal rather than a warning. This is important to help people identify the cause of the problem.

Sat Jan 31 18:22:13 GMT 2009  Duncan Coutts <[email protected]>
- Do the check for foreign libs after running configure
  This lets us pick up build info discovered by the ./configure script
  Sat Jan 31 18:40:16 GMT 2009  Duncan Coutts <[email protected]>
- Use the cc, cpp and ld options when checking foreign headers and libs
  In partiular this is needed for packages that use ./configure
  scripts to write .buildinfo files since they typically do not
  split the cpp/cc/ldoptions into the more specific fields.
  Sat Jan 31 18:48:13 GMT 2009  Duncan Coutts <[email protected]>
- Improve the error message for missing foreign libs and make it fatal
  The check should now be accurate enough that we can make it an
  error rather than just a warning.
  

The error message now includes a bit more advice:

Configuring foo-1.0...
setup: Missing dependencies on foreign libraries:
- Missing header file: foo.h
- Missing C libraries: foo, bar, baz
  This problem can usually be solved by installing the system packages that
  provide these libraries (you may need the "-dev" versions). If the libraries
  are already installed but in a non-standard location then you can use the
  flags --extra-include-dirs= and --extra-lib-dirs= to specify where they are.
  

Or a real example:

Configuring curl-1.3.4...
setup: Missing dependency on a foreign library:
- Missing header file: curl.h
- Missing C library: curl
  This problem can usually be solved by installing the system package that
  provides this library (you may need the "-dev" version). If the library is
  already installed but in a non-standard location then you can use the flags
  --extra-include-dirs= and --extra-lib-dirs= to specify where it is.
  

Thanks to Gleb Alexeyev for implementing most of this. It should result in many fewer confused users.

[bos]

(Imported comment by @dcoutts on 2009-01-31)

I've built nearly every package on hackage to look for regressions. Two things need fixing:

We do not look for .h files in the root directory of the package. This breaks HsPerl5-0.0.6, cedict-0.2.5 and halfs-0.2.

We do not add the include dirs for dependent packages. This breaks hscurses-1.3.0.2, plugins-1.4.0 and unix-compat-0.1.2.1. Of these plugins-1.4.0 is not a good test case because it is broken for other reasons (the .h file it #includes cannot be included on its own).

The solution to the first issue should be easy. Just add . into the list of include dirs. The second requires copying the code from the pre-processor module where we do the same thing for hsc2hs. Perhaps it's possible to share the code.

[bos]

(Imported comment by @dcoutts on 2009-02-16)

Fixed.

Wed Feb 18 11:05:20 GMT 2009  Duncan Coutts <[email protected]>
  * Use more cc options when checking for header files and libs
  Use -I. to simulate the search path that gets used when we tell ghc
  to -#include something. Also use the include dirs and cc options of
  dependent packages. These two changes fix about 3 packages each.