Add support for platform libraries

[edsko]

This PR adds support for building platform libraries. A stanza for a platform library looks something like

platform-library test-package
  type:                native-shared

  if os(Windows)
    options: standalone
    mod-def-file: TestPackage.def

  other-modules:       MyPlatformLib.Hello
                       MyPlatformLib.SomeBindings
  build-depends:       base >=4.7 && <4.9
  hs-source-dirs:      src
  c-sources:           csrc/MyPlatformLibWrapper.c
  default-language:    Haskell2010

where native-shared means that we want to build a native shared library (.so on Linux, .dylib on OSX, .dll on Windows). The parser also recognizes native-static but this is not currently supported anywhere. The standalone option means that the we merge all library dependencies into the dynamic library (i.e., ghc options -shared -static), rather than make the created dynamic library just record its dependencies (ghc options -shared -dynamic); it is currently compulsory on Windows and unsupported anywhere else. The mod-def-file can be used to specify a module definition file, and is also Windows specific.

This probably needs a fairly careful review, in particular in the definition of gbuild, which is the old buildOrReplExe and now deals with both executables and platform libraries.

Note: there will be a minor clash between this and the independent goals stuff (in particular tracking fine grained dependencies, #2514).

/cc @dcoutts

[23Skidoo]

Needs documentation. Will take a closer look soon.

[dcoutts]

Ah yes, user guide. @edsko and I will do that.

[23Skidoo]

Also would be nice to have a test. And there's a Travis failure.

[23Skidoo]

Looks suspicious.

[edsko]

Oh, whoops. Sorry about that. Will fix that (along with your other comments). Not yet sure why Travis is failing though.

[23Skidoo]

Not yet sure why Travis is failing though.

Looks like a warning (we use -Werror on Travis).

[23Skidoo]

Some general comments:

1. As a user, I find the notion of "platform library" as a first-class concept a bit confusing. While it is easy for a beginner to grasp what the other top-level entities (library, executable, test, bench etc.) in a .cabal file are, "platform library" is a non-standard Cabal-specific concept. The name is also a bit vague - without reading documentation, I couldn't have guessed that this refers to a DLL intended to be called from programs in other languages. Maybe "embedded library" would be better, though that could lead people to believe that it has something to do with embedded programming. Is there some accepted terminology for this concept? How do other languages/build tools implement it?
2. Platform libraries appear to be very similar to "normal" libraries. Have you considered making this functionality part of the existing library stanza? That is, something like:

library foo
    platform-library: True -- or embedded-library, or whatever
    if os(windows)
        standalone: True
        mod-def-file: path/to/file.def
    [...]

If the problem is that library must be unique in a package, maybe we should lift that restriction.

/cc @tibbe -- you might want to comment on this.

[dcoutts]

FYI: edsko's implementation here is based on my design.

I'm happy to consider other names that make it clearer. Suggestions? "system-library", "native-library", "foreign-library"?

As for whether it should be merged with normal libraries, no I think it should be a separate stanza, because it's building a qualitative different artefact. These system libs are inherently less portable, sometimes completely platform specific. They're also for a different purpose, they're not for consumption by more Haskell code, they don't actually export any Haskell symbols (in principle). They're for integration with other foreign code.

About other libraries, yes I think we should have "private" Haskell libraries within a package, but I would keep those as portable Haskell libs.

[edsko]

(Also, compared to "normal" libraries, other library specific flags such as exposed-modules do not apply to platform/system/native libraries.)

[dcoutts]

These libs also cannot be listed in build-depends in other packages (though in principle could be listed as extra-libraries:, like we depend on other C libs).

This new stanza is intended to be extensible with more types, for build artefacts like Windows assemblies, or OSX frameworks, or other platform-specific build artefacts that use platform-native foreign-code APIs.

[23Skidoo]

Re: bikeshedding

Suggestions? "system-library", "native-library", "foreign-library"?

None of this sounds quite right. "System library" feels like it refers to something like kernel32.dll,
"native" usually refers to native code, "foreign" in Haskell context means non-Haskell code.

[edsko]

Of course, "foreign" doesn't necessary refer to non-Haskell code, we have both foreign imports and foreign exports, and a "platform-library" is typically one that has foreign exports.

[dcoutts]

@23Skidoo I'm no claiming foreign is perfect, but the rationale is that it's for consumption by foreign code, the API it presents is an FFI one.

[23Skidoo]

Yes, "foreign" is probably the least bad sugggestion.

Of course, "foreign" doesn't necessary refer to non-Haskell code, we have both foreign imports and foreign exports.

True, but usually unqualified "foreign" means non-Haskell, as in "foreign pointer".

[edsko]

True, but usually unqualified "foreign" means non-Haskell, as in "foreign pointer".

Such a Haskell-centric view of the world.. oh wait :-P

[dcoutts]

I'm happy with "foreign-library"

[23Skidoo]

I sort of like "foreign export library", but it's a bit long. I looked into how other languages call this, but it appears that there's no accepted/widespread terminology.

[edsko]

I realized that the naming of the generated library isn't quite right, and we are also missing an implementation of copy; fixing both of those.

[edsko]

Started to look at implementing a test, but had some difficulty getting the existing package tests to work on my system. I've made some changes to the test driver so that they now work; they should also make it easier to modify these tests further later should we need to.

However, with 7.4.2 I'm still getting one test error (with or without my changes):

  MultipleSource
    finds second source of multiple source:                      FAIL (1.80s)
      expected: 'cabal install' should succeed
        output: "/Users/e/wt/projects/cabal/cabal-install/dist/7.4.2/build/cabal/cabal --config-file=/Users/e/wt/projects/cabal/cabal-install/tests/PackageTests/cabal-config install q" in PackageTests/MultipleSource
      Resolving dependencies...
      Notice: installing into a sandbox located at
      /Users/e/wt/projects/cabal/cabal-install/tests/PackageTests/MultipleSource/.cabal-sandbox
      Configuring q-0.1.0.0...
      Building q-0.1.0.0...
      Failed to install q-0.1.0.0
      Build log ( /Users/e/wt/projects/cabal/cabal-install/tests/PackageTests/MultipleSource/.cabal-sandbox/logs/q-0.1.0.0.log ):
      Configuring q-0.1.0.0...
      Building q-0.1.0.0...
      setup-Simple-Cabal-1.14.0-x86_64-osx-ghc-7.4.2:
      dist/dist-sandbox-3b4eb8e0/package.conf.inplace: inappropriate type
      cabal: Error: some packages failed to install:
      q-0.1.0.0 failed during the building phase. The exception was:
      ExitFailure 1

I don't know what that's about.

[dcoutts]

@23Skidoo happy enough with foreign-library? If so we'll change the code (and rename things in the code to use the same terminology). Not too late for better options.

[23Skidoo]

@dcoutts What do you think about foreign-export-library? IMO it's a bit clearer, if verbose.

[dcoutts]

Quick poll on #hackage didn't come up with a clear winner.

[simonmichael]

foreign-library was also my first guess, based on other usage of "foreign" in Haskell. One very slight issue, it supports the perception that Haskellers view the rest of the world with suspicion. Better if it could communicate "Haskell plays nicely with everything".

c-library, because library can be used by Haskell code, while this thing can be used by things which know the C ABI. Or portable-library/other-library, because C ABI is too limiting.

platform-library is not bad, makes sense to folks who know software dev jargon. It might be read as "the platform we are building on" which might be wrong.

native-library seems similar to platform-library, but clearer. It feels more capable and powerful than
foreign-library. Also the word native keeps recurring in the discussion. I don't quite get the objection to it above.

[edsko]

@23Skidoo , @dcoutts : Ok, added a test which builds and installs a platform library, and then attempts to link a C program against it. Verified that the test works on all supported platforms (Linux, OSX, Windows).

[23Skidoo]

Your test seems to fail with GHC < 7.10 on Linux.

[edsko]

Darn. Looking at this now. Seems some library paths are different. Bear with me.

[edsko]

Ugh, what a @£@% mess. I think it should now work on Linux for all versions tested on Travis. Here's what was going wrong (I've also added this as a note in the source code itself):

---

Suppose that the dynamic library depends on base, but not (directly) on integer-gmp (which, however, is a dependency of base). We will link the library as

gcc ... -lHSbase-4.7.0.2-ghc7.8.4 -lHSinteger-gmp-0.5.1.0-ghc7.8.4 ...

However, on systems (like Ubuntu) where the linker gets called with -as-needed by default, the linker will notice that integer-gmp isn't actually a direct dependency and hence omit the link.

Then when we attempt to link a C program against this dynamic library, the static linker will attempt to verify that all symbols can be resolved. The dynamic library itself does not require any symbols from integer-gmp, but base does. In order to verify that the symbols used by base can be resolved, the static linker needs to be able to find integer-gmp.

Finding the base dependency is simple, because the dynamic elf header (readelf -d) for the library that we have created looks something like

(NEEDED) Shared library: [libHSbase-4.7.0.2-ghc7.8.4.so]
(RPATH)  Library rpath: [/path/to/base-4.7.0.2:...]

However, when it comes to resolving the dependency on integer-gmp, it needs to look at the dynamic header for base. On modern ghc (7.8 and higher) this looks something like

(NEEDED) Shared library: [libHSinteger-gmp-0.5.1.0-ghc7.8.4.so]
(RPATH)  Library rpath: [$ORIGIN/../integer-gmp-0.5.1.0:...]

This specifies the location of integer-gmp in terms of the location of base (using the $ORIGIN) variable. But here's the crux: when the static linker attempts to verify that all symbols can be resolved, IT DOES NOT RESOLVE $ORIGIN. As a consequence, it will not be able to resolve the symbols and report the missing symbols as errors, even though the dynamic linker would be able to resolve these symbols. We can tell the static linker not to report these errors by using --unresolved-symbols=ignore-all and all will be fine when we run the program (indeed, this is what the gold linker does), but it makes the resulting library more difficult to use.

Instead what we can do is make sure that the generated dynamic library has explicit top-level dependencies on these libraries. This means that the static linker knows where to find them, and when we have transitive dependencies on the same libraries the linker will only load them once, so we avoid needing to look at the RPATH of our dependencies. We can do this by passing --no-as-needed to the linker, so that it doesn't omit any libraries.

Note that on older ghc (7.6 and before) the Haskell libraries don't have an RPATH set at all, which makes it even more important that we make these top-level dependencies.

Finally, we have to explicitly link against libffi for the same reason. For newer ghc this happens to be unnecessary on many systems because libffi is a library which is not specific to GHC, and when the static linker verifies that all symbols can be resolved it will find the libffi that is globally installed (completely independent from ghc). Of course, this may well be the wrong version of libffi, but it's quite possible that symbol resolution happens to work. This is of course the wrong approach, which is why we link explicitly against libffi so that we will find the right version of libffi.

[edsko]

So, the official status is now:

         7.4          7.6          7.8  7.10
Linux    OK           OK           OK   OK
Windows  Unknown      Unknown      OK   OK
OSX      Unsupported  Unsupported  OK   OK

7.4 and 7.6 on OSX have some path problems, somewhere in the package DB there are hardcoded references to /usr/local/lib even though I did not install anything there; however, we don't have the resources to address this right now so I suggest we open an issue and leave these as unsupported, unless somebody else wants to step up and fix it.

[23Skidoo]

There should be a warning/error if the user tries to build a package containing platform libraries on Windows/OSX with GHC < 7.8. Aside from that, is there anything else to do here besides bikeshedding?

[edsko]

Not as far as I'm concerned. Although I'm not sure if @dcoutts has tested this yet for his original use-case (and it would be nice to get confirmation from @pawelp7 that this does in fact help him build his library (#2589).

[23Skidoo]

Looking forward to merging this!

[23Skidoo]

I probably won't have time to revive this before the 1.24 release, so postponing for 1.26.

[23Skidoo]

@hvr told me that GHC 8 final will likely be delayed until the end of March, so maybe we'll have time to get this in.

[blogle]

Just wanted to revive this thread to see if there was a time frame on when this might make it into release?

[ezyang]

I'll milestone it for 2.0, I don't see why we can't merge it for then.

[ezyang]

The conflicts get bigger and bigger. We should try to merge this sooner rather than later.

[crocket]

Why don't you call it c-library? It is explicit.

[ezyang]

Cleaning this up to merge for 2.0.

[ezyang]

@edsko, @dcoutts: If a package has foreign libraries but no libraries, should headers be installed?

[ezyang]

Subsumed by #4043.

[phadej]

So standalone option isn't supported?

[phadej]

To expand previous: Is adding standalone for Linux difficult task? Or why it isn't there in the first place? I wonder if the resulting single .so could be a little smaller, as unused code can be purged in static linking of haskell libs, or is this just a wishful thinking?

See #4827