Stack's 'ubuntu-latest' CI takes 3x longer than 'macos-13' CI

[mpilgrem]

Currently, GitHub workflow integration-tests.yml has very different performance on one runner compared to the others:

• 'ubuntu-latest' (in an Alpine Linux Docker container) 120 mins
• 'macos-13' 42 mins
• 'windows-latest' 39 mins (skips some tests)

and, on AArch64:

• `macos-latest' 23 mins

'ubuntu-latest' v 'macos-13' has some stark contrasts, the worst example being:

• 4783-doctest-deps 548 seconds v 39 seconds

Could something be going wrong with caching?

[mpilgrem]

Stack's online documentation for its Docker integration states:

With Docker enabled, most stack sub-commands will automatically launch themselves in an ephemeral Docker container (the container is deleted as soon as the command completes). The project directory and ~/.stack are volume-mounted into the container, so any build artifacts are "permanent" (not deleted with the container).

and

Note: ~/.stack is separately volume-mounted, and is left alone during reset [stack docker reset].

As snapshots are in the Stack root, I've always assumed that caching the Stack root is sufficient.

[mpilgrem]

After adding log dumping on success too:

case ec of
  ExitSuccess -> do
    logInfo "Success! Dumping log\n\n"
    withSourceFile logfp $ \src ->
      runConduit $ src .| stderrC
    logInfo $ "\n\nEnd of log for " <> fromString name
  _ -> do
    logError "Failure, dumping log\n\n"
    withSourceFile logfp $ \src ->
      runConduit $ src .| stderrC
    logError $ "\n\nEnd of log for " <> fromString name

it seems that 'macos-13' (40 s) and 'ubuntu-latest' (588 s) are reporting exactly the same steps for integration test 4783, albeit in a different order.

[mpilgrem]

It seems to be something to do with linking. There are four places in integration test 4783 where linking occurs: (a) Stack's Setup shim for local acme-dont-copy, (b) ghc-paths-0.1.0.12's setup, (c) doctest, and (d) foo's doctest test suite.

• 'macos-13' (41 s): (a) 1.1 s (b) 0.8 a (c) 3.8 s (d) 1.5 s
• 'ubuntu-latest' (in Alpine Linux Docker container) (596 s): (a) 52.4 s (b) 0.4 s (c) 240.9 s (d) 235.5 s

The linking delays at (a), (c) and (d) are accounting for almost all of the additional duration of 'ubuntu-latest'.

[mpilgrem]

@benz0li, as described above, linking during integration tests seems to be much slower (4 mins v 4 secs) in the Linux/x86_64 Docker container than on macOS/x86_64. Would you happen to know why?

I queried ChatGPT-4o and it offered up this summary:

Cause	Impact	Fix
ld.bfd is slow	🔥🔥🔥 Huge	Use lld or gold
Static linking w/ musl	🔥🔥 Big	Use dynamic if possible
Docker file I/O	🔥 Medium	Mount to tmpfs or build outside
CPU limits in Docker	🔥 Medium	Use more cores

[benz0li]

Cross references:

• https://discourse.haskell.org/t/good-way-of-installing-specific-cabal-and-ghc-inside-a-docker-container/9930/13 ff
  • Alternatives to a self-hosted runner to build statically-linked Stack for Linux/AArch64? #6531 (comment)
• Disable GHCs aggressive selection of ld.gold haskell/ghcup-hs#1032

[benz0li]

IMHO ld.bfd is the safe choice.

[benz0li]

None of the final image(s) have binutils-gold installed

– https://gitlab.haskell.org/ghc/ghc/-/issues/25093#note_579337

[benz0li]

If you want to use ld.gold, install binutils-gold (i.e. command apk add --no-cache binutils-gold) before building Stack and link with the GNU gold linker instead of ld.