Skip to content

Say something in Troubleshooting about the GLIBC issue #3247

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
michaelpj opened this issue Oct 5, 2022 · 7 comments
Open

Say something in Troubleshooting about the GLIBC issue #3247

michaelpj opened this issue Oct 5, 2022 · 7 comments
Labels
documentation type: enhancement New feature or request

Comments

@michaelpj
Copy link
Collaborator

It comes up a lot, it would be good to have it written down in one place so @hasufell doesn't have to keep repeating himself.

@hasufell could you give me a quick summary and I'll add it?
@michaelpj michaelpj added type: enhancement New feature or request documentation labels Oct 5, 2022
@hasufell
Copy link
Member

hasufell commented Oct 8, 2022 •
edited
Loading

The problem

For unknown linux

  1. GHC 9.4.x only provide a fedora 33 bindist, not a fedora 27: https://downloads.haskell.org/~ghc/9.4.2/
  2. so far we've used fedora bindists as fallback for Linux_UnknownLinux distros, because debian bindists have caused more trouble (e.g. wrt ncurses/tinfo): https://github.com/haskell/ghcup-metadata/blob/448e6b65a19b5b9b5e9d5e67d896088142d0c239/ghcup-0.0.7.yaml#L2588-L2589
  3. HLS needs to be in sync with a specific GHC bindist due to ABI issues. The HLS release CI uses ghcup to install GHC and then build the release binaries with it. HLS CI builds fedora bindist on fedora 33, because otherwise it could not build the GHC-9.4.2 corresponding binary.
  4. Fedora 33 glibc is very new (GLIBC_2.32) and doesn't work with a lot of distros.

For known linuxes

Every distro that has an entry for GHC (e.g. Mint, Debian, Ubuntu) also needs an explicit bindist for HLS, so that we can guarantee both correct glibc and that ABI actually matches. For linux mint this is already not the case:

  1. For GHC 9.2.4 it installs ghc-9.2.4-x86_64-fedora27-linux.tar.xz
  2. For GHC 9.4.2 it installs ghc-9.4.2-x86_64-deb10-linux.tar.xz
  3. It installs HLS haskell-language-server-1.8.0.0-x86_64-linux-fedora33.tar.xz
    • this HLS is built with ghc-9.4.2-x86_64-fedora33-linux.tar.xz though
    • this causes both ABI mismatch and possibly glibc issues

What has been done so far

For ubuntu, we fixed this by adding another HLS release that specifically builds for ubuntu:

Solutions

For unknown linuxes

We could switch the default bindist for Linux_UnknownLinux to deb9 or deb10 for GHC. That would fix the GHC portion.

https://github.com/haskell/ghcup-metadata/blob/448e6b65a19b5b9b5e9d5e67d896088142d0c239/ghcup-0.0.7.yaml#L2588-L2589

This will cause a number of issues though:

  1. it will break cabal cache (because GHC ABI is different and cabal doesn't distinguish that) if the user already has build artifacts and reinstalls GHC. This can also break github CI.
  2. it will require users to re-install GHC
  3. will cause further potential ABI breakage with existing HLS release (probably only 1.8.0.0, because these issues started with GHC-9.4.x)
  4. Debian is less cross-distro compatible than Fedora (except for the ancient glibc), so we may be opening the door to ncurses/libtinfo issues

This however is only half of the solution. I realized that we in fact need another HLS release bindist that is built on a distro that is "unknown" in order to trigger ghcup to pick the right bindist for Linux_UnknownLinux. Currently we pick haskell-language-server-1.8.0.0-x86_64-linux-fedora33.tar.xz for unknown linuxes, which is built on Fedora 33.

That means we only get ABI matches as long as Linux_UnknownLinux and Linux_Fedora coincide (they often do) and otherwise we get fatal ABI mismatch failure. Oops.

My idea here is to actually build on Fedora 27 and do some hacks to make ghcup think it's not Fedora.

For known linuxes

Just like ubuntu, provide more bindists for known linuxes. Basically every linux distro entry that exists in GHC metadata needs a dedicated HLS release job.

Workarounds

  1. compile HLS from source: https://www.haskell.org/ghcup/guide/#hls
  2. or downgrade to an older HLS, e.g. via this in VScode:
    "haskell.toolchain": {
        "hls": "1.7.0.0"
    }

@hasufell
Copy link
Member

hasufell commented Oct 8, 2022

@wz1000 I suggest we fix this with 1.8.0.1

This was referenced Oct 8, 2022
Closed
Closed
@hasufell
Copy link
Member

hasufell commented Oct 8, 2022

argument against deb bindists as default: https://bugzilla.redhat.com/show_bug.cgi?id=1875587

Causes lots of warnings on invoking ghc:

[ghc@1de2333cecae ~]$ ghc --version
/home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/./ghc-9.4.2: /lib64/libtinfo.so.5: no version information available (required by /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHShaskeline-0.8.2-ghc9.4.2.so)
/home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/./ghc-9.4.2: /lib64/libtinfo.so.5: no version information available (required by /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-9.4.2-ghc9.4.2.so)
/home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/./ghc-9.4.2: /lib64/libtinfo.so.5: no version information available (required by /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSterminfo-0.4.1.5-ghc9.4.2.so)
The Glorious Glasgow Haskell Compilation System, version 9.4.2

@hasufell
Copy link
Member

hasufell commented Oct 8, 2022

I'm leaning towards Cento7 bindists, although Centos7 has EOL 2024.

[ghc@1de2333cecae ~]$ objdump -T /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/ghc | grep GLIBC
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 __libc_start_main
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 memset
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 getuid
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 getlogin
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.14  memcpy
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 memchr

[ghc@1de2333cecae ~]$ ldd /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/ghc
	linux-vdso.so.1 (0x00007fff42f06000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fc0e668a000)
	libHShaskeline-0.8.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHShaskeline-0.8.2-ghc9.4.2.so (0x00007fc0e6a06000)
	libHSghc-9.4.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-9.4.2-ghc9.4.2.so (0x00007fc0e0bdd000)
	libHSterminfo-0.4.1.5-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSterminfo-0.4.1.5-ghc9.4.2.so (0x00007fc0e0b9c000)
	libHSprocess-1.6.15.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSprocess-1.6.15.0-ghc9.4.2.so (0x00007fc0e0b5b000)
	libHShpc-0.6.1.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHShpc-0.6.1.0-ghc9.4.2.so (0x00007fc0e0b24000)
	libHSghci-9.4.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghci-9.4.2-ghc9.4.2.so (0x00007fc0e08ae000)
	libHSghc-heap-9.4.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-heap-9.4.2-ghc9.4.2.so (0x00007fc0e07e7000)
	libHSghc-boot-9.4.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-boot-9.4.2-ghc9.4.2.so (0x00007fc0e0714000)
	libHSbinary-0.8.9.1-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSbinary-0.8.9.1-ghc9.4.2.so (0x00007fc0e064a000)
	libHSexceptions-0.10.5-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSexceptions-0.10.5-ghc9.4.2.so (0x00007fc0e060e000)
	libHSstm-2.5.1.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSstm-2.5.1.0-ghc9.4.2.so (0x00007fc0e05ec000)
	libHSmtl-2.2.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSmtl-2.2.2-ghc9.4.2.so (0x00007fc0e05b6000)
	libHStransformers-0.5.6.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHStransformers-0.5.6.2-ghc9.4.2.so (0x00007fc0e0468000)
	libHSdirectory-1.3.7.1-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSdirectory-1.3.7.1-ghc9.4.2.so (0x00007fc0e0403000)
	libHSunix-2.7.3-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSunix-2.7.3-ghc9.4.2.so (0x00007fc0e031d000)
	libHStime-1.12.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHStime-1.12.2-ghc9.4.2.so (0x00007fc0e0180000)
	libHSfilepath-1.4.2.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSfilepath-1.4.2.2-ghc9.4.2.so (0x00007fc0e015a000)
	libHScontainers-0.6.6-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHScontainers-0.6.6-ghc9.4.2.so (0x00007fc0dfda0000)
	libHSbytestring-0.11.3.1-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSbytestring-0.11.3.1-ghc9.4.2.so (0x00007fc0dfc2e000)
	libHStemplate-haskell-2.19.0.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHStemplate-haskell-2.19.0.0-ghc9.4.2.so (0x00007fc0df852000)
	libHSpretty-1.1.3.6-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSpretty-1.1.3.6-ghc9.4.2.so (0x00007fc0df7da000)
	libHSghc-boot-th-9.4.2-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-boot-th-9.4.2-ghc9.4.2.so (0x00007fc0df79a000)
	libHSdeepseq-1.4.8.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSdeepseq-1.4.8.0-ghc9.4.2.so (0x00007fc0df77e000)
	libHSarray-0.5.4.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSarray-0.5.4.0-ghc9.4.2.so (0x00007fc0df6f6000)
	libHSbase-4.17.0.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSbase-4.17.0.0-ghc9.4.2.so (0x00007fc0dec3e000)
	libHSghc-bignum-1.3-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-bignum-1.3-ghc9.4.2.so (0x00007fc0debeb000)
	libHSghc-prim-0.9.0-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSghc-prim-0.9.0-ghc9.4.2.so (0x00007fc0de6ee000)
	libHSrts-1.0.2_thr-ghc9.4.2.so => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libHSrts-1.0.2_thr-ghc9.4.2.so (0x00007fc0de634000)
	libffi.so.8 => /home/ghc/.ghcup/ghc/9.4.2/lib/ghc-9.4.2/bin/../lib/x86_64-linux-ghc-9.4.2/libffi.so.8 (0x00007fc0de425000)
	libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007fc0de1fb000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fc0ddff3000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007fc0dddf0000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fc0ddbec000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc0dd9ce000)
	libgmp.so.10 => /lib64/libgmp.so.10 (0x00007fc0dd756000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fc0dd3a0000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc0e69d5000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc0dd187000)

@hasufell
Copy link
Member

hasufell commented Oct 8, 2022

https://gitlab.haskell.org/ghc/ghc/-/issues/22268

@hasufell hasufell mentioned this issue Oct 8, 2022
Closed
@hasufell hasufell mentioned this issue Nov 27, 2022
Closed
@chreekat
Copy link

fedora27 bindists are now available for 9.2.5 and 9.4.4:

@hasufell
Copy link
Member

fedora27 bindists are now available for 9.2.5 and 9.4.4:

We don't have corresponding HLS binaries.

@hasufell hasufell mentioned this issue Dec 31, 2022
Closed
@hasufell hasufell mentioned this issue Jan 29, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation type: enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add explicit job for unknown linux hasufell/haskell-language-server
3 participants
@chreekat @hasufell @michaelpj