[ATfL][build.sh] Test C++ libraries with LLD without overwriting config files

[amilendra]

Adds custom libc++, libc++abi, and libunwind test configurations to select LLD for C++ tests, replacing the invasive approach of overwriting Clang’s global configuration files.

[pawosm-arm]

This is a great approach, but I need to ask two questions first:

• how this will behave when facing any upstream changes to the related CMake files?
• would this approach be also feasible to cover the problem with CMake runtimes not being able to find runtimes built in a previous stage? see: #1047

[amilendra]

This is a great approach, but I need to ask two questions first:

* how this will behave when facing any upstream changes to the related CMake files?

I had been thinking about the same issue the couple of days as well.
A better solution would be to overlay on top of the upstream CMake files and only add the -fuse-ld=lld to that..
e.g. In that case atfl-libc++-shared.cfg would look something like this,

lit_config.load_config(config, 'llvm-libc++-shared.cfg')
config.substitutions.append(('%{link_flags}',
    '-fuse-ld=lld {}'.format(' '.join(link_flags))
))

But that would need out ATfL builds to be CMake based instead of the bash-script based approach.
So at the moment the only solution is to keep these test configuration files and copy over any changes to the upstream CMake files when needed. I expect the libcxx tests will fail and that will let us know when the copying needs to happen.

* would this approach be also feasible to cover the problem with CMake runtimes not being able to find runtimes built in a previous stage? see: #1047

I'll try to reproduce the problem first and have a think.

[pawosm-arm]

So at the moment the only solution is to keep these test configuration files and copy over any changes to the upstream CMake files when needed. I expect the libcxx tests will fail and that will let us know when the copying needs to happen.

I had similar dilemma while introducing the bolt-related cmake files. Eventually I decided not to create new ones, but instead copy existing, piggyback new or overriding flags of them, and use sed for adjusting things that could not be addressed by just adding new lines. With time, this sed part turned out not to be necessary, so now I'm only piggybacking on copies of existing files, and then instruct CMake to use the copies:

    mkdir -p "${cmake_caches}"
    cp "${SOURCES_DIR}/flang/cmake/caches/BOLT.cmake" "${cmake_caches}"
    { print_forced_cmake_flags_cache "COMMON_CMAKE_FLAGS"
      print_forced_cmake_flags_cache "USE_RPATH_CMAKE_FLAGS"
      print_forced_cmake_flags_cache "COMPILER_CMAKE_FLAGS"
      print_forced_cmake_flags_cache "COMPILER_RT_CMAKE_FLAGS"
      print_forced_cmake_flags_cache "FLANG_RT_CMAKE_FLAGS"
      print_forced_cmake_flags_cache "LIBOMP_SHARED_CMAKE_FLAGS"
      print_forced_cmake_flags_cache "LIBUNWIND_SHARED_CMAKE_FLAGS"
      print_forced_cached_flag "CMAKE_EXE_LINKER_FLAGS:STRING" "\"${libs} ${RELOCS_LINKER_FLAGS}\""
      print_forced_cached_flag "CMAKE_MODULE_LINKER_FLAGS:STRING" "\"${libs} ${RELOCS_LINKER_FLAGS}\""
      print_forced_cached_flag "CMAKE_SHARED_LINKER_FLAGS:STRING" "\"${libs} ${RELOCS_LINKER_FLAGS}\""
      print_forced_cached_flag "LLVM_ENABLE_RUNTIMES:STRING" "\"compiler-rt;flang-rt;libunwind;openmp\""
      print_forced_cached_flag "RUNTIMES_CMAKE_ARGS:STRING" "\"-DCMAKE_C_COMPILER=${ATFL_DIR}/bin/clang;-DCMAKE_CXX_COMPILER=${ATFL_DIR}/bin/clang++;-DCMAKE_Fortran_COMPILER=${ATFL_DIR}/bin/flang;-DCMAKE_CXX_FLAGS=-stdlib++-isystem${ATFL_DIR}/include/c++/v1 -D_LIBCPP_VERBOSE_ABORT_NOT_NOEXCEPT;-DCMAKE_EXE_LINKER_FL
    } >> ${cmake_caches}/BOLT.cmake

...

cmake "${CMAKE_ARGS[@]}" -G Ninja "${SOURCES_DIR}/llvm" \
        -DPGO_BUILD_CONFIGURATION="${cmake_caches}/BOLT.cmake" \

[amilendra]

I thought about this a bit more. All our builds except for RHEL-8 pass without this -fuse-ld=lld workaround.

/opt/rh/gcc-toolset-14/root/usr/bin/ld: /workspace/build/stage/libcpp_build/libcxx/test-suite-install/lib/libc++experimental.a(libunwind.cpp.o): undefined reference to symbol 'dladdr@@GLIBC_2.17'
/opt/rh/gcc-toolset-14/root/usr/bin/ld: /lib64/libdl.so.2: error adding symbols: DSO missing from command line

So looking at the reason for that, the root cause is due to RHEL-8 having GLIBC < 2.34 https://lwn.net/Articles/864920/
All other platforms have GLIBC >= 2.34. With GLIBC >= 2.34,

New applications do not need to link with -lpthread, -ldl, -lutil, -lanl anymore.

However application linking with GLIBC older than that should pass -ldl which our libcxx tests are not doing.
I raised an upstream PR for that llvm/llvm-project#222797.
If it gets merged we may not need the -fuse-ld=lld workaround or this custom configuration hack.

[pawosm-arm]

I raised an upstream PR for that

Yeah, this should have been done like that from the start. Thanks for addressing it properly.