Add sycl post link to clang sycl linker #5

asudarsa · 2025-04-03T00:03:14Z

This PR adds the following changes:

Add sycl-post-link (code splitting) to clang-sycl-linker
Create OffloadBinary device images and package them into a single file.

Thanks

Signed-off-by: Arvind Sudarsanam <[email protected]>

…gic to bundle images Signed-off-by: Arvind Sudarsanam <[email protected]>

asudarsa · 2025-04-03T00:05:25Z

clang/tools/clang-sycl-linker/ClangSYCLLinker.cpp

+    StringRef(BinaryData.begin(), BinaryData.size())))
+    return E;
+
+  {


This block of code unbundles the bundled file and then dumps each of the offload binary into a file. Thic block exists here only for testing and a similar code will reside in clang-linker-wrapper.

jzc · 2025-04-03T06:47:40Z

clang/tools/clang-sycl-linker/ClangSYCLLinker.cpp

+  }
+
+  SmallVector<char, 1024> BinaryData;
+  raw_svector_ostream OS(BinaryData);


Should we use raw_fd_ostream and write out to a file incrementally instead of holding all the data in memory?

jzc · 2025-04-03T06:50:08Z

clang/tools/clang-sycl-linker/ClangSYCLLinker.cpp

+        return createFileError(File, EC);
+    }
+
+    std::scoped_lock<decltype(ImageMtx)> Guard(ImageMtx);


Unnecessary mutex?

jzc · 2025-04-03T06:50:48Z

clang/tools/clang-sycl-linker/ClangSYCLLinker.cpp

+  // sycl-post-link step
+  auto SplitModules = runSYCLSplitModule(std::move(*LinkedModule), Args);
+  if (!SplitModules)
+    reportError(SplitModules.takeError());


Suggested change

reportError(SplitModules.takeError());

return SplitModules.takeError();

jzc · 2025-04-03T06:51:45Z

clang/tools/clang-sycl-linker/ClangSYCLLinker.cpp

+    SPVFile.append(utostr(I));
+    auto Err = runSPIRVCodeGen((*SplitModules)[I].ModuleFilePath, Args, SPVFile, C);
+    if (Err)
+      return std::move(Err);


Suggested change

return std::move(Err);

return Err;

jzc · 2025-04-03T06:53:53Z

clang/tools/clang-sycl-linker/ClangSYCLLinker.cpp


+/// Get a temporary filename suitable for output.
+Expected<StringRef> createOutputFile(const Twine &Prefix, StringRef Extension) {
+  std::scoped_lock<decltype(TempFilesMutex)> Lock(TempFilesMutex);


Unnecessary mutex?

…ctor-bits=128." (llvm#134997) Reverts llvm#134068 Caused a stage 2 build failure: https://lab.llvm.org/buildbot/#/builders/41/builds/6016 ``` FAILED: lib/Support/CMakeFiles/LLVMSupport.dir/Caching.cpp.o /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage1.install/bin/clang++ -DGTEST_HAS_RTTI=0 -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/lib/Support -I/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/llvm/lib/Support -I/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/include -I/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/llvm/include -mcpu=neoverse-512tvb -mllvm -scalable-vectorization=preferred -mllvm -treat-scalable-fixed-error-as-warning=false -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Werror=global-constructors -O3 -DNDEBUG -std=c++17 -UNDEBUG -fno-exceptions -funwind-tables -fno-rtti -MD -MT lib/Support/CMakeFiles/LLVMSupport.dir/Caching.cpp.o -MF lib/Support/CMakeFiles/LLVMSupport.dir/Caching.cpp.o.d -o lib/Support/CMakeFiles/LLVMSupport.dir/Caching.cpp.o -c /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/llvm/lib/Support/Caching.cpp Opcode has unknown scale! UNREACHABLE executed at ../llvm/llvm/lib/Target/AArch64/AArch64InstrInfo.cpp:4530! PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script. Stack dump: 0. Program arguments: /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage1.install/bin/clang++ -DGTEST_HAS_RTTI=0 -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/lib/Support -I/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/llvm/lib/Support -I/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/include -I/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/llvm/include -mcpu=neoverse-512tvb -mllvm -scalable-vectorization=preferred -mllvm -treat-scalable-fixed-error-as-warning=false -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Werror=global-constructors -O3 -DNDEBUG -std=c++17 -UNDEBUG -fno-exceptions -funwind-tables -fno-rtti -MD -MT lib/Support/CMakeFiles/LLVMSupport.dir/Caching.cpp.o -MF lib/Support/CMakeFiles/LLVMSupport.dir/Caching.cpp.o.d -o lib/Support/CMakeFiles/LLVMSupport.dir/Caching.cpp.o -c /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/llvm/lib/Support/Caching.cpp 1. <eof> parser at end of file 2. Code generation 3. Running pass 'Function Pass Manager' on module '/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/llvm/lib/Support/Caching.cpp'. 4. Running pass 'AArch64 load / store optimization pass' on function '@"_ZNSt17_Function_handlerIFN4llvm8ExpectedISt8functionIFNS1_ISt10unique_ptrINS0_16CachedFileStreamESt14default_deleteIS4_EEEEjRKNS0_5TwineEEEEEjNS0_9StringRefESB_EZNS0_10localCacheESB_SB_SB_S2_IFvjSB_S3_INS0_12MemoryBufferES5_ISH_EEEEE3$_0E9_M_invokeERKSt9_Any_dataOjOSF_SB_"' #0 0x0000b6eae9b67bf0 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage1.install/bin/clang+++0x81c7bf0) #1 0x0000b6eae9b65aec llvm::sys::RunSignalHandlers() (/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage1.install/bin/clang+++0x81c5aec) #2 0x0000b6eae9acd5f4 CrashRecoverySignalHandler(int) CrashRecoveryContext.cpp:0:0 #3 0x0000f16c1aff28f8 (linux-vdso.so.1+0x8f8) #4 0x0000f16c1aacf1f0 __pthread_kill_implementation ./nptl/pthread_kill.c:44:76 #5 0x0000f16c1aa8a67c gsignal ./signal/../sysdeps/posix/raise.c:27:6 #6 0x0000f16c1aa77130 abort ./stdlib/abort.c:81:7 llvm#7 0x0000b6eae9ad6628 (/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage1.install/bin/clang+++0x8136628) llvm#8 0x0000b6eae72e95a8 (/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage1.install/bin/clang+++0x59495a8) llvm#9 0x0000b6eae74ca9a8 (anonymous namespace)::AArch64LoadStoreOpt::findMatchingInsn(llvm::MachineInstrBundleIterator<llvm::MachineInstr, false>, (anonymous namespace)::LdStPairFlags&, unsigned int, bool) AArch64LoadStoreOptimizer.cpp:0:0 llvm#10 0x0000b6eae74c85a8 (anonymous namespace)::AArch64LoadStoreOpt::tryToPairLdStInst(llvm::MachineInstrBundleIterator<llvm::MachineInstr, false>&) AArch64LoadStoreOptimizer.cpp:0:0 llvm#11 0x0000b6eae74c624c (anonymous namespace)::AArch64LoadStoreOpt::optimizeBlock(llvm::MachineBasicBlock&, bool) AArch64LoadStoreOptimizer.cpp:0:0 llvm#12 0x0000b6eae74c429c (anonymous namespace)::AArch64LoadStoreOpt::runOnMachineFunction(llvm::MachineFunction&) AArch64LoadStoreOptimizer.cpp:0:0 ```

…reporting (llvm#131756) ### Description This PR resolves a deadlock between AddressSanitizer (ASan) and LeakSanitizer (LSan) that occurs when both sanitizers attempt to acquire locks in conflicting orders across threads. The fix ensures safe lock acquisition ordering by preloading module information before error reporting. --- ### Issue Details **Reproducer** ```cpp // Thread 1: ASan error path int arr[1] = {0}; std::thread t([&]() { arr[1] = 1; // Triggers ASan OOB error }); // Thread 2: LSan check path __lsan_do_leak_check(); ``` **Lock Order Conflict**: - Thread 1 (ASan error reporting): 1. Acquires ASan thread registry lock (B) 1. Attempts to acquire libdl lock (A) via `dl_iterate_phdr` - Thread 2 (LSan leak check): 1. Acquires libdl lock (A) via `dl_iterate_phdr` 1. Attempts to acquire ASan thread registry lock (B) This creates a circular wait condition (A -> B -> A) meeting all four Coffman deadlock criteria. --- ### Fix Strategy The root cause lies in ASan's error reporting path needing `dl_iterate_phdr` (requiring lock A) while already holding its thread registry lock (B). The solution: 1. **Preload Modules Early**: Force module list initialization _before_ acquiring ASan's thread lock 2. **Avoid Nested Locking**: Ensure symbolization (via dl_iterate_phdr) completes before error reporting locks Key code change: ```cpp // Before acquiring ASan's thread registry lock: Symbolizer::GetOrInit()->GetRefreshedListOfModules(); ``` This guarantees module information is cached before lock acquisition, eliminating the need for `dl_iterate_phdr` calls during error reporting. --- ### Testing Added **asan_lsan_deadlock.cpp** test case: - Reproduces deadlock reliably without fix **under idle system conditions** - Uses watchdog thread to detect hangs - Verifies ASan error reports correctly without deadlock **Note**: Due to the inherent non-determinism of thread scheduling and lock acquisition timing, this test may not reliably reproduce the deadlock on busy systems (e.g., during parallel `ninja check-asan` runs). --- ### Impact - Fixes rare but severe deadlocks in mixed ASan+LSan environments - Maintains thread safety guarantees for both sanitizers - No user-visible behavior changes except deadlock elimination --- ### Relevant Buggy Code - Code in ASan's asan_report.cpp ```cpp explicit ScopedInErrorReport(bool fatal = false) : halt_on_error_(fatal || flags()->halt_on_error) { // Acquire lock B asanThreadRegistry().Lock(); } ~ScopedInErrorReport() { ... // Try to acquire lock A under holding lock B via the following path // #4 0x000071a353d83e93 in __GI___dl_iterate_phdr ( // callback=0x5d1a07a39580 <__sanitizer::dl_iterate_phdr_cb(dl_phdr_info*, unsigned long, void*)>, // data=0x6da3510fd3f0) at ./elf/dl-iteratephdr.c:39 // #5 0x00005d1a07a39574 in __sanitizer::ListOfModules::init (this=0x71a353ebc080) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:784 // #6 0x00005d1a07a429e3 in __sanitizer::Symbolizer::RefreshModules (this=0x71a353ebc058) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp:188 // llvm#7 __sanitizer::Symbolizer::FindModuleForAddress (this=this@entry=0x71a353ebc058, // address=address@entry=102366378805727) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp:214 // llvm#8 0x00005d1a07a4291b in __sanitizer::Symbolizer::SymbolizePC (this=0x71a353ebc058, addr=102366378805727) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cpp:88 // llvm#9 0x00005d1a07a40df7 in __sanitizer::(anonymous namespace)::StackTraceTextPrinter::ProcessAddressFrames ( // this=this@entry=0x6da3510fd520, pc=102366378805727) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cpp:37 // llvm#10 0x00005d1a07a40d27 in __sanitizer::StackTrace::PrintTo (this=this@entry=0x6da3510fd5e8, // output=output@entry=0x6da3510fd588) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cpp:110 // llvm#11 0x00005d1a07a410a1 in __sanitizer::StackTrace::Print (this=0x6da3510fd5e8) // at llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cpp:133 // llvm#12 0x00005d1a0798758d in __asan::ErrorGeneric::Print ( // this=0x5d1a07aa4e08 <__asan::ScopedInErrorReport::current_error_+8>) // at llvm-project/compiler-rt/lib/asan/asan_errors.cpp:617 current_error_.Print(); ... } ``` - Code in LSan's lsan_common_linux.cpp ```cpp void LockStuffAndStopTheWorld(StopTheWorldCallback callback, CheckForLeaksParam *argument) { // Acquire lock A dl_iterate_phdr(LockStuffAndStopTheWorldCallback, &param); } static int LockStuffAndStopTheWorldCallback(struct dl_phdr_info *info, size_t size, void *data) { // Try to acquire lock B under holding lock A via the following path // #3 0x000055555562b34a in __sanitizer::ThreadRegistry::Lock (this=<optimized out>) // at llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_thread_registry.h:99 // #4 __lsan::LockThreads () at llvm-project/compiler-rt/lib/asan/asan_thread.cpp:484 // #5 0x0000555555652629 in __lsan::ScopedStopTheWorldLock::ScopedStopTheWorldLock (this=<optimized out>) // at llvm-project/compiler-rt/lib/lsan/lsan_common.h:164 // #6 __lsan::LockStuffAndStopTheWorldCallback (info=<optimized out>, size=<optimized out>, data=0x0, // data@entry=0x7fffffffd158) at llvm-project/compiler-rt/lib/lsan/lsan_common_linux.cpp:120 ScopedStopTheWorldLock lock; DoStopTheWorldParam *param = reinterpret_cast<DoStopTheWorldParam *>(data); StopTheWorld(param->callback, param->argument); return 1; } ```

Fixes llvm#123300 What is seen ``` clang-repl> int x = 42; clang-repl> auto capture = [&]() { return x * 2; }; In file included from <<< inputs >>>:1: input_line_4:1:17: error: non-local lambda expression cannot have a capture-default 1 | auto capture = [&]() { return x * 2; }; | ^ zsh: segmentation fault clang-repl --Xcc="-v" (lldb) bt * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8) * frame #0: 0x0000000107b4f8b8 libclang-cpp.19.1.dylib`clang::IncrementalParser::CleanUpPTU(clang::PartialTranslationUnit&) + 988 frame #1: 0x0000000107b4f1b4 libclang-cpp.19.1.dylib`clang::IncrementalParser::ParseOrWrapTopLevelDecl() + 416 frame #2: 0x0000000107b4fb94 libclang-cpp.19.1.dylib`clang::IncrementalParser::Parse(llvm::StringRef) + 612 frame #3: 0x0000000107b52fec libclang-cpp.19.1.dylib`clang::Interpreter::ParseAndExecute(llvm::StringRef, clang::Value*) + 180 frame #4: 0x0000000100003498 clang-repl`main + 3560 frame #5: 0x000000018d39a0e0 dyld`start + 2360 ``` Though the error is justified, we shouldn't be interested in exiting through a segfault in such cases. The issue is that empty named decls weren't being taken care of resulting into this assert https://github.com/llvm/llvm-project/blob/c1a229252617ed58f943bf3f4698bd8204ee0f04/clang/include/clang/AST/DeclarationName.h#L503 Can also be seen when the example is attempted through xeus-cpp-lite. ![image](https://github.com/user-attachments/assets/9b0e6ead-138e-4b06-9ad9-fcb9f8d5bf6e)

# Symptom We have seen SIGSEGV like this: ``` * thread #1, name = 'lldb-server', stop reason = SIGSEGV frame #0: 0x00007f39e529c993 libc.so.6`__pthread_kill_internal(signo=11, threadid=<unavailable>) at pthread_kill.c:46:37 ... * frame #5: 0x000056027c94fe48 lldb-server`lldb_private::process_linux::GetPtraceScope() + 72 frame #6: 0x000056027c92f94f lldb-server`lldb_private::process_linux::NativeProcessLinux::Attach(int) + 1087 ... ``` See [full stack trace](https://pastebin.com/X0d6QhYj). This happens on Linux where LLDB doesn't have access to `/proc/sys/kernel/yama/ptrace_scope`. A similar error (an unchecked `Error`) can be reproduced by running the newly added unit test without the fix. See the "Test" section below. # Root cause `GetPtraceScope()` ([code](https://github.com/llvm/llvm-project/blob/328f40f408c218f25695ea42c844e43bef38660b/lldb/source/Plugins/Process/Linux/Procfs.cpp#L77)) has the following `if` statement: ``` llvm::Expected<int> lldb_private::process_linux::GetPtraceScope() { ErrorOr<std::unique_ptr<MemoryBuffer>> ptrace_scope_file = getProcFile("sys/kernel/yama/ptrace_scope"); if (!*ptrace_scope_file) return errorCodeToError(ptrace_scope_file.getError()); ... } ``` The intention of the `if` statement is to check whether the `ptrace_scope_file` is an `Error` or not, and return the error if it is. However, the `operator*` of `ErrorOr` returns the value that is stored (which is a `std::unique_ptr<MemoryBuffer>`), so what the `if` condition actually do is to check if the unique pointer is non-null. Note that the method `ErrorOr::getStorage()` ([called by](https://github.com/llvm/llvm-project/blob/328f40f408c218f25695ea42c844e43bef38660b/llvm/include/llvm/Support/ErrorOr.h#L162-L164) `ErrorOr::operator *`) **does** assert on whether or not `HasError` has been set (see [ErrorOr.h](https://github.com/llvm/llvm-project/blob/328f40f408c218f25695ea42c844e43bef38660b/llvm/include/llvm/Support/ErrorOr.h#L235-L243)). However, it seems this wasn't executed, probably because the LLDB was a release build. # Fix The fix is simply remove the `*` in the said `if` statement.

asudarsa added 2 commits April 2, 2025 16:59

sycl-post-link patch cherry-picked

7911c2a

Signed-off-by: Arvind Sudarsanam <[email protected]>

[clang-sycl-linker][sycl-post-link] Include sycl-post-link and add lo…

ddcb918

…gic to bundle images Signed-off-by: Arvind Sudarsanam <[email protected]>

asudarsa commented Apr 3, 2025

View reviewed changes

jzc reviewed Apr 3, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add sycl post link to clang sycl linker #5

Add sycl post link to clang sycl linker #5

Uh oh!

asudarsa commented Apr 3, 2025

Uh oh!

asudarsa Apr 3, 2025

Uh oh!

jzc Apr 3, 2025

Uh oh!

jzc Apr 3, 2025

Uh oh!

jzc Apr 3, 2025

Uh oh!

jzc Apr 3, 2025

Uh oh!

jzc Apr 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	reportError(SplitModules.takeError());
	return SplitModules.takeError();

Add sycl post link to clang sycl linker #5

Are you sure you want to change the base?

Add sycl post link to clang sycl linker #5

Uh oh!

Conversation

asudarsa commented Apr 3, 2025

Uh oh!

asudarsa Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

jzc Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

jzc Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

jzc Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

jzc Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

jzc Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants