WebAssembly compile crashes on Windows with assertions enabled

[aykevl]

Whatever I try, it appears that WebAssembly on my LLVM build just doesn't work with assertions enabled. I always hit an assertion in the WebAssembly CFG Sort pass. When I disable assertions, the resulting code seems to work fine.

It appears to happen with basically any code, but here is one example:

target triple = "wasm32-unknown-unknown"

define void @integer() {
  ret void
}

When I compile this file, it results in a crash:

$ llvm-build.testwasm/bin/llc test.ll
Assertion failed!

Program: C:\Users\Ayke\tinygo\llvm-build.testwasm\bin\llc.exe
File: C:/Users/Ayke/tinygo/llvm-project/llvm/lib/Target/WebAssembly/WebAssemblyCFGSort.cpp, Line 374

Expression: OnStack.count(SRI.getRegionFor(&MBB)) && "Blocks must be nested in their regions"
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: C:\\Users\\Ayke\\tinygo\\llvm-build.testwasm\\bin\\llc.exe test.ll
1.      Running pass 'Function Pass Manager' on module 'test.ll'.
2.      Running pass 'WebAssembly CFG Sort' on function '@integer'
0x0000000001DC60E0 (0x0000000000000016 0x00000000033CE970 0x0000000000000016 0x00007FFD556746B8)
0x00007FFD5560ADFB (0x0000000000000001 0x00007FFD55631EFF 0x000000000566F170 0x00000000056D1FB0), raise() + 0x21B bytes(s)
0x00007FFD5560F1FB (0x00007FFD5566FA60 0x000000000219E230 0x00000000056D4810 0x0000000000000020), abort() + 0x1B bytes(s)
0x0000000000403B46 (0x00000000056BBFC8 0x00000000056D1FB0 0x00000000033CEC60 0x00000000056BBFC8)
0x0000000000403BF6 (0x00000000033CECB8 0x0000000100000020 0x0000000000000000 0x00000000056BBFC8)
0x000000000044A68D (0x00000000056BB650 0x00007FFD4E954DF3 0x0000000005674950 0x000000000564FF60)
0x000000000093E25B (0x00000000000000B7 0x00007FFD4E9557F6 0x00000000000000B7 0x0000000005649E70)
0x00000000012F97EA (0x000000000564A008 0x0000000005672110 0x00000000056488C0 0x0000000000000000)
0x0000000000DF3CBD (0x00000000033CF400 0x000000000564A090 0x000000000564A0B0 0x0000000005649E50)
0x0000000000DF392C (0x000000000564E768 0x0000000005643250 0x000000000564B180 0x0000000001187103)
0x0000000001677866 (0x000000000564D480 0x000000000564A280 0x0000000005648100 0x00000000000000E0)
0x00000000004697FA (0x000000000578D260 0x00000000033CFC80 0x00000000033CFD10 0x0000000000000000)
0x0000000001D771C9 (0x0000000000000002 0x000000000578D260 0x00000000023BE378 0x0000000000000000)
0x00000000004013C7 (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000)
0x00000000004014FB (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000)
0x00007FFD553F7034 (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000), BaseThreadInitThunk() + 0x14 bytes(s)
0x00007FFD572C2651 (0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000), RtlUserThreadStart() + 0x21 bytes(s)
Illegal instruction

That backtrace is hard to read, so here is one from GDB:

#0  0x0000000001dc60e0 in HandleAbort.cold.264 ()
#1  0x00007ffd5560adfb in raise () from C:\Windows\System32\msvcrt.dll
#2  0x00007ffd5560f1fb in msvcrt!abort () from C:\Windows\System32\msvcrt.dll
#3  0x0000000000403b46 in _wassert ()
#4  0x0000000000403bf6 in _assert ()
#5  0x000000000044a68d in sortBlocks(llvm::MachineFunction&, llvm::MachineLoopInfo const&, llvm::WebAssemblyExceptionInfo const&, llvm::MachineDominatorTre
e const&) ()
#6  0x000000000093e25b in (anonymous namespace)::WebAssemblyCFGSort::runOnMachineFunction(llvm::MachineFunction&) ()
#7  0x00000000012f97ea in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) ()
#8  0x0000000000df3cbd in llvm::FPPassManager::runOnFunction(llvm::Function&) ()
#9  0x0000000000df392c in llvm::FPPassManager::runOnModule(llvm::Module&) ()
#10 0x0000000001677866 in llvm::legacy::PassManagerImpl::run(llvm::Module&) ()
#11 0x00000000004697fa in compileModule(char**, llvm::LLVMContext&) ()
#12 0x0000000001d771c9 in main ()

So apparently this assert is triggered:

llvm-project/llvm/lib/Target/WebAssembly/WebAssemblyCFGSort.cpp

Lines 370 to 371 in 0e27d08

	assert(OnStack.count(SRI.getRegionFor(&MBB)) &&
	"Blocks must be nested in their regions");

I'm not at all familiar with this code or what it's supposed to do.

This is the CMake line I've used:

cmake -G Ninja ../llvm-project/llvm -DLLVM_TARGETS_TO_BUILD=WebAssembly -DCMAKE_BUILD_TYPE=Release -DLIBCLANG_BUILD_STATIC=ON -DLLVM_ENABLE_TERMINFO=OFF -DLLVM_ENABLE_ZLIB=OFF -DLLVM_ENABLE_LIBEDIT=OFF -DLLVM_ENABLE_Z3_SOLVER=OFF -DLLVM_ENABLE_OCAMLDOC=OFF -DLLVM_ENABLE_LIBXML2=OFF -DLLVM_ENABLE_PROJECTS=clang;lld -DLLVM_TOOL_CLANG_TOOLS_EXTRA_BUILD=OFF -DCLANG_ENABLE_STATIC_ANALYZER=OFF -DCLANG_ENABLE_ARCMT=OFF -DLLVM_ENABLE_ASSERTIONS=ON

OS: Windows 10 (21H2) x64
Compiler: g++.exe (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0, installed via scoop install mingw
I'm doing the build from the Git Bash shell.

Any hints how I can debug this would be appreciated.

[llvmbot]

@llvm/issue-subscribers-bug

[llvmbot]

@llvm/issue-subscribers-backend-webassembly

[fhahn]

That's strange, building the empty function on godbolt works: https://llvm.godbolt.org/z/orzzKP3nv

[aykevl]

Yeah, I'm strongly suspecting it has something to do with my build environment. Otherwise this would have been noticed a long time ago.
...unfortunately, that also makes it a lot harder to reproduce.

Note also that this only happens on Windows. Linux is not affected (didn't test macOS).

[asb]

I don't have an easy way to reproduce this issue, but as you say it's quite surprising. With Wasm being a default target, the windows buildbots will be compiling a whole bunch of WebAssembly .ll when running the LLVM unit tests.

[aykevl]

If anybody wants to reproduce this:

• start with a clean Windows 10 Pro VM. You can easily download the installer ISO from the Microsoft website, no activation needed for a test. I've used KVM (virt-manager) with a 40GB virtual disk, which works well enough for me.
• install Git Bash
• start the Git Bash shell for the following commands
• scoop install ninja python cmake make mingw (see https://scoop.sh/)
• download llvm-project using git at the release/14.x branch (I've tested 0e27d08 but IIRC this bug was already present in release/13.x so it probably doesn't matter much)
• build llvm-project in a new directory (e.g. llvm-build) using cmake -G Ninja ../llvm-project.upstream/llvm -DLLVM_TARGETS_TO_BUILD=WebAssembly -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON and ninja llc
• run llvm-build/bin/llc test.ll (with test.ll being the code snippet I posted earlier) and observe the crash

If I comment out all the assertions in the #ifndef NDEBUG block, it works fine.

[asb]

Thanks for such clear instructions on reproducing.

One thing that might be interesting to compare is the output of llc with -debug -print-after-all on windows vs Linux for the failing input.

[aykevl]

One thing that might be interesting to compare is the output of llc with -debug -print-after-all on windows vs Linux for the failing input.

Thank you! I did that but unfortunately there is no difference at all until the crash. The last part of that (before "PLEASE submit a bug report to ...") is this:

********** Register Coloring **********
********** Function: integer
Interesting register intervals:

Coloring register intervals:
# *** IR Dump After WebAssembly Register Coloring (wasm-reg-coloring) ***:
# Machine code for function integer: NoPHIs, TracksLiveness, TiedOpsRewritten
Function Live Ins: $arguments

bb.0 (%ir-block.0):
  liveins: $arguments
  RETURN implicit-def dead $arguments

# End machine code for function integer.

********** Exception Info Calculation **********
********** Function: integer
********** CFG Sorting **********
********** Function: integer