Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests segfault in Mono

[danmoseley]

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-81202-merge-135c923415104d6c85/Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests/1/console.6c3784b3.log?helixlogtype=result

net8.0-linux-Debug-arm64-Mono_release-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8

Possible stack trace for the crash

Thread 9 (Thread 0xff7909f741c0 (LWP 34)):
#0  0x0000ff791595bd5c in __waitpid (pid=<optimized out>, stat_loc=0xff7909f6e9f0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x0000ff79153519e8 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xff7909f6f450, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x0000ff79153105c0 in mono_handle_native_crash (signal=0xff791510e1d2 "SIGSEGV", mctx=0xff7909f6f450, info=0xff7909f6f7b0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
#4  0x0000ff7915279b90 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xff7909f6f7b0, context=0xff7909f6f830, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3749
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x0000ff7908de1d18 in ?? ()
#8  0x0000ff7914ad9848 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

---

(original issue description note: this is not the problematic stack trace)

Not sure how it indicates which thread failed, but I'm guessing it's this one?

Thread 10 (Thread 0xff7909d431c0 (LWP 35)):
#0  monoeg_g_hash_table_insert_replace (hash=0xab9b66ab1fb0, key=0xff78e945dca0, value=0xff78e945dca0, replace=0) at /__w/1/s/src/mono/mono/eglib/ghashtable.c:246
#1  0x0000ff79151467b4 in mono_class_inflate_generic_method_full_checked (method=<optimized out>, klass_hint=<optimized out>, context=0xff78e8e928f8, error=<optimized out>) at /__w/1/s/src/mono/mono/metadata/class.c:1312
#2  0x0000ff79151588f0 in mono_class_setup_vtable_ginst (klass=0xff78e8e92920, in_setup=<optimized out>) at /__w/1/s/src/mono/mono/metadata/class-setup-vtable.c:1363
#3  mono_class_setup_vtable_general (klass=0xff78e8e92920, overrides=<optimized out>, onum=<optimized out>, in_setup=<optimized out>) at /__w/1/s/src/mono/mono/metadata/class-setup-vtable.c:1723
#4  0x0000ff7915158484 in mono_class_setup_vtable_full (klass=0xff78e8e92920, in_setup=0xff78e945a130) at /__w/1/s/src/mono/mono/metadata/class-setup-vtable.c:960
#5  0x0000ff791529ca68 in mono_method_to_ir (cfg=<optimized out>, method=<optimized out>, start_bblock=0xff78e9443078, end_bblock=0xff78e94431b8, return_var=0x0, inline_args=<optimized out>, inline_offset=<optimized out>, is_virtual_call=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7924
#6  0x0000ff7915271728 in mini_method_compile (method=<optimized out>, opts=374417919, flags=JIT_FLAG_RUN_CCTORS, parts=0, aot_method_index=-1) at /__w/1/s/src/mono/mono/mini/mini.c:3495
#7  0x0000ff791527412c in mono_jit_compile_method_inner (method=0xff78e8e89940, opt=-381297504, error=0xff7909d3c970) at /__w/1/s/src/mono/mono/mini/mini.c:4129
#8  0x0000ff7915278cb0 in mono_jit_compile_method_with_opt (method=0xff78e8e89940, opt=374417919, jit_only=0, error=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2709
#9  jit_compile_method_with_opt_cb (arg=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2764
#10 jit_compile_method_with_opt (params=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2780
#11 0x0000ff79152780dc in mono_jit_compile_method (method=<optimized out>, error=0xff7909d3c970) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2799
#12 0x0000ff7915312a2c in common_call_trampoline (regs=0xff7909d3ca00, code=0xff79082dc1ec "", m=0xff78e8e89940, vt=0x0, vtable_slot=<optimized out>, error=0xff7909d3c970) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:618
#13 0x0000ff791531254c in mono_magic_trampoline (regs=0xff7909d3ca00, code=0xff79082dc1ec "", arg=0xff78e93e72a0, tramp=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:759
#14 0x0000ff7914ee8304 in ?? ()
#15 0x0000ff7914bdf620 in ?? ()
#16 0x0000ff7914b219c0 in ?? ()
#17 0x0064002e00740073 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

there is a core dump
https://dev.azure.com/dnceng-public/public/_build/results?buildId=148909&view=ms.vss-test-web.build-test-results-tab&runId=3170018&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=200309

[stephentoub]

There's another issue representing a crash in source generators on mono that's been plaguing us in CI:
#81123
I don't know if it's the same thing.

[danmoseley]

@janvorli would you or someone on the VM team be interested in the dump here before it vanishes? at least #81123 seems fairly common and this might be related.

[janvorli]

@danmoseley this is mono, so it would be better to point some mono folks to this. The coreclr runtime team doesn't have experience debugging mono.

[danmoseley]

@janvorli - doh - of course, brainstorm. Thanks.

@SamMonoRT there's a dump here if you want it, not sure how long before they time out.

[SamMonoRT]

@lambdageek - can you take an initial look

[lambdageek]

Thread 10 looks ok, actually. it's just doing some work. Thread 9 looks like it crashed:

Thread 9 (Thread 0xff7909f741c0 (LWP 34)):
#0  0x0000ff791595bd5c in __waitpid (pid=<optimized out>, stat_loc=0xff7909f6e9f0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x0000ff79153519e8 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xff7909f6f450, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x0000ff79153105c0 in mono_handle_native_crash (signal=0xff791510e1d2 "SIGSEGV", mctx=0xff7909f6f450, info=0xff7909f6f7b0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
#4  0x0000ff7915279b90 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xff7909f6f7b0, context=0xff7909f6f830, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3749
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x0000ff7908de1d18 in ?? ()
#8  0x0000ff7914ad9848 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

and some output from mono_dump_native_crash_info

=====================================================
instruction pointer is NULL, skip dumping
=====================================================

Frames 7 and 8 look like JITed code. we'll need to try repro locally.

I'll try to grab the coredump once I'm on my work computer Core dump expired.

[steveisok]

@lambdageek the links to builds in #81123 might contain coredumps.

[lewing]

@lambdageek as another data point the crashes only seem to happen on arm64