Segmentation Fault of node debugger/inspector with Chrome Memory Devtools

[Kmaschta]

• Version: v8.9.4
• Platform:
  • (docker) Linux e2bd997af1ec 4.9.60-linuxkit-aufs deps: update openssl to 1.0.1j #1 SMP Mon Nov 6 16:00:12 UTC 2017 x86_64 GNU/Linux
  • (host) Darwin air-marmelab.lan 16.7.0 Darwin Kernel Version 16.7.0: Thu Jan 11 22:59:40 PST 2018; root:xnu-3789.73.8~1/RELEASE_X86_64 x86_64
• Subsystem:

In order to reproduce, I just need to run a node server with node --inspect index.js (with an express server) with 10 concurrent requests (sent with siege).

I connect to the inspector thanks to a Chrome browser (version 63), in a the chrome://inspect, and when I take a Memory snapshot or record, I get a SIGSEGV signal.

Here is a report generated by the segfault-handler module after a catched segfault:

PID 21 received SIGSEGV for address: 0x2
/app/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x1a7b)[0x7fecdddfaa7b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7fecf7229890]
node /app/build/index.js(_ZN2v88internal14V8HeapExplorer8AddEntryEPNS0_10HeapObjectE+0x13)[0x1062fc3]
node /app/build/index.js(_ZN2v88internal14V8HeapExplorer19SetContextReferenceEPNS0_10HeapObjectEiPNS0_6StringEPNS0_6ObjectEi+0xf0)[0x10654f0]
node /app/build/index.js(_ZN2v88internal14V8HeapExplorer24ExtractContextReferencesEiPNS0_7ContextE+0x2370)[0x10678a0]
node /app/build/index.js(_ZN2v88internal14V8HeapExplorer22ExtractReferencesPass2EiPNS0_10HeapObjectE+0xa8)[0x10679b8]
node /app/build/index.js(_ZN2v88internal14V8HeapExplorer27IterateAndExtractSinglePassIXadL_ZNS1_22ExtractReferencesPass2EiPNS0_10HeapObjectEEEEEbv+0x275)[0x106d075]
node /app/build/index.js(_ZN2v88internal14V8HeapExplorer27IterateAndExtractReferencesEPNS0_14SnapshotFillerE+0x298)[0x106da18]
node /app/build/index.js(_ZN2v88internal21HeapSnapshotGenerator16GenerateSnapshotEv+0x12a)[0x106dbca]
node /app/build/index.js(_ZN2v88internal12HeapProfiler12TakeSnapshotEPNS_15ActivityControlEPNS_12HeapProfiler18ObjectNameResolverE+0x5c)[0x1058f0c]
node /app/build/index.js(_ZN12v8_inspector23V8HeapProfilerAgentImpl16takeHeapSnapshotENS_8protocol5MaybeIbEE+0xab)[0xabd3bb]
node /app/build/index.js(_ZN12v8_inspector8protocol12HeapProfiler14DispatcherImpl16takeHeapSnapshotEiSt10unique_ptrINS0_15DictionaryValueESt14default_deleteIS4_EEPNS0_12ErrorSupportE+0x189)[0xa69819]
node /app/build/index.js(_ZN12v8_inspector8protocol12HeapProfiler14DispatcherImpl8dispatchEiRKNS_8String16ESt10unique_ptrINS0_15DictionaryValueESt14default_deleteIS7_EE+0xe6)[0xa6c086]
node /app/build/index.js(_ZN12v8_inspector8protocol14UberDispatcher8dispatchESt10unique_ptrINS0_5ValueESt14default_deleteIS3_EE+0x55c)[0xa54bfc]
node /app/build/index.js(_ZN12v8_inspector22V8InspectorSessionImpl23dispatchProtocolMessageERKNS_10StringViewE+0x22)[0xac3e42]
node /app/build/index.js[0x12ba03c]
node /app/build/index.js(_ZN4node12NodePlatform28FlushForegroundTasksInternalEv+0x1f4)[0x1273624]
node /app/build/index.js[0x143e44b]
node /app/build/index.js[0x144ffa8]
node /app/build/index.js(uv_run+0x156)[0x143edd6]
node /app/build/index.js(_ZN4node5StartEP9uv_loop_siPKPKciS5_+0xc8d)[0x122c1bd]
node /app/build/index.js(_ZN4node5StartEiPPc+0x163)[0x1224d03]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fecf6e90b45]
node /app/build/index.js[0x8aee41]

Here there anything I can do?

[Kmaschta]

Certainly related to #18223 !

[Kmaschta]

I just have the same error on linux, inside the same docker:

• Version: v8.9.4
• Platform:
  • (docker) Linux e2bd997af1ec 4.9.60-linuxkit-aufs deps: update openssl to 1.0.1j #1 SMP Mon Nov 6 16:00:12 UTC 2017 x86_64 GNU/Linux
  • (host) Linux kmaschta-marmelab 4.13.0-16-lowlatency Multiple persistent child-process-related test failures on Windows when running as a Service #19-Ubuntu SMP PREEMPT Wed Oct 11 19:51:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Report:

PID 22 received SIGSEGV for address: 0x2
/app/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x1b19)[0x7f8e9a7d2b19]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f8eb3d72890]
node /app/build/index.js(_ZN2v88internal14V8HeapExplorer8AddEntryEPNS0_10HeapObjectE+0x13)[0x10629c3]
node /app/build/index.js(_ZN2v88internal14V8HeapExplorer19SetContextReferenceEPNS0_10HeapObjectEiPNS0_6StringEPNS0_6ObjectEi+0xf0)[0x1064ef0]
node /app/build/index.js(_ZN2v88internal14V8HeapExplorer24ExtractContextReferencesEiPNS0_7ContextE+0x2370)[0x10672a0]
node /app/build/index.js(_ZN2v88internal14V8HeapExplorer22ExtractReferencesPass2EiPNS0_10HeapObjectE+0xa8)[0x10673b8]
node /app/build/index.js(_ZN2v88internal14V8HeapExplorer27IterateAndExtractSinglePassIXadL_ZNS1_22ExtractReferencesPass2EiPNS0_10HeapObjectEEEEEbv+0x275)[0x106ca75]
node /app/build/index.js(_ZN2v88internal14V8HeapExplorer27IterateAndExtractReferencesEPNS0_14SnapshotFillerE+0x298)[0x106d418]
node /app/build/index.js(_ZN2v88internal21HeapSnapshotGenerator16GenerateSnapshotEv+0x12a)[0x106d5ca]
node /app/build/index.js(_ZN2v88internal12HeapProfiler12TakeSnapshotEPNS_15ActivityControlEPNS_12HeapProfiler18ObjectNameResolverE+0x5c)[0x105890c]
node /app/build/index.js(_ZN12v8_inspector23V8HeapProfilerAgentImpl16takeHeapSnapshotENS_8protocol5MaybeIbEE+0xab)[0xabcdbb]
node /app/build/index.js(_ZN12v8_inspector8protocol12HeapProfiler14DispatcherImpl16takeHeapSnapshotEiSt10unique_ptrINS0_15DictionaryValueESt14default_deleteIS4_EEPNS0_12ErrorSupportE+0x189)[0xa69219]
node /app/build/index.js(_ZN12v8_inspector8protocol12HeapProfiler14DispatcherImpl8dispatchEiRKNS_8String16ESt10unique_ptrINS0_15DictionaryValueESt14default_deleteIS7_EE+0xe6)[0xa6ba86]
node /app/build/index.js(_ZN12v8_inspector8protocol14UberDispatcher8dispatchESt10unique_ptrINS0_5ValueESt14default_deleteIS3_EE+0x55c)[0xa545fc]
node /app/build/index.js(_ZN12v8_inspector22V8InspectorSessionImpl23dispatchProtocolMessageERKNS_10StringViewE+0x22)[0xac3842]
node /app/build/index.js[0x12b89fc]
node /app/build/index.js(_ZN4node12NodePlatform28FlushForegroundTasksInternalEv+0x1f4)[0x1272174]
node /app/build/index.js[0x145796b]
node /app/build/index.js[0x14694c8]
node /app/build/index.js(uv_run+0x156)[0x14582f6]
node /app/build/index.js(_ZN4node5StartEP9uv_loop_siPKPKciS5_+0xc75)[0x122af15]
node /app/build/index.js(_ZN4node5StartEiPPc+0x163)[0x1223b73]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f8eb39d9b45]
node /app/build/index.js[0x8ae7c1]

[addaleax]

@Kmaschta Can you provide a core dump (which would already be very helpful), or maybe even try to reproduce this with a debug build of Node? (Or provide code to reproduce this?)

/cc @nodejs/v8

[hashseed]

@Kmaschta could you check whether this is indeed a duplicate of #18223 by running a debug build of Node.js?

[Kmaschta]

I've hard time generating a core dump with Docker ...
I'll try to build a node locally, but I can't promise a result.

[Kmaschta]

Hi there,

I forked Node, cherry-picked the commit that seems to fix the bug in #18223
(here https://github.com/Kmaschta/node/commits/v8.x) and tested my app with a freshly built Node.
But my app still crashes when I take a heap snapshot.

Did I make a mistake ? How can I help you now ?
Do you know how can I get a core dump somewhat easily ?

[Kmaschta]

@addaleax @hashseed Hey, I managed to get a core dump ! Should I upload it here?

Here is the backtrace found in the core:

(llnode) v8 bt
 * thread #1: tid = 27572, 0x00007ffe34e23fc3 node`v8::internal::V8HeapExplorer::AddEntry(v8::internal::HeapObject*) + 19, name = 'node', stop reason = signal SIGSEGV
  * frame #0: 0x00007ffe34e23fc3 node`v8::internal::V8HeapExplorer::AddEntry(v8::internal::HeapObject*) + 19
    frame #1: 0x00007ffe34e264f0 node`v8::internal::V8HeapExplorer::SetContextReference(v8::internal::HeapObject*, int, v8::internal::String*, v8::internal::Object*, int) + 240
    frame #2: 0x00007ffe34e288a0 node`v8::internal::V8HeapExplorer::ExtractContextReferences(int, v8::internal::Context*) + 9072
    frame #3: 0x00007ffe34e289b8 node`v8::internal::V8HeapExplorer::ExtractReferencesPass2(int, v8::internal::HeapObject*) + 168
    frame #4: 0x00007ffe34e2e075 node`bool v8::internal::V8HeapExplorer::IterateAndExtractSinglePass<&(v8::internal::V8HeapExplorer::ExtractReferencesPass2(int, v8::internal::HeapObject*))>() + 629
    frame #5: 0x00007ffe34e2ea18 node`v8::internal::V8HeapExplorer::IterateAndExtractReferences(v8::internal::SnapshotFiller*) + 664
    frame #6: 0x00007ffe34e2ebca node`v8::internal::HeapSnapshotGenerator::GenerateSnapshot() + 298
    frame #7: 0x00007ffe34e19f0c node`v8::internal::HeapProfiler::TakeSnapshot(v8::ActivityControl*, v8::HeapProfiler::ObjectNameResolver*) + 92
    frame #8: 0x00007ffe3487e3bb node`v8_inspector::V8HeapProfilerAgentImpl::takeHeapSnapshot(v8_inspector::protocol::Maybe<bool>) + 171
    frame #9: 0x00007ffe3482a819 node`v8_inspector::protocol::HeapProfiler::DispatcherImpl::takeHeapSnapshot(int, std::unique_ptr<v8_inspector::protocol::DictionaryValue, std::default_delete<v8_inspector::protocol::DictionaryValue> >, v8_inspector::protocol::ErrorSupport*) + 393
    frame #10: 0x00007ffe3482d086 node`v8_inspector::protocol::HeapProfiler::DispatcherImpl::dispatch(int, v8_inspector::String16 const&, std::unique_ptr<v8_inspector::protocol::DictionaryValue, std::default_delete<v8_inspector::protocol::DictionaryValue> >) + 230
    frame #11: 0x00007ffe34815bfc node`v8_inspector::protocol::UberDispatcher::dispatch(std::unique_ptr<v8_inspector::protocol::Value, std::default_delete<v8_inspector::protocol::Value> >) + 1372
    frame #12: 0x00007ffe34884e42 node`v8_inspector::V8InspectorSessionImpl::dispatchProtocolMessage(v8_inspector::StringView const&) + 34
    frame #13: 0x00007ffe3507b03c node`node::inspector::InspectorIo::DispatchMessages() (.part.70) + 492
    frame #14: 0x00007ffe35034624 node`node::NodePlatform::FlushForegroundTasksInternal() + 500
    frame #15: node`uv__async_io(loop=<unavailable>, w=<unavailable>, events=<unavailable>) at async.c:118
    frame #16: node`uv__io_poll(loop=<unavailable>, timeout=<unavailable>) at linux-core.c:400
    frame #17: node`uv_run(loop=<unavailable>, mode=<unavailable>) at core.c:368
    frame #18: 0x00007ffe34fed1bd node`node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) + 3213
    frame #19: 0x00007ffe34fe5d03 node`node::Start(int, char**) + 355
    frame #20: libc.so.6`__libgcc_s_init at unwind-resume.c:34

[Kmaschta]

Another nice message that I got from a segfault:

node[7304]: ../src/async-wrap.cc:132:v8::RetainedObjectInfo* node::WrapperInfo(uint16_t, v8::Local<v8::Value>): Assertion `(nullptr) != (wrap)' failed.
 1: node::Abort() [node]
 2: 0x121a6bb [node]
 3: node::WrapperInfo(unsigned short, v8::Local<v8::Value>) [node]
 4: v8::internal::HeapProfiler::ExecuteWrapperClassCallback(unsigned short, v8::internal::Object**) [node]
 5: v8::internal::GlobalHandlesExtractor::VisitPersistentHandle(v8::Persistent<v8::Value, v8::NonCopyablePersistentTraits<v8::Value> >*, unsigned short) [node]
 6: v8::internal::GlobalHandles::IterateAllRootsWithClassIds(v8::PersistentHandleVisitor*) [node]
 7: 0x106bc37 [node]
 8: v8::internal::HeapSnapshotGenerator::GenerateSnapshot() [node]
 9: v8::internal::HeapProfiler::TakeSnapshot(v8::ActivityControl*, v8::HeapProfiler::ObjectNameResolver*) [node]
10: v8_inspector::V8HeapProfilerAgentImpl::takeHeapSnapshot(v8_inspector::protocol::Maybe<bool>) [node]
11: v8_inspector::protocol::HeapProfiler::DispatcherImpl::takeHeapSnapshot(int, std::unique_ptr<v8_inspector::protocol::DictionaryValue, std::default_delete<v8_inspector::protocol::DictionaryValue> >, v8_inspector::protocol::ErrorSupport*) [node]
12: v8_inspector::protocol::HeapProfiler::DispatcherImpl::dispatch(int, v8_inspector::String16 const&, std::unique_ptr<v8_inspector::protocol::DictionaryValue, std::default_delete<v8_inspector::protocol::DictionaryValue> >) [node]
13: v8_inspector::protocol::UberDispatcher::dispatch(std::unique_ptr<v8_inspector::protocol::Value, std::default_delete<v8_inspector::protocol::Value> >) [node]
14: v8_inspector::V8InspectorSessionImpl::dispatchProtocolMessage(v8_inspector::StringView const&) [node]
15: 0x12ba03c [node]
16: v8::internal::Isolate::InvokeApiInterruptCallbacks() [node]
17: v8::internal::StackGuard::HandleInterrupts() [node]
18: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [node]
19: 0x3e6c5b38463d

[addaleax]

@addaleax @hashseed Hey, I managed to get a core dump ! Should I upload it here?

@Kmaschta If you program handles data that is not public, it will likely end up in the core dump, so you might want to upload it somewhere and email us links (they are in the https://github.com/nodejs/node/ README). If not, feel free to post a link publicly.

Either way, yes, that is going to be very helpful!

[bnoordhuis]

@Kmaschta That last one is #18256. I have it on my radar.

[jasnell]

is this still an issue?