neo4j-driver causing Node.js process to crash silently

[acjohnso25]

We're using neo4j-driver 1.5.2 in a Node.js service and have been seeing mysterious crashes for months now. We tried catching uncaught exceptions and even wrapping process.exit() to try to track them down, but no luck; the process always just dies silently. Finally we decided to turn on some profiling and saw this on the next crash:

==== JS stack trace =========================================

Security context: 0x3deeb22b4629
    1: stringify(aka stringify) [native json.js:157] [pc=0x13b372f3a284] (this=0x3deeb22041b9 ,u=0x28cad263c799 >,v=0x3deeb22041b9 ,I=0x3deeb22041b9 )
    2: arguments adaptor frame: 1->3
    3: /* anonymous */(aka /* anonymous */) [/opt/FillerService+r_0.0.346/server/node_modules/neo4j-driver/node_modules/babel-runtime/node_modules/core-js/library/modules/es6.symbol.js:209] [pc=0x13b37317fdcf] (this=0x3deeb22041b9 )
    4: /* anonymous */(aka /* anonymous */) [/opt/FillerService+r_0.0.346/server/node_modules/neo4j-driver/node_modules/babel-runtime/node_modules/core-js/library/modules/_FAILs.js:3] [pc=0x13b3731653bb] (this=0x3deeb22041b9 ,exec=0x28cad263c281 )
    5: /* anonymous */ [/opt/FillerService+r_0.0.346/server/node_modules/neo4j-driver/node_modules/babel-runtime/node_modules/core-js/library/modules/es6.symbol.js:204] [pc=0x13b37317a99b] (this=0x28cad25bbfe1 ,exports=0x28cad25bbfe1 ,require=0x28cad25c3a91 ,module=0x28cad25bbf91 ,__filename=0x28cad25ba711 ,__dirname=0x28cad25c3a29 )
    7: _compile [module.js:409] [pc=0x13b372f4b469] (this=0x28cad25bbf91 ,content=0x28cad25bec49 ,filename=0x28cad25ba711 )
    8: .js [module.js:~414] [pc=0x13b37302ae6b] (this=0x2a7f02ef6821 ,module=0x28cad25bbf91 ,filename=0x28cad25ba711 )
    9: load [module.js:343] [pc=0x13b372f422bb] (this=0x28cad25bbf91 ,filename=0x28cad25ba711 )
   10: _load [module.js:300] [pc=0x13b372f37be4] (this=0x2a7f02ef6859 ,request=0x3e1a00fdf229 ,parent=0x28cad25b4879 ,isMain=0x3deeb22041b9 )
   11: arguments adaptor frame: 2->3
   12: require(aka require) [internal/module.js:~11] [pc=0x13b372fb691b] (this=0x3deeb22041b9 ,path=0x3e1a00fdf229 )
   13: /* anonymous */ [/opt/FillerService+r_0.0.346/server/node_modules/neo4j-driver/node_modules/babel-runtime/node_modules/core-js/library/fn/symbol/index.js:1] [pc=0x13b3731789bb] (this=0x28cad25b48c9 ,exports=0x28cad25b48c9 ,require=0x28cad25b7f71 ,module=0x28cad25b4879 ,__filename=0x28cad25b2e21 ,__dirname=0x28cad25b7f09 )
   15: _compile [module.js:409] [pc=0x13b372f4b469] (this=0x28cad25b4879 ,content=0x28cad25b7901 ,filename=0x28cad25b2e21 )
   16: .js [module.js:~414] [pc=0x13b37302ae6b] (this=0x2a7f02ef6821 ,module=0x28cad25b4879 ,filename=0x28cad25b2e21 )
   17: load [module.js:343] [pc=0x13b372f422bb] (this=0x28cad25b4879 ,filename=0x28cad25b2e21 )
   18: _load [module.js:300] [pc=0x13b372f37be4] (this=0x2a7f02ef6859 ,request=0x3e1a00fde7c9 ,parent=0x28cad25accd9 ,isMain=0x3deeb22041b9 )
   19: arguments adaptor frame: 2->3
   20: require(aka require) [internal/module.js:~11] [pc=0x13b372fb691b] (this=0x3deeb22041b9 ,path=0x3e1a00fde7c9 )
   21: /* anonymous */ [/opt/FillerService+r_0.0.346/server/node_modules/neo4j-driver/node_modules/babel-runtime/core-js/symbol.js:1] [pc=0x13b373178794] (this=0x28cad25acd29 ,exports=0x28cad25acd29 ,require=0x28cad25af479 ,module=0x28cad25accd9 ,__filename=0x28cad25abac9 ,__dirname=0x28cad25af411 )
   23: _compile [module.js:409] [pc=0x13b372f4b469] (this=0x28cad25accd9 ,content=0x28cad25aef19 ,filename=0x28cad25abac9 )
   24: .js [module.js:~414] [pc=0x13b37302ae6b] (this=0x2a7f02ef6821 ,module=0x28cad25accd9 ,filename=0x28cad25abac9 )
   25: load [module.js:343] [pc=0x13b372f422bb] (this=0x28cad25accd9 ,filename=0x28cad25abac9 )
   26: _load [module.js:300] [pc=0x13b372f37be4] (this=0x2a7f02ef6859 ,request=0x3e1a00fc4d89 ,parent=0x28cad24a74c9 ,isMain=0x3deeb22041b9 )
   27: arguments adaptor frame: 2->3
   28: require(aka require) [internal/module.js:~11] [pc=0x13b372fb691b] (this=0x3deeb22041b9 ,path=0x3e1a00fc4d89 )
   29: /* anonymous */ [/opt/FillerService+r_0.0.346/server/node_modules/neo4j-driver/node_modules/babel-runtime/helpers/typeof.js:9] [pc=0x13b37316b0ba] (this=0x28cad24a7519 ,exports=0x28cad24a7519 ,require=0x28cad24aa551 ,module=0x28cad24a74c9 ,__filename=0x28cad24a62b9 ,__dirname=0x28cad24aa4e9 )
   31: _compile [module.js:409] [pc=0x13b372f4b469] (this=0x28cad24a74c9 ,content=0x28cad24a97a9 ,filename=0x28cad24a62b9 )
   32: .js [module.js:~414] [pc=0x13b37302ae6b] (this=0x2a7f02ef6821 ,module=0x28cad24a74c9 ,filename=0x28cad24a62b9 )
   33: load [module.js:343] [pc=0x13b372f422bb] (this=0x28cad24a74c9 ,filename=0x28cad24a62b9 )
   34: _load [module.js:300] [pc=0x13b372f37be4] (this=0x2a7f02ef6859 ,request=0x3e1a00fc43b1 ,parent=0x28cad24a1201 ,isMain=0x3deeb22041b9 )
   35: arguments adaptor frame: 2->3
   36: require(aka require) [internal/module.js:~11] [pc=0x13b372fb691b] (this=0x3deeb22041b9 ,path=0x3e1a00fc43b1 )
   37: /* anonymous */ [/opt/FillerService+r_0.0.346/server/node_modules/neo4j-driver/node_modules/babel-runtime/helpers/possibleConstructorReturn.js:5] [pc=0x13b37316ad5b] (this=0x28cad24a1251 ,exports=0x28cad24a1251 ,require=0x28cad24a3df9 ,module=0x28cad24a1201 ,__filename=0x28cad249ffb1 ,__dirname=0x28cad24a3d91 )
   39: _compile [module.js:409] [pc=0x13b372f4b469] (this=0x28cad24a1201 ,content=0x28cad24a34e9 ERROR("this hasn't been initialised - super() hasn't been called");\n  }\n\n  return call && ((typeof call === "undefined" ? "undefined" : (0, _typeof3.default)(call)) === "object" || typeof call === "function") ? call : self;\n};>,filename=0x28cad249ffb1 )
   40: .js [module.js:~414] [pc=0x13b37302ae6b] (this=0x2a7f02ef6821 ,module=0x28cad24a1201 ,filename=0x28cad249ffb1 )
   41: load [module.js:343] [pc=0x13b372f422bb] (this=0x28cad24a1201 ,filename=0x28cad249ffb1 )
   42: _load [module.js:300] [pc=0x13b372f37be4] (this=0x2a7f02ef6859 ,request=0x3e1a00fbaba1 ,parent=0x28cad2441319 ,isMain=0x3deeb22041b9 )
   43: arguments adaptor frame: 2->3
   44: require(aka require) [internal/module.js:~11] [pc=0x13b372fb691b] (this=0x3deeb22041b9 ,path=0x3e1a00fbaba1 )
   45: /* anonymous */ [/opt/FillerService+r_0.0.346/server/node_modules/neo4j-driver/lib/v1/ERROR.js:16] [pc=0x13b373166e3c] (this=0x28cad2441369 ,exports=0x28cad2441369 ,require=0x28cad2445171 ,module=0x28cad2441319 ,__filename=0x28cad24403b1 ERROR.js>,__dirname=0x28cad2445109 )
   47: _compile [module.js:409] [pc=0x13b372f4b469] (this=0x28cad2441319 ,content=0x28cad2443381 ,filename=0x28cad24403b1 ERROR.js>)
   48: .js [module.js:~414] [pc=0x13b37302ae6b] (this=0x2a7f02ef6821 ,module=0x28cad2441319 ,filename=0x28cad24403b1 ERROR.js>)
   49: load [module.js:343] [pc=0x13b372f422bb] (this=0x28cad2441319 ,filename=0x28cad24403b1 ERROR.js>)
   50: _load [module.js:300] [pc=0x13b372f37be4] (this=0x2a7f02ef6859 ,request=0x3e1a00fa79c1 ERROR>,parent=0x28cad23950d9 ,isMain=0x3deeb22041b9 )
   51: arguments adaptor frame: 2->3
   52: require(aka require) [internal/module.js:~11] [pc=0x13b372fb691b] (this=0x3deeb22041b9 ,path=0x3e1a00fa79c1 ERROR>)
   53: /* anonymous */ [/opt/FillerService+r_0.0.346/server/node_modules/neo4j-driver/lib/v1/integer.js:16] [pc=0x13b37316064b] (this=0x28cad2395129 ,exports=0x28cad2395129 ,require=0x28cad23a61e9 ,module=0x28cad23950d9 ,__filename=0x28cad2394161 ,__dirname=0x28cad23a6181 )
   55: _compile [module.js:409] [pc=0x13b372f4b469] (this=0x28cad23950d9 ,content=0x28cad2397221 ,filename=0x28cad2394161 )
   56: .js [module.js:~414] [pc=0x13b37302ae6b] (this=0x2a7f02ef6821 ,module=0x28cad23950d9 ,filename=0x28cad2394161 )
   57: load [module.js:343] [pc=0x13b372f422bb] (this=0x28cad23950d9 ,filename=0x28cad2394161 )
   58: _load [module.js:300] [pc=0x13b372f37be4] (this=0x2a7f02ef6859 ,request=0x3e1a00fa7949 ,parent=0x28cad2389f71 ,isMain=0x3deeb22041b9 )
   59: arguments adaptor frame: 2->3
   60: require(aka require) [internal/module.js:~11] [pc=0x13b372fb691b] (this=0x3deeb22041b9 ,path=0x3e1a00fa7949 )
   61: /* anonymous */ [/opt/FillerService+r_0.0.346/server/node_modules/neo4j-driver/lib/v1/index.js:8] [pc=0x13b37315e670] (this=0x28cad2389fc1 ,exports=0x28cad2389fc1 ,require=0x28cad2391b51 ,module=0x28cad2389f71 ,__filename=0x28cad2388d31 ,__dirname=0x28cad2391ae9 )
   63: _compile [module.js:409] [pc=0x13b372f4b469] (this=0x28cad2389f71 ,content=0x28cad238c0a9 ,filename=0x28cad2388d31 )
   64: .js [module.js:~414] [pc=0x13b37302ae6b] (this=0x2a7f02ef6821 ,module=0x28cad2389f71 ,filename=0x28cad2388d31 )
   65: load [module.js:343] [pc=0x13b372f422bb] (this=0x28cad2389f71 ,filename=0x28cad2388d31 )
   66: _load [module.js:300] [pc=0x13b372f37be4] (this=0x2a7f02ef6859 ,request=0x3e1a00fa6fa1 ,parent=0x28cad2383ef1 ,isMain=0x3deeb22041b9 )
   67: arguments adaptor frame: 2->3
   68: require(aka require) [internal/module.js:~11] [pc=0x13b372fb691b] (this=0x3deeb22041b9 ,path=0x3e1a00fa6fa1 )
   69: /* anonymous */ [/opt/FillerService+r_0.0.346/server/node_modules/neo4j-driver/lib/index.js:8] [pc=0x13b37315e227] (this=0x28cad2383f41 ,exports=0x28cad2383f41 ,require=0x28cad2386d11 ,module=0x28cad2383ef1 ,__filename=0x28cad2382b51 ,__dirname=0x28cad2386ca9 )
...

I'm not really sure what's going on here, or even how to reproduce it. It's very intermittent, doesn't seem to happen in any predictable way. (Believe me, I've tried.)

If anybody needs more info or has ideas for ways to repro, I'm all ears.

[lutovich]

Hi @acjohnso25,

We have never seen problem like this before.
Looks like your whole Node process crashes and leaves this stacktrace behind, right? Please attach the full crash report, if possible.

Which version of Node are you using?
Do you think it's possible that process ran out of memory and crashed because of this?

[acjohnso25]

We are using Node.js version 4.8.0. Our Neo4j cluster is running 3.2.7.

The possibility of running out of memory occurred to me previously, so I already hacked in a bit of monitoring -- basically spitting out the results of process.memoryUsage() every 60s -- figuring that would show me if the footprint was growing steadily leading up to each crash. However, it's not; the size up to each crash seems to fluctuate normally, and (more importantly) isn't noticeably different than other instances running at the same time (that didn't crash).

I suppose it's possible that something goes haywire inside that 60s interval so it's not showing up there. However, we always run four instances of this process on each machine, so if one was suddenly chewing up all the resources, I'd expect to see all four (or at least more than one) crash around the same time...but this has not happened once.

I'll post the complete crash report in a separate comment.

[acjohnso25]

crashdump-MicroService-13405.txt

[acjohnso25]

BTW, the crash happened again this morning, on a different machine. The stack trace from that crash is basically the same as the one I pasted initially.

[lutovich]

Is that a full crash dump? It looks cut at the end.

[acjohnso25]

Hm...perhaps. That was what was on the machine; the only thing I removed were some additional logs at the end that contained our service and hostnames. I'll find and post another one...

[acjohnso25]

crashdump-MicroService-13416.txt

[acjohnso25]

Here are the dependency versions installed on one of our production instances, from npm ls, if that helps...

├─┬ [email protected]
│ └─┬ [email protected]
│   ├── [email protected]
│   └── [email protected]

[lutovich]

@acjohnso25 thanks for uploading the file. Stacktrace points to transpiled code and ends in driver's error.js file on line

var _possibleConstructorReturn2 = require('babel-runtime/helpers/possibleConstructorReturn');

it's only babel's code from then on.

We have not seen such errors before, so they might be environment dependent.

Could you please increase memory limit for Node to 4GB? I know you have monitoring and we see nothing about memory in the crashdump, but increasing heap will help us to completely rule out memory usage issue.

Is upgrading Node an option for you?

[acjohnso25]

@lutovich Neither of those is trivial to do in our production environment (which is where we are seeing this) at the moment. I'll provide an update if we get the chance to do that.

I'm taking a look through babel-runtime issues to see if there are any similar reports there...but there are 460 issues open...which is a lot... :(

[acjohnso25]

@lutovich Is there an older version of the driver that we could try, prior to it using babel-runtime? I've tried to figure out what release incorporated this change, but github's UI doesn't seem to make this easy.

The best I've been able to figure out is that babel-runtime was added with this commit: e6c5ce1

But figuring out what version that commit wound up in is not obvious to a GH neophyte like me... (I've used Bitbucket more.)

Update: I found the PR, #162, which suggests the change was merged into 1.1 branch.

[lutovich]

Hi @acjohnso25,

Did you have a chance to try the pre-babel driver?
Is it possible for you to use node-report with the app so that it generates a different and hopefully more useful crash dump?

[lutovich]

Hi @acjohnso25,

Do you still experience this issue? Did you try newer Node version or maybe have more info about how can we reproduce the problem?

[acjohnso25]

Hi @lutovich, sorry for not getting back to you last time. Busy couple of weeks.

I never did try the pre-babel driver; it appeared that I'd have to step back to 1.0 or a 0.x release to do so, which I figured would probably introduce all kinds of other problems.

node-report is a good suggestion, as is updating Node.js. Those are both on our to-do list, but require changes to our common platform, so we simply haven't had a chance to try them yet.

Sorry, I don't have any more info on reproducing the problem at this time.

[zhenlineo]

Close due to lack of information. Pls be free to reopen it once more information is available.