orca graph can't parse very large JSON files

[etpinard]

From plotly/plotly.js#3386

• download https://1drv.ms/u/s!AiKoo8nP3bRiaZALJrLSl4lM8I8
• run orca graph heatmap.graph.json --debug
• and get export error 422 for heatmap - json parse error

I suspect that we're hitting the Node readFile to-utf-8 limit here:

orca/src/app/runner/get-body.js

Line 39 in b2d6556

fs.readFile(p, 'utf-8', done)

That is,

var fs = require('fs')
fs.readFileSync('./heatmap.graph.json', 'utf-8')

errors, see stacktrace

--- Last few GCs --->

[20617:0x43cfdc0] 72029 ms: Mark-sweep 1113.0 (1122.9) -> 1113.0 (1117.4) MB, 4.5 / 0.0 ms (average mu = 0.877, current mu = 0.006) last resort GC in old space requested
[20617:0x43cfdc0] 72033 ms: Mark-sweep 1113.0 (1117.4) -> 1113.0 (1117.4) MB, 3.8 / 0.0 ms (average mu = 0.833, current mu = 0.008) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 0x31eb9abdc01d]

Security context: 0x051a9ab11cb9
1: handleWriteReq(aka handleWriteReq) [0x2f2a210028f9] [internal/stream_base_commons.js:20] [bytecode=0x17a0c39bf409 offset=139](this=0x2e88262826f1 ,req=0x2f2a2100e179 ,data=0x2f2a2100e101 <Very long string[580987116]>,encoding=0x051a9ab24059 <String[4]: utf8>)
2: writeGeneric(aka writeGeneric) [0x2f2a210028...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: 0x8c02c0 node::Abort() [node]
2: 0x8c030c [node]
3: 0xad15de v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xad1814 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xebe752 [node]
6: 0xecdfbf v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
7: 0xe962a5 [node]
8: 0xe9db3a v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [node]
9: 0xfe704d v8::internal::String::SlowFlatten(v8::internal::Handlev8::internal::ConsString, v8::internal::PretenureFlag) [node]
10: 0xacebe4 v8::internal::String::Flatten(v8::internal::Handlev8::internal::String, v8::internal::PretenureFlag) [node]
11: 0xadc6d0 v8::String::Utf8Length() const [node]
12: 0x96b2f8 node::StringBytes::Size(v8::Isolate*, v8::Localv8::Value, node::encoding) [node]
13: 0x97246b int node::StreamBase::WriteString<(node::encoding)1>(v8::FunctionCallbackInfov8::Value const&) [node]
14: 0x9798c1 void node::StreamBase::JSMethod<node::LibuvStreamWrap, &(int node::StreamBase::WriteString<(node::encoding)1>(v8::FunctionCallbackInfov8::Value const&))>(v8::FunctionCallbackInfov8::Value const&) [node]
15: 0xb5996f [node]
16: 0xb5a4d9 v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) [node]
17: 0x31eb9abdc01d
Aborted (core dumped)

whereas

var fs = require('fs')
fs.readFileSync('./heatmap.graph.json')

works (it returns a buffer)

---

Potential solutions:

• use fs.readFile() w/o the utf-8 arg and parse Buffer into an object some other way
• use streams instead on large files