OOM on large monorepo project

[avatarneil]

Search terms

OOM, Out Of Memory, Heap Allocation, Crash, Performance

Expected Behavior

I expected pnpm typedoc --packages 'packages/*' to run without crashing!

Actual Behavior

pnpm typedoc --packages 'packages/*' crashes when running on >3 packages within our monorepo. It's worth noting, however, that TypeDoc executes perfectly fine when attempting to generate docs for only a handful of packages within the monorepo (e.g. pnpm typedoc --packages 'packages/mds-types' --packages 'packages/mds-agency').

Stack Trace

<--- Last few GCs --->

[45163:0x118008000] 52690 ms: Mark-sweep 4047.0 (4134.4) -> 4036.0 (4139.4) MB, 1383.4 / 0.0 ms (average mu = 0.423, current mu = 0.122) allocation failure scavenge might not succeed
[45163:0x118008000] 54171 ms: Mark-sweep 4052.5 (4140.0) -> 4040.8 (4144.0) MB, 1434.9 / 0.0 ms (average mu = 0.269, current mu = 0.032) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
1: 0x101390285 node::Abort() (.cold.1) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
2: 0x1000c5e59 node::Abort() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
3: 0x1000c5fbf node::OnFatalError(char const*, char const*) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
4: 0x100242fd7 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
5: 0x100242f73 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
6: 0x1003fe015 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
7: 0x10045b674 v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
8: 0x100442ceb void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
9: 0x100442816 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
10: 0x100442516 v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
11: 0x1004603ae v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
12: 0x100418102 v8::internal::ItemParallelJob::Task::RunInternal() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
13: 0x100418588 v8::internal::ItemParallelJob::Run() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
14: 0x100444625 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
15: 0x1004441e7 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
16: 0x10042ef87 v8::internal::MarkCompactCollector::Evacuate() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
17: 0x10042c84b v8::internal::MarkCompactCollector::CollectGarbage() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
18: 0x1003fe734 v8::internal::Heap::MarkCompact() [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
19: 0x1003fb723 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
20: 0x1003f95cd v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
21: 0x10040722a v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
22: 0x1004072b1 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
23: 0x1003d41dd v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
24: 0x100759ebf v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
25: 0x100ad6939 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/neil/.nvm/versions/node/v15.11.0/bin/node]
 ERROR  Command was killed with SIGABRT (Aborted): typedoc --packages packages/*

It seems like this could be pretty tricky to trace down, as there are no pointers in the stack trace to TypeDoc code (at least as far as I can tell) :/

I wonder if this could simply be due to over-parallelization, and if that's the case (perhaps this is a dumb suggestion), could there be an option added to generate documentation serially instead of in parallel to lighten the memory pressure?

Steps to reproduce the bug

Apologies in advance for not being able to create a minimal-repro project for this; I believe this is an issue that arises due to the size of a project.

• Clone mds-core
• git checkout feature/neil/upgrade-typedoc-to-0.21.0
• Ensure pnpm is installed (this is a pnpm based project, YMMV using npm/yarn)
• pnpm clean && pnpm build (clears any local build artifacts that may exist, installs dependencies, builds JS artifacts)
• pnpm typedoc --packages 'packages/*'

Environment

• Typedoc version: 0.21.0
• TypeScript version: 4.2.4
• Node.js version: 15.11.0
• OS: MacOS 12.0b1

[Gerrit0]

Hmm... does typedoc crash before or after creating all the programs? (Does rendering json instead of HTML work?) Unfortunately typedoc relies on having program info throughout the rendering process, so there isn't currently a good way to switch it around to only create one ts.Program at a time. Fixing this would nearly enable generating docs from TypeDoc's json output, which I'd like to be able to someday anyways...

[avatarneil]

@Gerrit0 It seems like with the default memory allocated to node on my machine, emitting to JSON also fails; however, when I bump node's memory limit up to ~32GB, after about 10 minutes it almost succeeds! It seems like memory usage begins dropping, but prior to the emission of the JSON I am faced with a different (I imagine likely unrelated) error 😿 :

TypeDoc exiting with unexpected error:
TypeError: Cannot read property 'map' of undefined
    at Object.convertType (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/types.js:254:57)
    at convertType (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/types.js:96:34)
    at /Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/types.js:254:71
    at Array.map (<anonymous>)
    at Object.convertType (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/types.js:254:57)
    at Object.convertType (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/types.js:96:34)
    at Converter.convertType (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/converter.js:61:24)
    at Object.createSignature (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/factories/signature.js:37:41)
    at convertVariableAsFunction (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/symbols.js:441:21)
    at Object.convertVariable (/Users/neil/Documents/mds.nosync/mds-core/node_modules/.pnpm/typedoc@0.21.0_typescript@4.2.4/node_modules/typedoc/dist/lib/converter/symbols.js:404:16)

Thanks for all your great work on this project and for the quick response, we used to use TypeDoc for generating our project documentation (older version of Typescript & pre-0.20.0 TypeDoc), and our team is eager to be able to use TypeDoc again!

[Gerrit0]

Unfortunately the convertType there means that you still haven't gotten through conversion, so TypeDoc is still working on converting one of the projects.

I think fixing this requires doing a couple things:

1. Refactor such that resolution happens in stages, and after resolution, no references to compiler nodes or symbols are kept. This is necessary since if we don't do this, then we have to keep every program in memory until all resolution is complete
2. Refactor to create only one ts.Program at a time

This is a rather large change... it's possible, but by no means trivial.

[danni]

I've run into what looks like the same bug. Medium-sized mono-repo using project references to build, publishes an acyclic graph of packages. Uses a yarn workspace and lerna.

Runs out of memory in packages mode. It does however complete if I give it a list of entrypoints in resolve mode. It doesn't seem to make any difference if I give a list of packages entrypoints in packages mode.

This project was only recently ported to project references (from a pure lerna setup) and I have memory of my pre-project references setup working correctly. But I foolishly didn't keep that branch, if you're interested I can recreate it.

[jlarmstrongiv]

Any workarounds besides increasing memory?

Is it possible to build each project’s typedoc individually and then merge them at the end?

Giving it a list of entrypoints in resolve mode did not work for me.

[Gerrit0]

Is it possible to build each project’s typedoc individually and then merge them at the end?

Once 0.23 is out, this is what I plan to work on next, and is how I'd like packages mode to always work, but it's not there yet.

[jlarmstrongiv]

Cool! Looking forward to it

That’s similar to how merging jest coverage reports with nyc works