Recommendation for multi-module setup? Memory leak with 18+ Rollup watchers

[marijnh]

I have a project with a bunch (18) separate (TS) modules, each of which is compiled down to its own .js file with rollup and this plugin. I'd like to set up a single rollup process to watch and recompile the lot of them, which I'm currently doing with a rollup config file that instantiates this plugin 18 times, each time with a different include setting, because otherwise the plugin will emit .d.ts files for all the files in each project's dist directory.

This regularly (but only after a bunch of recompile cycles) leads to node running out of memory:

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x134e879]
Security context: 0x2ed275600919 <JSObject>
    1: getDeclarationName(aka getDeclarationName) [0x3270c86f3011] [/home/marijn/src/cm6/node_modules/typescript/lib/typescript.js:~27802] [pc=0x34f390cf2ea2](this=0x2b31f95404d1 <undefined>,0x2df9bfc13a39 <NodeObject map = 0x2513288541d9>)
    2: declareSymbol(aka declareSymbol) [0x3270c86f3091] [/home/marijn/src/cm6/node_modules/typescript/lib/typescript.js:~27...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

I guess that has something to do with the TypeScript compiler building up its type/code graphs multiple times, once for every instance of the plugin. Is that correct?

Is there a way to set up the plugin once, but use it to build several projects, each of which emits .d.ts files only for its own files?

Or is there maybe some other recommended way to approach a use case like this?

[ezolenko]

You could try using single instance of typescript with multiple instances of the plugin. See typescript option:

const ts = require("typescript");
// ...
[
    rpt2_1 ( { typescript: ts } ),
]

// ...
[
    rpt2_2 ( { typescript: ts } ),
]

Let me know if that helps. If it doesn't consider making a repo with reproduction, I'll see if there is anything on the plugin end that can be doe for this scenario.

[marijnh]

Since require will return the already-loaded module when called multiple times on the same package, I'm not sure how requiring typescript in my rollup.config is any different from the require("typescript") that the plugin itself does.

You can set up the process I'm working with cloning https://github.com/codemirror/codemirror.next, and running npm install && npm watch in it. If you then touch, for example, view/src/index.ts a few times, you see the memory consumption of the node process grow every rebuild cycle, and then eventually the whole thing crashes. (Or at least, it does on my machine, after about 20 cycles.)

[ezolenko]

Indeed.

Looks like there is a memory leak somewhere. Rollup had a leak in watch mode before, but it seems to be fixed. (rollup/rollup#883)

Each instance of the plugin will create it's own instance of typescript.LanguageService and will not reset it during watch mode. This is how IDEs are operating I think, so typescript itself shouldn't have obvious leaks, or somebody would notice.

You can try running plugin with clean: true, this disables cache (but I don't see what could be leaking there either).

[zouhangwithsweet]

This problem is also bothering me. I have 60+ ui components to build. Actually, I just can build 10 components one time.

[zouhangwithsweet]

I find way to bypass the problem. Use child_process to build a small amount components, then kill this process. With a recursion call, everything is ok. :)

main.js

const buildChild = (a, b) => {
  const cp = require('child_process')
  const c1 = cp.spawn('node', ['./build/build.component.js', a, b])
  c1.stdout.on('data', function (data) {
    spinner.info(`${chalk.blue(data)}`)
  })

  c1.stderr.on('data', function (data) {
    spinner.warn(`${chalk.red(data)}`)
  })

  c1.on('close', function (code) {
    a += 5
    b += 5
    if (a > pkgs.length && b > pkgs.length) {
      spinner.succeed(`${chalk.green('Build done. Exit code ' + code)}`)
      return
    }
    buildChild(a, b)
  })
}

build.component.js

const runBuild = async () => {
  let index = 0
  const pkgs = await getPackages()
  const inputs = pkgs
    .map(pkg => pkg.name)
    .slice(process.argv[2], process.argv[3])

  build(inputs[index])

  async function build(name) {
    const inputOptions = {
      input: pathToEntry,
      plugins: [
        nodeResolve(),
        css(),
        vue({
          target: 'browser',
          css: false,
        }),
        typescript({
          tsconfigOverride: {
            compilerOptions: {
              declaration: false,
            },
            'exclude': [
              'node_modules',
              '__tests__',
            ],
          },
          abortOnError: false,
        }),
      ],
      external(id) {
        return /^vue/.test(id)
          || deps.some(k => new RegExp('^' + k).test(id))
      },
    }
    const outOptions = {
      format: 'es',
      file: [filename],
    }

    const bundle = await rollup.rollup(inputOptions)
    console.log(name, 'done')
    await bundle.write(outOptions)
    index++
    if (index < inputs.length) {
      await build(inputs[index])
    } else {
      console.log('batch done')
    }
  }
}

runBuild()