ENOMEM with exec/spawn - child process tries to reserve as much mem as parent

[zbjornson]

• Version: v8.12.0 64-bit
• Platform: Linux 4.15.0-1026 (Ubuntu 16.04)
• Subsystem: child_process

A tiny exec call when Node.js is using at least half of the otherwise-available memory causes spawn ENOMEM:

// If your system has more than 4 GB of mem, repeat these two lines until >50% of memory is used:
let x = Buffer.allocUnsafe(2e9);
x.fill(2); // virtual -> reserved
// Causes ENOMEM even if there's >1 GB of available memory:
require("child_process").exec("pwd", console.log)

# (strace)
[pid  3017] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ffae3ea7a10) = -1 ENOMEM (Cannot allocate memory)

Apparently this is a common problem when using fork()/clone(), and using posix_spawn(3) avoids it.

• Good write up of the issue: https://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html
• Related glibc bug discussion from JDK devs working on same issue https://sourceware.org/bugzilla/show_bug.cgi?id=10311
• JDK bug that goes with above https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6850720

I don't see any upstream issues in libuv for this.

[zbjornson]

This could be the underlying cause of this very popular SO issue for some users: https://stackoverflow.com/questions/26193654/node-js-catch-enomem-error-thrown-after-spawn

[mscdex]

I think something like this would be handled/solved at the libuv level, not node. The libuv issue tracker is here.

[vtjnash]

Duplicate of #14917?

[zbjornson]

Not really a dupe. Both are caused by using fork(), but different symptoms (ENOMEM vs. blocked loop), and this issue isn't addressed by MADV_DONTFORK as suggested in #14917. I just tried my repro with a buffer created from madvise(..., MADV_DONTFORK)'ed memory, and the fork() syscall still failed with ENOMEM. Even if the kernel doesn't make the memory available, it apparently still requests the allocation (or goes through some of the same assertions).

Edit May 23, 2019 tried madvise(MADV_DONTFORK) with a 5.1+ kernel and it still causes ENOMEM.

Edit: in Aug 2019 it does seem to work: #25382 (comment)

[bnoordhuis]

What does sysctl vm.overcommit_memory print on your system?

As you've probably gathered from the issues linked in libuv/libuv#2133, this is not something that is solvable, easily or at all. vfork() and clone() have their own share of issues.

[vtjnash]

@bnoordhuis Can you point me in the right direction for the issues you've encountered with using vfork? I've tried to search the issue tracker and commit history, but seemed hard to discover back that far. I've seen a couple references to cases where it was seen to have issues (such as libc developers arguing about it https://sourceware.org/bugzilla/show_bug.cgi?id=10354), but fork seems to share most of the same limitations (posix also specifies that "Consequently, [after fork] to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called."), but additionally (on linux) has been observed to be slower and to fail in low-memory situations. Additionally on linux, libuv could use clone directly to avoid some of the issues (with the vfork flag, but providing a separate stack). I did find this interesting graph though, which suggests that the problems with fork might be linux-specific: https://github.com/rtomayko/posix-spawn

It's been a couple years, but when I looked into the linux kernel code for fork, my recollection is that it does not respect the overcommit_memory flag(s) and just enforces that process memory < physical memory (plus probably some extra constant fudge factors). I couldn't really say what memory it counts against the limit however (e.g. if it includes MADV_DONTFORK).

[bnoordhuis]

The biggest issue for libuv is that vfork() doesn't run pthread_atfork() handlers. That might be surmountable for Node.js but add-ons might be a problem.

It's been a couple years, but when I looked into the linux kernel code for fork, my recollection is that it does not respect the overcommit_memory flag(s)

Linux's overcommit logic has been overhauled several times in the last years. I wouldn't know where exactly it stands now without checking the source (and also of past releases.) :-)

[vtjnash]

They aren't run though because they shouldn't often be needed. Although I don't know what Node.js does (or promises to do) in this case. From the rationale of pthread_atfork, running them is necessary to work around design issues with fork arising due to the COW semantics in multi-threaded programs that try to also use "unsafe" functions. But we can audit uv_spawn and know that it only uses the async-safe functions then execv (per fork documentation). Did someone actually need their pthread_atfork handler to run, or is that just a hypothetical concern that someone might try to do something more in their pthread_atfork handler?

OTOH, libuv now does register a pthread_atfork handler which does some extra operations that adds a couple of additional unnecessary syscalls, and thus may increase the cost of launching a process without vfork. I have no idea whether the impact of that code is measurable though.

But if the overcommit-related logic has changed, perhaps it might be faster now too? I see the OP is on an older Ubuntu 16.04. But if the kernel can do fork (almost) as fast and now as vfork and without hitting ENOMEM limits (as it seems may be true on mach, so perhaps possible at the hardware level), that would great! (and would save me from actually needing to ever upstream the vfork support to libuv, haha)

[zbjornson]

@bnoordhuis the default overcommit_memory was 0. The OP repro run with the three options:

0 - ENOMEM
1 - okay
2 - ENOMEM (with additional difficulty allocating the buffer in the first place)

Aside from that workaround (possibly viable) or adding swap (not viable), for my specific situation I'm looking into a PR to make https://github.com/googleapis/google-auth-library-nodejs either not exec at all, or exec earlier before the process grows.

[bnoordhuis]

Although I don't know what Node.js does (or promises to do) in this case.

@vtjnash There is (or was) at least one shared memory add-on module that uses pthread_atfork() so I don't think it's out of the question that other add-ons exist that, directly or indirectly, rely on the current behavior.

I say "indirectly" because add-on X might depend on shared library Y, which in turn depends on a functioning pthread_atfork(). Since that's impossible to audit for, I'd be uncomfortable changing the default, but, as discussed in libuv/libuv#141, an opt-in should be acceptable.

@zbjornson Thanks for checking! I figured that DWIM mode (vm.overcommit_memory=1) would end up working around this.

It's curious that madvise(MADV_DONTFORK) didn't work and I wonder if that's a kernel bug. It's not the behavior I'd expect at any rate.