Ninja and RAM/Memory usage

[rubyFeedback]

Hey folks.

I am using meson + ninja since ... well, 3 years or so, largely because other devs
switched to it increasingly. Meson and ninja is an awesome combination. Hopefully
we can abandon the GNU configure legacy one day. But anyway, this is a side
issue.

I am very happy with the speed of meson + ninja, which is great. There is, however
had, one concern that I have had in the past and this is in regards to the memory /
RAM usage.

This may be my old computer here from 2017 failing slowly, or something. I have
16GB RAM and it is a fairly fast computer. I compile everything from source.

But in the last, say, 12 months or so I am noticing memory/RAM issues.

This was the case in the past too but it also deadlocks and freezes my
computer. Again, this may be hardware related but ... the thing is, it only
freezes so in certain projects, most of which use ninja.

For instance, an hour ago I tried to compile webkitgtk using these
instructions:

https://www.linuxfromscratch.org/blfs/view/svn/x/webkitgtk.html

This one is using ninja.

A few hours before that I had a similar issue with compiling the
most recent node from source.

https://www.linuxfromscratch.org/blfs/view/svn/general/nodejs.html

This one does not use ninja, though, so it may well be that
something is wrong with my computer here. I am, however had,
not really writing this issue from the point of view of my computer
alone, because the secondary problem I have is that these
almost consistently keeps on happening whenever I try to
compile a larger program. It also happened to me in LLVM

• cmake recently, and perhaps 2 years ago, on the same
  machine, I did not have that issue. GCC versions changed
  too, binutils as well so perhaps my hardware is ok but the
  software is not so much ok.

Anyway.

So what does this have to do with ninja?

Well - my suspicion is that this all kind of has to do with
one way or another with RAM or perhaps a CPU is
semi-flawed or something.

Would it be possible to add more "fine-tuned" control
towards ninja itself, directly? For instance, the bigger projects
I don't mind if ninja compiles without maximum speed. I
understand your rationale for that as the default - after
all a ninja is fast and ninja should be fast. But a ninja
should not die in the middle of the mission and get
intercepted! No noob ninjas please. Translation: no
fails when RAM or other issues arise.

Perhaps there are some conditionals or config values
I could shift around, e. g. I could then set it before
trying to compile, say, webkitgtk. Where ninja is very
careful and not too aggressive. For smaller programs
I never run into any of these issues by the way. So not
sure what is up (or perhaps my CPU gets overloaded,
I have to clean this machine up soon anyway).

Could there be more control added towards RAM usage
and how greedy ninja is? I understand this is not a
trivial issue to work with, so please feel free to close it,
but it would be nice to get some feedback about this. If
I am the only one with any such issues then that's ok, but
perhaps others also have encountered problems (and,
as said, it is hard to say who is at fault - could be meson
too after all. But at the end of the day, even if these are
separate projects or separate problems, they may indirectly
affect ninja too, if people on lower cost machines can not
easily compile stuff).

[jhasse]

In the last couple of years linkers required more RAM due to LTO. Also the number of available threads grew (SMT, Ryzen, ...) while RAM didn't grow as much. This resulted in Ninjas assumption (number of threads +2 for the default of jobs) to be optimistic for many builds.

What to do? In the end we could add more monitoring to Ninja to check if the memory usage exceeds a threshold and start to freeze jobs.

But the correct solution would be to fix this at the kernel level. Tell the kernel "hey here are a bunch of processes, they don't need to run in parallel if there isn't enough memory".

[bmcpt]

@jhasse An easy solution is to just not spawn new tasks if there is less than, say, 2GB ram free. This will avoid the absolute worst case scenario. Build tasks are one of the few things that use a lot of ram, avoiding the system crash is partly their responsibility as well.

[chmorgan]

Just started using ninja here. Kept coming back to my terminal window being gone. Confirmed it's the OOM killing the process. I love the idea of maxing out compute resources to reduce build time but ninja should really do something smart regarding spawning new processes once the system memory has been exhausted. It took four tries to complete a rebuild here for a relatively complex c++ project (Kicad) in a 16GB qemu instance on an 8 core i9 x86 MacBook. I bet lots of other people are hitting the same issue and giving up on using ninja without looking into what's up.

[LeChatP]

Same for me, I try to compile llvm with ninja, Even with VM with 10 Threads high performance, 8GB RAM + 36GB Swap (NVME PCI4) it is still being killed by OOMKiller.... Ninja just does memory leak.

[bmx666]

I'm really frustrated—3, 4, 5 years and nothing has changed. Many embedded devices have 4-8 cores, but limited with RAM 1,2 or 4 GB only. I ended up implementing a simple max used memory limitation:
master...bmx666:ninja:max-used-memory

[bmx666]

Successfully tested build qtwebengine for Qt5 and Qt6 with ICECC -j 24 on Yocto kirkstone.

Build server (hypervised) - 2 (4 virtual) cores, 14GB RAM, 8GB swap on SSD.
Docker container - with flags --cpu=3 --memory=12g --memory-swap=18g, leaving some RAM for server's processes.
ICECC workers 1 - 8 cores, 16GB RAM, 20GB ICECC cache, nice lvl 5
ICECC workers 2 - 8 cores, 32GB RAM, 20GB ICECC cache, nice lvl 5
ICECC build server Docker worker - 1 core, 1GB ICECC cache, no remote, nice lvl 5
ICECC-scheduler on build server

Set max used memory 4GB, run ninja -j32 -m 4096. This will prevent new jobs from starting if there are many link stages, and many jobs go to more powerful ICECC workers anyway.

Compile stage of qtwebengine package takes 6 hours only.
Regular build of qtwebengine package on build server without ICECC, and 3 threads takes 42 hours.

### icecc.conf - for docker

ICECC_NICE_LEVEL="5"
ICECC_LOG_FILE="/var/log/iceccd.log"
ICECC_NETNAME="docker"
ICECC_MAX_JOBS="1"
ICECC_ALLOW_REMOTE="no"
ICECC_BASEDIR="/var/cache/icecc"
ICECC_SCHEDULER_LOG_FILE="/var/log/icecc_scheduler.log"
ICECC_SCHEDULER_HOST="172.17.0.1"
ICECC_CACHE_LIMIT="1024"

### icecc.conf - for powerful workers

ICECC_NICE_LEVEL="5"
ICECC_LOG_FILE="/var/log/iceccd.log"
ICECC_NETNAME="worker1"
ICECC_MAX_JOBS="8"
ICECC_ALLOW_REMOTE="yes"
ICECC_BASEDIR="/work/icecc"
ICECC_SCHEDULER_LOG_FILE="/var/log/icecc_scheduler.log"
# BUILD SERVER IP - 10.0.0.10
ICECC_SCHEDULER_HOST="10.0.0.10"
ICECC_CACHE_LIMIT="20480"

### local.conf - extra envs

BB_NUMBER_THREADS = "3"                                                                                                                                                                                     
PARALLEL_MAKE = "-j 3"                                                                                                                                                                                      
BB_PRESSURE_MAX_CPU = "25000"                                                                                                                                                                               
BB_PRESSURE_MAX_IO = "15000"                                                                                                                                                                                
BB_PRESSURE_MAX_MEMORY = "12000"                                                                                                                                                                            
INHERIT += "icecc"
ICECC_PARALLEL_MAKE = "-j 24"
ICECC_PATH = "/usr/bin/icecc"
NINJA_MAX_USED_MEMORY = "-m 4096"

### qtwebengine_%.bbappend - for QT5

export NINJAFLAGS="${PARALLEL_MAKE} ${NINJA_MAX_USED_MEMORY}"

do_configure:prepend () {
    if [ "${@use_icecc(bb, d)}" = "yes" ]
    then
        export CC="icecc gcc"
        export CXX="icecc g++"
    fi
}

do_compile:prepend() {
    if [ "${@use_icecc(bb, d)}" = "yes" ]
    then
        export CC="icecc gcc"
        export CXX="icecc g++"
    fi
}

python () {
    if 'icecc' in d.getVar('INHERIT', True):
        d.appendVar('EXTRA_OECMAKE', " \
            -DCMAKE_CXX_COMPILER_LAUNCHER=icecc \
            -DCMAKE_C_COMPILER_LAUNCHER=icecc \
        ")
}

### qtwebengine_%.bbappend - for QT6

EXTRA_OECMAKE_BUILD += " ${NINJA_MAX_USED_MEMORY} "

[fish2bird]

g has changed. Many e

Nice, I want this.