Geth 1.8.1 -syncmode light eats up all available system memory

[b-f-]

geth -syncmode light --cache=1024 uses more than 5gb ram and growing.

System information

Geth version: 1.8.1, mist 0.9.3
OS & Version: win 10, 10gb memory

Expected behaviour

• geth memory usage does not cause swapping
• command line switch to limit peak mem usage of geth process on windows

Actual behaviour

every app on system starts swapping

Steps to reproduce the behaviour

• sync with --fast until blockheight reached
• sync fullmode for awhile for several days, restart geth several times in between
• sync with -syncmode light
• run geth -syncmode light --cache=1024
• run "Ethereum Wallet.exe" -node-light

[karalabe]

Could you please attach to teh geth instance (geth attach /path/to/your/ipc/socket/within/geths/datadir/geth.ipc), and do a debug.stacks()? There might be some goroutines leaking out, that cause memory not to be deallocated.

[b-f-]

EDIT: removed --node-datadir="" from geth commands.
Here's how to repeat the issue:

1 See geth_full.txt (contains command, mem usage, stack dump); wait until "Imported block headers" messages start appearing in log, then terminate batch with ctrl+c, then Y.
geth_full.txt

2 See geth_light.txt (contains command, mem usage, two stack dumps; second dump was too long for console buffer). Memory just keeps increasing.
geth_light.txt

Note: Start ethereum wallet process several seconds after starting geth light:
"Ethereum Wallet.exe" -node-light

UPDATE: this same issue seems to be related to running Mist right after restarting geth.

[karalabe]

If you specify --node-datadir="", then you are explicitly requesting an in-memory database, so all the data Geth downloads and would store normally ends up in memory only. Either drop this flag or specify a proper directory and it should work.

[b-f-]

Just removed the actual path from parameter for privacy. I never use exact string --node-datadir=""

[riceke]

Hello!

Unfortunately this issue is still present also in version 1.8.2 of Geth.

I am running a light Geth node on one machine and connecting with EthereumWallet (version 0.9.3) from another machine via HTTP RPC using port forwarding over ssh.

Once I start EthereumWallet and it connects to the node the memory usage on the node shoots up and all available memory is used up.

Looking at the stack trace (attached in the dubug info file) there is an explosion in go-routines being spawned that end up hanging like this:

goroutine 8715 [chan receive]:
github.com/ethereum/go-ethereum/les.(*sentReq).stateStopped(0x20a0fd10, 0x154133d8)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/les/retrieve.go:254 +0x54
github.com/ethereum/go-ethereum/les.(*sentReq).(github.com/ethereum/go-ethereum/les.stateStopped)-fm(0x154133d8)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/les/retrieve.go:221 +0x1c
github.com/ethereum/go-ethereum/les.(*sentReq).retrieveLoop(0x20a0fd10)
/home/travis/gopath/src/github.com/ethereum/go-ethereum/les/retrieve.go:187 +0x58
created by github.com/ethereum/go-ethereum/les.(*retrieveManager).sendReq
/home/travis/gopath/src/github.com/ethereum/go-ethereum/les/retrieve.go:161 +0x260

This is exactly the same thing that happens in the stack trace that was submitted by 'b-f-' previously.

According to the trace file (also attached in the debug info file) there was some 1896 go-routines hanging like this and eating up memory.

This does not seem right.

Thanks!

debug-info.zip

Any updates? This is a ridiculous memory leak, especially for a coin of this magnitude. I'm running geth on a dedicated server and it ended up eating 50GB of RAM.

[riceke]

Hello!

I submitted a pull request with a fix for this problem some time ago but it is still waiting to be accepted.

I am running a version of latest geth patched with my fix and I am able to successfully run a light node an a 2GB ASUS Tinkerboard with memory left to spare.

[zsfelfoldi]

I confirm that what @riceke found was actually a bug in the retriever that caused a goroutine leak in case of first sending to an unresponsive server. Hopefully #16776 will fix the current leak, I'll add automated tests for leaks in the future.