Geth nodes crashes, memory leak

[Phistr90]

System information

Geth version: 1.4.10-stable-5f55d95a
OS & Version: Arch

Expected behaviour

Actual behaviour

Geth nodes crashes after a few minutes.

Steps to reproduce the behaviour

Start geth node, try to sync

Log:

http://pastebin.com/kVjhA2vU

I guess the problem is the VM codehash da002555

[bobfeldbauer]

I can confirm seeing the same issue with geth version 1.4.11-stable-fed692f6 on Debian. The problematic block is block #2283416. Possibly the payload in this transaction: https://etherchain.org/tx/0x5c19695f50a30abbadfeef201d695d9c95c254534019e2f6a7a590e9ef246e82

[chrisfranko]

yikes

[gddrx]

[MicahZoltu]

Be careful what you read on the internet everyone. Not everyone posting comments here is a geth developer. Safest to assume they aren't and they are trying to pump/dump unless you have evidence to the contrary.

[Gustav-Simonsson]

Please wait for official security alert to be published at https://blog.ethereum.org/ - until then be careful what people claim.

So far we can confirm a memory leak in go-ethereum master branch (and probably earlier releases). More info soon.

[devktor]

Here are some GDB crash logs :

Thread 4 "geth" received signal SIGPIPE, Broken pipe.

[Switching to Thread 0x7ffff64e5700 (LWP 6055)]

syscall.Syscall () at /usr/lib/go-1.6/src/syscall/asm_linux_amd64.s:27

27      CMPQ    AX, $0xfffffffffffff001

(gdb) bt

#0  syscall.Syscall () at /usr/lib/go-1.6/src/syscall/asm_linux_amd64.s:27

#1  0x000000000063812f in syscall.write (fd=492, p=..., n=119, err=...) at /usr/lib/go-1.6/src/syscall/zsyscall_linux_amd64.go:1064

#2  0x000000000063649d in syscall.Write (fd=492, p=..., n=32, err=...) at /usr/lib/go-1.6/src/syscall/syscall_unix.go:180

#3  0x00000000009f51a7 in net.(*netFD).Write (fd=0xc82d318ee0, p=..., nn=0, err=...) at /usr/lib/go-1.6/src/net/fd_unix.go:328

#4  0x0000000000a09284 in net.(*conn).Write (c=0xc82d7f8048, b=..., ~r1=0, ~r2=...) at /usr/lib/go-1.6/src/net/net.go:184

#5  0x00000000008fd1b3 in github.com/ethereum/go-ethereum/p2p.(*rlpxFrameRW).WriteMsg (rw=0xc8267ec900, msg=..., ~r1=...)
    at /opt/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:598

#6  0x00000000008f667a in github.com/ethereum/go-ethereum/p2p.(*rlpx).WriteMsg (t=0xc82c9d0e70, msg=..., ~r1=...)
    at /opt/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:96

#7  0x000000000090878b in github.com/ethereum/go-ethereum/p2p.(*conn).WriteMsg (this=0xc82abaa500, anon0=..., ~r1=...) at <autogenerated>:14

#8  0x00000000008f14e1 in github.com/ethereum/go-ethereum/p2p.Send (w=..., msgcode=2, data=..., ~r3=...)
    at /opt/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/message.go:97

#9  0x00000000008f15b7 in github.com/ethereum/go-ethereum/p2p.SendItems (w=..., msgcode=2, elems=..., ~r3=...)
    at /opt/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/message.go:110

#10 0x00000000008f2e9f in github.com/ethereum/go-ethereum/p2p.(*Peer).pingLoop (p=0xc82eb1bac0)
    at /opt/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/peer.go:201

#11 0x0000000000479491 in runtime.goexit () at /usr/lib/go-1.6/src/runtime/asm_amd64.s:1998

#12 0x000000c82eb1bac0 in ?? ()

#13 0x0000000000000000 in ?? ()

[Atrides]

I confirm the problem with geth 1.4.11

[bobfeldbauer]

Seems to be the same on develop branch HEAD too.

[colm]

@devktor It seems to be an invalid msg at message.go:110 which fails to send at peer.go:201. Could you log the msg that fails and share?

EDIT : Im wrong here. Plz ignore. Its at pingMsg in pingLoop that causes the crash. This just keeps the peers connected. There is not data from any TX sent here. I have an intuation the bug is in the code that tries to seperate ETC and ETH networks.

@devktor can you give us max verbosity logs?

[sandakersmann]

I have an intuation the bug is in the code that tries to seperate ETC and ETH networks.

I think you are on to something here.

[colm]

@sandakersmann we can confirm this by seeing if geth 1.4.9 cashes as it was before the HF and does not have this. Do you ahve a synced chain to test this?

[trapp]

@colm, 1.4.9 would not sync the ETH chain. It would sync the ETC chain where the transaction in question does not exist.

[colm]

@trapp haha yeah my mistake.

[devktor]

here are full logs with GDB:

#0  syscall.Syscall () at /usr/lib/go-1.6/src/syscall/asm_linux_amd64.s:27
No locals.
#1  0x000000000063812f in syscall.write (fd=800, p=..., n=119, err=...) at /usr/lib/go-1.6/src/syscall/zsyscall_linux_amd64.go:1064
No locals.
#2  0x000000000063649d in syscall.Write (fd=800, p=..., n=32, err=...) at /usr/lib/go-1.6/src/syscall/syscall_unix.go:180
No locals.
#3  0x00000000009f51a7 in net.(*netFD).Write (fd=0xc8354c5f10, p=..., nn=0, err=...) at /usr/lib/go-1.6/src/net/fd_unix.go:328
        n = 859774696608
        ~r0 = {tab = 0x0, data = 0x20}
        syscall·2 = 0xc82e90bca0 ""
        err·3 = {tab = 0x20, data = 0xc822830a80}
        err = {tab = 0x0, data = 0x0}
        err = {tab = 0x0, data = 0x0}
#4  0x0000000000a09284 in net.(*conn).Write (c=0xc836e83398, b=..., ~r1=0, ~r2=...) at /usr/lib/go-1.6/src/net/net.go:184
        n = 32
        err = {tab = 0x10, data = 0x10}
#5  0x00000000008fd1b3 in github.com/ethereum/go-ethereum/p2p.(*rlpxFrameRW).WriteMsg (rw=0xc838543800, msg=..., ~r1=...)
    at /opt/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:598
        fsize = 3
        ~r0 = {tab = 0xc82194fa48, data = 0xeb2da0}
        text·2 = 0x7ffff7f480a8 "\340\267\373\000\000\000\000\000\000\323\376", '\000' <repeats 21 times>, "@L^\000\000\000\000\000\060\212^\000\000\000\000\000\340J^\000\000\000\000\000\340T^\000\000\000\000\000\060\214^\000\000\000\000\000@\213^\000\000\000\000\000 V^\000\000\000\000\000\240Y^\000\000\000\000\000PL^\000\000\000\000\000\240Z^\000\000\000\000\000\300[^\000\000\000\000\000\340\\^\000\000\000\000\000\340\210^\000\000\000\000\000\360]^\000\000\000\000\000\200U^\000\000\000\000\000P_^\000\000\000\000\000`L^\000\000\000\000\000p`^\000\000\000\000\000\020Q^\000\000\000\000\000\060S^\000\000\000\000\000\220T^\000\000\000\000\000"...
        err = {tab = 0x0, data = 0x0}
        err = {tab = 0x0, data = 0x4}
        err = {tab = 0x8, data = 0x0}
        err = {tab = 0x0, data = 0x9}
        ptype = {array = 0xc83478cd98 "\001", len = 1, cap = 1}
        mac = {array = 0xc800000000 <error: Cannot access memory at address 0xc800000000>, len = 4364455, cap = 860003851584}
        headbuf = {
          array = 0xc82e90bc20 "q\371`\336\347 e\247^\360s\b\323\300\237\001qf\017)\267q\027Pݍj\030\as\337\001h})\f\360\237{\260\350\\\240\361'\226\062\064۴\017MD\215\307T\235\304^T\313\034\202\316h})\f\360\237{\260\350\\\240\361'\226\062\064۴\017MD\215\307T\235\304^T\313\034\202\316qf\017)\267q\027Pݍj\030\as\337\001<H\360\030\\kr2\321\332l\033z'\331jqf\017)\267q\027Pݍj\030\as\337\001<H\360\030\\kr2\321\332l\033z'\331j\204\202#", <incomplete sequence \310>, len = 32, cap = 32}
        fmacseed = {array = 0x0, len = 0, cap = 860003851584}
        b = {
          array = 0xc82e90bc20 "q\371`\336\347 e\247^\360s\b\323\300\237\001qf\017)\267q\027Pݍj\030\as\337\001h})\f\360\237{\260\350\\\240\361'\226\062\064۴\017MD\215\307T\235\304^T\313\034\202\316h})\f\360\237{\260\350\\\240\361'\226\062\064۴\017MD\215\307T\235\304^T\313\034\202\316qf\017)\267q\027Pݍj\030\as\337\001<H\360\030\\kr2\321\332l\033z'\331jqf\017)\267q\027Pݍj\030\as\337\001<H\360\030\\kr2\321\332l\033z'\331j\204\202#", <incomplete sequence \310>, len = 32, cap = 32}
        tee = {S = {tab = 0xf1ac20, data = 0xc83c395d40}, W = {tab = 0xc82e90bc00, data = 0x0}, Err = {tab = 0x0, data = 0x2}}
#6  0x00000000008f14e1 in github.com/ethereum/go-ethereum/p2p.Send (w=..., msgcode=1, data=..., ~r3=...)
    at /opt/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/message.go:97
        r = {tab = 0x7ffff7e038e8, data = 0xc83c395d40}
        err = {tab = 0x0, data = 0x0}
#7  0x00000000008f15b7 in github.com/ethereum/go-ethereum/p2p.SendItems (w=..., msgcode=1, elems=..., ~r3=...)
    at /opt/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/message.go:110
No locals.
#8  0x00000000008f69fb in github.com/ethereum/go-ethereum/p2p.(*rlpx).close (t=0xc82fcbe930, err=...)
    at /opt/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/rlpx.go:106
        r = 4
        ~r0 = {sec = 63609829155, nsec = 375050915, loc = 0x1a08fe0 <time.localLoc>}
        t·2 = {sec = 63609829155, nsec = 375050915, loc = 0x1a08fe0 <time.localLoc>}
#9  0x0000000000903e3e in github.com/ethereum/go-ethereum/p2p.(*Server).setupConn (srv=0xc820059680, fd=..., flags=4, dialDest=0x0)
    at /opt/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:636
        running = true
        phs = 0xc836cab5c0
        c = 0xc834dc4a00
        err = {tab = 0x7ffff4469d98, data = 0xc83478cd60}
        err = {tab = 0x7ffff7e038e8, data = 0xc836cab5c0}
        err = {tab = 0x0, data = 0x0}
#10 0x00000000009072e7 in github.com/ethereum/go-ethereum/p2p.(*Server).listenLoop.func1 (srv=0xc820059680, fd=..., slots=0xc82147cfc0)
    at /opt/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/p2p/server.go:595
No locals.
#11 0x0000000000479491 in runtime.goexit () at /usr/lib/go-1.6/src/runtime/asm_amd64.s:1998
No locals.
---Type <return> to continue, or q <return> to quit---
#12 0x000000c820059680 in ?? ()
No symbol table info available.
#13 0x00007ffff4469188 in ?? ()
No symbol table info available.
#14 0x000000c836e83398 in ?? ()
No symbol table info available.
#15 0x000000c82147cfc0 in ?? ()
No symbol table info available.
#16 0x0000000000000062 in ?? ()
No symbol table info available.
#17 0x0000000000000000 in ?? ()
No symbol table info available.

[colm]

@devktor Do you have a lot of running byte VM da002555 in your logs? A few others have this which suggests the bug is in the EVM.