LND crashed (last words are about updating routing policies)

[bockha]

Background

LND crashed just right after updating routing policies.

Your environment

• Version: 0.5.1-beta commit=v0.5.1-beta-138-g5451211d1947de5b2376aff5eb39c6e9f969cbbb, build=production, logging=default
• Linux odroid 4.14.66-147 Fix name typo in README #1 SMP PREEMPT Wed Aug 22 13:24:49 -03 2018 armv7l armv7l armv7l GNU/Linux
• bitciond v0.16.3

Steps to reproduce

log

2018-12-08 19:34:29.635 [INF] ATPL: Processing new external signal
2018-12-08 19:34:29.642 [INF] DISC: Updating routing policies for chan_points=(map[wire.OutPoint]struct {}) (len=1) {
 (wire.OutPoint) iRemovedTheChannelOutpointHere:0: (struct {}) {
 }
}

2018-12-08 19:34:29.712 [INF] ATPL: Processing new external signal
2018-12-08 19:34:29.719 [INF] DISC: Updating routing policies for chan_points=(map[wire.OutPoint]struct {}) (len=1) {
 (wire.OutPoint) iRemovedTheChannelOutpointHere:1: (struct {}) {
 }
}
---crash---
2018-12-08 21:19:04.970 [INF] LTND: Version: 0.5.1-beta commit=v0.5.1-beta-138-g5451211d1947de5b2376aff5eb39c6e9f969cbbb, build=production, logging=default
2018-12-08 21:19:04.971 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2018-12-08 21:19:05.043 [INF] CHDB: Checking for schema update: latest_version=7, db_version=7
2018-12-08 21:19:05.054 [INF] RPCS: password RPC server listening on 127.0.0.1:10007
2018-12-08 21:19:05.054 [INF] RPCS: password gRPC proxy started at [::]:8087
2018-12-08 21:19:05.054 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to unlock an existing wallet, or `lncli changepassword` to change the password of an existing wallet and unlock it.
2018-12-08 21:20:00.581 [INF] LNWL: Opened wallet

stderr

runtime: pointer 0xd002e03 to unallocated span span.base()=0xcfea000 span.limit=0xcff9f80 span.state=3
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

full stderr output can be found here https://gist.github.com/bockha/4032ecbca81c59299b07cf12b01a9d39

Expected behaviour

I expect it to not crash.

Actual behaviour

it crashes.

[Roasbeef]

fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

Which version of Go are you running?

[Roasbeef]

Perhaps looks related to golang/go#26243

[bockha]

$ go version
go version go1.11.1 linux/arm
$ go env
GOARCH="arm"
GOBIN=""
GOCACHE="/home/lnd-mainnet/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/lnd-mainnet/gocode"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go-1.11"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.11/pkg/tool/linux_arm"
GCCGO="gccgo"
GOARM="6"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build361969253=/tmp/go-build -gno-record-gcc-switches"

[bockha]

It happened again this morning, unfortunately i do not have stderr this time:

2018-12-11 00:35:42.893 [INF] ATPL: Processing new external signal
2018-12-11 00:35:42.901 [INF] DISC: Updating routing policies for chan_points=(map[wire.OutPoint]struct {}) (len=1) {
 (wire.OutPoint) channelIdRemoved:0: (struct {}) {
 }
}

2018-12-11 06:09:50.712 [INF] LTND: Version: 0.5.1-beta commit=v0.5.1-beta-138-g5451211d1947de5b2376aff5eb39c6e9f969cbbb, build=production, logging=default
2018-12-11 06:09:50.713 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2018-12-11 06:09:50.908 [INF] CHDB: Checking for schema update: latest_version=7, db_version=7
2018-12-11 06:09:50.922 [INF] RPCS: password RPC server listening on 127.0.0.1:10007
2018-12-11 06:09:50.922 [INF] RPCS: password gRPC proxy started at [::]:8087
2018-12-11 06:09:50.922 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to unlock an existing wallet, or `lncli changepassword` to c
hange the password of an existing wallet and unlock it.

Last words are again about updating channel policies, which is done by a python script. It runs regularly and updates every single channels policy.

[halseth]

@bockha I would try compiling with the newest go compiler v1.11.3.

[bockha]

I throttled the speed of channelupdatepolicy requests of my python script. I since had no more crash. I will try to reproduce the crash by switching off the throttling.

[bockha]

Hmm, the crash does no longer occur.

[wpaulino]

Closing for now, can reopen if the issue arises again.