Panic: Segmentation Violation, doesn't shutdown gracefully

[priom]

System information

Geth version: 1.11.3
OS & Version: Ubuntu 18.04

Expected behaviour

Should shutdown gracefully by safely saving the state and db

Actual behaviour

Throws Panic with Segmentation Violation

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=0x9bfb72]

goroutine 865 [running]:
github.com/ethereum/go-ethereum/core.(*HeaderChain).SetCurrentHeader(0xc0004c66c0, 0x0)
	/home/travis/gopath/src/github.com/ethereum/go-ethereum/core/headerchain.go:483 +0xa2
github.com/ethereum/go-ethereum/core.(*BlockChain).Rollback(0xc000bc2000, 0xc02082a000, 0xe0a, 0xe0a)
	/home/travis/gopath/src/github.com/ethereum/go-ethereum/core/blockchain.go:915 +0x379
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processHeaders.func1(0xc0039d7d90, 0xc0009b8380)
	/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1422 +0x1e5
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processHeaders(0xc0009b8380, 0x93d41, 0x24997d, 0xc028b69380, 0x14d14e0, 0xc0001b91a0)
	/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1523 +0xc20
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func6(0xc01b300f70, 0x0)
	/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:526 +0x48
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1(0xc0009b8380, 0xc028e0f3e0, 0xc028fae0c0)
	/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:543 +0x63
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync
	/home/travis/gopath/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:543 +0xaf

Steps to reproduce the behaviour

Run core-geth and kill by ctrl+c and see the error

[meowsbits]

Thanks for this report! Looks like there's a missing nil check in there. I'll propose a fix today.

[meowsbits]

I believe this issue has already been resolved in the master branch.
I believe it was a regression of the downloader shutdown mechanism or one of the fixes listed below. We're releasing v1.11.4 today, which will include the fix, so please try that and see if the issue is resolved for you.

Probably rel ethereum/go-ethereum#20988 or ethereum/go-ethereum#20988 or ethereum/go-ethereum#20695

I was able to reproduce the issue at v1.11.3 with this script, and unable to reproduce it at v1.11.4.

#!/usr/bin/env bash

# This script attempts to cause geth to panic on shutdown because
# of a nil current header number.
#
# The script waits for geth to be in the 'middle' of downloading,
# waiting for a log line matching a receipts import, 
# and then kills geth with SIGINT (ie ctrl-c).
#
# I used it to sucessfully reproduce the issue at
# https://github.com/etclabscore/core-geth/issues/101 
# with core-geth version v1.11.3,
# and unsuccessfully reproduce at core-geth v1.11.4.

make geth
./build/bin/geth version |& tee geth.version.log

rm -rf ./tempdd

trial=0
run_and_interrupt(){
	>geth.$trial.log
	(
	./build/bin/geth --datadir ./tempdd |& tee geth.$trial.log
	)&
	gethpid=$!
	echo "Watching geth pid: $gethpid"
	tail -f geth.$trial.log | while read line; do
		if echo "$line" | grep -q -E 'Imported .* receipts'; then
			echo "Saw receipts, killing $gethpid"
			pkill --signal 2 geth
			return
		fi
	done
}

grep_log_for_panic(){
	[[ ! -f geth.$trial.log ]] && return 0
	if cat geth.$trial.log | grep -q panic; then
		echo "Found panic"
		return 1
	else
		echo "Did not find panic"
		return 0
	fi
}

while grep_log_for_panic; do
	if [[ $trial -gt 10 ]]; then
		echo "Max trials, exiting."
		exit 0
	fi
	trial=$((trial+1))
	echo "Running. Trials=$trial"
       	run_and_interrupt
	sleep 5
done

[meowsbits]

@priom I'm going to assume this is resolved as of https://github.com/etclabscore/core-geth/releases/tag/v1.11.4 and close the issue. Please reopen if the issue persists for you.

[priom]

Thank you @meowsbits. I'll retry and if the issue persists then I'll reopen this issue.