Skip to content

Syncing stucked and can't be gracefully shutdown #24159

Closed
@bitrocks

Description

@bitrocks

System information

Geth version: 1.10.14-stable
OS & Version: Linux

Expected behaviour

  1. sync to the latest height
  2. gracefully shutdown when kill with SIGTERM signal

Actual behaviour

  1. the node stopped syncing in block height 13842763, (latest: 13885049), also with warn like Failed to decode block body like I reported in keeps showing Failed to decode block body, error=EOF #24158
INFO [12-26|12:33:56.111] Deep froze chain segment                 blocks=554 elapsed=241.616ms number=13,752,279 hash=2e59b3..f96a84
INFO [12-26|12:33:58.431] Imported new chain segment               blocks=73  txs=17086 mgas=1164.054 elapsed=8.387s    mgasps=138.779 number=13,842,299 hash=342930..98e0ef age=5d23h4m   dirty=2.50GiB
INFO [12-26|12:34:06.553] Imported new chain segment               blocks=71  txs=13764 mgas=1099.929 elapsed=8.122s    mgasps=135.415 number=13,842,370 hash=14483f..2641fd age=5d22h49m  dirty=2.50GiB
INFO [12-26|12:34:14.642] Imported new chain segment               blocks=61  txs=11946 mgas=939.153  elapsed=8.089s    mgasps=116.100 number=13,842,431 hash=f9d338..04b378 age=5d22h34m  dirty=2.50GiB
INFO [12-26|12:34:22.731] Imported new chain segment               blocks=63  txs=11259 mgas=945.075  elapsed=8.088s    mgasps=116.838 number=13,842,494 hash=229347..6716fb age=5d22h23m  dirty=2.50GiB
INFO [12-26|12:34:30.758] Imported new chain segment               blocks=64  txs=11538 mgas=1013.797 elapsed=8.027s    mgasps=126.296 number=13,842,558 hash=8b58b7..6b6467 age=5d22h9m   dirty=2.50GiB
INFO [12-26|12:34:38.848] Imported new chain segment               blocks=71  txs=13084 mgas=1109.530 elapsed=8.089s    mgasps=137.161 number=13,842,629 hash=7050b9..e01c99 age=5d21h55m  dirty=2.50GiB
INFO [12-26|12:34:46.864] Imported new chain segment               blocks=68  txs=12332 mgas=1040.302 elapsed=8.016s    mgasps=129.776 number=13,842,697 hash=7bf296..8d08e2 age=5d21h42m  dirty=2.50GiB
INFO [12-26|12:34:54.922] Imported new chain segment               blocks=60  txs=11309 mgas=929.221  elapsed=8.058s    mgasps=115.316 number=13,842,757 hash=f210aa..664ab5 age=5d21h29m  dirty=2.49GiB
INFO [12-26|12:34:55.780] Imported new chain segment               blocks=6   txs=1647  mgas=118.826  elapsed=858.637ms mgasps=138.389 number=13,842,763 hash=608650..5b1793 age=5d21h26m  dirty=2.50GiB
INFO [12-26|12:34:55.787] Downloader queue stats                   receiptTasks=0 blockTasks=32901 itemSize=71.65KiB  throttle=3659
WARN [12-26|12:34:56.108] Failed to decode block body              block=11,305,880 error=EOF
INFO [12-26|12:44:55.200] New local node record                    seq=17107 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=60493 tcp=30303
WARN [12-26|12:45:13.302] Checkpoint challenge timed out, dropping id=e9b288e40fe09256 conn=dyndial addr=212.102.60.119:30303 type=Geth/v1.10.9-stable/...
INFO [12-26|13:06:51.372] New local node record                    seq=17108 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=1158  tcp=30303
INFO [12-26|13:12:04.523] New local node record                    seq=17109 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=60493 tcp=30303
INFO [12-26|13:20:20.199] New local node record                    seq=17110 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=1158  tcp=30303
INFO [12-26|13:23:30.618] Writing clean trie cache to disk         path=/data/chain-data/eth/geth/triecache threads=1
INFO [12-26|13:23:30.618] Regenerated local transaction journal    transactions=0 accounts=0
INFO [12-26|13:23:31.912] Persisted the clean trie cache           path=/data/chain-data/eth/geth/triecache elapsed=1.294s
  1. I try to gracefully shut it down with SIGTERM signal, but it shows:
WARN [12-27|03:38:37.882] Already shutting down, interrupt more to panic. times=9
INFO [12-27|03:46:35.320] New local node record                    seq=17209 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=51291 tcp=30303
INFO [12-27|03:52:10.772] New local node record                    seq=17210 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=1158  tcp=30303
INFO [12-27|03:54:48.611] New local node record                    seq=17211 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=51291 tcp=30303
INFO [12-27|04:02:42.190] New local node record                    seq=17212 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=1158  tcp=30303
WARN [12-27|04:02:57.649] Already shutting down, interrupt more to panic. times=8
INFO [12-27|04:05:15.901] New local node record                    seq=17213 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=51291 tcp=30303
INFO [12-27|04:09:01.191] New local node record                    seq=17214 id=9ed7c028b0b3fb36 ip=125.34.62.110 udp=1158  tcp=30303

similar issue #21273 , is it some new kind of deadlock?

Steps to reproduce the behaviour

Backtrace

https://gist.github.com/bitrocks/77db0474aa319839520b3808832574f5

When submitting logs: please submit them as text and not screenshots.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions