Database Inconsistency while restarting the Reth Pod in Kubernetes.

[rabindraregmi]

I am trying to run the Reth(v1.0.6) + Lighthouse client for our private testnet of our own L2 chain. I am using the Helm Chart provided in this repository for both the Reth and Lighthouse. Since the persistence was not enabled in the helm chart, I am using kustomize overlay to override some of the configuration as follows:

image:
  repository: ghcr.io/paradigmxyz/reth
  tag: v1.0.6
persistence:
  enabled: true
  storageClassName: hostpath
  size: 20Gi

terminationGracePeriodSeconds: 30

extraArgs:
  - --chain /network-configs/genesis.json
  - --p2p-secret-key /keys/reth-bootnode-p2p.key
  - --rpc.eth-proof-window 100 
  - --rpc.proof-permits 1000
  - --http.api admin,net,eth,web3,debug,txpool,trace
  - --ws.api=net,eth
  - --discovery.port=30303

Everything running fine and blocks are being progressed and all. But when I restart the pod which is running the reth client, I get this issue:

│ 2025-02-05T06:27:36.842816Z  INFO Verifying storage consistency.                                                                                                                                                   
│ 2025-02-05T06:27:36.886484Z  INFO Setting unwind target. checkpoint_block_number=10 unwind_target=0 segment=Transactions                                                                                           │
│ thread 'main' panicked at /project/crates/node/builder/src/launch/common.rs:408:13:                                                                                                                                
│ assertion `left != right` failed: A static file <> database inconsistency was found that would trigger an unwind to block 0                                                                                        
│   left: Unwind(0)                                                                                                                                                                                                  
│  right: Unwind(0)                                                                                                                                                                                                  
│ stack backtrace:                                                                                                                                                                                                   
 2025-02-05T06:27:36.893101Z  INFO Setting unwind target. checkpoint_block_number=10 unwind_target=0 segment=Receipts               

and it only works after I delete the pvc attached with beacon, validator and reth node. After deleting the PVCs, it again sync from genesis block and then it works.

[skylenet]

Hey @rabindraregmi 👋

The terminationGracePeriodSeconds: 30 is probably too low. Try to increase it to 300 . Reth needs some time to shut down cleanly. In your case it will force kill the process after 30s. We normally use 5min (300s) as a good default.

[rabindraregmi]

Facing the same issue even with setting the value 300. When I delete the pod with kubectl delete pod <pod_name>, service restarts the pod and it's stuck at database inconsistency issue.