dash 20.0.1 stop sync at verifying last block

[lcgogo]

Expected behavior

I run some dash in docker, after docker stop/start, several dash do not sync with followed log. The others restart are ok. But I am not sure whether they are ok at next restart.
Never found this issue in dash 19.x.x

2023-11-28T14:34:09Z LoadBlockIndexDB: timestamp index disabled
2023-11-28T14:34:09Z LoadBlockIndexDB: spent index disabled
2023-11-28T14:34:09Z Opening LevelDB in /data/dash/chainstate
2023-11-28T14:34:09Z Opened LevelDB successfully
2023-11-28T14:34:09Z Using obfuscation key for /data/dash/chainstate: f6f230e5bd669995
2023-11-28T14:34:10Z Loaded best chain: hashBestChain=000000000000002cb95fcd940f3fd3e9aa0648b39e3bb2630a0d93fa52885eb8 height=1978082 date=2023-11-27T02:15:08Z progress=0.999671
2023-11-28T14:34:10Z CDeterministicMNManager::MigrateDBIfNeeded -- upgrading DB to migrate MN type
2023-11-28T14:34:10Z CDeterministicMNManager::MigrateDBIfNeeded -- migration already done. skipping.
2023-11-28T14:34:10Z CDeterministicMNManager::MigrateDBIfNeeded2 -- upgrading DB to migrate MN state bls version
2023-11-28T14:34:10Z CDeterministicMNManager::MigrateDBIfNeeded2 -- migration already done. skipping.
2023-11-28T14:34:10Z init message: Verifying blocks...
2023-11-28T14:34:10Z AppInitMain: bls_legacy_scheme=0
2023-11-28T14:34:10Z Verifying last 6 blocks at level 3
^C

To reproduce

System information

[lcgogo]

@PastaPastaPasta Please help to provide some suggestion.

[qwizzie]

If this is Dashmate related (you mentioned running dash in docker) , you could contact pshenmic on dash.org/forum through 'conversation' or post your Dashmate issue here : https://www.dash.org/forum/index.php?threads/dashmate-discussion.53951/#post-235627

To be used if you don't get a timely response here....

[UdjinM6]

@lcgogo I believe I had an issue like that once but I couldn't reproduce it. Pls try removing following folders from .dashcore/llmq one by one in the exact order I specified below. Try starting a node each time you deleted a folder to check which one is potentially causing the issue:

1. isdb
2. dkgdb
3. recsigdb

[UdjinM6]

Ideally, if you want to help debugging this, backup .dashcore/llmq folder before removing internals to restore it later and test if removing subfolders in another order solves the issue on the exact same one again (if it does at all ofc).

[lcgogo]

@lcgogo I believe I had an issue like that once but I couldn't reproduce it. Pls try removing following folders from .dashcore/llmq one by one in the exact order I specified below. Try starting a node each time you deleted a folder to check which one is potentially causing the issue:

1. isdb
2. dkgdb
3. recsigdb

Thanks, will try it. But i am not sure 100% reproduce it because I tried to manually restart some dash containers, but not meet the issue.

[lcgogo]

Ideally, if you want to help debugging this, backup .dashcore/llmq folder before removing internals to restore it later and test if removing subfolders in another order solves the issue on the exact same one again (if it does at all ofc).

Meet the issue again, after rm -rf llmq/dkgdb/ and start the dash can continue to sync. Many thanks!