Trie check / repair tool

[holiman]

Related to various issues where trie nodes go missing, a.k.a data corruption. Examples include #20845

We should have a geth ... repairtrie, or geth .. checktrie. What that command would do, is

• Load up the chain,( not initiate any synch or peering)
• Start at the trie root, and verify the hash.
• Walk down the trie and verify each hash.
• If a node cannot be verified, flag the missing hash/path for download.
• Continue until trie is done
• Generate report about trie health, missing nodes/paths.

At some point, also download the msising nodes from a peer, via fastsync protocol..

[holiman]

• If a node cannot be verified, flag the missing hash/path for download.
• Continue until trie is done

While walking the trie, I think that the only error that will occur, in practice, is that we find that resolving a certain hash fails: a missing trie node. At that point, we cannot continue walking that path further down, so unless we fetch it on the fly, we cannot "continue until done".

(Theoretically, I guess we could also find that hash(rlp(node)) != expected_hash, but I suspect we won't run into that. I guess it could happen if a flle system corruption actually flipped a bit in a leaf node, e.g modding a byte in the balance or nonce-field or whatever)

So perhaps a better solution is to fix the trie while walking it, otherwise the user would have to re-run the same 10+ hour-long process again and again for every missing trie node on the same path

[holiman]

Hm, I guess an alternative approach could be:

• Stop at first error, output

missing trie node: `0x...NODE`
path :`0x...PATH`

Then we could add

debug.fetchTrieNode(`0x..NODE`)

And, eventually, you rerun it:

geth --repairtrie --start=0x...PATH

Just thinking it would be nice to keep it kind of compartmentalized, and keep the tool simple....

[karalabe]

An alternative would be to fetch missing stuff on the fly, use it to continue iteration, but don't inject into the database. Then at the end offer to fix or abort? The issue however is that this needs network access, and by the time you find a missing trie node the network might have already moved on, so fixing it is not trivial.

[holiman]

Hm.. If we download it, why wouldn't we fix it too? I don't see what we would gain by not injecting it -- isn't the only drawback that we have to release our iterator and then resume it again ?

[karalabe]

Yeah, you could fix it. If the thing is still available in the network. But what if not? And also this requires network access, which we don't have infra for currently (offline chain but online handler).

[AusIV]

My team has developed a tool to do limited state trie verification that may be of interest here. Our streaming replication solution for blockchain data is a bit more error prone than conventional node management, so we frequently run state trie verifications on offline data. A full verification (walking the entire state trie and account tries checking every hash) can take days, which is impractical in our use case, but we've found that checking a sample of nodes from around the state trie can very reliably catch the sort of problems we encounter in a much more reasonable timeframe.

You're welcome to take a look if interested: https://github.com/NoteGio/go-ethereum/blob/support/1.9.12/cmd/geth/chaincmd.go#L786

[holiman]

Yeah, you could fix it. If the thing is still available in the network. But what if not?

We can do a first "optimistic" solution, and see how well it works.

Optimistic: doesn't move pivot points, or even care about pivot points, just tries to find the missing state on the network.

[holiman]

Implemented in #21650 .

[holiman]

Closing this, for the reasons given in #21650. Plus, most times, it's better for the user to resync from scratch rather than "heal", since the state will have shuffled along.