Support `type: "closestDescendantMerkleValue"` for `chainHead_v1_storage`

[voliva]

We recently released polkadot-api@1.8, introducing a new feature: storage.watchEntries.

This feature uses the JSON-RPC spec for storage with the closestDescendantMerkleValue type. This provides the Merkle value for a partial key, allowing us to avoid re-requesting the entire list of entries on every block. Instead, it can compare the Merkle value (essentially a hash of all the nodes below that storage entry in the Merkle tree), and recompute the entries only if it changes.

Currently, Chopsticks' chainHead_v1_storage supports value and descendantValues but doesn't support closestDescendantMerkleValue yet.

I attempted to start implementing this, but I couldn't find any reference to the merkle tree data structure of the chain. I'm interested in contributing to this feature, but I would appreciate some guidance, or if it would need too big of a refactor.
Any pointers or suggestions on how to approach this would be greatly appreciated.

[ermalkaleci]

Chopsticks keeps storage diff for each block and doesn't have merkle tree and I don't see how we can implement it.

[xlc]

Presumably smoldot implements this RPC so we should be able to reuse its implementation. We should check smoldot's implementation to see the tree details. However, because Chopsticks isn't actually using merkle tree to store the states, the implementation won't be efficient.

[voliva]

I'm just thinking out loud, but in case it might makes thing substantially easier - polkadot-api doesn't need the actual "correct" merkle value, it just needs the property that it has to change if any of the nodes below have changed.

It wouldn't be spec-compliant, but if chopsticks only has diffs per block, maybe it could make it easier to increment an id.

If it's still complex, or if it still has to go through all the storage entries under that key, then maybe it's better to go the spec-compliant route of calculating the actual hash.

[xlc]

we can easily return a mock value. eg the block hash