`DecodeAll` check for `post_migrate` hook

[ggwpez]

There are a few cases when migrations did not verify that the resulting state of a pallet is decodable with the new runtime.
This can easily happen when using StorageMap::translate since it just ignores un-decodable elements and carries them over into the new runtime.

We could add a check that tries to decode the whole state after a migration (or batch thereof) to verify that the state is not corrupted.
Sure there could still be un-decodable data in storage in the form of blobs (eg pallet-preimage) but decoding errors needs to be handled in these cases anyway. This would help but not completly solve with the introduction of BoundedVec.
Migrations still need to truncate any unbounded vectors or use WeakBoundedVec instead since otherwise someone could front-run the migration. cc @kianenigma

[kianenigma]

cc @Ank4n

[kianenigma]

struct DecodeEntireState;

impl OnRuntimeUpgrade for DecodeEntireState {
  fn post_upgrade() {
     sp_io::next_key() and decode() 
  }
  
  fn on_runtime_upgrade() {
    0 
  } 
}

And we put this in the runtime.

[kianenigma]

also related paritytech/substrate#12199, we should run try_state after all migrations as well. I already fixed this in paritytech/substrate#12537. maybe someone wants to backport the changes to master earlier?

[ggwpez]

Referencing your comment: https://github.com/paritytech/substrate/issues/12825#issuecomment-1337482936

IMO we need a new tuple type like AllStorageWithSystem analogous to AllPalletsWithSystem or something similar to access the storage values including their correct key/value types. The we can do assert_eq!(Map::decode_len(), Map::iter().count()) for maps and know that they are not corrupted. Analogous for all other storage types.

[kianenigma]

Technically AllStorageWithSystem should be present in the metadata, which is available at the top level runtime. So it is 100% possible to read all of them storage keys + types from there.

[bkchr]

When we are expanding the pallet macro, we also have access to all storage items. This expansion also contains the implementation of OnRuntimeUpgrade. So, we could just add to each pallet post_upgrade the decode all check. As we will call each pallet post_upgrade we will get the requested check for "free".

[bkchr]

But then we could not "control" if this should be enabled or not, but we could also solve this. We could for example introduce the mentioned DecodeEntireState and let it implement OnRuntimeUpgrade, but we set some special storage value to enable the pallet level decode all check.