Can't build blocks on testing runtime Westend Penpal (polkadot-sdk master)

[karolk91]

I'm experiencing issue similar to #958 (using chopsticks 1.2.2 that should already have the fix for #958 ) when running Westend Penpal (built from polkadot-sdk master) on chopsticks. Not sure what is specific to Penpal (maybe worth noting its not a system-chain), as there is no issue with other runtimes like Asset Hub or People that were also built from master

So building Penpal from polkadot-sdk master:

cargo build --release -p penpal-runtime

and using config:

endpoint: wss://westend-penpal-rpc.polkadot.io
mock-signature-host: true
db: ./db_pen.sqlite
runtime-log-level: 5
wasm-override: ../../polkadot-sdk/target/release/wbuild/penpal-runtime/penpal_runtime.compact.compressed.wasm

import-storage:
  System:
    Account:
      -
        -
          - 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY # Alice
        - providers: 1
          data:
            free: 1000000000000000
  Assets:
    Account:
      - [[1984, 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY], { balance: 1000000000 }]
    Asset: [[[1984], { supply: 1000000000 }]]
  Sudo:
    Key: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY # Alice

starting network like (Relay + some chains and Penpal)

npx @acala-network/chopsticks@1.2.2 xcm -r configs/westend-override.yaml -p configs/westend-asset-hub-override.yaml -p configs/westend-penpal-override.yaml -p configs/westend-people-override.yaml

when I'm using wasm-override, its not possible to build new blocks on Penpal with errors like (any tx like balance transfer):

        chopsticks::executor  TRACE: [24] Calling BlockBuilder_apply_extrinsic
                     runtime  ERROR: [24] panicked at /polkadot-sdk/cumulus/pallets/parachain-system/src/lib.rs:150:13:
Relay chain block number needs to strictly increase between Parachain blocks!
.
.
.
[11:04:49.624] WARN (parachain-validation-data): Failed to get validation data from block 1441244 Error: createType(GenericExtrinsic):: createType(ExtrinsicV4):: createType(Call):: Call: failed decoding parachainSystem.setValidationData:: Struct: failed on args: {"data":"{\"validationData\":\"PolkadotPrimitivesV8PersistedValidationData\",\"relayChainState\":\"SpTrieStorageProof\",\"relayParentDescendants\":\"Vec<SpRuntimeHeader>\",\"collatorPeerId\":\"Option<B
endpoint: wss://westend-penpal-rpc.polkadot.io
ytes>\"}","inbound_messages_data":"{\"downwardMessages\":\"{\\\"fullMessages\\\":\\\"Vec<PolkadotCorePrimitivesInboundDownwardMessage>\\\",\\\"hashedMessages\\\":\\\"Vec<CumulusPrimitivesParachainInherentHashedMessage>\\\"}\",\"horizontalMessages\":\"CumulusPalletParachainSystemParachainInherentAbridgedInboundMessagesCollection\"}"}:: decodeU8aStruct: failed at 0x9103aad5951957944d367249648b89a2… on data (index 1/2): {"validationData":"PolkadotPrimitivesV8PersistedValidationData","relayChainState":"SpTrieStorageProof","relayParentDescendants":"Vec<SpRuntimeHeader>","collatorPeerId":"Option<Bytes>"}:: decodeU8aStruct: failed at 0x04e803000000… on collatorPeerId (index 4/4): Option<Bytes>:: Bytes: required length less than remainder, expected at least 59, found 5
.
.
.
    err: {
      "type": "Error",
      "message": "Failed to apply inherents",
      "stack":
          Error: Failed to apply inherents
              at initNewBlock (/Users/kkdev/.npm/_npx/a02e1b5f0b9eea63/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:213:19)
              at async buildBlock (/Users/kkdev/.npm/_npx/a02e1b5f0b9eea63/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:320:44)
              at async TxPool.buildBlock (/Users/kkdev/.npm/_npx/a02e1b5f0b9eea63/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:352:43)
              at async TxPool.buildBlockIfNeeded (/Users/kkdev/.npm/_npx/a02e1b5f0b9eea63/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:328:9)
    }

at the same time, when NOT using wasm-override and starting with current live state of Penpal (its some older runtime and working fine with current chopsticks) then updating with extrinsic system.setCode, I'm still getting this warning in chopstics (below) but its only for the first block after the upgrade, further blocks are build succesfully and without warnings or other issues

[10:58:59.860] WARN (parachain-validation-data): Failed to get validation data from block 1441035 Error: createType(GenericExtrinsic):: createType(ExtrinsicV4):: createType(Call):: Call: failed decoding parachainSystem.setValidationData:: Struct: failed on args: {"data":"{\"validationData\":\"PolkadotPrimitivesV8PersistedValidationData\",\"relayChainState\":\"SpTrieStorageProof\",\"relayParentDescendants\":\"Vec<SpRuntimeHeader>\",\"collatorPeerId\":\"Option<Bytes>\"}","inbound_messages_data":"{\"downwardMessages\":\"{\\\"fullMessages\\\":\\\"Vec<PolkadotCorePrimitivesInboundDownwardMessage>\\\",\\\"hashedMessages\\\":\\\"Vec<CumulusPrimitivesParachainInherentHashedMessage>\\\"}\",\"horizontalMessages\":\"CumulusPalletParachainSystemParachainInherentAbridgedInboundMessagesCollection\"}"}:: decodeU8aStruct: failed at 0x9103037244a08080d594c888c7bbb116… on data (index 1/2): {"validationData":"PolkadotPrimitivesV8PersistedValidationData","relayChainState":"SpTrieStorageProof","relayParentDescendants":"Vec<SpRuntimeHeader>","collatorPeerId":"Option<Bytes>"}:: decodeU8aStruct: failed at 0x08e803000000ec03000000… on collatorPeerId (index 4/4): Option<Bytes>:: Bytes: required length less than remainder, expected at least 59, found 10