state root mismatch in mainnet block

[advaita-saha]

DBG 2025-06-08 15:21:00.844+02:00 Processing JSON-RPC request                id=6124 name=engine_newPayloadV4
DBG 2025-06-08 15:21:00.849+02:00 Validating block                           topics="forked chain" blkHash=0xa10def2bab9e0be77481642451b4aeda1c3c9d6573000dc2b2837a4ca632184c blk="(parentHash: 0xcf3c9219dde8edf0128df7d6141a5cc162f9b85d8e9d9ae50ac4ef1e0292824b, coinbase: 0x4838b106fce9647bdf1e7877bf73ce8b0bad5f97, stateRoot: 0x21d542193a43b9e641cfd84417208886f523d6ce8e5ee9e531000b2fe51f8fb4, transactionsRoot: 0x8cbf60d429a823a8c0fec48bf4d5e62ecf3399ab79ef7e45013e8eb46b2441fe, receiptsRoot: 0x19a281c1df718341df4010d3bbb357cb606eaa12d048737a91946f4f9a4de94e, number: 22660130, gasLimit: 35929690, gasUsed: 21497674, nonce: 0x0000000000000000, baseFeePerGas: ok(459109746), withdrawalsRoot: ok(0x0ff9a54af9abf6e39044e540775f4b839753ff60b60d9adec9d78474d412505d), blobGasUsed: ok(393216), excessBlobGas: ok(0), parentBeaconBlockRoot: ok(0x121b002a4151e78f770513326fdfd912650cf73e99e09c21de1d3f0a98ee09fa), requestsHash: ok(0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855))"
INF 2025-06-08 15:21:01.272+02:00 New payload received and validated         topics="beacon engine" number=22660130 hash=a10def32184c parent=cf3c9292824b txs=278 gasUsed=21497674 blobGas=393216
DBG 2025-06-08 15:21:01.279+02:00 Returning JSON-RPC response                id=6124 name=engine_newPayloadV4 len=128
DBG 2025-06-08 15:21:01.315+02:00 Amount of peers                            topics="eth p2p peer_pool" amount=333
DBG 2025-06-08 15:21:01.316+02:00 Accept RlpxTransport error                 topics="eth p2p rlpx" remoteAddress=208.115.202.194:41934 err="Could not decode handshake auth message: auth: ECIES encryption/decryption error"
DBG 2025-06-08 15:21:01.443+02:00 Received JSON-RPC request                  topics="JSONRPC-HTTP-SERVER" address=127.0.0.1:50208 len=343
DBG 2025-06-08 15:21:01.443+02:00 Processing JSON-RPC request                id=6125 name=engine_forkchoiceUpdatedV3
INF 2025-06-08 15:21:01.443+02:00 Fork choice updated                        topics="beacon engine" requested=22660130 hash=a10def32184c head=22660130 base=22659999 baseHash=4429cf0b8659
DBG 2025-06-08 15:21:01.443+02:00 Returning JSON-RPC response                id=6125 name=engine_forkchoiceUpdatedV3 len=163
DBG 2025-06-08 15:21:12.320+02:00 Processing JSON-RPC request                id=6126 name=engine_newPayloadV4
DBG 2025-06-08 15:21:12.326+02:00 Validating block                           topics="forked chain" blkHash=0x33a5929f31c0cb0242e97b5e0475768ba4459d769f82937575b774cdcee1dd69 blk="(parentHash: 0xa10def2bab9e0be77481642451b4aeda1c3c9d6573000dc2b2837a4ca632184c, coinbase: 0x4838b106fce9647bdf1e7877bf73ce8b0bad5f97, stateRoot: 0xe21455de43c63e1dc1465abcd9f79ba8de22c747c9b392ce549981cb8ba41efb, transactionsRoot: 0xcbbce6461946955524fc393cd57bd9b2c3a98bd506e686d732881711a3d6aeb9, receiptsRoot: 0x68d544f2d752fc1c56ff654a7ac0c0f72e45973bc071827df1f502ced7fdcae2, number: 22660131, gasLimit: 35894604, gasUsed: 15281856, nonce: 0x0000000000000000, baseFeePerGas: ok(470395372), withdrawalsRoot: ok(0x65123b73071e03033f0c9af92110fd3dfe658ba9a5a7b2000c64fc666a528884), blobGasUsed: ok(0), excessBlobGas: ok(0), parentBeaconBlockRoot: ok(0xc00c4b829acc9669ae569f0c085feea3273d8406f20cabb22f8724c7b764b6ce), requestsHash: ok(0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855))"
DBG 2025-06-08 15:21:12.643+02:00 wrong state root in block                  blockNumber=22660131 blockHash=0x33a5929f31c0cb0242e97b5e0475768ba4459d769f82937575b774cdcee1dd69 parentHash=0xa10def2bab9e0be77481642451b4aeda1c3c9d6573000dc2b2837a4ca632184c expected=0xe21455de43c63e1dc1465abcd9f79ba8de22c747c9b392ce549981cb8ba41efb actual=0xe60944cdff401a2458c547e64a47ece627de2467cda4a090757cda15bbb892e1 arrivedFrom=0x21d542193a43b9e641cfd84417208886f523d6ce8e5ee9e531000b2fe51f8fb4
WRN 2025-06-08 15:21:12.644+02:00 Error importing block                      topics="beacon engine" number=22660131 hash=33a592e1dd69 parent=a10def32184c error="stateRoot mismatch, expect: 0xe21455de43c63e1dc1465abcd9f79ba8de22c747c9b392ce549981cb8ba41efb, got: 0xe60944cdff401a2458c547e64a47ece627de2467cda4a090757cda15bbb892e1"

Not during sync, so most likely not async issue. But needs to be identified what happened

[arnetheduck]

Notably, invalid state has been written to the database (a restart doesn't clear the problem) so the actual error probably happened during earlier processing)

[advaita-saha]

INF 2025-06-13 11:05:21.198+00:00 Import error (skip remaining)              topics="beacon sync" n=0 iv=#10241..#10280 nBlocks=40 nthBn=#10241 nthHash=2a6379f55a9e base=#10240 head=#10240 blkFailCount=2 error="stateRoot mismatch, expect: 0x4aaa5f94be376794ff77b2fbd43ae6004f265e257ef4a0c06a3e980505af9dce, got: 0xe72aac42a906e9d8d13fac60f1dfd4b953dbe4e5ad6b160b6c1a3faf69313cdb"

state root mismatch issue came up in devnet

[jangko]

state root mismatch issue came up in devnet

What branch and what devnet? @advaita-saha

[arnetheduck]

restarting from an earlier database snapshot, the same node just synced past the failing block, so it's not a block processing issue.

[advaita-saha]

restarting from an earlier database snapshot, the same node just synced past the failing block, so it's not a block processing issue.

this points us in the direction of async re-entrancy issue, but reviewing the asyncQueue code, I am unable to find any logical problems or situations.

[jangko]

Finally I'm able to reproduce state root mismatch offline without FC. After trying various combination of txFrame usage. the most promising sequence:

• import block 1
• skip block 2 or execute block 3 using block 1 txFrame
• import block 3, the block execution is not validating the txFrame readiness, it just produce state root mismatch.

Now we need to locate where in the FC module this kind of out of place txFrame.
Extra txFrame validation can be added to FC, but it must be able to tell why the anomaly happen.

But the most important is reproduce the bug using FC module offline.

[jangko]

Potentially fixed by #3441

Let it run through the rest of fusaka-devnet-2 life time, and see if this happen again or not before closing this issue.