Multiple copies of a transaction included in a block

[joepetrowski]

On block 56592, two copies of the same transaction (0x6cc...69a) were included in the block.

Related: polkadot-js/tools#77

{
    "number": 56592,
    "hash": "0xe0fc619d34b50148f6a89bf82fe340771f6cc93b467a28af88fafbfc0e2a5019",
    "parentHash": "0xda823678b85d1d42801ba4931ae95f864f204d7c788610e3999adb16b41becd2",
    "stateRoot": "0x3549f852fdb1d4222f4480d407461adaabb2519ba5e45f6669c46388ca8cc652",
    "extrinsicsRoot": "0xcc770108aa7b591e43876d4ca3780462ff4f5812ac7bfd3c99f6d72ae2e4c628",
    "logs": [
        {
            "type": "PreRuntime",
            "index": 6,
            "value": [
                1161969986,
                "0x02260000002637a60f00000000"
            ]
        },
        {
            "type": "Seal",
            "index": 5,
            "value": [
                1161969986,
                "0x261f9f7bd78af25a4ab7dad25d4fe8554b2b7a83f29a1c7ca633c347c4d9242f668a6cec59e6c948743e8c60a37c312cbe031553d60bb124703c192304ae3289"
            ]
        }
    ],
    "extrinsics": [
        {
            "method": "timestamp.set",
            "signature": null,
            "nonce": 0,
            "args": [
                1575308004000
            ],
            "tip": 0,
            "hash": "0x40f663e25cbcc09a081a4f6956918823e325cf02d4693892db40dc8592c155da",
            "info": {},
            "events": [
                {
                    "method": "system.ExtrinsicSuccess",
                    "data": [
                        {
                            "weight": 10000,
                            "class": "Operational",
                            "paysFee": true
                        }
                    ]
                }
            ],
            "success": true
        },
        {
            "method": "finalityTracker.finalHint",
            "signature": null,
            "nonce": 0,
            "args": [
                56590
            ],
            "tip": 0,
            "hash": "0x20502068781f0ba1befdbbec79cf6b7e28a6baa32802d0a0ea3795e43a52bccf",
            "info": {},
            "events": [
                {
                    "method": "system.ExtrinsicSuccess",
                    "data": [
                        {
                            "weight": 10000,
                            "class": "Normal",
                            "paysFee": true
                        }
                    ]
                }
            ],
            "success": true
        },
        {
            "method": "parachains.setHeads",
            "signature": null,
            "nonce": 0,
            "args": [
                []
            ],
            "tip": 0,
            "hash": "0xcf52705d1ade64fc0b05859ac28358c0770a217dd76b75e586ae848c56ae810d",
            "info": {},
            "events": [
                {
                    "method": "system.ExtrinsicSuccess",
                    "data": [
                        {
                            "weight": 1000000,
                            "class": "Normal",
                            "paysFee": true
                        }
                    ]
                }
            ],
            "success": true
        },
        {
            "method": "sudo.sudo",
            "signature": {
                "signature": "0xaa39a466975f2169ec1950f783f5808dccbe17257b679ed0e0b994e4d354a62818499ddd94fdc14bc89ba148e6dc7231a353bcc7209da2c524f3cf107dd2d48d",
                "signer": "DwGa4E65nbGafaGirjMH5kXbtT7jHuAcSXHLLkTePnqw95p"
            },
            "nonce": 0,
            "args": [
                {
                    "callIndex": "0x0001",
                    "args": {
                        "_remark": "0x11"
                    }
                }
            ],
            "tip": 0,
            "hash": "0x6ccf7be305e5b2a53f28af48f8a8d25bc7acf88b29882fe2f2a280ac0023169a",
            "info": {},
            "events": [
                {
                    "method": "balances.ReapedAccount",
                    "data": [
                        "DwGa4E65nbGafaGirjMH5kXbtT7jHuAcSXHLLkTePnqw95p"
                    ]
                },
                {
                    "method": "system.ExtrinsicFailed",
                    "data": [
                        {
                            "module": 24,
                            "error": 0
                        },
                        {
                            "weight": 0,
                            "class": "Operational",
                            "paysFee": false
                        }
                    ]
                }
            ],
            "success": false
        },
        {
            "method": "sudo.sudo",
            "signature": {
                "signature": "0xaa39a466975f2169ec1950f783f5808dccbe17257b679ed0e0b994e4d354a62818499ddd94fdc14bc89ba148e6dc7231a353bcc7209da2c524f3cf107dd2d48d",
                "signer": "DwGa4E65nbGafaGirjMH5kXbtT7jHuAcSXHLLkTePnqw95p"
            },
            "nonce": 0,
            "args": [
                {
                    "callIndex": "0x0001",
                    "args": {
                        "_remark": "0x11"
                    }
                }
            ],
            "tip": 0,
            "hash": "0x6ccf7be305e5b2a53f28af48f8a8d25bc7acf88b29882fe2f2a280ac0023169a",
            "info": {},
            "events": [
                {
                    "method": "balances.ReapedAccount",
                    "data": [
                        "DwGa4E65nbGafaGirjMH5kXbtT7jHuAcSXHLLkTePnqw95p"
                    ]
                },
                {
                    "method": "system.ExtrinsicFailed",
                    "data": [
                        {
                            "module": 24,
                            "error": 0
                        },
                        {
                            "weight": 0,
                            "class": "Operational",
                            "paysFee": false
                        }
                    ]
                }
            ],
            "success": false
        }
    ]
}

[tomusdrw]

It's a valid state transition, because the first transaction causes the account balance to drop below ED, so the account is reaped. Then the second (identical) transaction is valid again, cause it also have nonce 0.

The actual curious thing is how did we produce a block with two identical transactions right after another, as mentioned polkadot-js/tools#77 it might be some bug in either transaction pool (storing the same transaction twice) or ReadyIterator returning the same transaction twice.
I've checked the code couple of times already, and added a bunch of tests, but I still can't figure out how is it possible with current version. If it ever happens again we really need traces with sc_basic_authorship=trace,txpool=trace on.

[shawntabrizi]

Does it make sense for an account that has a balance be bootstrapped to have a nonce of at least 1?

i.e. I transfer some funds to an non-existent account. The OnNewAccount hook is called, and we increment the nonce to 1.

[joepetrowski]

Does it make sense for an account that has a balance be bootstrapped to have a nonce of at least 1?

i.e. I transfer some funds to an non-existent account. The OnNewAccount hook is called, and we increment the nonce to 1.

Not sure if nonce=0 is part of what causes this, although the 2 examples we have of it are nonce=0. Maybe worth a script to search for duplicate txs to see if this has happened elsewhere. Sudo does not pay fees, and in the other case the first successful execution reaped the account, so it couldn't pay any more fees. I think the risk here is multiple inclusions of a txs that (a) pays fees and (b) does not reap the account. Would executive charge the length/weight fees prior to realizing that it's invalid (or charge them nonetheless)?

[joepetrowski]

Wrote a script to go through blocks and save those that had duplicate txs.

https://drive.google.com/file/d/1plyBzuLKHLoOroqAsi2etgUa7-3DUgHH/view?usp=sharing

Some findings:

1. This went up to block 1.23 M, but the last occurrence of duplicate txs happened in block 461,362. This may have been fixed in a release since then.
2. Some txs that are duplicated seem to be duplicated again in other blocks. See duplicates-in-many-blocks.json for a map of tx-hash to a list of blocks that contained duplicates.

TODO: See what all these txs have in common / why they might have been included multiple times. Posting here now in case anyone wants to start going through the data.