Description
System information
Geth version: Geth/v1.9.25-stable-e7872729/linux-amd64/go1.15.6
OS & Version: Linux Ubuntu 18.04 LTS
Commit hash : (if develop)
We have a Highly Available configuration with 2 Geth Node and AWS Application Load Balancer.
Our software (several workers) sending quite a high number of transactions that are balanced across the nodes.
Sometimes the network fees rising high, so several transactions stuck in a pending state. We limit this number to 10.
When it occurs we bumping the gas limit to push them into the network.
Expected behaviour
The transactions should stay in the queue and emit to the network ether after increasing the gas limit, or at the moment when the gas price goes down
Actual behaviour
In some very rare case under heavy network load a node drops all transactions and start to emit new with the changing nounce.
There is a message in the log: Demoting invalidated transaction
Steps to reproduce the behaviour
It is quite a rare case, but it seems to become more frequent with the increase of a load of a node ( a number TX per interval) and with the growth of network fees
I am not sure if it is a bug or node misconfiguration but here is a key point of opening this issue:
Line 1438 in 2778667
if list.Len() > 0 && list.txs.Get(nonce) == nil {
gapped := list.Cap(0)
for _, tx := range gapped {
hash := tx.Hash()
log.Error("Demoting invalidated transaction", "hash", hash)
// Internal shuffle shouldn't touch the lookup set.
pool.enqueueTx(hash, tx, false, false)
}
So it definitely goes into this function and drops pending transactions which should be never happened according to the comment.
I wonder if any setting could protect from such an event? Is it a software issue due to the loaded network?