Once out of sync, Prysm struggles to catch up because of enqueued (aggregated) attestations processing. #16160
Description
If Prysm receives an (aggregated) attestation for an unknown block, it enqueues it, and processes it when the corresponding block is received.
Receiving an attestation for an unknown block is a common situation when the node falls out of sync.
Processing enqueued (aggregated) attestations can take a lot of time, sometimes more than 12 seconds.
In the following graph, the blue curve represents how far the node is out of sync, and the yellow curve represent the pending (aggregated) attestations processing time.
The blue curve is plotted from the sinceSlotStartTime field of the following log:
[2025-12-18 19:13:49.63] DEBUG blockchain: Synced new block block=0x1b2040b8... chainServiceProcessedTime=345.940662ms dataAvailabilityWaitedTime=33.254µs epoch=62166 finalizedEpoch=62164 finalizedRoot=0x1a4982e2... justifiedEpoch=62165 justifiedRoot=0xc30edf74... parentRoot=0x199659b8... sinceSlotStartTime=1.633639852s slot=1989319 slotInEpoch=7 version=fulu
The orange curve is plotted from the duration field of the following log:
[2025-12-17 13:13:21.49] DEBUG sync: Verified and saved pending attestations to pool blockRoot=00df1ddeec58 duration=9.944169547s pendingAttsCount=1661 prefix=sync
This situation typically happens at start, when the node is synced to the latest finalized block but not yet to the head.
However, this situation may also happen when, for any reason, the node fell out of sync, and now struggles to reach the tip of chain again.