Summary
In 0.5.0-rc.1, zaino loses nearly all sync progress when the process restarts. After syncing millions of blocks, a restart causes it to resume from ~101k (the golden snapshot baseline) instead of from where it left off. This is a regression from rc/0.4.0 / PR #1238, where restarts correctly resume near the last committed height.
Related: #1260 (OOM during accumulator rebuild) — these two bugs compound: the OOM crashes the process, and this bug ensures no progress is retained across those crashes.
Environment
Observed behavior
0.5.0-rc.1 (broken)
After each OOM restart (45 so far), zaino re-syncs from height ~101k:
zainod v0.4.0
write_blocks_to_height: syncing finalised blocks 101162..=3382533 on Mainnet
write_blocks_to_height: syncing height 103361 / 3382528 on Mainnet
write_blocks_to_height: syncing height 102154 / 3382528 on Mainnet
Every restart shows the same 101162..=3382533 range — no progress is retained.
PR #1238 (works correctly)
After each OOM restart, zaino resumes from near chain tip:
write_blocks_to_height: syncing finalised blocks 3382524..=3382530 on Mainnet
write_blocks_to_height: committed batch to height 3382530 (7 blocks)
Only the last few blocks need re-syncing — LMDB state is properly persisted.
Root cause analysis
The regression was introduced by PRs #1249 and #1250, which added a background sync architecture with ephemeral passthrough (EphemeralMode/EphemeralReference in router.rs).
When sync_to_height encounters a range larger than LONG_RUNNING_SYNC_THRESHOLD, it spawns the sync as a background task and serves reads from a validator passthrough. In PR #1238 (without this architecture), sync writes happen inline/synchronously, so each LMDB batch commit is durable and survives crashes.
With the background sync approach, when the process is killed (e.g., OOM), the spawned task's uncommitted work is lost. On restart, the persistent DB only has data up to ~101k (the snapshot baseline), because the background task hadn't committed its batches to durable storage.
The module path change from finalised_state::db::v1::write_core to finalised_state::finalised_source::v1::write_core is cosmetic — the real difference is the router layer and background sync spawning.
Expected behavior
Sync progress should be durable across restarts, regardless of whether sync is inline or background. Either:
- The background sync should commit batches incrementally to LMDB (as the inline path does), or
- The background sync should checkpoint progress periodically so restarts don't lose everything
Impact
Combined with #1260, this creates a permanent crash loop with zero forward progress: sync → reach tip → OOM during accumulator rebuild → restart from ~101k → repeat. After 45 restarts over hours, the deployment is still at height ~101k.
Summary
In
0.5.0-rc.1, zaino loses nearly all sync progress when the process restarts. After syncing millions of blocks, a restart causes it to resume from ~101k (the golden snapshot baseline) instead of from where it left off. This is a regression fromrc/0.4.0/ PR #1238, where restarts correctly resume near the last committed height.Related: #1260 (OOM during accumulator rebuild) — these two bugs compound: the OOM crashes the process, and this bug ensures no progress is retained across those crashes.
Environment
0.5.0-rc.1(regression), vs PR rc-0.4.0: Speed up finalised-state sync & v1.1→v1.2 migration (sandblast-height stall fix) #1238 /rc_0_4_0_slow_sync_plus_updates(works correctly)Observed behavior
0.5.0-rc.1 (broken)
After each OOM restart (45 so far), zaino re-syncs from height ~101k:
Every restart shows the same
101162..=3382533range — no progress is retained.PR #1238 (works correctly)
After each OOM restart, zaino resumes from near chain tip:
Only the last few blocks need re-syncing — LMDB state is properly persisted.
Root cause analysis
The regression was introduced by PRs #1249 and #1250, which added a background sync architecture with ephemeral passthrough (
EphemeralMode/EphemeralReferenceinrouter.rs).When
sync_to_heightencounters a range larger thanLONG_RUNNING_SYNC_THRESHOLD, it spawns the sync as a background task and serves reads from a validator passthrough. In PR #1238 (without this architecture), sync writes happen inline/synchronously, so each LMDB batch commit is durable and survives crashes.With the background sync approach, when the process is killed (e.g., OOM), the spawned task's uncommitted work is lost. On restart, the persistent DB only has data up to ~101k (the snapshot baseline), because the background task hadn't committed its batches to durable storage.
The module path change from
finalised_state::db::v1::write_coretofinalised_state::finalised_source::v1::write_coreis cosmetic — the real difference is the router layer and background sync spawning.Expected behavior
Sync progress should be durable across restarts, regardless of whether sync is inline or background. Either:
Impact
Combined with #1260, this creates a permanent crash loop with zero forward progress: sync → reach tip → OOM during accumulator rebuild → restart from ~101k → repeat. After 45 restarts over hours, the deployment is still at height ~101k.