State sync up to a specific block.

[shamil-gadelshin]

What is the correct (easiest) way to sync the blockchain state to a specific block in Substrate?

We'd like to develop a way to fast-sync our Substrate-based blockchain similar to warp-sync or similar techniques.
This is our current idea:

• Get a block close to the head (it could be a thousand blocks away). How we choose the block and justify the security of this choice is out of the scope of this question because it is specific to our blockchain design.
• Insert this block into the blockchain bypassing the checks. Download and insert the state for this block.
• Resume normal Substrate sync.

We developed a solution based on Substrate codebase.

1. We configure a node with SyncMode::LightState to allow bypassing some checks.
2. Insert the target block using a combination of the lock_import_and_run and apply_block.
3. Reuse StateSync syncing strategy to download and insert the state by utilizing a simplified form of the syncing-engine.
4. Set internal data structures to the correct values to resume normal syncing:
   a) Remove block gap data from the blockchain to skip downloading the previous history.
   b) Reinitialize the syncing engine and its strategies with SyncMode::Full to download the full blocks.
   c) Update the known common block values with the target block number when nodes are synced themselves.

The core of this functionality (2. inserting a target block and 3. obtaining the correct state) is rather straightforward and requires only exporting some Substrate API as public.
However, additional "hacks" like "1. Use SyncMode::LightState" and "4. Set internal data structures" seem invasive. Actually, "4.b) Reinitialize the syncing engine to use SyncMode::Full" and "4.c) Update the known common block values" are redundant because Substrate ChainSync strategy can auto-recover from these events but it produces error messages in the process - so it's UX improvement.

1.Node configuration with SyncMode::LightState allows bypassing some checks - specifically the check for NonCanonicalOverlay. The error seems to occur because SyncMode::Full sets commit_state variable to true when starting from genesis in contrast to LightState and try_commit_operation operation goes to the branch where it triggers canonicalize_block which in turn goes to NonCanonicalOverlay and triggers the final InvalidBlockNumber error.

4.a) Block gap is set during the try_commit_operation when the blockchain detects the difference between the best block and the block we're trying to insert is greater than 1.

So, how do we insert a specific block in the blockchain and download the state for it without triggering errors? How to disable gap sync download properly? Is there a simple way to achieve our goal without the described techniques?

We appreciate your comments.

[nazar-pc]

Tl;DR is basically the following:

1. Chain is at Genesis
2. We implemented downloading of block and its state
3. We need to find a way to insert this downloaded block with state (useally using normal ImportQueueService::import_blocks such that we don't hit any errors and afterwards everything looks like a normal pruned node

@bkchr may I bother you to help with this a bit? Or maybe there is someone with good knowledge of this part of the codebase.
I suspect this is probably not doable with nice public APIs right now, but maybe there is a small gap to bridge so we can get what we need here?

UPD: Think of it like warp sync, but without gap sync that follows.

[shamil-gadelshin]

These PRs resolve the issue:

• Allow to disable gap creation during block import #5343
• Syncing strategy refactoring (part 2) #5666
• Syncing strategy refactoring #5469