Confusing blocks-pruning description

[crystalin]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

Default is to keep all finalized blocks. otherwise, all blocks can be kept (i.e 'archive'), or for all canonical blocks (i.e 'archive-canonical'), or for the last N blocks (i.e a number).

NOTE: only finalized blocks are subject for removal!

If only the finalized blocks are subject to removal, and all finalized blocks are canonical (maybe I'm wrong here?), what is the difference between parameter default (all finalized blocks) and archive-canonical ?

Steps to reproduce

No response

[lexnv]

Hi,

all finalized blocks are canonical (maybe I'm wrong here?)

The consensus engine may not revert a finalized block,
while the canonical block (theoretically) can be reverted.
Therefore, a finalized block also implies that the block is canonical
(but not necessarily the other way around).

what is the difference between parameter default (all finalized blocks) and archive-canonical ?

The default option is BlocksPruning::KeepFinalized (see appendix section) which
is equivalent to the archive-canonical CLI flag.

The BlocksPruning::KeepAll is equivalent to the archive CLI flag.

The archive-canonical option implies that ONLY the finalized blocks are kept around
and displaced branches (blocks from forks) are eventually pruned (ie removed).

[block 0] -- [block 1 ] -- [block 2 ] -- [block 3]
         |
          -- [block 1'] -- [block 2']

In this example, when block 3 is finalized, the displaced branch (the fork containing 1' and 2')
is pruned (presuming the node is started with archive-canonical).

NOTE: only finalized blocks are subject for removal!

I'm not quite sure if this is right. I supposed that was true at some point and
we should update the documentation for this.

// CC @arkpar @cheme @bkchr

As opposed to archive option that implies that ALL the blocks are kept around in the DB.

Appendix

The BlocksPruning that you mention is defined in the client/db code:

substrate/client/db/src/lib.rs

Lines 322 to 331 in 54b7395

	/// Block pruning settings.
	#[derive(Debug, Clone, Copy, PartialEq)]
	pub enum BlocksPruning {
	/// Keep full block history, of every block that was ever imported.
	KeepAll,
	/// Keep full finalized block history.
	KeepFinalized,
	/// Keep N recent finalized blocks.
	Some(u32),
	}

The CLI exposes this option to end-users here:

substrate/client/cli/src/params/pruning_params.rs

Lines 33 to 42 in 54b7395

	/// Specify the blocks pruning mode, a number of blocks to keep or 'archive'.
	///
	/// Default is to keep all finalized blocks.
	/// otherwise, all blocks can be kept (i.e 'archive'),
	/// or for all canonical blocks (i.e 'archive-canonical'),
	/// or for the last N blocks (i.e a number).
	///
	/// NOTE: only finalized blocks are subject for removal!
	#[arg(alias = "keep-blocks", long, value_name = "COUNT")]
	pub blocks_pruning: Option<String>,

While the user input flag is parsed here:

substrate/client/cli/src/params/pruning_params.rs

Lines 66 to 76 in 54b7395

	Some(bp) => match bp.as_str() {
	"archive" => Ok(BlocksPruning::KeepAll),
	"archive-canonical" => Ok(BlocksPruning::KeepFinalized),
	bc => bc
	.parse()
	.map_err(|_| {
	error::Error::Input("Invalid blocks pruning mode specified".to_string())
	})
	.map(BlocksPruning::Some),
	},
	None => Ok(BlocksPruning::KeepFinalized),

Then, here is where the pruning actually happens:

substrate/client/db/src/lib.rs

Lines 1738 to 1750 in 54b7395

	BlocksPruning::KeepAll => {},
	BlocksPruning::Some(blocks_pruning) => {
	// Always keep the last finalized block
	let keep = std::cmp::max(blocks_pruning, 1);
	if finalized >= keep.into() {
	let number = finalized.saturating_sub(keep.into());
	self.prune_block(transaction, BlockId::<Block>::number(number))?;
	}
	self.prune_displaced_branches(transaction, finalized, displaced)?;
	},
	BlocksPruning::KeepFinalized => {
	self.prune_displaced_branches(transaction, finalized, displaced)?;
	},

[crystalin]

Thank you for the clear explanation. I think the confusion came from the "NOTE: only finalized blocks are subject for removal!" which as you are suggesting might not be true anymore.

[cheme]

As I understand it, the note is here to underline that the pruning (when a pruning range is defined) is done against finalized the blocks in db, with range applying over the latest finalized block number.
But it should maybe say "within the finalized block number range" or something similar.
But when using range pruning only canonical blocks are stored in state so it is not fundamentally false (only in archive mode we do prune forks in state).