shardtree: discard stale cap roots on truncation

[gustavovalverde]

Checkpoint truncation could retain a cached cap hash covering positions after the target checkpoint. After inserting replacement commitments, root queries could return the pre-truncation root.

The cap is non-authoritative: shard data remains the source of truth. This change retains cache nodes wholly before the checkpoint and discards cached leaves and parent annotations that cross its boundary, forcing later root queries to recompute from the retained and replacement shards.

Shard and checkpoint truncation semantics are unchanged. The regression reproduces the relevant wallet sequence through public ShardTree operations: insert subtree roots, cache their combined root, scan to a valid checkpoint, truncate, and insert replacement commitments.

[nuttycom]

@gustavovalverde this looks like it now conflicts with main

[gustavovalverde]

@nuttycom rebased

[nuttycom]

truncate_to_checkpoint could report success without changing the tree when the target position fell inside a pruned node. That left shards, cap roots, and checkpoints retaining hashes that committed to data beyond the requested cut, so a caller could rescan into deterministic InsertionError::Conflict failures.

This sounds to me like a problem in the pruning logic, not one that we should work around by truncating to a different point than the one that was requested. It should always be possible to truncate to exactly any checkpoint leaf; if that invariant isn't preserved by some paths, then the correct option is to find that bug rather than try to work around it.

[nuttycom]

Blocking on figuring out how it's possible to get into the described failure state in the first place. Checkpoints should be allowed to be aggregated up the tree only in the case that a checkpoint corresponds to the right-hand leaf of some filled subtree. If there is a path to a checkpoint being constructed (or retained after pruning) for a pruned leaf, that's definitely a bug.

[gustavovalverde]

@nuttycom the fix is now more targeted. The PR body was updated alongside the changelog to represent the latest code

[gustavovalverde]

Thanks, you were right: the earlier tests manufactured invalid checkpoints and did not establish a real path to that state. I removed that behavior and narrowed the PR to the stale cap cache issue.

The regression now uses only a valid checkpoint and public operations: import 2 shard roots, cache their combined hash, scan the first shard, truncate at its checkpoint, then insert a replacement second shard. On main, the final root incorrectly remains abcdefghijklmnop...; with this patch it becomes abcdefghIJKLMNOP....

The fix does not change shard or checkpoint truncation semantics. It only discards cap nodes and annotations that commit beyond a successful truncation boundary, so later roots recompute from authoritative shard data.