Orchard commitment-tree root frozen while Sapling tracks the chain, blocking spends with TreeRootsDiverged

[gustavovalverde]

Summary

A testnet wallet holding legacy Orchard notes cannot spend them. Sync aborts with WalletError::TreeRootsDiverged: the wallet's Orchard note-commitment-tree root never tracks the chain, while the same wallet's Sapling tree tracks correctly. The light-wallet server serves a valid, non-empty Orchard frontier and the relevant Orchard actions; the gap is between those inputs and zally's populated Orchard ShardTree.

Environment

Component	Pin	Notes
zally	ec5b213	zally-wallet, zally-storage, zally-chain
zinder	74abb4b	zinder-client, zinder-proto (Ironwood mains)
librustzcash	235d581	[patch.crates-io], PR #2539 (Ironwood scan/storage) + PR #2412 (target_expiry_height)

Chain: Zcash testnet. NU6.3/Ironwood activates at height 4,133,000. The affected wallet's notes are pre-activation legacy Orchard at heights 4,056,276 and 4,057,384. Account birthday is height 4,050,200. The unified address carries receiver_flags = 13 (transparent + Sapling + Orchard).

Symptom

• Sync returns WalletError::TreeRootsDiverged.
• The Orchard commitment-tree root does not advance to match the chain, while Sapling matches at every scanned height.
• The behaviour persists across a full wallet.db wipe and a fresh rescan from the birthday, so it is not local database corruption.
• The frozen Orchard root is a real non-empty value. It matches neither the canonical empty-tree root nor zinder's Orchard frontier at the birthday height, so the Orchard tree receives some state and then stops advancing.

The chain feed is correct

zinder serves a valid, non-empty Orchard frontier at the wallet birthday. TreeStateAtHeight at height 4,050,200 returns Orchard finalRoot = 873948679049069b6a463703c5f776d1c6f71eee96542f41066e51c63c00ff1c alongside a populated finalState, and orchardCommitmentTreeSize advances normally (236,668 at the tip used for this reproduction). zinder also serves the legacy Orchard actions (lightwalletd CompactBlock field 6) that carry this wallet's notes, consensus-verified canonical against zebra.

librustzcash 235d581 is not the cause

At the pinned revision the scan path builds the Orchard tree from scanned legacy actions, and the reference storage path appends those commitments to the Orchard ShardTree. Both stages are Orchard-complete with no post-activation skip:

• Scan emits Orchard leaves unconditionally. scan_block_with_runners reads each CompactTx's actions (legacy Orchard, proto field 6), trial-decrypts under OrchardDomain, and pushes one MerkleHashOrchard::from_cmx(&output.cmx()) per action into orchard_note_commitments. This is gated only by the orchard cargo feature, not by any branch-id or NU6.3 conditional (zcash_client_backend/src/scanning/compact.rs:412-447).
• Ironwood is a separate parallel pool, not a replacement. Ironwood commitments come from ironwood_actions (proto field 9) into a distinct tree and distinct ScannedBundles (compact.rs:450-485, compact.rs:528-543). Post-activation blocks do not divert legacy Orchard actions away from the Orchard tree.
• The reference storage path appends the Orchard leaves. put_blocks collects block.into_commitments().orchard, builds subtrees via build_subtrees::<_, ORCHARD_SHARD_HEIGHT> seeded at the caller-supplied ChainState Orchard frontier, and calls with_orchard_tree_mut(|orchard_tree| update_tree("Orchard", from_state.final_orchard_tree(), ..., orchard_tree, ...)) (zcash_client_backend/src/data_api/ll/wallet.rs:467-508, wallet.rs:584-604).
• Inconsistent inputs fail loudly, not silently. A mismatch between the supplied frontier and a block's reported tree size returns PutBlocksError::NonSequentialBlocks (wallet.rs:237-246), and a missing orchardCommitmentTreeSize against present actions underflows to TreeSizeInvalid (compact.rs:690-706).
• The only NU6.3 default-skip is Ironwood, not Orchard. On the WalletCommitmentTrees trait, with_orchard_tree_mut and put_orchard_subtree_roots are required methods, while with_ironwood_tree_mut has a default returning Ok(None) (zcash_client_backend/src/data_api.rs:3740-3778). Only Ironwood can be silently dropped by a non-overriding backend, and this wallet's notes are pre-Ironwood.

librustzcash HEAD (9eb1f86) sits ahead of the pin but does not change this conclusion for the Orchard scan path.

Root cause locus

zally owns the Orchard ShardTree population. It reads the Orchard root straight out of librustzcash-owned shardtree storage and never computes the tree itself: commitment_tree_roots returns orchard = with_orchard_tree_mut(|tree| tree.root_at_checkpoint_depth(Some(0)))...map(|node| node.to_bytes()) (crates/zally-storage/src/sqlite.rs:701-726). The tree is written only through librustzcash's WalletWrite inside storage.scan_blocks (crates/zally-wallet/src/sync.rs:1558) and through put_orchard_subtree_roots (sqlite.rs:653).

zally does wire Orchard into the pipeline: the chain source maps Orchard to a supported pool and rejects only Ironwood (crates/zally-chain/src/zinder_source.rs:308-314); backfill_subtree_roots iterates both Sapling and Orchard (sync.rs:1461-1497); and the scan-start frontier is carried whole into the scan via from_state (sync.rs:1361-1364). So the Orchard tree is fed a frontier and leaves, yet its root freezes at a non-empty value that never reaches the chain root.

That narrows the fault to zally's Orchard append and checkpoint seam: the scanned Orchard commitments and the backfilled subtree roots are not reconstructing the chain Orchard root the way the equivalent Sapling path does.

Decisive measurement to pin the mechanism

Compare the frozen wallet Orchard root against the Orchard finalRoot that zinder's TreeStateAtHeight returns at scan start (873948679049069b6a463703c5f776d1c6f71eee96542f41066e51c63c00ff1c at height 4,050,200). If they are equal, the frontier was seeded but per-block appends never ran. If they differ, the frontier itself is wrong. This measurement selects the fix location. Candidate mechanisms:

1. The scanned ScannedBlock.orchard() commitments are not appended through with_orchard_tree_mut and update_tree("Orchard", ...) the way Sapling is, so per-block Orchard appends never land.
2. The Orchard ChainState frontier passed to the scan does not match the real Orchard tree size at scan start, so appends build from the wrong position and the reconstructed root diverges.
3. The Orchard subtree-root backfill (sync.rs:1461-1497) does not complete the region below the birthday, leaving the tree unable to reconstruct the full chain root.

Suggested direction

Confirm zally's WalletWrite/WalletCommitmentTrees implementation drives Orchard exactly as the librustzcash reference put_blocks does, rather than treating Orchard like Ironwood's Ok(None) default. Ensure the from_state Orchard frontier matches the tree size at scan start and that put_orchard_subtree_roots covers the pre-birthday region.

Evidence

Live tree-state (zinder 74abb4b, testnet, TreeStateAtHeight).

• Height 4,050,200 (birthday): Orchard finalRoot = 873948679049069b6a463703c5f776d1c6f71eee96542f41066e51c63c00ff1c, Sapling finalRoot = 2809bfce6bb818928f263252f9c72ca149d8a6c32913709cd9f2e11fd6358300, with populated finalState for both.
• Chain metadata: orchardCommitmentTreeSize = 236668, saplingCommitmentTreeSize = 370762 at the tip used for this reproduction (height 4,147,234).
• Height 1,000,000 (pre-NU5): no Orchard finalRoot in the tree-state.

Reported wallet root. 78bdaf718b1d566946420dbc7d755c35e31587d600767d6cda46154429b77403, constant across scanned heights. It does not equal the birthday Orchard frontier above in either byte order, and it is not the canonical empty Orchard tree root.

Reproduction.

1. Fetch a compact block carrying the Orchard actions and confirm the pool contents:
   grpcurl -plaintext 127.0.0.1:19101 zinder.v1.wallet.WalletQuery/CompactBlock -d '{"height":4056276}', base64-decode payloadBytes, then protoc --decode=cash.z.wallet.sdk.rpc.CompactBlock and count actions (Orchard, field 6) versus outputs (Sapling, field 5).
2. Fetch the Orchard frontier at scan start:
   grpcurl -plaintext 127.0.0.1:19101 zinder.v1.wallet.WalletQuery/TreeStateAtHeight -d '{"height":4050200}', base64-decode payloadBytes, read orchard.commitments.finalRoot.
3. Cross-check against consensus with zebra z_gettreestate at the same heights (zebra returns roots in reversed byte order).
4. Sync the wallet from the birthday and read the Orchard root via commitment_tree_roots; observe it frozen while Sapling advances.

Related, lower-priority findings from the same investigation

Two latent issues share the storage-boundary code and are worth folding into this same fix pass rather than filing separately:

verify_tree_roots contradicts its documented empty-tree skip. A wallet whose Orchard tree is legitimately empty at a checkpoint (Sapling correct, no Orchard leaves yet) faults sync with TreeRootsDiverged and no escape hatch, contradicting the documented contract that empty trees are skipped. commitment_tree_roots returns Some(empty_root) rather than None for a scanned-but-empty Orchard tree, because scanning creates a checkpoint every block, so root_at_checkpoint_depth(Some(0)) sees that checkpoint (crates/zally-storage/src/sqlite.rs:715-722). verify_tree_roots skips only when both pools are None; its success arm passes when sapling_match != Some(false) && orchard_match != Some(false), and its fallthrough diverges whenever either pool is Some(false) (crates/zally-wallet/src/sync.rs:1672-1712). A Sapling-correct, Orchard-empty state yields orchard_match == Some(false) and hits the divergence arm, contradicting the doc comment at sync.rs:1631-1633. Suggested direction: at the storage boundary return None for a provably empty (zero-leaf) tree so the existing (None, ...) guard skips it, applied symmetrically to Sapling. Do not relax the non-empty mismatch arm; a genuinely divergent non-empty root must still fault.

decode_tree_state silently defaults a missing Orchard/Ironwood frontier. When a tree-state response omits the Orchard (or Ironwood) finalState, zally seeds an empty frontier into the scan instead of faulting. For a wallet whose birthday precedes an Orchard tree in the served tree-state, this quietly produces a wrong tree that the root check later blames, hiding the true cause. decode_tree_state fills orchard_tree from the /orchard/commitments/finalState JSON pointer but falls back to unwrap_or_default() (crates/zally-chain/src/zinder_source.rs:379-383). zinder omits the Orchard tree entirely from the tree-state at pre-activation heights (confirmed: TreeStateAtHeight at height 1,000,000 returns no Orchard finalRoot), so this default path is reachable. Suggested direction: replace unwrap_or_default() with a hard MalformedCompactBlock error at post-Orchard-activation heights, so an un-fed frontier faults loudly rather than seeding a silent empty tree. Apply the same to ironwood_tree once Ironwood is wired.

[gustavovalverde]

Issues B and C are fixed on main.

• aab23a8 (storage: report no checkpoint root for zero-leaf pools): commitment_tree_roots now reports None for a pool whose latest retained checkpoint commits zero leaves, using leaf presence as the discriminator so a real tree can never be misclassified. verify_tree_roots' existing None guard then skips the pool, honoring the documented empty-tree contract. The non-empty mismatch arm is unchanged and still faults. Covered by a wallet integration test that syncs to tip with an Orchard-empty wallet against a chain serving a non-empty Orchard root.
• cfbdfd4 (chain: refuse tree states missing an active pool frontier): decode_tree_state now returns MalformedCompactBlock when a pool's finalState is missing or empty at a height where the pool is active on the source's network (Sapling, Orchard/NU5, Ironwood/NU6.3), and keeps the benign empty default below activation. Covered by unit tests for the pre-activation, post-activation, empty-string, and fully-populated cases.

Issue A (the frozen Orchard root) remains open: it needs the decisive measurement against the affected wallet data, which is not reproducible from a fresh wallet.

[gustavovalverde]

Closing after a full reproduction-based investigation. Verdict: fixed on main by the shardtree checkpoint-pruning fix family; the mechanism required owned (marked) notes, which is why it cannot be re-triggered from public chain data alone.

Reproduction evidence (live, against a synced testnet zinder serving the exact documented region)

Scenario	Pre-fix rev (ec5b213, the rev in this issue's table, built --locked)	Current main (cfbdfd4)
Fresh wallet, birthday 4,050,200, scan through both legacy note heights to 4,058,210	converges, all roots match	converges, all roots match
Same, extended through the orchard 196,608-leaf shard boundary to 4,133,900 (~84k blocks)	converges, boundary crossed cleanly (189,586 to 198,188 leaves)	converges
Wallet with owned Orchard notes in the region	faults (this issue's report; corroborated by the faucet incident cycle)	works in production (fauzec wallet, full rescan from 4,009,000 with many owned notes, settling payments daily)

The chain feed served the exact documented inputs (orchard finalRoot 87394867... at 4,050,200, size 185,321), and per-block orchard appends landed correctly even at the pre-fix rev: the frozen root was never a frontier/append defect.

Mechanism

Stock shardtree 0.6.2's prune_excess_checkpoints corrupts tree state when clearing flags on folded leaves. Folded leaves exist only where the wallet has marked leaves, i.e. owned notes; chunked scanning churns hundreds of checkpoint prunes per 1000-block chunk, so a wallet with owned Orchard notes in the scanned region deterministically corrupts its Orchard tree, yielding a stable wrong root (matching the observed 78bdaf71...: non-empty, equal to neither the birthday frontier nor the empty root) that a wallet.db wipe + rescan reproduces identically. Sapling survived because this wallet's owned notes were Orchard-pool. A markless wallet never folds, which is why both fresh-wallet reproductions converge at every rev.

Fix attribution

• Upstream: the forked shardtree/incrementalmerkletree 48b5297 (folded-leaf clear-flags fix + retained-checkpoint pruning fix; upstream as incrementalmerkletree PR #192).
• zally adoption: 4ec16bd (folded-leaf clear-flags), c2011d1 (pruning-safe shardtree), 8f6e536 (full Ironwood stack alignment).
• Related hardening from this issue's B and C sections: aab23a8, cfbdfd4.

Regression guard

43a5705 adds a live test (orchard_root_converges_from_legacy_birthday, ci-live profile) that syncs a fresh wallet from birthday 4,050,200 through the legacy note region asserting wallet-vs-chain orchard root convergence at every step, with ZALLY_TEST_SCAN_TARGET extending it through the shard boundary for deep runs. It guards the frontier-seeding and append paths and the zero-leaf None semantics; by the mechanism above, a marked-leaf guard additionally requires a funded fixture wallet, which is the one follow-up this investigation could not automate.

Reopen if a wallet with owned notes in a legacy region diverges on the current stack; that would falsify the mechanism attribution.

[gustavovalverde]

Correction to the closing comment's mechanism attribution, now proven decisively.

The mechanism, verified by value identity

The frozen root 78bdaf718b1d566946420dbc7d755c35e31587d600767d6cda46154429b77403 is, byte for byte, the depth-32 MerkleHashOrchard root over exactly zinder's three backfilled Orchard subtree caps with empty padding:

• subtree 0: 25934a8c8cde7b4ba7e51d78f2321c7e286d140811a192f692f29d3f0ecce510 (completes 3,364,755)
• subtree 1: a7d4af61ae5f9c5a63dd74d7bb541f42ca227c79c30ed360c569167f7dedfc1e (completes 3,861,020)
• subtree 2: 78fccfdeba6dcd684fa879326e6822f59cd0a22c0aaa6df1e5596a877f870128 (completes 4,094,022)

Recomputed independently against the workspace's exact orchard/incrementalmerkletree pins; the identity holds.

So the wallet's Orchard tree was fine and per-block appends landed correctly. The bad actor was the ROOT READ: 65069ed (2026-06-04) changed the read to root_at_checkpoint_depth(None), which computes over the full tree including backfilled caps. The birthday frontier (tree size 185,321) sits inside shard 2, whose cap completes at 4,094,022, above everything the wallet had scanned, so the read reported the caps-only root: constant, non-empty, equal to neither the frontier nor the empty root, and deterministically identical after a wipe. Sapling escaped because all five of its caps complete below the birthday. This refutes the original doc's mechanisms 1 and 2 outright and inverts 3.

Corrected fix attribution

• 04edf80 (2026-07-05) fixed the read: roots report at the latest checkpoint, truncating past-frontier caps out of the computation. This, not the shardtree pruning fork, is the fix for THIS issue; the pruning fixes named in the closing comment address the separate conflict/panic family (Repair ladder never escalates on a recurring same-region conflict fault #5 et al.).
• aab23a8 keeps zero-leaf pools out of verification entirely (defense in depth).
• 0b35587 (new) closes the remaining seam this investigation exposed: subtree backfill installed every cap the chain served with no frontier gate; it now defers caps completing above the fully scanned height to a later backfill pass, so no read can ever span a folded, unscanned cap regardless of read semantics. Covered by unit tests; the live legacy-birthday regression test (43a5705) passes with the filter active.

Why the earlier falsification run converged at ec5b213: that rev (2026-07-06) already contains 04edf80; the frozen-root evidence in the report was captured against the pre-04edf80 read window. The owned-notes hypothesis in the closing comment is withdrawn: no marks are needed for this mechanism, and the caps-root identity explains every recorded observation.