[perfectra1n]

Multi-cluster shared kopia repository support: a first-class cluster identity, own/foreign-aware discovery, collision-free maintenance leases — plus two standalone data-safety fixes this work uncovered.

Why

Sharing one kopia repository across clusters (the volsync-style "back up on cluster A, restore on cluster B" workflow) currently fights the operator on three fronts:

1. Discovery assumes hostname == namespace: other clusters' snapshots warn DiscoveredSnapshotUnplaced or pile into fallbackNamespace.
2. Identity collisions are a real data-loss vector: two clusters with same-named namespaces+policies write the same username@hostname:path; kopia dedups identical manifests, so one cluster's GFS prune can delete a manifest the other cluster's Snapshot CR still references.
3. Maintenance leases collide: the managed lease was derived from the CR name only, so same-named ClusterRepositories on two clusters both "own" maintenance and run concurrent content GC.

What ships

One knob: identityDefaults.cluster (both repo kinds; RFC1123 label ≤32, dot-free). Setting it flips, coherently:

• Default identity hostname becomes <namespace>.<cluster> (CEL gets a cluster variable; explicit overrides still win). Namespaces can't contain ., so the scheme is placement-invertible.
• Discovery classifies own vs foreign (Bare / OwnCluster / ForeignCluster), with a typed catalog.foreignSnapshots: Ignore | Fallback policy. Foreign snapshots are counted (status.catalog.foreignSnapshotCount + kopiur_repo_foreign_snapshots gauge), never silently invisible — and never warned about when they're expected. The mover pre-filters foreign suffixes before its listing cap so a busy peer can't crowd your own snapshots out.
• Maintenance leases become cluster-qualified and injective (kopiur/{cluster}/…, dot-joined kopia identity reserved to the kopiur prefix), with ownerAliases so the legacy owner upgrades automatically on the first claim, and RestampPolicy: OwnFormatsOnly so a cluster never clobbers a foreign owner. Exactly one cluster runs maintenance; others yield visibly.
• mode: ReadOnly is now a safe consumer posture: bootstrap no longer stamps (or clobbers) the maintenance owner and connects with kopia's --readonly.
• Repository identityDefaults edits are guarded at admission: identity re-resolves live on every run (the "pinned at admission" story in the docs was wrong), so an unguarded edit silently re-identified every consumer. Edits now require the allow-identity-change annotation when consumers have history, and list the affected policies as an admission warning.

Cross-cluster restore needed nothing new — Restore.spec.source.identity / fromPolicy already restore without a Snapshot CR; the new scenario page ties it together.

Standalone data-safety fixes (worth calling out — they affect every deployment, not just multi-cluster)

• kopia's create-time retention was silently deleting CR-tracked snapshots. kopia snapshot create unconditionally applies the source's retention policy — and with none set, kopia's defaults (keep-latest 10 / hourly 48 / daily 7 / weekly 4 / monthly 24 / annual 3) delete manifests that Succeeded Snapshot CRs still reference, for any GFS window wider than those defaults. The mover now pins effectively-infinite kopia-side retention at the identity scope (CR-driven GFS is the sole deleter, as designed) and passes --no-auto-maintenance. Verified against the pinned kopia binary: 12 creates → 10 surviving manifests without the fix, 12 with it.
• Snapshot matching was identity-blind. Delete self-heal, deep-verify resolution, and filesystem success resolution matched by source path only — /pvc/<name> collides across namespaces (and clusters). All matchers now filter by identity, with a same-path/different-hostname regression test at unit and e2e tiers.

Migration

docs/scenarios/shared-repository-multi-cluster.md documents the full live-migration story — the order is load-bearing (upgrade all clusters first; disable non-owner maintenance and remove takeoverPolicy: Force before any identity flip; the residual pre-migration dedup window and its deletionPolicy: Orphan mitigation). Setting cluster is unset-by-default and byte-for-byte back-compatible: every cluster-off code path is covered by fixture tests proving plans identical to before.

Verification

• Full hermetic gate green at every one of the 17 commits (mise run test / clippy / complexity-check / gen-check), with each milestone independently re-verified.
• New e2e suite (crates/e2e/tests/multi_cluster.rs): own-suffix placement, foreign ignore+count with zero unplaced warnings, legacy bare placement, lease yield + anti-restamp, legacy-owner alias upgrade, Fallback flip + restore-by-identity of foreign data, and the same-path/different-identity survival regression — plus the webhook acknowledgment flow. Green on fresh kind clusters (7/7 twice: the implementation run and an independent full-suite run).
• Local full-suite runs show the four known machine-specific failures on the dev box (NFS pair, health-probe wipe detection, leak-sweep timing) — verified identical on a main-baseline worktree run, so pre-existing/environmental, not from this branch. CI is the arbiter for those suites.
• kopia-binary integration test for the retention fix.

Related

• Rejects the "archive Snapshot CRs" idea and splits the CR-count levers into Bound Snapshot CR reconcile cost at scale; document the math; no archive mode #249 (with rationale).
• Follow-ups worth their own issues (from the final review): identity-guard degradation under Role-scoped installs (403 → same-namespace LIST fallback); small cleanups (matcher predicate unification, RestoreSource exhaustive-match refactor, restampPolicy serialization skip, milestone-marker comment sweep).

[greptile-apps]

Greptile Summary

This PR adds multi-cluster support for shared Kopia repositories. The main changes are:

• Cluster-aware repository identity through identityDefaults.cluster.
• Own-vs-foreign catalog discovery with foreign snapshot policy and counts.
• Cluster-qualified maintenance leases with alias-based migration support.
• Safer read-only repository bootstrap behavior.
• Identity-aware snapshot matching for restore, verify, and delete paths.
• Admission guards for repository identity default changes.

Confidence Score: 5/5

This PR appears safe to merge.

No actionable bugs were identified in the reviewed identity, catalog, maintenance, mover, or webhook paths.

No files require special attention.

T-Rex Logs

What T-Rex did

• The run captured tool-availability.log to surface missing external tools, including the command, working directory, and exit evidence.
• Identity tests were executed and completed with exit code 0, as shown by the identity tests log.
• Maintenance tests were executed and completed with exit code 0, as shown by the maintenance tests log.
• The E2E compile harness validation ran in compile-only mode and finished with exit code 0.
• Retention tests and related discovery were executed, including retention-test-list, retention-tests, and matcher-tests, all finishing with exit code 0.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
crates/api/src/identity.rs	Adds cluster-aware identity resolution and hostname classification.
crates/api/src/validate/repository.rs	Extends repository validation for cluster identity and foreign snapshot policy combinations.
crates/controller/src/catalog.rs	Implements own/foreign catalog placement, filtering, and counting.
crates/controller/src/io/maintenance.rs	Adds cluster-qualified maintenance owner planning, aliases, and read-only suppression.
crates/mover/src/main.rs	Pins Kopia retention during snapshot creation and honors read-only bootstrap behavior.
crates/mover/src/bootstrap.rs	Adds foreign-suffix prefiltering before the catalog materialization cap.
crates/mover/src/resolve.rs	Adds identity-aware manifest matching to avoid same-path cross-identity collisions.
crates/webhook/src/identity_repo_edit.rs	Adds the repository identityDefaults edit guard and acknowledgement flow.
crates/webhook/src/handlers.rs	Wires identity default edit checks and warnings into repository admission handlers.
crates/controller/src/repository.rs	Propagates cluster, foreign snapshot, read-only, and maintenance settings into bootstrap.
crates/controller/src/cluster_repository.rs	Adds cluster repository bootstrap and catalog support for shared repositories.
crates/api/src/maintenance.rs	Adds injective cluster-qualified maintenance lease identity derivation.

_{Reviews (1): Last reviewed commit: "docs(controller): correct PlacementPass:..." | Re-trigger Greptile}