[perfectra1n]

Why

A user's kro tooling briefly deleted and recreated their SnapshotSchedules. Every scheduled
Snapshot carries an ownerReference to its schedule, so Kubernetes GC cascade-deleted roughly 600
to 700 Snapshot CRs in one burst. Each CR defaulted to deletionPolicy: Delete, and each finalizer
spawned its own mover Job running kopia snapshot delete with no concurrency bound anywhere. The
result was hundreds of pods actively destroying backup history, stopped only because the user
noticed in time. Deleting all of your backups while making the cluster suffer should not be the
default behavior of a backup operator.

There were three uncapped delete surfaces: external cascades (this incident, but also GitOps prunes
and bulk kubectl deletes), the GFS retention prune, and the failed-history prune. Only the
namespace-deletion axis had a guard (onNamespaceDelete, default Orphan).

What

One discriminator plus three guards, all decided in pure, exhaustively matched functions
(ADR-0006 has the full rationale):

1. pruned-by discriminator. The operator's own prune paths now stamp
   kopiur.home-operations.com/pruned-by: retention|failed-history before deleting, so the finalizer can
   tell Kopiur's lifecycle from an external deletion. Unrecognized values are treated as external
   (fail-safe).

2. Schedule-deletion cascade guard. New SnapshotSchedule.spec.deletion.onScheduleDelete
   (default Retain), stamped onto every produced Snapshot and propagated on edit. When a Snapshot
   is deleted externally and its owning schedule is gone, terminating, or was replaced (UID
   mismatch), an effective Delete downgrades to a retain: the finalizer releases with zero repo
   contact, a Warning event is emitted, and the catalog rediscovers the kopia snapshots as
   origin: discovered (forced Retain) within the catalog interval. The pre-upgrade fleet is
   protected retroactively because an absent stamp resolves to Retain. Deliberate cascade remains
   available by setting onScheduleDelete: Delete.

3. Per-repository mass-deletion breaker. New deletionProtection.threshold on both repository
   kinds (default 10, 0 disables). When pending external destructive deletions reach the
   threshold, they hold: no delete work happens, the Snapshot gets a DeletionHeld condition whose
   message contains the exact ack command and the value to copy, and the repository mirrors a
   MassDeletionHeld condition. Release is a timestamp-valued annotation
   (kopiur.home-operations.com/allow-mass-deletion) that only frees deletions requested at or before the
   acked instant, so a stale ack committed to Git is inert against any later wave. Operator prunes
   bypass the breaker by design; retention keeps working during a held wave.

4. Batched, optionally capped execution. Deletions now aggregate per repository into one mover
   Job (up to 200 manifest ids over a single repo connect, ~10s accumulation window) instead of one
   Job per Snapshot. A lone delete is a batch of one. The old per-CR delete Job is retired (an
   adoption shim honors in-flight legacy Jobs across the upgrade, removal tracked for next
   release). A global concurrency cap exists as an opt-in backstop
   (maxConcurrentDeleteJobs, default uncapped) since batching already collapses the incident
   scenario to about three pods; a small default cap would only add head-of-line blocking.

The counting and firing filters deliberately diverge: breaker counting over-includes (trips
earlier, fail-safe), batch firing under-includes (held, namespace-terminating-orphan-destined,
unpinned-peer, and unsynced-schedule-owned members are excluded and simply retry). An excluded
member requeues; it is never silently dropped.

Bugs found and fixed along the way

The e2e tiers caught three real bugs during development, each now guarded by a regression test:

• A spawned wait_until_ready() on the controller's own store races the applier's getter
  (kube-rs DelayedInit holds a single waker); the synced flag never flipped and every external
  deletion deferred forever. Fixed by flipping the flag from reconcile, which kube-runtime already
  gates behind the store's init swap.
• The no-overlap exclusion originally covered only live Jobs, so members of an already-succeeded
  batch could be re-enrolled before their finalizers released, serializing mass drains at scale.
• The batch fire set originally reused the breaker's inclusive counting filter, letting a routine
  retention prune sweep held members' data into its batch without an ack. The flagship e2e scenario
  (held wave survives a concurrent prune untouched) now pins the fix.

Testing

• Hermetic: full workspace green; cognitive-complexity ratchet tightened 31 -> 29.
• e2e (fresh kind clusters): 8 new mass-deletion scenarios including the incident's regression
  guard (schedule delete leaves kopia intact, zero delete Jobs, rediscovery), breaker hold/ack
  drain with real kopia-side count assertions, concurrent-wave disjointness, throttle cap,
  repo-outage retry convergence, and held-wave-vs-prune. Full suite green (known local-only NFS
  failures excepted, unchanged from main).
• Adversarially reviewed: per-milestone reviews plus a whole-branch review that walked every path
  ending in kopia snapshot delete and every finalizer release that skips deletion.

Upgrade notes

• Existing Snapshots need no migration: absent stamps resolve to Retain, and startup backfills the
  repository pin for pre-Snapshot deletion remains stuck when its credential-projecting SnapshotPolicy was deleted first #255 CRs so per-repo breaker counting sees them.
• The breaker is ON by default (threshold 10). Repositories with legitimately more than 10
  concurrent external CR deletions should raise it or set 0.
• Anything matching on {name}-delete Job names (dashboards, alerts) should switch to the
  snapdel-* batch Jobs and the new kopiur_snapshot_deletions* metrics.
• Old-operator prune waves in flight during the upgrade auto-reclassify via the terminating-CR
  stamp; a very large wave may need one ack.

Closes the incident class discussed in Discord; supersedes nothing.

[greptile-apps]

Greptile Summary

This PR adds mass-deletion protection for Snapshot cleanup paths. The main changes are:

• Schedule deletion now defaults to retaining produced kopia snapshots unless explicitly configured to cascade.
• Repository and ClusterRepository specs now include a mass-deletion breaker with timestamp-based acknowledgement.
• Snapshot deletes are batched per repository through snapdel-* mover Jobs instead of one Job per Snapshot.
• Operator prune paths stamp a pruned-by annotation so retention and failed-history cleanup bypass external-deletion guards.
• Controller watches, CRDs, Helm values, docs, metrics, and e2e coverage were updated for the new deletion flow.

Confidence Score: 5/5

Safe to merge based on the reviewed changed paths.

No verified runtime, data-loss, or security issues remained after reviewing the deletion planning, batching, schedule stamping, prune annotation, mover execution, and repository acknowledgement wiring.

No files require special attention.

T-Rex Logs

What T-Rex did

• Verified the environment readiness by confirming cargo and rustc are available, while docker, kind, kubectl, and helm are not.
• Ran the mass deletion controller snapshot tests using the documented command and working directory; the test completed with EXIT_CODE: 0.
• Ran the mass deletion mover workspec tests using the documented command and working directory; the test completed with EXIT_CODE: 0.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
crates/controller/src/snapshot/mod.rs	Wires live deletion facts into the finalizer and dispatches repository-scoped batch delete Jobs.
crates/controller/src/snapshot/batch.rs	Adds batch membership, breaker counting, fire eligibility, throttling, and batch Job lifecycle helpers.
crates/controller/src/snapshot/plan.rs	Centralizes deletion decisions for cascade retention, namespace deletion, prune bypass, and breaker holds.
crates/controller/src/snapshot_schedule.rs	Stamps and propagates schedule deletion policy onto produced Snapshots.
crates/controller/src/io/prune.rs	Adds stamp-then-delete helpers for retention and failed-history prunes.
crates/mover/src/main.rs	Runs batch snapshot deletes and reports batch Jobs through log-only status handling.
crates/mover/src/workspec/mod.rs	Adds the batch snapshot-delete work-spec payload.
crates/e2e/tests/mass_deletion.rs	Adds e2e coverage for cascade retention, breaker hold/ack, batching, throttling, and prune interaction.

_{Reviews (1): Last reviewed commit: "chore(mise): revert unintended node lock..." | Re-trigger Greptile}