[Bug]: GC CleanupRepo re-reads all manifests per deleted blob (quadratic on remote storage)

[thees]

zot version

2.1.18

Describe the bug

CleanupRepo checks every candidate blob against the repo index by re-reading all manifests from storage once per blob (IsBlobReferenced performs a full manifest traversal per digest). This makes a single GC run cost O(deleted blobs × manifests) storage reads.

On remote storage (GCS/S3) with many tags this is catastrophic:

• In our example, a repo with 519 manifests took ~20s per deleted blob (~500 sequential manifest GETs each).
• The repo lock is held for the entire run, so a single cleanup can stretch into a multi-day operation and blocks all pushes to that repo in the meantime.

To reproduce

1. Configuration: zot with a remote storage backend (GCS or S3) and GC enabled.
2. Client tool used: push many tags/manifests into a single repo (e.g. ~500 manifests), then delete enough tags to leave a large batch of orphan blobs to be collected.
3. Seen error: GC (CleanupRepo) crawls at tens of seconds per deleted blob; the repo lock stays held for hours, and concurrent pushes to that repo block until GC finishes.

Expected behavior

The referenced-blob set should be computed once per repo per cleanup run (refreshed only when a manifest deletion actually rewrites index.json), so GC cost is roughly O(manifests + deleted blobs) instead of quadratic, and the repo lock is released in a timely manner.

Additional context

Proposed fix:

• Add common.GetReferencedBlobs, which walks index.json and reads each (possibly nested) manifest exactly once to build the full set of referenced digests (manifest, config, and layer digests). It mirrors IsBlobReferencedInImageIndex's traversal and skips unreadable/missing manifests identically.
• Rework CleanupRepo to build that set once and reuse it for every blob, refreshing it only after a manifest deletion rewrites index.json (blobs referenced solely by the deleted manifest then become collectible).
• Introduce deleteBlobChecked(repo, digest, isReferenced func() bool) so GC can pass the precomputed set, while the DeleteBlob API path keeps the per-blob IsBlobReferenced check unchanged.

Behavior preserved: same referencing semantics and same handling of unreadable manifests; only the read pattern changes.

Tests added:

• TestCleanupRepoReadsEachManifestOnlyOnce — asserts each manifest is read exactly once regardless of the number of deleted blobs (regression guard against the quadratic reads).
• TestCleanupRepoDeletesBlobsOrphanedByManifestDelete — verifies blobs orphaned by a manifest deleted in the same run are still cleaned up (validates the refresh-after-deletion logic).