[Feat]: provide flags to reduce overhead of referrer sync

[jzhn]

Is your feature request related to a problem? Please describe.

Currently, GET /referrers endpoint at Zot is extremely slow.

time regctl artifact tree <upstream>/image@sha256:aaa
regctl artifact tree   0.02s user 0.03s system 3% cpu 1.359 total

time regctl artifact tree <zot>/image@sha256:aaa
regctl artifact tree   0.01s user 0.02s system 0% cpu 14.348 total

for a simple tree of OCI image

Digest: sha256:aaa
Children:
  - sha256:bbb [linux/amd64]
    Referrers:
      - sha256:ccc: sha256-bbb.att
      - sha256:ddd: sha256-bbb.sig
  - sha256:eee [linux/arm64]
    Referrers:
      - sha256:fff: sha256-eee.att
      - sha256:ggg: sha256-eee.sig
Referrers:
  - sha256:hhh: sha256-aaa.sig

While it’s understandable that OCI 1.1 referrers API is designed to be dynamic so it offsets the benefit of a pull through cache, looking at Zot’s code, there are a few things that seriously slows down Zot:

• It recursively syncs the tree of all referrers, and the referrers or referrers…
• It also fetches all tags in attempt to the discover old cosign compatibility tags. if matched, the tree of those are also synced.
• there’s no parallelism when syncing multiple trees discovered from above.

Describe the solution you'd like

adding feature flags per sync repostory, to tune down the aggressiveness of referrer sync.

• conditionally not to sync the cosign compatibility tags - those can be synced later when the client requests for the manifests.
• conditionally not to sync recursively - only sync the root-level referrers.

Describe alternatives you've considered

• consider the referrer API so dynamic that we just always proxy the call. this would solve the problem nicely - we still get the nice caching benefits of content-addressable objects, but leave the dynamic objects simply proxied.

Upon discussing with Zot community at Slack, it is determined that it's unsafe because there could be referrers being pushed to Zot referring a mirrored manifest/index. https://cloud-native.slack.com/archives/C03EGRE4QGH/p1772485455490499

Additional context

No response

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[andaaron]

Fixed in #3842