[Bug]: Pull-through cache re-triggers upstream pulls, exhausting rate limits and downloading all archs

[lgtm-dude]

zot version

ghcr.io/project-zot/zot-linux-amd64:v2.1.3-rc5

Describe the bug

Please don't get me wrong, I really, really like the ZOT registry its a cool piece of software, thanks for developing it :-)

Description of my deployment:

• Internal zot registry, authenticated Users (Keycloak / oidc) are able to delete images from cache
  • Even unauthenticated users can pull Docker images from my intranet
• The ZOT registry works as a pull through cache, it's the only kind that will work in my situation.

Current available Versions:
The problem is critical in v2.1.2 and earlier, ghcr.io/project-zot/zot-linux-amd64:v2.1.3-rc5 seems to reduce the impact between v2.1.2 and v2.1.3-rc5 it's still awful.

Problems:
I am using the ZOT Image Registry to cache docker.io Images because I do not want to run into the PULL Limits within the first 60 Minutes of the day (many users try to pull behind same NAT including my CI/CD). As ma internet connection is quite low, useless fetches are a huge problem as well.

I need to be able to fetch all possible Images from docker.io (quay, gcr as well, without restriction) so I cannot reduce the surface given by prefix: "**" and even if I do so, the Problem is still there but with reduced collision area.

1. The main problem is, as soon an docker.io Image is being cached and gets refetched by a client, ZOT starts to reevaluate all existing images within its registry and starts downloading the images again. That naturally eliminates my RATE limits (200 right now) within a pull of 3-8 already cached images, that's not what I need a Image registry for.

2. When it starts retransmitting the Images, it also downloads ALL architectures, it literately DDOS'es my Internet connection.

3. State of v2.1.3-rc5 i cannot pass a list of architectures to ZOT that i need which also nukes my storage ;-)

Possible fix?

• As docker.io rate limits must not be wasted please consider:
  • Only pull the needed architectures of an image
  • Only pull an update of just the one image that is being requested and only if it really had an update
    • You may use insted of a PULL or GET /v2/reponame/manifests/ ,which counts toward pull limits, something different like HEAD /v2/reponame/manifests/ or GET /v2/reponame/tags/list or whatever possible (as far I know).
  • At least it would be fine having a switch that will not check for updates on an existing image (retagging images is not fine anyways, despite latest)

I really don't want to switch to harbor, nexus or whatever tools are out there and stick with ZOT but the time I can spend on ZOT is very limited, runs out and I might never ever come back again if that can't be fixed in a timely manner.

This is a real show stopper to me and I don't think I'm the only human out there that has that specific problem, please help!

To reproduce

1. Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: zot-config
  namespace: zot
data:
  config.yaml: |
    storage:
      rootDirectory: "/var/lib/zot"
      dedupe: true
      commit: true
      gc: true
      gcDelay: "1h"
      gcInterval: "24h"
      retention: {
        dryRun: false,
        delay: "1h",
        policies: [{
          repositories: ["docker-images/**", "k8s-images/**"],
          deleteReferrers: true,
          deleteUntagged: true,
          keepTags: [{
            patterns: [".*"],
            pulledWithin: "720h"
            }]
          }
        ]
      }
    log:
      level: "info"
    http:
      address: "0.0.0.0"
      port: 5000
      externalUrl: "https://zot.example.com"
      auth: {
        failDelay: 10,
        openid: {
          providers: {
            oidc: {
              name: "Keycloak",
              clientid: "myzotkeycloakclientid",
              clientsecret: ,
              keypath: "",
              issuer: "https://keycloak.example.com/realms/example",
              scopes: ["openid"]
            }
          }
        }
      }
      accessControl: {
        repositories: {
          "**": {
            defaultPolicy: ["read", "create", "update", "delete"],
            anonymousPolicy: ["read"]
          }
        }
      }
    extensions: {
      ui: {
        enable: true
      },
      scrub: {
        enable: true,
        interval: "24h"
      },
      search: {
        enable: true,
        cve: {
          updateInterval: 4h
        }
      },
      sync: {
        enable: true,
        credentialsFile: "/config/credentials.json",
        registries: [
          {
            urls: ["https://k8s.gcr.io"],
            maxRetries: 1,
            retryDelay: "10s",
            content: [
              {
                prefix: "**",
                destination: "/k8s-images"
              }
            ],
            onDemand: true,
            tlsVerify: true
          },
          {
            urls: ["https://docker.io"],
            maxRetries: 2,
            retryDelay: "10s",
            content: [
              {
                prefix: "**",
                destination: "/docker-images"
              }
            ],
            onDemand: true,
            tlsVerify: true
          },
          {
            urls: ["https://quay.io"],
            maxRetries: 1,
            retryDelay: "10s",
            content: [
              {
                prefix: "**",
                destination: "/quay-images"
              }
            ],
            onDemand: true,
            tlsVerify: true
          }
        ]
      },
      metrics: {
        enable: true,
        prometheus: {
          path: "/metrics"
        }
      }
    }
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: zot-credentials
  namespace: zot
data:
  credentials.json: |
    {
      "docker.io": {
        "username": "mydockeruser",
        "password": ""
      }
    }

3. Client tool used
4. Seen error

Expected behavior

• ZOT checks for updates without consuming docker.io rate limits.
• ZOT is able to fetch only images of an given array of architectures
• ZOT is able to just auto update the requested Image, not everything stored in ZOT (no useless traffic, no waste of limits)
  • As an alternative, zot does not auto update images already within the ZOT store
• ZOT Devs thaught about that limitations if they do also apply to quay.io and gcr.io

Screenshots

No response

Additional context

ZOT is a cool tool, keep it up and thank you for creating it :-)

[rchincha]

@lgtm-dude have you looked at this example?

https://github.com/project-zot/zot/blob/main/examples/config-docker-compat-sync.json

especially the "preserveDigest: true" part?

v2.1.3 documentation is pending a final release.

[lgtm-dude]

Dear @rchincha

I've set preserveDigest: true and had to add compat: ["docker2s2"] to the http block of the config aswell.

I tried these versions:
ghcr.io/project-zot/zot-linux-amd64:v2.1.3-rc5
ghcr.io/project-zot/zot-linux-amd64:v2.1.3-rc6

What changed is that now I can pull an already cached Image again, despite being docker.io rate limited.
But still, each time something pulls out of the Registry, the pull rate limit of docker.io gets reduced and the images get pulled too.

A local pull (when hosted in Docker, not k8s) shows same behavior.

[andaaron]

The issue is without reaching out to docker.io, we don't know if the tag is pointing to the same manifest, or it is pointing a newer manifest. For example alpine:3.21 may have moved from 3.21.2 to 3.21.3.

@eusebiu-constantin-petu-dbk, @rchincha, maybe we could make this behavior configurable? Does this make sense?

[lgtm-dude]

Idea: Maybe creating a custom image on hub.docker.com as a reference to test it? (or using one that you are 100% sure it’s what you want to compare so then there is no need to reach out to hub.docker.com / docker.io if we do so) For the sake of trying it? Greetings Christoph Von: Andrei Aaron ***@***.***> Gesendet: Montag, 5. Mai 2025 12:24 An: project-zot/zot ***@***.***> Cc: Christoph Schimana ***@***.***>; Mention ***@***.***> Betreff: Re: [project-zot/zot] [Bug]: Pull-through cache re-triggers upstream pulls, exhausting rate limits and downloading all archs (Issue #3129) [https://avatars.githubusercontent.com/u/25718736?s=20&v=4]andaaron left a comment (project-zot/zot#3129)<#3129 (comment)> The issue is without reaching out to docker, we don't know if the tag is pointing to the same manifest, or it is pointing a newer manifest. For example alpine:3.21 may have moved from 3.21.2 to 3.21.3. @eusebiu-constantin-petu-dbk<https://github.com/eusebiu-constantin-petu-dbk>, @rchincha<https://github.com/rchincha>, maybe we could make this behavior configurable? Does this make sense? — Reply to this email directly, view it on GitHub<#3129 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BRQPIVR7SIVIODN24EUUTED24432FAVCNFSM6AAAAAB4JSYTL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNJQGU2TKNBUG4>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>

[andaaron]

I mean zot needs to know internally if it needs to "invalidate" the cache, in case there is a newer manifest available using the same tag at docker.io.

How to do this without hitting the rate limit?

[lgtm-dude]

No idea if it will work but my ChatGPT gave me the following hint:

---

Recommended Strategy (English Summary for Zot Developers)

To detect if a tag (e.g., latest) has changed without hitting Docker Hub’s rate limits, cache the current Docker-Content-Digest for the tag and later perform a HEAD request to the same manifest endpoint with the If-None-Match header set to that digest. If the digest has not changed, the registry returns 304 Not Modified; otherwise, 200 OK with a new digest in the Docker-Content-Digest response header. This works reliably across tags and digest-based pulls, is OCI-compliant, lightweight, and easy to implement in Go. Authentication is required via Bearer token (scoped for pull access).

Minimal GO Example:

package main

import (
	"fmt"
	"net/http"
)

func checkTagChanged(tag string, cachedDigest string, token string) (string, bool, error) {
	url := fmt.Sprintf("https://registry-1.docker.io/v2/library/nginx/manifests/%s", tag)

	req, err := http.NewRequest("HEAD", url, nil)
	if err != nil {
		return "", false, err
	}
	req.Header.Set("Authorization", "Bearer "+token)
	req.Header.Set("Accept", "application/vnd.docker.distribution.manifest.v2+json")

	if cachedDigest != "" {
		req.Header.Set("If-None-Match", fmt.Sprintf("\"%s\"", cachedDigest))
	}

	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		return "", false, err
	}
	defer resp.Body.Close()

	if resp.StatusCode == http.StatusNotModified {
		return cachedDigest, false, nil // Not changed
	}
	if resp.StatusCode != http.StatusOK {
		return "", false, fmt.Errorf("unexpected status: %s", resp.Status)
	}

	newDigest := resp.Header.Get("Docker-Content-Digest")
	return newDigest, newDigest != cachedDigest, nil
}

Maybe it works, mayby not, the idea might be useful.

(Don't have much time right now, so I've pasted this alike "fire and forget")

[lgtm-dude]

Is there any progress?
Something I can help with?

Greetings
Christoph

[rchincha]

@lgtm-dude we are looking at this issue.

[eusebiu-constantin-petu-dbk]

I've set preserveDigest: true and had to add compat: ["docker2s2"] to the http block of the config aswell.

Hello, using the above should have worked and not trigger a pull on docker, it is indeed a bug and I'll fix it very soon.

Using If-None-Match header works, but it doesn't help us because some information from the docker image config is lost when converting to oci format, so I can not convert back to docker image to know the digest before making the request.

Caching it would work, but sync doesn't persist anything.

We can make it configurable from my point view.

[lgtm-dude]

Any other info you need to make progress?