[Feat]: sync only specified architecture / operating system images

[deric]

Is your feature request related to a problem? Please describe.

Would it be possible to synchronize only a subset OS/architecture images?

Describe the solution you'd like

Support configuration of synchronized architectures / OS, e.g.:

  extensions:
    sync:
      enable: true
      registries:
        - urls:
            - https://registry.k8s.io
           architecture:
             - amd64
            os:
             - linux

Describe alternatives you've considered

It might be possible to filter out architecture based on tags, if the upstream repo uses consistent tags.

Additional context

Currently all architectures/OSes are synchronized.

REPOSITORY    TAG       OS/ARCH         DIGEST      SIGNED      SIZE        
k8s.io/pause  3.8       *               a9627f21    false       853MB       
                        linux/amd64     8a9e1be3    false       308kB       
                        linux/arm/v7    9a28d3e8    false       206kB       
                        linux/arm64     d895231c    false       265kB       
                        linux/ppc64le   4c69178e    false       357kB       
                        linux/s390x     02052591    false       254kB       
                        windows/amd64   dfbedbf6    false       270MB       
                        windows/amd64   637963cd    false       274MB       
                        windows/amd64   4b935553    false       308MB

[rchincha]

@deric, thanks for trying out zot.
One of the goals of sync feature was to make a content-preserving copy (similar to git clone).
If this is not done, then quite a few things will be broken such as signatures etc.

[andaaron]

Hi @deric,

The issue is the list of os/arch manifests themselves is an image index with its own digest, in your example a9627f21.
If we exclude some of the os/arch manifests, then the digest of the index containing them would need to change. If the digest of the index changes, any referrers of the old digest value would be broken.
For example if someone signed the entire index/manifest list, the tooling would not be able to verify the signature.

[deric]

Right, for the whole signature it makes sense. But each OS has independent set of layers and its own digest, right? It doesn't make sense to put linux layer on top of windows. In this example the linux build has 308kB (because it's dummy image) while windows build totals to 852MB. It terms of network bandwidth and storage it might make quite a difference.

[andaaron]

Each OS/Arch combination has its own manifest json which references the layers specific for that architecture.
All of these manifests are referenced by digest in the same index json file. The index has its own digest.
The index is the one tagged with k8s.io/pause:3.8 in your example.

Most client tools know what OS/Arch they are running locally when they do the download, they read the tagged index json, and and only download that manifest and layers for the layers they are interested in. That specific logic for picking one manifest over the others or the entire index and all manifests as a whole is on client side.

Zot itself is acting as a client in case of sync, but unlike the clients which are concerned with running a container locally, zot needs to accommodate any OS/Arch combinations may be needed in future, also if multiple architectures need to be downloaded it can't have multiple manifests with the same k8s.io/pause:3.8 tag. The tag needs to be on an index.
But if only certain OS/Arch combinations are downloaded on zot, then the index would be different compared to the original upstream one (and implicitly have different digests), which would break any app relying on the old index digest.

[rchincha]

Closing this issue until we revisit.

[crmky]

Is that possible to pull all the metadata, then fetch the necessary layers on demand based on user requests?

[andaaron]

We are currently tracking this in #2997.

[rchincha]

This requires supporting "sparse" image indexes.