Loading saved image results in: Digest of source image's manifest would not match destination reference

[matejonnet]

Issue Description

Cannot load local image saved using podman save when the image pulled and saved is referenced by sha256 and not manually tagged after the pull.

Steps to reproduce the issue

$ podman images
REPOSITORY  TAG         IMAGE ID    CREATED     SIZE

$ podman pull --arch amd64 registry.access.redhat.com/ubi9/ubi@sha256:f3b9841ea7615b38fd271cf18ae235d8228d3369036eb175ded171c936808255
Trying to pull registry.access.redhat.com/ubi9/ubi@sha256:f3b9841ea7615b38fd271cf18ae235d8228d3369036eb175ded171c936808255...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 35d6a5cce6a1 done   | 
Copying config 4461e0f894 done   | 
Writing manifest to image destination
Storing signatures
4461e0f8948d70db7614912a1d3d4b0b180d4a3044a08058bec55dbbb941b1f3

$ podman images
REPOSITORY                           TAG         IMAGE ID      CREATED     SIZE
registry.access.redhat.com/ubi9/ubi  <none>      4461e0f8948d  7 days ago  217 MB

$ podman save --format=oci-archive -o saved-image.tar ubi9/ubi@sha256:f3b9841ea7615b38fd271cf18ae235d8228d3369036eb175ded171c936808255
Copying blob 0f4901ccb757 done   | 
Copying config 4461e0f894 done   | 
Writing manifest to image destination

$ podman pull oci-archive:saved-image.tar
Error: unable to perform copy: unable to copy from source oci-archive:saved-image.tar:: Digest of source image's manifest would not match destination reference

However it works, if I save the image using a tag.

$ podman tag 4461e0f8948d registry.access.redhat.com/ubi9/ubi:my-tag

$ podman save --format=oci-archive -o saved-image-using-tag.tar ubi9/ubi:my-tag
Copying blob 0f4901ccb757 done   | 
Copying config 4461e0f894 done   | 
Writing manifest to image destination

$ podman pull oci-archive:saved-image-using-tag.tar
Getting image source signatures
Copying blob 35d6a5cce6a1 skipped: already exists  
Copying config 4461e0f894 done   | 
Writing manifest to image destination
4461e0f8948d70db7614912a1d3d4b0b180d4a3044a08058bec55dbbb941b1f3


### Describe the results you received

Unable to pull image from local archive

Error: unable to perform copy: unable to copy from source oci-archive:saved-image.tar:: Digest of source image's manifest would not match destination reference

### Describe the results you expected

Being able to use `podman pull oci-archive:saved-image.tar`

### podman info output

```yaml
$ podman version
Client:        Podman Engine
Version:       5.6.2
API Version:   5.6.2
Go Version:    go1.24.7
Git Commit:    9dd5e1ed33830612bc200d7a13db00af6ab865a4
Built:         Tue Sep 30 02:00:00 2025
Build Origin:  Fedora Project
OS/Arch:       linux/amd64

$ rpm -q podman
podman-5.6.2-1.fc42.x86_64

$ podman info
host:
  arch: amd64
  buildahVersion: 1.41.5
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 98.41
    systemPercent: 0.34
    userPercent: 1.25
  cpus: 22
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "42"
  eventLogger: journald
  freeLocks: 2048
  hostname: mana
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.16.10-200.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 4527820800
  memTotal: 33043333120
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.1-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: crun
    package: crun-1.24-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.24
      commit: 54693209039e5e04cbe3c8b1cd5fe2301219f0a1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250919.g623dbf6-1.fc42.x86_64
    version: |
      pasta 0^20250919.g623dbf6-1.fc42.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 1h 32m 6.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/mlazar/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/mlazar/.local/share/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 51282993152
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/mlazar/.local/share/containers/storage/volumes
version:
  APIVersion: 5.6.2
  BuildOrigin: Fedora Project
  Built: 1759190400
  BuiltTime: Tue Sep 30 02:00:00 2025
  GitCommit: 9dd5e1ed33830612bc200d7a13db00af6ab865a4
  GoVersion: go1.24.7
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.2

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[mtrmac]

At a higher level, I recommend against using tar-like image archives. Due to the linear structure, no cross-image reuse (making full copies of all base layers, over and over again), no parallelism, and format restrictions, they are inherently inefficient. Generally I recommend using registries, and registry-to-registry copies.

(Even to the point of setting up a “transfer registry” in a container, copying to that registry, then moving the backing store of that registry across to a different site, again setting up “the same” registry in a container, and copying from the “transfer registry” to the ultimate destination.)

Or use a dedicated mirroring tool, like maybe oc-mirror.

---

At a fundamental level, what is happening here is that podman pull decompressed the on-registry image and discarded the compressed version. The compressed version can’t be reliably recreated from the decompressed version, so podman save can not reliably create a manifest matching the original digest. (It’s completely impossible in the case of this image, which does not use an OCI format, and is forcibly converted to OCI via podman save --format=oci-archive — that conversion changes the digest — but it would still be impossible to do reliably if the input were OCI-formatted, in general.)

In oci-archive, there is no concept of a “digest reference” for image names, so creating such archives, where the @sha256: part does not match the contents, is valid for oci-archive:. But, on pull, we try to store the image into local storage based on the name in the archive, and in local storage we do have a concept of a “digest reference”, and we do expect the manifest to match. And, because the compressed version is discarded, that is ~never going to be the case.

---

Any workflow that starts with podman pull is going to change digests.

If you must use a file-based format (and not a registry-to-registry copy), a direct copy from the registry would be a minimal starting point: skopeo copy docker://… oci-archive:… . But even that would not work here, where that would trigger a schema2→OCI conversion. You could use a dir: format to represent an image, unmodified, as it exists on the registry.

But, again, my primary recommendation is to use a registry-to-registry copy, or a dedicated mirroring tool.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[praiskup]

Thank you for the info, @mtrmac.

The use case is to use an existing image from the registry for the Mock (RPM build tool) bootstrapping process. In short, we need to deliver the native version of dnf from the target distribution and use that version of dnf to install the RPM build environment.

We want to make all of this work in a hermetic environment. In short:

1. We note the image registry URL at the start (one that contains the correct version of dnf).
2. We prefetch the image using a trusted tool within the Konflux environment (to be done).
3. We extract the filesystem tree (including the /bin/dnf binary) from that image.
4. We need to make /bin/dnf accessible for Mock.

We run Mock in a hermetic environment (podman --network=none). Step 4 needs to be done inside that environment. Since it was the most convenient approach previously, we also perform step 3 inside: we bind-mount the OCI image as a tarball into the container.

This is somewhat an aesthetic issue anyway because we want to implement a checksum check to ensure that the reimported image (from the tarball) in the hermetic environment matches the original checksum.

But, again, my primary recommendation is to use a registry-to-registry copy, or a dedicated mirroring tool.

I think we can consider using dir: because we don’t want to create and maintain a dedicated mirroring tool. Are there any other mirroring tools we could look into?

[mtrmac]

Of skopeo sync and oc-mirror, oc-mirror has a much larger dedicated set of maintainers. There may be many other relevant tools, I don’t follow this space much.

---

On the Podman side, I think it might make sense for podman pull and podman load to have a --local-tag option, letting the user choose the local name independently from the registry location or the name in an archive.