Support Docker’s non-OCI health check in OCI images?

[akostadinov]

Issue Description

This is maybe related to #3525 and #17817

Basically when starting this image, the built-in health check appears ignored:

ghcr.io/immich-app/immich-server:v1.128.0

Steps to reproduce the issue

Steps to reproduce the issue

1. podman run ghcr.io/immich-app/immich-server:v1.128.0
2. podman inspect ...

Describe the results you received

no status of healthcheck

Describe the results you expected

healthcheck reported

podman info output

host:
  arch: amd64
  buildahVersion: 1.37.3
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-2.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 95.85
    systemPercent: 1.13
    userPercent: 3.01
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    version: "40"
  eventLogger: journald
  freeLocks: 2036
  hostname: catcentre
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 655360
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 655360
      size: 65536
  kernel: 6.10.12-200.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1526951936
  memTotal: 16448086016
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.2-2.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.17-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.17
      commit: 000fa0d4eeed8938301f3bcf8206405315bc1017
      rundir: /run/user/1002/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240906.g6b38f07-1.fc40.x86_64
    version: |
      pasta 0^20240906.g6b38f07-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1002/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 4294438912
  swapTotal: 4294963200
  uptime: 1171h 31m 7.00s (Approximately 48.79 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/containers/.config/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 8
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /media/.../containers
  graphRootAllocated: 2146418302976
  graphRootUsed: 1092298436608
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 30
  runRoot: /run/user/1002/containers
  transientStore: false
  volumePath: /media/.../containers/volumes
version:
  APIVersion: 5.2.3
  Built: 1727136000
  BuiltTime: Tue Sep 24 03:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

actually running the container via quadlet. podman inspect shows the following command:

[Unit]
Requires=immich-redis.service immich-pgsql.service
After=immich-redis.service immich-pgsql.service
Wants=immich-machine-learning.service

[Container]
ContainerName=immich_server
EnvironmentFile=/media/.../immich.env
Image=ghcr.io/immich-app/immich-server:v1.128.0
PublishPort=8222:2283
Network=immich
StopTimeout=900
GroupAdd=keep-groups
AddDevice=/dev/dri:/dev/dri
SecurityLabelLevel=s0:c555,c666
Volume=/media/...
Volume=/etc/localtime:/etc/localtime:ro

[Install]
WantedBy=multi-user.target default.target

[Service]
TimeoutStartSec=900
TimeoutStopSec=950
Restart=always

Additional information

No response

[Luap99]

The image is an OCI image and oci images do not have an healtcheck config so it is impossible to specify one there.

I don't know what immich does but if their image contains a healtcheck it must be in docker format otherwise it will be ignored as healtchecks are not part of an OCI image.

[akostadinov]

It apparently works with docker-compose.

[Luap99]

Mhh, looking at the oci spec it seems to list healtcheck as reserved in the config now, but only because docker uses it. It doesn't seem to be actually standardized, opencontainers/image-spec#749

As such podman could in theory parse that value (and builds could set it instead of ignoring it).
skopeo inspect --config --raw docker://ghcr.io/immich-app/immich-server:v1.128.0 | jq .config shows it is actually defined while skopeo inspect --config docker://ghcr.io/immich-app/immich-server:v1.128.0 | jq .config shows that c/image is not parsing that (likely because it is not actually defined in the spec)

cc @mtrmac

[mtrmac]

Docker has, in the past few years, introduced https://github.com/moby/docker-image-spec/blob/main/specs-go/v1/image.go , an “extension” of OCI which includes health checks and other fields.

(Note that DockerOCIImageMediaType is the old v2s2 value, and it is not actually referenced anywhere in the moby/moby codebase.)

This puts strict OCI consumers into an annoying position. Pragmatically, I guess there is demand for supporting that…

[boekabart]

How strange to just close this issue. At my workplace, we have the following issue:

When building an image using docker buildx build (...) --push, the healthchecks are picked up by podman. But when we use docker buildx build (...) --builder <dedicated_builder> --push, they do NOT. The dedicated builder is created without any special flags, and the reason we sometimes NEED to use it, is to have support for multiplatform images.

So given this, we're stuck with either a single-platform image built by the (outdated) built-in docker build builder , and have healthchecks working, OR we use multi-platform images built by the (up-to-date) builder, but lack healthchecks.

With the market for ARM64 dev+cloud machines having grown so large, there is a serious demand for multiplatform images; which apparently (we do nothing out of the ordinary) don't fully work with podman... I can imagine that being a reason to revert to docker at some point in time. Manually having to re-add health checks in our k8s/compose/service files is maintenance hell...

[mtrmac]

Reopening to let Podman maintainers possibly reconsider. I don’t have a strong opinion on this.

[Honny1]

There seems to be a connection:

• Podman fails to recognize HEALTHCHECK in certain BuildKit images #18904

[mtrmac]

Yes, that looks like a duplicate.