Watchtower with cleanup doesn't free image space (run system prune)

[Christoph-AK]

Describe the bug
After running watchtower with cleanup for a while, my /var/lib/docker/overlay2 directory filled up and wasn't cleaned.

I needed to run docker system prune for the space to be freed.

To Reproduce
Run watchtower with cleanup for a while with a frequently updating service, like grafana-oss.

Expected behavior
The space of images gets correctly cleaned up after updates.

Environment
Ubuntu on AWS, Docker version 20.10.14, build a224086

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[MrYutz]

This was also an issue for us last week on the same set of specs. AWS Ubuntu with --cleanup enabled did not actually prune the disk, causing an upgrade to fail. Still digging to make sure the upgrade failed due to only disk space, but there were a bunch of previous upgrades that had not been pruned by watchtower.

[pewter77]

Seems like this was an issue before here: #662 might be related.
I'm having this issue. Filled up to 90+GB. I only update using this on a weekly basis though so I imagine this is a huge annoyance for others.

[piksel]

Could someone show the logs of a run? If you are running with --cleanup there should be a log entry for every removed image.

[pewter77]

_watchtower_logs.txt

services:

  watchtower:
    image:  containrrr/watchtower
    container_name: watchtower
    hostname: watchtower
    environment:
      - TZ=${TZ}
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_ROLLING_RESTART=true
      #- WATCHTOWER_DEBUG=true
      - WATCHTOWER_SCHEDULE=* 2 6 * * 5
      #- WATCHTOWER_RUN_ONCE=true
      ## Watchtower autodetects discord urls and uses shoutrr##
      - WATCHTOWER_NOTIFICATIONS=slack 
      - WATCHTOWER_NOTIFICATION_SLACK_HOOK_URL=${WATCHTOWER_WEBHOOK_URL}/slack
      - WATCHTOWER_NOTIFICATION_SLACK_IDENTIFIER=*removed*
      - WATCHTOWER_NOTIFICATION_SLACK_ICON_EMOJI=":whale:"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: always

Not sure if this helps, but here's my logs and my compose file. The images do get uploaded, the containers do restart. But old images aren't removed.

formatted watchtower log

2022-10-09T10:53:04Z [I] Watchtower 1.4.0
2022-10-09T10:53:04Z [I] Using notifications: discord
2022-10-09T10:53:04Z [I] Checking all containers (except explicitly disabled with label)
2022-10-09T10:53:04Z [I] Scheduling first run: 2022-10-14 06:02:00 +0000 UTC
2022-10-09T10:53:04Z [I] Note that the first check will be performed in 115 hours, 8 minutes, 55 seconds
2022-10-14T06:02:14Z [I] Found new boredazfcuk/icloudpd:latest image (3197a7ceac91)
2022-10-14T06:04:29Z [I] Found new linuxserver/calibre:latest image (cdf8e27a4d35)
2022-10-14T06:05:01Z [I] Found new linuxserver/bookstack:latest image (763857427ff7)
2022-10-14T06:05:47Z [I] Found new linuxserver/bazarr:latest image (be7a0f0d0360)
2022-10-14T06:07:09Z [I] Found new linuxserver/jellyfin:latest image (b7cdd9a3e0be)
2022-10-14T06:08:02Z [I] Found new linuxserver/calibre-web:latest image (df01e6a5e09f)
2022-10-14T06:09:00Z [I] Found new linuxserver/mariadb:latest image (5dd6d3328713)
2022-10-14T06:09:24Z [I] Found new linuxserver/sonarr:latest image (7ea0c1d905c3)
2022-10-14T06:10:12Z [I] Found new fallenbagel/jellyseerr:latest image (6d1c33b5983a)
2022-10-14T06:10:41Z [I] Found new fbonalair/traefik-crowdsec-bouncer:latest image (3e297f1f2afa)
2022-10-14T06:11:05Z [I] Found new apache/tika:latest image (14d779cbcee5)
2022-10-14T06:11:27Z [I] Stopping /tika (3562a2585f43) with SIGTERM
2022-10-14T06:11:28Z [I] Creating /tika
2022-10-14T06:11:29Z [I] Stopping /crowdsec-traefik-bouncer (d940031c49fd) with SIGTERM
2022-10-14T06:11:29Z [E] Error response from daemon: Cannot kill container: d940031c49fd71adc6a66358b9431b4eb0cdcca06aea496cf3a4a43f0a42bc21: No such container: d940031c49fd71adc6a66358b9431b4eb0cdcca06aea496cf3a4a43f0a42bc21
2022-10-14T06:11:29Z [I] Stopping /jellyseerr (3efb49189a90) with SIGTERM
2022-10-14T06:11:29Z [E] Error response from daemon: Cannot kill container: 3efb49189a905712c6ca5c1f894275065523b95ae944f2f349ddf3a0ab11a7d6: No such container: 3efb49189a905712c6ca5c1f894275065523b95ae944f2f349ddf3a0ab11a7d6
2022-10-14T06:11:29Z [I] Stopping /sonarr (bcb3ee1ed32a) with SIGTERM
2022-10-14T06:11:29Z [E] Error response from daemon: Cannot kill container: bcb3ee1ed32abaa8ed66ce8ee1cbcf4ca59681127ab0462eb22c63508accf562: No such container: bcb3ee1ed32abaa8ed66ce8ee1cbcf4ca59681127ab0462eb22c63508accf562
2022-10-14T06:11:29Z [I] Stopping /bookstack-db (0b26c538e75e) with SIGTERM
2022-10-14T06:11:29Z [E] Error response from daemon: Cannot kill container: 0b26c538e75eaaf101c36bd6ca92f46b142819232585e1111c4184dbb9dce818: No such container: 0b26c538e75eaaf101c36bd6ca92f46b142819232585e1111c4184dbb9dce818
2022-10-14T06:11:29Z [I] Stopping /calibre-web (b856389be4d9) with SIGTERM
2022-10-14T06:11:29Z [E] Error response from daemon: Cannot kill container: b856389be4d9a63ba4ebed73d4604fa28b43f0fcdc78fa9d7a3fbbe20fd93eaa: No such container: b856389be4d9a63ba4ebed73d4604fa28b43f0fcdc78fa9d7a3fbbe20fd93eaa
2022-10-14T06:11:29Z [I] Stopping /jellyfin (a075d136bbd6) with SIGTERM
2022-10-14T06:11:29Z [E] Error response from daemon: Cannot kill container: a075d136bbd6c6cb0fb4acd70b50240bddd00d5f2dcd8e3dd0166bd25a5ee3f3: No such container: a075d136bbd6c6cb0fb4acd70b50240bddd00d5f2dcd8e3dd0166bd25a5ee3f3
2022-10-14T06:11:29Z [I] Stopping /bazarr (fd4693ef4d3a) with SIGTERM
2022-10-14T06:11:29Z [E] Error response from daemon: Cannot kill container: fd4693ef4d3a1c2e04482019044033022752ba5602ee6e8af75f38168da7e354: No such container: fd4693ef4d3a1c2e04482019044033022752ba5602ee6e8af75f38168da7e354
2022-10-14T06:11:29Z [I] Stopping /bookstack (b8e2e9862017) with SIGTERM
2022-10-14T06:11:29Z [E] Error response from daemon: Cannot kill container: b8e2e9862017c270b522a35a14e52dff52c7ba7a895fde5379edfc269b214341: No such container: b8e2e9862017c270b522a35a14e52dff52c7ba7a895fde5379edfc269b214341
2022-10-14T06:11:29Z [I] Stopping /calibre (63634c02ab16) with SIGTERM
2022-10-14T06:11:29Z [E] Error response from daemon: Cannot kill container: 63634c02ab16c826df1982b0a44c1f95527f8b529055bde6474ac2bec6eec800: No such container: 63634c02ab16c826df1982b0a44c1f95527f8b529055bde6474ac2bec6eec800
2022-10-14T06:11:29Z [I] Stopping /icloudpd (267a1e59f39b) with SIGTERM
2022-10-14T06:11:29Z [E] Error response from daemon: Cannot kill container: 267a1e59f39b911e5b2e10b7d8454b9b6e4fa9bbade81a41740ba6a94ae4fe6e: No such container: 267a1e59f39b911e5b2e10b7d8454b9b6e4fa9bbade81a41740ba6a94ae4fe6e
2022-10-14T06:11:29Z [I] Removing image 3a8c80c1ea4c
2022-10-14T06:11:29Z [I] Session done
                         Failed: 10
                         Scanned: 36
                         Updated: 1
                         notify: no
2022-10-17T15:18:19Z [I] Waiting for running update to be finished...
2022-10-17T15:19:05Z [I] Watchtower 1.4.0
2022-10-17T15:19:05Z [I] Using notifications: discord
2022-10-17T15:19:05Z [I] Checking all containers (except explicitly disabled with label)
2022-10-17T15:19:05Z [I] Scheduling first run: 2022-10-21 06:02:00 +0000 UTC
2022-10-17T15:19:05Z [I] Note that the first check will be performed in 86 hours, 42 minutes, 54 seconds
2022-10-21T06:02:27Z [I] Found new fallenbagel/jellyseerr:latest image (35c487516c47)
2022-10-21T06:02:46Z [I] Found new linuxserver/bookstack:latest image (4512f4b60a63)
2022-10-21T06:03:14Z [I] Found new linuxserver/calibre-web:latest image (c62f06aace35)
2022-10-21T06:03:30Z [I] Found new linuxserver/mariadb:latest image (a92ed9a9db3a)
2022-10-21T06:05:00Z [I] Found new linuxserver/jellyfin:latest image (d099b467fa7b)
2022-10-21T06:05:27Z [I] Found new linuxserver/bazarr:latest image (84ff69510384)
2022-10-21T06:05:58Z [I] Found new linuxserver/calibre:latest image (2e477cb2b8a0)
2022-10-21T06:06:23Z [I] Found new linuxserver/nzbget:latest image (5d93dda0858d)
2022-10-21T06:06:40Z [I] Found new linuxserver/lidarr:nightly image (086f3ce3f7ce)
2022-10-21T06:06:59Z [I] Found new hotio/qflood:latest image (cccfd7d953e7)
2022-10-21T06:07:15Z [I] Found new linuxserver/radarr:develop image (d94d80e22318)
2022-10-21T06:07:58Z [I] Found new linuxserver/prowlarr:develop image (ee0371f4fbed)
2022-10-21T06:08:30Z [I] Found new cloudflare/cloudflared:latest image (21d29910565b)
2022-10-21T06:08:53Z [I] Found new deluan/navidrome:latest image (2f581c2753cc)
2022-10-21T06:09:28Z [I] Found new gitea/gitea:latest image (8376639274f8)
2022-10-21T06:10:37Z [I] Found new thecodingmachine/gotenberg:latest image (e7a3b8014c7f)
2022-10-21T06:10:37Z [I] Stopping /gotenberg (fc251c52c2bb) with SIGTERM
2022-10-21T06:10:38Z [I] Creating /gotenberg
2022-10-21T06:10:38Z [I] Stopping /gitea (724da654169b) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: 724da654169b555d814e234b5374df5bed2277cf492e34cef7c4eca7ed7c40a0: No such container: 724da654169b555d814e234b5374df5bed2277cf492e34cef7c4eca7ed7c40a0
2022-10-21T06:10:38Z [I] Stopping /navidrome (5bcf1563e960) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: 5bcf1563e960564fcc6a1d961a9603b701a11e645136898d0cf1a1e94bf31aab: No such container: 5bcf1563e960564fcc6a1d961a9603b701a11e645136898d0cf1a1e94bf31aab
2022-10-21T06:10:38Z [I] Stopping /cloudflared (9993f76ab301) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: 9993f76ab301bd2fbe47f9671cfdb40a72975073c550ce70e59e931536ae921c: No such container: 9993f76ab301bd2fbe47f9671cfdb40a72975073c550ce70e59e931536ae921c
2022-10-21T06:10:38Z [I] Stopping /prowlarr (e4ae2448a13d) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: e4ae2448a13d84ad7c909a08183f155b3de93e963213ab42dc2ddd8b825e9d9b: No such container: e4ae2448a13d84ad7c909a08183f155b3de93e963213ab42dc2ddd8b825e9d9b
2022-10-21T06:10:38Z [I] Stopping /radarr (a509a935a0d6) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: a509a935a0d6296b5c046ec50a41fcf3b7506771c048736ff7d4c3a55d5140a8: No such container: a509a935a0d6296b5c046ec50a41fcf3b7506771c048736ff7d4c3a55d5140a8
2022-10-21T06:10:38Z [I] Stopping /qflood (629b4432c8bd) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: 629b4432c8bd73650c4a40625913d0924ca839f5dd59eff5186addad9d8eeb5f: No such container: 629b4432c8bd73650c4a40625913d0924ca839f5dd59eff5186addad9d8eeb5f
2022-10-21T06:10:38Z [I] Stopping /lidarr (d2653533b194) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: d2653533b194d5202de05833dc8d420ebb7c9df83d15ba7c2bddb6d92c4d2c74: No such container: d2653533b194d5202de05833dc8d420ebb7c9df83d15ba7c2bddb6d92c4d2c74
2022-10-21T06:10:38Z [I] Stopping /nzbget (ad0f82b6e3f5) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: ad0f82b6e3f5ea5a3050f7d1120f551378469e8fea9be456b94c5d68466902cd: No such container: ad0f82b6e3f5ea5a3050f7d1120f551378469e8fea9be456b94c5d68466902cd
2022-10-21T06:10:38Z [I] Stopping /calibre (96dd2af84908) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: 96dd2af84908e23449ad3403e467e740b0e3a1d9f37e84c841c6ed83734443f5: No such container: 96dd2af84908e23449ad3403e467e740b0e3a1d9f37e84c841c6ed83734443f5
2022-10-21T06:10:38Z [I] Stopping /bazarr (c8075fa6e508) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: c8075fa6e5080bb8dca1d8be033030a81f8725057caf74d8e00dc0fba27002d4: No such container: c8075fa6e5080bb8dca1d8be033030a81f8725057caf74d8e00dc0fba27002d4
2022-10-21T06:10:38Z [I] Stopping /jellyfin (abe814a24227) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: abe814a242278db43a46007aab6c5f8a8fe03362ea747ac0ff7a3b07aef8eff1: No such container: abe814a242278db43a46007aab6c5f8a8fe03362ea747ac0ff7a3b07aef8eff1
2022-10-21T06:10:38Z [I] Stopping /bookstack-db (bc0b5e6322fa) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: bc0b5e6322fa8b84398f04162b25b36c560e1110fbd2be05c095748d98d63a13: No such container: bc0b5e6322fa8b84398f04162b25b36c560e1110fbd2be05c095748d98d63a13
2022-10-21T06:10:38Z [I] Stopping /calibre-web (861caf49c61c) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: 861caf49c61c0aa635f00948fa858d258cf424711337f88dc01816936b50ccc3: No such container: 861caf49c61c0aa635f00948fa858d258cf424711337f88dc01816936b50ccc3
2022-10-21T06:10:38Z [I] Stopping /bookstack (ae4db2c0eb5c) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: ae4db2c0eb5c1fc143c12b7734f00ef4b924971c38c7428139b4121750cb8389: No such container: ae4db2c0eb5c1fc143c12b7734f00ef4b924971c38c7428139b4121750cb8389
2022-10-21T06:10:38Z [I] Stopping /jellyseerr (80893d2dac2a) with SIGTERM
2022-10-21T06:10:38Z [E] Error response from daemon: Cannot kill container: 80893d2dac2a6aef33b806860130b95ce9de6eaa3e7abc95577906ef8ecaf0a1: No such container: 80893d2dac2a6aef33b806860130b95ce9de6eaa3e7abc95577906ef8ecaf0a1
2022-10-21T06:10:38Z [I] Removing image ba3c7d7ad873
2022-10-21T06:10:39Z [I] Session done
                         Failed: 15
                         Scanned: 36
                         Updated: 1
                         notify: no
2022-10-21T06:10:49Z [E] Failed to send shoutrrr notification
                         error: failed to send using discord: timed out
                         index: 0
                         notify: no
                         service: discord
2022-10-28T09:52:09Z [I] Watchtower 1.4.0
2022-10-28T09:52:09Z [I] Using notifications: discord
2022-10-28T09:52:09Z [I] Checking all containers (except explicitly disabled with label)
2022-10-28T09:52:09Z [I] Scheduling first run: 2022-11-04 06:02:00 +0000 UTC
2022-10-28T09:52:09Z [I] Note that the first check will be performed in 164 hours, 9 minutes, 50 seconds
2022-10-28T12:40:42Z [I] Waiting for running update to be finished...
2022-10-28T12:41:23Z [I] Watchtower 1.4.0
2022-10-28T12:41:23Z [I] Using notifications: discord
2022-10-28T12:41:23Z [I] Checking all containers (except explicitly disabled with label)
2022-10-28T12:41:23Z [I] Scheduling first run: 2022-11-04 06:02:00 +0000 UTC
2022-10-28T12:41:23Z [I] Note that the first check will be performed in 161 hours, 20 minutes, 36 seconds
2022-10-28T12:45:22Z [I] Waiting for running update to be finished...
2022-10-28T12:46:03Z [I] Watchtower 1.4.0
2022-10-28T12:46:03Z [I] Using notifications: discord
2022-10-28T12:46:03Z [I] Checking all containers (except explicitly disabled with label)
2022-10-28T12:46:03Z [I] Scheduling first run: 2022-11-04 06:02:00 +0000 UTC
2022-10-28T12:46:03Z [I] Note that the first check will be performed in 161 hours, 15 minutes, 56 seconds
2022-11-04T06:03:03Z [I] Found new linuxserver/emby:latest image (acd160e3bdc5)
2022-11-04T06:04:34Z [I] Found new thecodingmachine/gotenberg:latest image (32d50612a4d9)
2022-11-04T06:04:54Z [I] Found new gitea/gitea:latest image (54e1c09fe090)
2022-11-04T06:05:07Z [I] Found new cloudflare/cloudflared:latest image (0ea591232359)
2022-11-04T06:05:28Z [I] Found new linuxserver/prowlarr:develop image (b0fe2a3cfbf2)
2022-11-04T06:05:45Z [I] Found new linuxserver/radarr:develop image (51cd50a0d4a4)
2022-11-04T06:06:08Z [I] Found new linuxserver/lidarr:nightly image (788ce3f924e6)
2022-11-04T06:06:25Z [I] Found new linuxserver/nzbget:latest image (cab6ac2862fb)
2022-11-04T06:06:40Z [I] Found new hotio/qflood:latest image (8ea92066f024)
2022-11-04T06:07:51Z [I] Found new linuxserver/calibre:latest image (20e655056867)
2022-11-04T06:08:17Z [I] Found new linuxserver/bazarr:latest image (0f1d18c244f6)
2022-11-04T06:09:13Z [I] Found new linuxserver/jellyfin:latest image (15b37dfad42a)
2022-11-04T06:09:31Z [I] Found new linuxserver/bookstack:latest image (5e8a21c8a6e3)
2022-11-04T06:09:59Z [I] Found new linuxserver/calibre-web:latest image (14b7fa082068)
2022-11-04T06:10:39Z [I] Found new apache/tika:latest image (c0dc805a621e)
2022-11-04T06:11:31Z [I] Found new linuxserver/sonarr:latest image (6763490b50cb)
2022-11-04T06:11:54Z [I] Found new boredazfcuk/icloudpd:latest image (65908edfadef)
2022-11-04T06:12:10Z [I] Found new linuxserver/readarr:develop image (55f9e7b66f43)
2022-11-04T06:12:53Z [I] Found new linuxserver/code-server:latest image (599871d7c050)
2022-11-04T06:13:17Z [I] Found new traefik:latest image (288889429bec)
2022-11-04T06:13:43Z [I] Found new authelia/authelia:latest image (699801b4deb3)
2022-11-04T06:14:04Z [I] Found new portainer/portainer-ee:latest image (0bfd1d8b18bc)
2022-11-04T06:14:07Z [I] Stopping /portainer (bb75cb00ed90) with SIGTERM
2022-11-04T06:14:07Z [I] Creating /portainer
2022-11-04T06:14:07Z [I] Stopping /authelia (2ecaf46291e3) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 2ecaf46291e36e3692f2bff03373b77c911447ec43962a18d076bff5269f38b2: No such container: 2ecaf46291e36e3692f2bff03373b77c911447ec43962a18d076bff5269f38b2
2022-11-04T06:14:07Z [I] Stopping /traefik (77a44cd51d5f) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 77a44cd51d5f35b96371aa7465ff67943040aad3643201837e0457586c873e1a: No such container: 77a44cd51d5f35b96371aa7465ff67943040aad3643201837e0457586c873e1a
2022-11-04T06:14:07Z [I] Stopping /code-server (6d882f1e850b) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 6d882f1e850b1942dfbc4884361ee3ad9d1473afdadb7df19d27e0eb0246fd0a: No such container: 6d882f1e850b1942dfbc4884361ee3ad9d1473afdadb7df19d27e0eb0246fd0a
2022-11-04T06:14:07Z [I] Stopping /readarr (f78e43778484) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: f78e43778484799a2a95a9ba1aadbe9dcd3291113b2b8f52d80a4866f12f2409: No such container: f78e43778484799a2a95a9ba1aadbe9dcd3291113b2b8f52d80a4866f12f2409
2022-11-04T06:14:07Z [I] Stopping /icloudpd (631b36cbe5bb) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 631b36cbe5bb180d4d4cb2d52c1e2ba6e1ce8e0e986c12ebc86527c32743fde0: No such container: 631b36cbe5bb180d4d4cb2d52c1e2ba6e1ce8e0e986c12ebc86527c32743fde0
2022-11-04T06:14:07Z [I] Stopping /sonarr (4af0cac7918f) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 4af0cac7918fa0a1c92bb95723293495f233bae925376bd7baf80b99fb4ff95f: No such container: 4af0cac7918fa0a1c92bb95723293495f233bae925376bd7baf80b99fb4ff95f
2022-11-04T06:14:07Z [I] Stopping /tika (25c826909194) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 25c826909194f37dd1fe089ef8e122cb2bc1903a2dc20947694c02124184ffa5: No such container: 25c826909194f37dd1fe089ef8e122cb2bc1903a2dc20947694c02124184ffa5
2022-11-04T06:14:07Z [I] Stopping /calibre-web (4aac6391d22a) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 4aac6391d22acaf517c3982e8a76dac9b10d6e2a50641dccc57260d951dc0a35: No such container: 4aac6391d22acaf517c3982e8a76dac9b10d6e2a50641dccc57260d951dc0a35
2022-11-04T06:14:07Z [I] Stopping /bookstack (8feeac227e58) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 8feeac227e5810418d4cdd9543a55814c99ee1dac8c5263d4c674aa2279e081f: No such container: 8feeac227e5810418d4cdd9543a55814c99ee1dac8c5263d4c674aa2279e081f
2022-11-04T06:14:07Z [I] Stopping /jellyfin (cd755850590a) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: cd755850590a039b14ebdf971ac46996d9ab3520779d3a2e1c16e30b6f949e0d: No such container: cd755850590a039b14ebdf971ac46996d9ab3520779d3a2e1c16e30b6f949e0d
2022-11-04T06:14:07Z [I] Stopping /bazarr (b603b3bb6963) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: b603b3bb6963b8e23a5a455ac6658ff824f07ec462c933984d8132ba1bd5e5e3: No such container: b603b3bb6963b8e23a5a455ac6658ff824f07ec462c933984d8132ba1bd5e5e3
2022-11-04T06:14:07Z [I] Stopping /calibre (0ff8f22aec98) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 0ff8f22aec9869cfb297d3cd4a2108810d4625412acb39a885a3d35f54ee1ffd: No such container: 0ff8f22aec9869cfb297d3cd4a2108810d4625412acb39a885a3d35f54ee1ffd
2022-11-04T06:14:07Z [I] Stopping /qflood (2d1dff624d15) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 2d1dff624d15bffc52c28d2ce373e3ceb986626ff8259823d36da76244e02d5a: No such container: 2d1dff624d15bffc52c28d2ce373e3ceb986626ff8259823d36da76244e02d5a
2022-11-04T06:14:07Z [I] Stopping /nzbget (8e2fb1e0ad23) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 8e2fb1e0ad234ad174686dbf79e2869b78dc8d6215c2ac8d420b38f4f1948dbf: No such container: 8e2fb1e0ad234ad174686dbf79e2869b78dc8d6215c2ac8d420b38f4f1948dbf
2022-11-04T06:14:07Z [I] Stopping /lidarr (d296a2923dd1) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: d296a2923dd170f9417d4816f00678cc0770792b00c3c876fb9a398a4b1b8cc8: No such container: d296a2923dd170f9417d4816f00678cc0770792b00c3c876fb9a398a4b1b8cc8
2022-11-04T06:14:07Z [I] Stopping /radarr (83a64b4ab990) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 83a64b4ab990aaa4112cfb6508f81828464a7dff062dd51f2b0a760ed90e5214: No such container: 83a64b4ab990aaa4112cfb6508f81828464a7dff062dd51f2b0a760ed90e5214
2022-11-04T06:14:07Z [I] Stopping /prowlarr (93c362479cc6) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 93c362479cc650cc4b274ebd109f8c5a7ca6b049944cb4066d22e6f9bf0ca610: No such container: 93c362479cc650cc4b274ebd109f8c5a7ca6b049944cb4066d22e6f9bf0ca610
2022-11-04T06:14:07Z [I] Stopping /cloudflared (a3239bec497d) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: a3239bec497d480d64a3314dccbc8a786a6e43c75cce934156ebce79a3d3e1ba: No such container: a3239bec497d480d64a3314dccbc8a786a6e43c75cce934156ebce79a3d3e1ba
2022-11-04T06:14:07Z [I] Stopping /gitea (291897fae836) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 291897fae836df4a5dc81a29f83ee852a43ac86a5029a26938d9c34284c29705: No such container: 291897fae836df4a5dc81a29f83ee852a43ac86a5029a26938d9c34284c29705
2022-11-04T06:14:07Z [I] Stopping /gotenberg (26904818792b) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 26904818792bddeeaec3e3c42aecf13174556ca4d7a94cc72bac4bdfdb31adf2: No such container: 26904818792bddeeaec3e3c42aecf13174556ca4d7a94cc72bac4bdfdb31adf2
2022-11-04T06:14:07Z [I] Stopping /emby (506cac5fe01d) with SIGTERM
2022-11-04T06:14:07Z [E] Error response from daemon: Cannot kill container: 506cac5fe01d15db0f4b7d5edd6f11c77a472922f783acf0030cc9e5f44f95c8: No such container: 506cac5fe01d15db0f4b7d5edd6f11c77a472922f783acf0030cc9e5f44f95c8
2022-11-04T06:14:07Z [I] Removing image 1528f5b692fe
2022-11-04T06:14:08Z [I] Session done
                         Failed: 21
                         Scanned: 36
                         Updated: 1
                         notify: no

[Christoph-AK]

So far I only got the behavior on one production system. As it is important that that one runs well, I included the gc container into my compose.

  gc:
    image: drone/gc
    restart: unless-stopped
    container_name: gc
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - GC_INTERVAL=1h

With that all images get cleaned as expected.

Unfortunately with that I can't really provide any useful logs on this.

The continuos log of watchtower: https://gist.github.com/Christoph-AK/052523cf83c093a3cba2dca63684c7b3

Thanks for looking into this!

[piksel]

@pewter77 so, the containers tika, gotenberg and portainer were updated, but their previous images (3a8c80c1ea4c, ba3c7d7ad873 and 1528f5b692fe respectively) were not removed?
The fact that there are 10-21 containers that seem to just disappear during the check also seems concerning...

Update: Perhaps the reason for your storage filling up is because of all those failed updates? For every check it would pull the new image, but since it cannot update the container it won't remove the old one.

[piksel]

@Christoph-AK Could you check what images are present in docker images and compare that to the images removed in the log?
Watchtower doesn't run system prune, it just tries to remove the old image when a new one was updated to.

Also, I added some more logging to #1466 so that all images that are removed (including parent images) are actually logged (using --debug).

[pewter77]

_watchtower_logs (1).txt
So watchtower updated to 1.5.1 and I got more logs, it seems there's some weird logic where it shuts down some containers, shows that container doesn't exist anymore and then doesn't finish deleting it? Again all my containers are up and running, but many different images are still there unused.

I should mention that I do have connectivity issues because of my location, I have to use obfuscating proxies and such to pass over country wide internet firewalls. Though they are fairly consistent, sometimes they timeout. But like I said these containers seem to be updated, just the leftover images are still here so I'm not sure this is the issue.