Watchtower gets a 404 when trying to update

[wrench12435678]

"Could not do a head request for "xavierh/goaccess-for-nginxproxymanager:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present""

Recently change detection had the same issue with buildx, docker/setup-buildx-action#187

It seems to be related to "provenance: false"

Is this the same issue for go access, or a different reason watchtower is failing to update?

Thanks

[xavier-hernandez]

Thanks for opening this issue and looking into it.

This does appear to be an issue with watchtower. I did recently update the actions on GitHub because there are phasing out the old actions in May/June which is why we're probably getting this error now.

Looks like they have pull request out there to fix the problem, containrrr/watchtower#1526 . I'll monitor it for a few days to see what progress they've made on approving it.

For now... docker pull xavierh/goaccess-for-nginxproxymanager:latest will work for you.

[Pwuts]

@wrench12435678 can you post a Watchtower debug log snippet, from Trying to load authentication credentials or Got image name to Pulling image? Pulling images from Docker Hub should currently work afaik.

[martadinata666]

watchtower  | time="2023-01-29T00:09:45+07:00" level=debug msg="Trying to load authentication credentials." container=/neededimg-moby-buildkit-1 image="moby/buildkit:buildx-stable-1"
watchtower  | time="2023-01-29T00:09:45+07:00" level=debug msg="No credentials for moby found" config_file=/config.json
watchtower  | time="2023-01-29T00:09:45+07:00" level=debug msg="Got image name: moby/buildkit:buildx-stable-1"
watchtower  | time="2023-01-29T00:09:45+07:00" level=debug msg="Checking if pull is needed" container=/neededimg-moby-buildkit-1 image="moby/buildkit:buildx-stable-1"
watchtower  | time="2023-01-29T00:09:45+07:00" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
watchtower  | time="2023-01-29T00:09:46+07:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
watchtower  | time="2023-01-29T00:09:46+07:00" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
watchtower  | time="2023-01-29T00:09:46+07:00" level=debug msg="Setting scope for auth token" image=moby/buildkit scope="repository:moby/buildkit:pull"
watchtower  | time="2023-01-29T00:09:46+07:00" level=debug msg="No credentials found."
watchtower  | time="2023-01-29T00:09:47+07:00" level=debug msg="Parsing image ref" host=index.docker.io image=moby/buildkit normalized="docker.io/moby/buildkit:buildx-stable-1" tag=buildx-stable-1
watchtower  | time="2023-01-29T00:09:47+07:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/moby/buildkit/manifests/buildx-stable-1"
watchtower  | time="2023-01-29T00:09:49+07:00" level=warning msg="Could not do a head request for \"moby/buildkit:buildx-stable-1\", falling back to regular pull." container=/neededimg-moby-buildkit-1 image="moby/buildkit:buildx-stable-1"
watchtower  | time="2023-01-29T00:09:49+07:00" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/neededimg-moby-buildkit-1 image="moby/buildkit:buildx-stable-1"
watchtower  | time="2023-01-29T00:09:49+07:00" level=debug msg="Pulling image" container=/neededimg-moby-buildkit-1 image="moby/buildkit:buildx-stable-1"
watchtower  | time="2023-01-29T00:09:53+07:00" level=debug msg="No new images found for /neededimg-moby-buildkit-1"

not goaccess image, but technically every image that build using buildx v0.10.0+ will result 404 with watchtower. I saw a PR supporting new OCI manifest containrrr/watchtower#1533 , but cant really tell if it working, as no watchtower development container image to test it out.

Ref:

• docker manifest does not work with output from buildx v0.10 docker/buildx#1509 (comment)

[Pwuts]

cant really tell if it working, as no watchtower development container image to test it out

Try containrrr/watchtower:latest-dev :)

[xavier-hernandez]

That appears to have worked. :)

I set the check interval to 10 seconds temporarily.

BEFORE

# docker container logs watchtower
time="2023-01-28T20:00:10-05:00" level=info msg="Watchtower 1.5.1"
time="2023-01-28T20:00:10-05:00" level=info msg="Using notifications: pushover"
time="2023-01-28T20:00:10-05:00" level=info msg="Checking all containers (except explicitly disabled with label)"
time="2023-01-28T20:00:10-05:00" level=info msg="Scheduling first run: 2023-01-28 20:00:20 -0500 EST"
time="2023-01-28T20:00:10-05:00" level=info msg="Note that the first check will be performed in 9 seconds"
time="2023-01-28T20:00:20-05:00" level=warning msg="Could not do a head request for \"xavierh/goaccess-for-nginxproxymanager:latest\", falling back to regular pull." container=/goaccess image="xavierh/goaccess-for-nginxproxymanager:latest"
time="2023-01-28T20:00:20-05:00" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/goaccess image="xavierh/goaccess-for-nginxproxymanager:latest"
time="2023-01-28T20:00:20-05:00" level=warning msg="Could not do a head request for \"xavierh/goaccess-docker-console:latest\", falling back to regular pull." container=/goaccess_docker_console image="xavierh/goaccess-docker-console:latest"
time="2023-01-28T20:00:20-05:00" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/goaccess_docker_console image="xavierh/goaccess-docker-console:latest"
time="2023-01-28T20:00:21-05:00" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no

AFTER

# docker container logs watchtower
time="2023-01-28T20:00:44-05:00" level=info msg="Watchtower v1.5.1-26-g8464e0d"
time="2023-01-28T20:00:44-05:00" level=info msg="Using notifications: pushover"
time="2023-01-28T20:00:44-05:00" level=info msg="Checking all containers (except explicitly disabled with label)"
time="2023-01-28T20:00:44-05:00" level=info msg="Scheduling first run: 2023-01-28 20:00:54 -0500 EST"
time="2023-01-28T20:00:44-05:00" level=info msg="Note that the first check will be performed in 9 seconds"
time="2023-01-28T20:00:54-05:00" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no
time="2023-01-28T20:01:04-05:00" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no
time="2023-01-28T20:01:14-05:00" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no
time="2023-01-28T20:01:24-05:00" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no
time="2023-01-28T20:01:34-05:00" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no
time="2023-01-28T20:01:44-05:00" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no

[wrench12435678]

As Watchtower has fixed the issue on their side (Latest-Dev) and recently have pushed out an update on Latest (Not sure if its part of it), I will close this issue as it's been resolved