“Hash Sum mismatch” from mono apt repositories

[ahmetb]

As of couple of hours ago apt-get updates started to fail due to hash sum mismatch from mono apt repos.

Step 0 : FROM mono:4.0.1
 ---> fb2241ca3202
Step 1 : ENV DNX_VERSION 1.0.0-beta6
 ---> Running in f61901c25920
 ---> 99acc3867408
Step 2 : ENV DNX_USER_HOME /opt/dnx
 ---> Running in 73d2f4cc436c
 ---> cefdc691c727
Step 3 : RUN apt-get -qq update && apt-get -qqy install unzip
 ---> Running in 8c57cbb4d887
W: Conflicting distribution: http://download.mono-project.com wheezy/snapshots/4.0.0 Release (expected wheezy/snapshots but got wheezy)
W: Failed to fetch http://download.mono-project.com/repo/debian/dists/wheezy/snapshots/4.0.0/main/binary-amd64/Packages  Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.

@directhex any ideas?

[akoeplinger]

Can you reproduce this locally? I just tried and it works fine here. Hash sum mismatch usually means the CDN still has an old copy of some files.

[ahmetb]

@akoeplinger this is happening on CircleCI. It's very likely you're right as I can't repro on my development machine.
https://circleci.com/gh/aspnet/aspnet-docker/119
https://circleci.com/gh/aspnet/aspnet-docker/120
https://circleci.com/gh/aspnet/aspnet-docker/121

[akoeplinger]

@ahmetalpbalkan doesn't seem to happen in builds 123-125, so I'd say something cached it on CircleCI's end?

[ahmetb]

Looks like that :-/ weird. It only happened for a small time window.

[ahmetb]

@akoeplinger it started to happen again in the past few builds https://circleci.com/gh/aspnet/aspnet-docker/129 weird thing is, apt-get update is happening inside mono:4.0.1 image and we are pulling the image as part of "docker build".

This makes me think unless something is changed on the APT repos very recently, there should be a problem somewhere. Do you know how long the TTL is approximately for the CDN edge nodes?

.ci/build-all-tags.sh .
[CI] Building image 'microsoft/aspnet:1.0.0-beta5'...
+ docker build -t microsoft/aspnet:1.0.0-beta5 ./1.0.0-beta5
Sending build context to Docker daemon  2.56 kB


Sending build context to Docker daemon 

Step 0 : FROM mono:4.0.1
4.0.1: Pulling from mono






mono:4.0.1: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:689eaa5094f8df32f260ff59363605b483cc07e038d126732ee6119e0a82fed0
Status: Downloaded newer image for mono:4.0.1
 ---> fb2241ca3202
Step 1 : ENV DNX_VERSION 1.0.0-beta5
 ---> Running in 8ab12dd2c489
 ---> 3aa68f747df0
Step 2 : ENV DNX_USER_HOME /opt/dnx
 ---> Running in 67957b87897a
 ---> 480d34cef597
Step 3 : RUN apt-get -qq update && apt-get -qqy install unzip
 ---> Running in 24f2fd63c1f1
W: Conflicting distribution: http://download.mono-project.com wheezy/snapshots/4.0.0 Release (expected wheezy/snapshots but got wheezy)
W: Failed to fetch http://download.mono-project.com/repo/debian/dists/wheezy/snapshots/4.0.0/main/binary-amd64/Packages  Hash Sum mismatch

E: Some index files failed to download. They have been ignored, or old ones used instead.

.ci/build-all-tags.sh . returned exit code 1

[akoeplinger]

@ahmetalpbalkan weird indeed. The only thing that recently changed was the release of Mono 4.0.4 on Friday, but that should've propagated everywhere by now unless CircleCI does some weird caching.

[akoeplinger]

FYI I got reports about this on Travis as well so looks like it wasn't only on CircleCI. I can't reproduce it there as well right now though...

[ahmetb]

Let's keep reopen just in case :)

[ahmetb]

It eventually passed on many retries I did and then intermittently come back... I know for sure CircleCI modifies the apt repos on the "host" but I am running the apt-get "inside the container" so that shouldn't be the case....

I think CDN can be blamed here.

[directhex]

Definitely a CDN issue. Sometimes Packages.gz and Release don't get expired at the same time, because Cloudfront is evil and wrong.

[ahmetb]

Not happening lately. Closing.

[directhex]

Last time it happened because the clock on origin-download was out of date, causing calls to expire the CDN cache to fail. Because apparently that's a thing.