proxy.golang.org 403 forbidden much like #48107

[martinrode]

Using Go 1.20 in our Github CI chain, we experience 403 errors from certain IPs of our CSP (dedicated runner at our CSPs) when accessing a specific package.

Using CURL or our laptop computers and most IPs from our ISP work fine.

go install github.com/programmfabrik/apitest@latest
go: downloading github.com/programmfabrik/apitest v1.13.0
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/programmfabrik/golib v0.0.0-20[23](https://github.com/programmfabrik/fylr/actions/runs/6074760588/job/16483034297#step:17:24)0614100546-9870ba66917d
go: downloading github.com/spf13/afero v1.9.5
go: downloading github.com/spf13/cobra v1.7.0
go: downloading github.com/sirupsen/logrus v1.9.3
go: downloading github.com/spf13/viper v1.16.0
go: downloading github.com/moul/http2curl v1.0.0
go: downloading github.com/tidwall/gjson v1.14.4
go: downloading github.com/Masterminds/sprig/v3 v3.2.3
go: downloading github.com/mattn/go-sqlite3 v1.14.17
go: downloading golang.org/x/mod v0.12.0
go: downloading golang.org/x/oauth2 v0.9.0
go: downloading github.com/PuerkitoBio/goquery v1.8.1
go: downloading github.com/clbanning/mxj v1.8.4
go: downloading github.com/tidwall/jsonc v0.3.2
go: downloading golang.org/x/net v0.11.0
go: downloading github.com/antchfx/xmlquery v1.3.17
go: downloading github.com/gabriel-vasile/mimetype v1.4.2
go: downloading github.com/gofrs/uuid v4.4.0+incompatible
go: downloading github.com/gorilla/mux v1.8.0
go: downloading github.com/logrusorgru/aurora v2.0.3+incompatible
go: downloading github.com/yuin/goldmark v1.4.13
go: downloading golang.org/x/crypto v0.10.0
go: downloading golang.org/x/text v0.10.0
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/fsnotify/fsnotify v1.6.0
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading github.com/spf13/cast v1.5.1
go: downloading github.com/spf13/jwalterweatherman v1.1.0
go: downloading golang.org/x/sys v0.9.0
go: downloading github.com/tidwall/match v1.1.1
go: downloading github.com/tidwall/pretty v1.2.0
go: downloading github.com/Masterminds/goutils v1.1.1
go: downloading github.com/Masterminds/semver/v3 v3.2.1
go: downloading github.com/google/uuid v1.3.0
go: downloading github.com/huandu/xstrings v1.4.0
go: downloading github.com/imdario/mergo v0.3.16
go: downloading github.com/mitchellh/copystructure v1.2.0
go: downloading github.com/shopspring/decimal v1.3.1
go: downloading github.com/andybalholm/cascadia v1.3.2
go: downloading github.com/antchfx/xpath v1.2.4
go: downloading github.com/golang/groupcache v0.0.0-202103312[24](https://github.com/programmfabrik/fylr/actions/runs/6074760588/job/16483034297#step:17:25)755-41bb18bfe9da
go: downloading github.com/subosito/gotenv v1.4.2
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.0.8
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/mitchellh/reflectwalk v1.0.2
Error: /home/github-runner/go/pkg/mod/github.com/programmfabrik/[email protected][30](https://github.com/programmfabrik/fylr/actions/runs/6074760588/job/16483034297#step:17:31)614100546-9870ba66917d/mime_type_reader.go:7:2: github.com/gabriel-vasile/[email protected]: reading https://proxy.golang.org/github.com/gabriel-vasile/mimetype/@v/v1.4.2.zip: [40](https://github.com/programmfabrik/fylr/actions/runs/6074760588/job/16483034297#step:17:41)3 Forbidden

Our CSP is unable to block assigning that IP (if you need it, I can dig it up) to their VMs.

[suzmue]

Hi @martinrode, would you be able to provide us with that IP or let us know which country that IP is originating from? Thanks!

[martinrode]

Hi, we caught at least one affected IP which was 167.235.252.67. Country of origin is probably Germany (or Finnland). BR Martin

[muhammedsaidkaya]

Is there any update?

[enescakir]

Hi @suzmue, we've encountered a similar issue. The affected IP address is 178.63.229.9, which is located in Germany. We have other servers in the same data center, but they remain unaffected.

I've attached the complete logs from the GODEBUG=http2debug=1 go mod download github.com/aws/[email protected] command.

The bottom of the logs shows that. How can we find the exact reason why it's not available?

http2: Transport received DATA stream=1 len=193 data="<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>We're sorry, but this service is not available in your location</Details></Error>"
go: github.com/aws/[email protected]: reading https://proxy.golang.org/github.com/aws/aws-sdk-go/@v/v1.51.21.zip: 403 Forbidden

http2-debug-aws.txt

[dmitshur]

CC @samthanawalla, @findleyr, @hyangah.

[findleyr]

@enescakir a geoip lookup for that IP reports a different, embargoed country.
That may be inaccurate, but likely explains the response.

[enescakir]

@enescakir a geoip lookup for that IP reports [EDIT(findleyr): a different, embargoed country] as the associated country. That may be inaccurate, but likely explains the response.

It’s an IP address from Germany (Hetzner Falkenstein)

[enescakir]

What tool do you use for geoip lookup? I've tried different online tools, but they all show Germany.

[findleyr]

@enescakir sorry, it is an internal tool, and I didn't realize it disagreed with external tooling. I've hidden the specific country in my and your comment out of an abundance of caution. The exact country is probably fine to share, but also isn't relevant for this discussion: the point is that our hosting infrastructure disagrees about the location of that IP, and thinks that it is in a restricted location.

Unfortunately, I also looked into filing a dispute for this IP, and it does not look feasible. Therefore, your options are to try to acquire a new IP for this machine, or wait to see if the issue resolves itself. Sorry for not being able to provide a more satisfactory solution.

[enescakir]

@enescakir sorry, it is an internal tool, and I didn't realize it disagreed with external tooling. I've hidden the specific country in my and your comment out of an abundance of caution. The exact country is probably fine to share, but also isn't relevant for this discussion: the point is that our hosting infrastructure disagrees about the location of that IP, and thinks that it is in a restricted location.

Unfortunately, I also looked into filing a dispute for this IP, and it does not look feasible. Therefore, your options are to try to acquire a new IP for this machine, or wait to see if the issue resolves itself. Sorry for not being able to provide a more satisfactory solution.

Thanks for the detailed explanation, @findleyr. We will indeed return this subnet and acquire a new one. However, the challenging question is, how can we prevent this from happening again with one of our IP subnets?

[ozgune]

Hey @findleyr - I had a quick question.

I read @martinrode and @enescakir's comments. It looks like your internal tool is blocking some of Hetzner's IPs. Hetzner is Europe's largest bare metal and cost-effective cloud provider. https://www.hetzner.com They have regions in Germany and Finland.

We'll ask to recycle the IPs on our end. However, given the internal tool, it looks like we're going to run into this issue again. What's a good way to remedy it?

[findleyr]

@ozgune based on my reading, it sounds like there's nothing we can really do for individual IPs. I would however hope that this is a rare occurrence, since the internal tool should meet a high standard for accuracy. If this proves to be a persistent problem, we can revisit.

[findleyr]

Closing this as (Unfortunately) unactionable. Will revisit if this proves to be a persistent problem.