net: Error when running concurrent DNS queries using cgo on MacOS

[cezarsa]

What version of Go are you using (go version)?

$ go version
go version go1.12 darwin/amd64

Does this issue reproduce with the latest release?

Yes, only on the latest release, it wasn't a problem with Go 1.11

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/cezarsa/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/cezarsa/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.12/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.12/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/18/22s9w_gd21979c67k0p0p7t00000gn/T/go-build808727368=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

What did you expect to see?

With the code available in https://play.golang.org/p/Za2kDnfq-wB on MacOS Mojave 10.14

The expected result for running it with both no environment variables and with PARALLEL=1 would be:

$ go run dns.go
remote addr for udp4: 127.0.0.1:27017
remote addr for udp6: [::1]:27017

What did you see instead?

Running it with no envs yields the expected result. However, running it with PARALLEL=1 causes the following output:

$ PARALLEL=1 go run dns.go
ERR for udp4: dial udp4: address localhost: no suitable address found
remote addr for udp6: [::1]:27017

The problem only seems to exist when running on MacOS, and only when using cgo resolver, using Go resolver or on Linux does not causes any errors:

$ GODEBUG=netdns=cgo+1 PARALLEL=1 go run dns.go
go package net: using cgo DNS resolver
ERR for udp4: dial udp4: address localhost: no suitable address found
remote addr for udp6: [::1]:27017

$ GODEBUG=netdns=go+1 PARALLEL=1 go run dns.go
go package net: GODEBUG setting forcing use of Go's resolver
remote addr for udp4: 127.0.0.1:27017
remote addr for udp6: [::1]:27017

[cezarsa]

I've been taking a look at the lookup code, and it looks like the problem is here:

go/src/net/lookup.go

Line 266 in ee6bec9

ch, called := r.getLookupGroup().DoChan(host, func() (interface{}, error) {

The lookup group will only allow 1 query to run even when they are using different values for the network protocol, and the underlying lookup implementation is only returning ip addresses valid to the chosen network.

A possible fix would be to change the key for the lookup group to be network + host, instead of only host. Testing locally this change does fix the problem, but I don't know if it's a valid solution or some other implications unknown to me of such change.

Also, from the Git history it looks like this didn't change for 1.12 so this does not explain why this used to work on 1.11.

[bcmills]

CC @mikioh @bradfitz @ianlancetaylor for net

[ianlancetaylor]

Changing the key to network + "\000" + host seems fine to me. It would be nice to have a test, though.

[cezarsa]

@ianlancetaylor Cool, if no one is working on this I'd like to try sending a CL including some tests. Further tracking down the problem I think the source of the regression is actually c659be4 but I think changing the lookup group key still makes more sense as a fix.

[gopherbot]

Change https://golang.org/cl/166037 mentions this issue: net: use network and host as singleflight key during lookupIP

[mikioh]

@gopherbot please consider this for backport to 1.12

[gopherbot]

Backport issue(s) opened: #31062 (for 1.12).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[mikioh]

As reported in #31058, this is a regression in Go 1.12 and there's no workaround for Windows and others that have no support for GODEBUG=netdns=go.