net: Resolver leads to too many DNS queries

[Shnitzelil]

Go version

go.1.23.3 windows

Output of go env in your module/workspace:

set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\<user>\AppData\Local\go-build
set GOENV=C:\Users\<user>\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\<user>\go\pkg\mod
set GONOPROXY=gitlab.otxlab.net
set GONOSUMDB=gitlab.otxlab.net
set GOOS=windows
set GOPATH=C:\Users\<user>\go
set GOPRIVATE=gitlab.otxlab.net
set GOPROXY=https://proxy.golang.org/cached-only
set GOROOT=C:\Users\<user>\go\pkg\mod\golang.org\t[email protected]
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLCHAIN=auto
set GOTOOLDIR=C:\Users\<user>\go\pkg\mod\golang.org\t[email protected]\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.23.3
set GODEBUG=
set GOTELEMETRY=local
set GOTELEMETRYDIR=C:\Users\<user>\AppData\Roaming\go\telemetry
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\scm\gitlab\<my project>\go.mod
set GOWORK=
set CGO_CFLAGS=-IC:/scm/ZMQ/include
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-LC:/scm/ZMQ/bin
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=C:\Users\<user>\AppData\Local\Temp\go-build2635385560=/tmp/go-build -gno-record-gcc-switches

What did you do?

Set http client enabling Go DNS resolver lead to too many queries.
2 for HTTPS and another 2 for HTTP.

By using OS DNS resolving I see only 1 request

This is the code

package main

import (
	"fmt"
	"net"
	"net/http"
	"time"
)

func main() {
	// Create a custom dialer
	dialer := &net.Dialer{
		Timeout:   5 * time.Second, // Set timeout for dialing a connection
		KeepAlive: 30 * time.Second,
		FallbackDelay: time.Duration(-1), // Only IPv4
		Resolver: &net.Resolver{
			PreferGo: true,
		},
	}

	// Configure transport with the custom dialer
	transport := &http.Transport{
		DialContext:         dialer.DialContext,
		DisableKeepAlives:   false, // Keep connections alive
		MaxIdleConns:        10,
		IdleConnTimeout:     90 * time.Second,
		TLSHandshakeTimeout: 10 * time.Second,
	}

	// Create an HTTP client with the custom transport
	client := &http.Client{
		Transport: transport,
		Timeout:   10 * time.Second, // Overall request timeout
	}

	// Make a request using the custom client
	resp, err := client.Get("https://golang.org/")
	if err != nil {
		fmt.Println(err)
		return
	}
	
	_ = resp.Body.Close()
	resp, err = client.Get("http://golang.org/")
	if err != nil {
		fmt.Println(err)
		return
	}
	_ = resp.Body.Close()

}

What did you see happen?

Capture of Wireshark when PreferGo is set to True

Capture of Wireshark when PreferGo is set to False

What did you expect to see?

To see only one for HTTP and HTTPS not duplicate queries.

Also if I understand correctly setting FallbackDelay: time.Duration(-1) should enable IPv4 only.

[gabyhelp]

Related Issues

• net: Resolver's lookup function behavior is oblivious to a custom Dial function #54244 (closed)
• net: cgo dns resolver doesn't work as expected. #66114 (closed)
• net: resolver dials multiple times in parallel, and retries same server before progressing #32873 (closed)
• net: Resolver doesn't use provided Dial function in all cases #60712
• net: DNS Lookups Very Slow (netgo) #49253 (closed)
• net/http: Client.Do blocks on DNS AAAA record until timeout even after A record succeeds #57697 (closed)
• net.Resolver Ignores Custom Dialer #33086 (closed)
• net: DNS default resolver vs custom resolver behaviors. #29621 (closed)
• net: resolver add unexpected bytes to udp packet when wrapping the net.Conn #49750 (closed)
• net: Resolver does not appear to use its dialer function on Windows #33621 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dmitshur]

CC @ianlancetaylor, @neild.

[mateusz834]

I see 4 requests no matter the selected resolver:

$ GODEBUG=netdns=cgo+2 go run .
go package net: confVal.netCgo = true  netGo = false
go package net: GODEBUG setting forcing use of the cgo resolver
go package net: hostLookupOrder(golang.org) = cgo
go package net: hostLookupOrder(go.dev) = cgo
go package net: hostLookupOrder(golang.org) = cgo
go package net: hostLookupOrder(go.dev) = cgo

$ GODEBUG=netdns=go+2 go run .
go package net: confVal.netCgo = false  netGo = true
go package net: GODEBUG setting forcing use of the Go resolver
go package net: hostLookupOrder(golang.org) = files,dns
go package net: hostLookupOrder(go.dev) = files,dns
go package net: hostLookupOrder(golang.org) = files,dns
go package net: hostLookupOrder(go.dev) = files,dns

EDIT: note that this requires PreferGo to be false, but has the same effect.

[seankhliao]

I believe 4 requests is expected: the test uses http.Get, which follow redirects, so there's:

• 1 for golang.org
• 1 for the redirect to go.dev
• 1 for golang.org (second time)
• 1 for another redirect to go.dev

go itself does not cache DNS queries, it relies on the host to do so, see #24796
the system resolver (cgo) may decide to do additional caching, this is not within our control.

closing as working as intended.

[mateusz834]

Also if the DNS requests were concurrent, we would only do one of them, as we use singleflight.

[Shnitzelil]

I believe 4 requests is expected: the test uses http.Get, which follow redirects, so there's:

• 1 for golang.org
• 1 for the redirect to go.dev
• 1 for golang.org (second time)
• 1 for another redirect to go.dev

go itself does not cache DNS queries, it relies on the host to do so, see #24796 the system resolver (cgo) may decide to do additional caching, this is not within our control.

closing as working as intended.

It should happen only one time to each domain/hostname...

[Shnitzelil]

I see 4 requests no matter the selected resolver:

$ GODEBUG=netdns=cgo+2 go run .
go package net: confVal.netCgo = true netGo = false
go package net: GODEBUG setting forcing use of the cgo resolver
go package net: hostLookupOrder(golang.org) = cgo
go package net: hostLookupOrder(go.dev) = cgo
go package net: hostLookupOrder(golang.org) = cgo
go package net: hostLookupOrder(go.dev) = cgo

$ GODEBUG=netdns=go+2 go run .
go package net: confVal.netCgo = false netGo = true
go package net: GODEBUG setting forcing use of the Go resolver
go package net: hostLookupOrder(golang.org) = files,dns
go package net: hostLookupOrder(go.dev) = files,dns
go package net: hostLookupOrder(golang.org) = files,dns
go package net: hostLookupOrder(go.dev) = files,dns
EDIT: note that this requires PreferGo to be false, but has the same effect.

Please re-test it on Windows.

[ianlancetaylor]

@Shnitzelil

It should happen only one time to each domain/hostname...

As noted above, Go expects the DNS resolver to do caching. The Go standard library does not cache DNS results itself. That is not its job.

Please re-test it on Windows.

You can show us the output on Windows by setting the environment variable yourself.

[mateusz834]

Please re-test it on Windows.

I expect there to be the same result, i guess that windows does some internal caching, thus we do not see additional requests in cgo resolver (but we still call to the OS do to the hostname lookup).