net/http: transport hang

[kstenerud]

What version of Go are you using (go version)?

$ go version
go version go1.13.9 linux/amd64

Does this issue reproduce with the latest release?

I can't reproduce it

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/karl/.cache/go-build"
GOENV="/home/karl/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/karl/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/karl/work/Devel/testgorm/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build561612903=/tmp/go-build -gno-record-gcc-switches"

What did you do?

It was just a standard HTTP post:

	request, err := http.NewRequest("POST", c.signingEndpoint.String(), nil)
	if err != nil {
		return "", err
	}
	request.Header.Add("Authorization", c.signingAuth)
	client := http.DefaultClient
	response, err = client.Do(request)

What did you expect to see?

I expected it to succeed or fail

What did you see instead?

It hung:

goroutine 1 [select]:
net/http.(*Transport).getConn(0xd12d60, 0xc000198300, 0x0, 0xc0003986c0, 0x5, 0xc000362a60, 0x10, 0x0, 0x0, 0x0, ...)
	/usr/local/go/src/net/http/transport.go:1239 +0x553
net/http.(*Transport).roundTrip(0xd12d60, 0xc0000fe000, 0xc0001c4900, 0xc0000f32f0, 0x700000000414378)
	/usr/local/go/src/net/http/transport.go:522 +0x603
net/http.(*Transport).RoundTrip(0xd12d60, 0xc0000fe000, 0xd12d60, 0x0, 0x0)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc0000fe000, 0xa446a0, 0xd12d60, 0x0, 0x0, 0x0, 0xc000010028, 0xd, 0x1, 0x0)
	/usr/local/go/src/net/http/client.go:250 +0x443
net/http.(*Client).send(0xd52fe0, 0xc0000fe000, 0x0, 0x0, 0x0, 0xc000010028, 0x0, 0x1, 0x7ff365c16490)
	/usr/local/go/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xd52fe0, 0xc0000fe000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:641 +0x3ce
net/http.(*Client).Do(...)
	/usr/local/go/src/net/http/client.go:509
github.com/agentclient/agent-client.(*agentClient).sendSigningRequest(0xc0000b2aa0, 0xc000318060, 0x0, 0x414378, 0x80)
	/home/karl/agent-client/agent_client.go:253 +0x24c
github.com/agentclient/agent-client.(*agentClient).CurrentAgentQuery(0xc0000b2aa0, 0x59e4c2, 0xc0001ce000, 0xa4cce0, 0xc0000aa078)
	/home/karl/agent-client/agent_client.go:69 +0xb6
github.com/agentclient/agent-client.(*agentClient).sendProfilePrologue(0xc0000b2aa0, 0xc0000f3c78, 0x4, 0x9a0201)
	/home/karl/agent-client/agent_client.go:141 +0x43
github.com/agentclient/agent-client.(*agentClient).SendProfile(0xc0000b2aa0, 0xc0003fc000, 0x9e06, 0xf000, 0x0, 0x0)
	/home/karl/agent-client/agent_client.go:223 +0x158
github.com/agentclient/agent-client.(*probe).endProfile(0xc0000fc600, 0x0, 0x0)
	/home/karl/agent-client/probe.go:455 +0x356
github.com/agentclient/agent-client.(*probe).End(0xc0000fc600, 0x0, 0x0)
	/home/karl/agent-client/probe.go:234 +0x402
github.com/agentclient/agent-client.End()
	/home/karl/agent-client/api.go:59 +0x2d
main.main()
	/home/karl/work/Devel/testapp/main.go:60 +0x2ec

Looking at the source, it seems to have hung while waiting for the transport queue to allow it to dial. This is a very simple, single goroutine program that was making a standard HTTP POST request.

It only happened once. I'd never encountered it before, and was unable to reproduce again, so it will be hard to track down what happened, but I suspect I hit an edge case in the locking and queue logic.

[andybons]

@bradfitz @neild

[davecheney]

@kstenerud would you be able to upgrade to Go 1.16 and confirm if the problem remains. Thank you.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[secmask]

@kstenerud would you be able to upgrade to Go 1.16 and confirm if the problem remains. Thank you.

I got same issue, and I can't reproduce it,
go version 1.16.12

92 @ 0x43cd45 0x44df57 0x71f1e9 0x71a88b 0x701295 0x96f5ed 0x9702d2 0x6be134 0x6bdb1f 0x6bfb9f 0x96e025 0x96e000 0x96e53e 0x96b655 0x96d745 0x970791 0x474101
#	0x71f1e8	net/http.(*Transport).getConn+0x588									/opt/hostedtoolcache/go/1.16.12/x64/src/net/http/transport.go:1368
#	0x71a88a	net/http.(*Transport).roundTrip+0x7ea									/opt/hostedtoolcache/go/1.16.12/x64/src/net/http/transport.go:579
#	0x701294	net/http.(*Transport).RoundTrip+0x34									/opt/hostedtoolcache/go/1.16.12/x64/src/net/http/roundtrip.go:17

[gland1]

It happens also with golang 17.
For me it happens after mutliple connection timeout errors.
eventually it hanged forever:

goroutine 27 [select, 1469 minutes]:
net/http.(*Transport).getConn(0xc0004443c0, 0xc00042c540, {{}, 0x0, {0xc00003e0f0, 0x5}, {0xc00061e060, 0x15}, 0x0})
/usr/local/go/src/net/http/transport.go:1372 +0x5d2
net/http.(*Transport).roundTrip(0xc0004443c0, 0xc0002cc000)
/usr/local/go/src/net/http/transport.go:581 +0x774
net/http.(*Transport).RoundTrip(0xc0003b4540, 0xc0002cc000)
/usr/local/go/src/net/http/roundtrip.go:18 +0x19
golang.org/x/oauth2.(*Transport).RoundTrip(0xc0006266c0, 0xc000446500)
/root/workspace/KumoScale-CSI-3.22/pkg/mod/golang.org/x/[email protected]/transport.go:55 +0x14a
net/http.send(0xc000446500, {0x15199c0, 0xc0006266c0}, {0x13304a0, 0x1, 0x0})
/usr/local/go/src/net/http/client.go:252 +0x5d8
net/http.(*Client).send(0xc00050f680, 0xc000446500, {0xc000446500, 0x2f, 0x0})
/usr/local/go/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0xc00050f680, 0xc000446500)
/usr/local/go/src/net/http/client.go:725 +0x908
net/http.(*Client).Do(...)
/usr/local/go/src/net/http/client.go:593