net/http: allow customizing proxy CONNECT url

[liuxingbaoyu]

Update May 5 2021: The current proposed API is in #45460 (comment) - rsc

---

What version of Go are you using (go version)?

go version go1.16 windows/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

set GO111MODULE=auto
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\流星暴雨\AppData\Local\go-build
set GOENV=C:\Users\流星暴雨\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=E:\Eternal\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=E:\Eternal\go;
set GOPRIVATE=
set GOPROXY=https://goproxy.io,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.16
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\ 流星暴雨\AppData\Local\Temp\go-build1194816388=/tmp/go-build -gno-record-gcc-switches

What did you do?

I want to use a proxy to access HTTPS websites and customize the backend IP address

What did you expect to see?

Is there some way to customize the URL of the CONNECT request

What did you see instead?

Both CONNECT request and SSL handshake use cm.targetAddr

go/src/net/http/transport.go

Lines 1667 to 1672 in 2ebe77a

	connectReq := &Request{
	Method: "CONNECT",
	URL: &url.URL{Opaque: cm.targetAddr},
	Host: cm.targetAddr,
	Header: hdr,
	}

go/src/net/http/transport.go

Lines 1725 to 1729 in 2ebe77a

	if cm.proxyURL != nil && cm.targetScheme == "https" {
	if err := pconn.addTLS(ctx, cm.tlsHost(), trace); err != nil {
	return nil, err
	}
	}

go/src/net/http/transport.go

Lines 1838 to 1846 in 2ebe77a

	// tlsHost returns the host name to match against the peer's
	// TLS certificate.
	func (cm *connectMethod) tlsHost() string {
	h := cm.targetAddr
	if hasPort(h) {
	h = h[:strings.LastIndex(h, ":")]
	}
	return h
	}

[seankhliao]

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

[liuxingbaoyu]

@seankhliao Oh no, I'm sorry that I didn't express it clearly. I read the relevant code and did not find a way to customize the CONNECT request. Perhaps this requires a new API.

[seankhliao]

what exactly do you want to do? change the address the of the proxy while keeping the hostname?

[liuxingbaoyu]

When an HTTPS request is sent through an HTTP proxy, it will first send a CONNECT request to establish a TCP connection, and then it will start a handshake with the SSL of the website. If a website has multiple resolution addresses, such as using a CDN, you can use the specified backend IP to access the website by specifying the CONNECT URL. I hope there is an API that can customize the CONNECT URL or customize the entire CONNECT request.
Similar to #13290

[seankhliao]

Right, different values for CONNECT xxx (for connecting) and Host: yyy (for TLS)

[seankhliao]

cc @bradfitz @neild @fraenkel

[rsc]

The code that sets up the CONNECT does:

connectReq := &Request{
	Method: "CONNECT",
	URL:    &url.URL{Opaque: cm.targetAddr},
	Host:   cm.targetAddr,
	Header: hdr,
}

If hdr["Host"] is set, then it sounds like we should use it instead of cm.targetAddr to set up connectReq.Host. That is, add:

if host, ok := hdr["Host"]; ok {
    if len(host) >= 1 {
        connectReq.Host = host[0]
    } else {
        connectReq.Host = ""
    }
}

Does anyone have any objections to this?

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[fraenkel]

Sorry but I am confused on what we are trying to solve.

We have a request to some target site, www.target.org, and we are going through some Proxy. The actual request has a URL/host of www.target.org. It really doesn't matter what the host header is. The Request will ignore the Host header and generate its own based, https://github.com/golang/go/blob/master/src/net/http/request.go#L570

A proxy is required to ignore the host header if it receives an absolute uri, https://tools.ietf.org/html/rfc7230#section-5.4

When a proxy receives a request with an absolute-form of
   request-target, the proxy MUST ignore the received Host header field
   (if any) and instead replace it with the host information of the
   request-target.

[rsc]

@fraenkel, that's for a standard HTTP proxy, not for some custom thing serving CONNECT requests.
Do you object to providing this functionality? It seems unobjectionable to us.

[fraenkel]

@rsc sorry my bad.
I missed the fact that we were talking about the ProxyConnectHeaders.
You will also have to reset connectReq.URL.Opaque = "" to have it appear on the CONNECT line.

[fraenkel]

Probably easier to rewrite the url since the host is only used of there is no opaque bit in the url.

[fraenkel]

Sorry I confused myself again. Ignore what I said. I need to double check my test case which didn't see the host header on the other side.

[fraenkel]

I forgot that ReadRequest deletes the host header. So as long as the other side isn't using Go, this should work.

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[gopherbot]

Change https://go.dev/cl/431756 mentions this issue: net/http: allow customizing proxy CONNECT url

[neild]

if host, ok := hdr["Host"]; ok {
    if len(host) >= 1 {
        connectReq.Host = host[0]
    } else {
        connectReq.Host = ""
    }
}

I don't believe this is correct.

RFC 9110 9.3.6 states that a proxy should connect to the "server identified by the request target". It doesn't say anything about the Host header that I can see, and I can't find any examples of using different values in the target and the Host header.

I did find a 2009 ietf-http-wg thread discussing the role of the Host header in CONNECT requests which seems to have concluded that the header is required, but that proxies should either ignore it or validate that it exactly matches the request target.

That said, the general idea of setting the target of the proxy request via the proxy connect headers seems reasonable; we should just set both the request target and the host header.