net/http: double escape of URL

[djgilcrease]

What version of Go are you using (go version)?

$ go version
go version go1.10.1 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/digitalxero/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/opt/projects/f5/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build355493993=/tmp/go-build -gno-record-gcc-switches"

What did you do?

url := "http://some.example.com/foo%2Fbar"
req, _ := http.NewRequest("POST", url, body)
resp, _ := client.Do(req)
fmt.Println(resp.Request.RequestURI)
# 

What did you expect to see?

"http://some.example.com/foo%2Fbar"

What did you see instead?

"http://some.example.com/foo%252Fbar"

[ianlancetaylor]

"foo%252Fbar" is of course the escaped form of "foo%2Fbar".

This is a question probably better directed to a forum rather than to the issue tracker. See https://golang.org/wiki/Questions . Thanks.

[djgilcrease]

This is a bug because foo%2Fbar is a valid path as is and does not need to be escaped again. It even passes the internal validEncodedPath check in url.go, but because it does a secondary check of unescaping the path and comparing it to the valid escaped path you get a double escaped path when you should not

[ianlancetaylor]

Can you show us a complete program demonstrating the bug? Thanks.

CC @bradfitz

[djgilcrease]

https://play.golang.org/p/DlQWHrX1cuD

package main

import (
	"fmt"
	"net/url"
)

func main() {
	expected := "https://httpbin.org/anything/foo%2Fbar/bla"
	good, _ := url.Parse(fmt.Sprintf("https://httpbin.org/anything/%s/bla", url.PathEscape("foo/bar")))
	bad, _ := url.Parse("https://httpbin.org")
	bad.Path = fmt.Sprintf("anything/%s/bla", url.PathEscape("foo/bar"))
	alsoBad, _ := url.Parse("https://httpbin.org")
	alsoBad.RawPath = fmt.Sprintf("anything/%s/bla", url.PathEscape("foo/bar"))

	workAround, _ := url.Parse("https://httpbin.org")
	workAround.Path = fmt.Sprintf("anything/%s/bla", url.PathEscape("foo/bar"))
	workAround.RawPath = workAround.Path
	workAround.Path, _ = url.PathUnescape(workAround.Path)

	if expected != good.String() {
		fmt.Printf("GOOD: %s != %s", expected, good.String())
	}

	if expected != bad.String() {
		fmt.Printf("BAD: %s != %s\n", expected, bad.String())
	}

	if expected != alsoBad.String() {
		fmt.Printf("ALSO BAD: %s != %s\n", expected, alsoBad.String())
	}

	if expected != workAround.String() {
		fmt.Printf("Worked Around: %s != %s", expected, workAround.String())
	}
}

[crvv]

I think the example doesn't show any bug.
The document(https://golang.org/pkg/net/url/#URL) says

the Path field is stored in decoded form: /%47%6f%2f becomes /Go/

If url.Path is /foo%2Fbar/bla, its encoded form is /foo%252Fbar/bla.

And the "workAround" is the right way to do this.
It is documented clearly at https://golang.org/pkg/net/url/#URL.EscapedPath

[bradfitz]

The first example shows fmt.Println(resp.Request.RequestURI) but that Request.RequestURI field is documented as:

        // RequestURI is the unmodified Request-URI of the
        // Request-Line (RFC 2616, Section 5.1) as sent by the client
        // to a server. Usually the URL field should be used instead.
        // It is an error to set this field in an HTTP client request.
        RequestURI string

Note that last sentence: "It is an error to set this field in an HTTP client request".

And indeed, the net/http code returns an error if it's set.

I don't see how that's possible. The sample code as provided seems like it would never happen or wouldn't return the results described.

Then the follow-up code shows completely unrelated (and working as expected) code that doesn't involve the RequestURI field.

I'm going to close this. If you have a better example of the bug, please post and we can reopen.