net/url: regression after #22907 fixed

[ayanamist]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version devel +76c4587 Tue Aug 28 09:26:45 2018 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/tianyang.yty/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/tianyang.yty/go"
GOPROXY=""
GORACE=""
GOROOT="/home/tianyang.yty/gobuild/go"
GOTMPDIR=""
GOTOOLDIR="/home/tianyang.yty/gobuild/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build461801575=/tmp/go-build -gno-record-gcc-switches"

What did you do?

https://play.golang.org/p/bScfUelhUmE

What did you expect to see?

2009/11/10 23:00:00 RawQuery array[]=first&array[]=second
2009/11/10 23:00:00 url.Values{"array[]":[]string{"first", "second"}}

What did you see instead?

2018/08/28 19:57:03 RawQuery array%5B%5D%3Dfirst%26array%5B%5D%3Dsecond
2018/08/28 19:57:03 url.Values{"array[]=first&array[]=second":[]string{""}}

Many javascript framework, like jquery, will append [] to param keys, in current and previous stable release, this works fine, but after 1040626 fix #22907 pushed to master, this backward compatibility has been broken.

[thinkerou]

gin web framework also have same question. thanks!

[appleboy]

See the example code and test result from travis

func TestQuery(t *testing.T) {
	u, err := url.Parse("http://bing.com/search?q=dotnet")
	if err != nil {
		t.Fatal(err)
	}

	if u.RawQuery != "q=dotnet" {
		t.Error("RawQuery error")
	}

	u, err = url.Parse("http://bing.com/search?k=v&id=main&id=omit&array[]=first&array[]=second&ids[i]=111&ids[j]=3.14")
	if err != nil {
		t.Fatal(err)
	}

	if u.RawQuery != "k=v&id=main&id=omit&array[]=first&array[]=second&ids[i]=111&ids[j]=3.14" {
		log.Println(u.RawQuery)
		t.Error("RawQuery error")
	}

	log.Println(u.Query())
}

[agnivade]

Well, according to RFC 3986 https://tools.ietf.org/html/rfc3986#section-3.4, a fragment can only contain pchar, which allows sub-delims, not gen-delims. So, as I understand, a [] is invalid and should be escaped.

@meirf

[ayanamist]

@agnivade Although rfc does not include gen-delims, however real-world clients often accept this.
e.g. Chrome, will treat [] as valid chars, you can see "Query String Parameters" in below screenshot.

[rsc]

It seems like we should roll back that CL. This has to keep working (with UTF8 value).

package main

import (
	"log"
	"net/url"
)

func main() {
	u, err := url.Parse("http://bing.com/search?utf8=☺&param=true")
	if err != nil {
		log.Fatal(err)
	}
	
	log.Println("RawQuery", u.RawQuery)

	log.Printf("%#v", u.Query())
}

[gopherbot]

Change https://golang.org/cl/137716 mentions this issue: Revert "net/url: escape URL.RawQuery on Parse if it contains invalid characters"

[shuLhan]

@rsc Thank you.

[gopherbot]

Change https://golang.org/cl/159157 mentions this issue: net/url, net/http: reject control characters in URLs

[gopherbot]

Change https://golang.org/cl/159478 mentions this issue: [release-branch.go1.10] net/url, net/http: reject control characters in URLs

[gopherbot]

Change https://golang.org/cl/160178 mentions this issue: net/url, net/http: relax CTL-in-URL validation to only ASCII CTLs

[gopherbot]

Change https://golang.org/cl/160678 mentions this issue: [release-branch.go1.10] net/http, net/url: reject control characters in URLs

[gopherbot]

Change https://golang.org/cl/160798 mentions this issue: [release-branch.go1.11] net/http, net/url: reject control characters in URLs

[gopherbot]

Change https://golang.org/cl/162960 mentions this issue: doc/go1.12: document net/url.Parse now rejecting ASCII CTLs

[gopherbot]

Change https://golang.org/cl/162826 mentions this issue: [release-branch.go1.12] doc/go1.12: document net/url.Parse now rejecting ASCII CTLs