net/http: invalid byte '"' in Cookie.Value; dropping invalid bytes #18627

sinylei · 2017-01-12T07:18:32Z

What version of Go are you using (`go version`)?

go root# go version
go version go1.7.1 darwin/amd64

What operating system and processor architecture are you using (`go env`)?

root# go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.7.1/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.7.1/libexec/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build065605285=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"

What did you do?

Hi,
I am trying to access a Web server using net/http.
But the request was rejected by the server due to the cookie not accepted, missing a pair of "".
My application report in console log as below:
net/http: invalid byte '"' in Cookie.Value; dropping invalid bytes

The cookie I sent in my request looks like this:
Cookie: _LSID=2543ebb4-1213-47f8-b31a-a17a31ec93d3,Fy4oMdFmUw5wt1vD/gm0jDBV3hje6omX1VDusKRcf/Bj5Qenv4uTaPE2pRWvt7JNXyO0bFdrQfd0w4TYzxIXGQ==

The cookie required and accepted by server:
cookie: _LSID="2543ebb4-1213-47f8-b31a-a17a31ec93d3,Fy4oMdFmUw5wt1vD/gm0jDBV3hje6omX1VDusKRcf/Bj5Qenv4uTaPE2pRWvt7JNXyO0bFdrQfd0w4TYzxIXGQ=="

The difference is a pair of char(") at the both sides of the Cookie value string.

What did you expect to see?

Could you remove the Cookie Value authentication for char '"'

What did you see instead?

The text was updated successfully, but these errors were encountered:

davecheney · 2017-01-12T08:52:58Z

Please attach a code sample that shows the problem.

…

On Thu, 12 Jan 2017, 18:18 sinylei ***@***.***> wrote: Please answer these questions before submitting your issue. Thanks! What version of Go are you using (go version)? go root# go version go version go1.7.1 darwin/amd64 What operating system and processor architecture are you using (go env)? root# go env GOARCH="amd64" GOBIN="" GOEXE="" GOHOSTARCH="amd64" GOHOSTOS="darwin" GOOS="darwin" GOPATH="" GORACE="" GOROOT="/usr/local/Cellar/go/1.7.1/libexec" GOTOOLDIR="/usr/local/Cellar/go/1.7.1/libexec/pkg/tool/darwin_amd64" CC="clang" GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build065605285=/tmp/go-build -gno-record-gcc-switches -fno-common" CXX="clang++" CGO_ENABLED="1" What did you do? Hi, I am trying to access a Web server using net/http. But the request was rejected by the server due to the cookie not accepted, missing a pair of "". My application report in console log as below: net/http: invalid byte '"' in Cookie.Value; dropping invalid bytes The cookie I sent in my request looks like this: Cookie: _LSID=2543ebb4-1213-47f8-b31a-a17a31ec93d3,Fy4oMdFmUw5wt1vD/gm0jDBV3hje6omX1VDusKRcf/Bj5Qenv4uTaPE2pRWvt7JNXyO0bFdrQfd0w4TYzxIXGQ== The cookie required and accepted by server: cookie: _LSID="2543ebb4-1213-47f8-b31a-a17a31ec93d3,Fy4oMdFmUw5wt1vD/gm0jDBV3hje6omX1VDusKRcf/Bj5Qenv4uTaPE2pRWvt7JNXyO0bFdrQfd0w4TYzxIXGQ==" The difference is a pair of char(") at the both sides of the Cookie value string. What did you expect to see? Could you remove the Cookie Value authentication for char '"' What did you see instead? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#18627>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcAxwPjr0i2-RUX813ZWcCil88Rvr7ks5rRdPJgaJpZM4LhcVu> .

kennygrant · 2017-01-12T09:00:18Z

See also #7243 which discusses these restrictions on cookie values.

sinylei · 2017-01-12T09:19:19Z

import (
    "github.com/mozillazg/request"
    "net/http"
)

func checkServer(user) {
    c := &http.Client{}
    
    req := request.NewRequest(c)
    req.Cookies = map[string]string {
//        "_LSID" : user.LSID,
        "_LSID" : `"` + user.LSID + `"`,
    }
...
}

My code looks above using a third package request which is a wrapper of original net/http.
The user.LSID is string like as below:
2543ebb4-1213-47f8-b31a-a17a31ec93d3,Fy4oMdFmUw5wt1vD/gm0jDBV3hje6omX1VDusKRcf/Bj5Qenv4uTaPE2pRWvt7JNXyO0bFdrQfd0w4TYzxIXGQ==

The server which is out of my control requires the cookie as below.
_LSID="2543ebb4-1213-47f8-b31a-a17a31ec93d3,Fy4oMdFmUw5wt1vD/gm0jDBV3hje6omX1VDusKRcf/Bj5Qenv4uTaPE2pRWvt7JNXyO0bFdrQfd0w4TYzxIXGQ=="

The server requires Double quotes in the cookie value and mandatory.

davecheney · 2017-01-12T09:23:52Z

@sinylei i meant a code sample that compiled.

sinylei · 2017-01-12T09:35:24Z

@davecheney sorry, I can not provide you a code sample since it is not a standalone app, it is a subsystem of a project, which is original written by nodejs. Now I am trying to rewrite some subsystem using golang to improve the performance.
Briefly, this subsystem can not work independently. Another process does the 'User Login' to the Web server and transfer the user session cookie to this subsystem. This subsystem initialized another request to the Web server using this cookie.

I understand that golang has the restrictions on cookie value to avoid any potential security issue for server-side application developed by golang. But for the client application, the language should provide the developer flexibility on cookie values that required by some server. Actually, nodejs has no such restrictions on cookie values. I read the source code of net/http and found the restriction is here:

https://golang.org/src/net/http/cookie.go#L320

ghost · 2017-01-13T01:28:49Z

The cookies with quoted values added via AddCookie are unquoted, those added via Header.Add are not.

package main

import (
	"fmt"
	"log"
	"net/http"
	"net/http/httptest"
	"net/http/httputil"
)

func main() {
	s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		data, _ := httputil.DumpRequest(r, false)
		fmt.Println(string(data))
	}))
	c := http.Cookie{
		Name:  "name1",
		Value: `"quoted"`,
		Path:  "/",
	}
	r, _ := http.NewRequest("GET", s.URL, nil)
	r.AddCookie(&c)
	r.Header.Add("Cookie", `name2="quoted"`)
	_, err := http.DefaultClient.Do(r)
	if err != nil {
		log.Fatal(err)
	}
}

Output:

2017/01/13 08:28:38 net/http: invalid byte '"' in Cookie.Value; dropping invalid bytes
GET / HTTP/1.1
Host: 127.0.0.1:42767
Accept-Encoding: gzip
Cookie: name1=quoted
Cookie: name2="quoted"
User-Agent: Go-http-client/1.1

bradfitz · 2017-01-13T01:31:45Z

If Go is violating RFCs, somebody should cite which RFCs we're violating, and then we can fix.

If Go is not violating RFCs, somebody should provide evidence that the RFCs are wrong (that popular clients & servers behave otherwise, reliably, with details). Then we can file bugs against the RFCs and add workarounds.

ghost · 2017-01-13T01:39:38Z

According to the RFC 6265 section 4.1.1,

cookie-value      = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )

where DQUOTE is a double quote.

So double-quoted cookie values are explicitly allowed.

bradfitz · 2017-01-13T02:03:06Z

@opennota, yes, but Request.Cookies contains the unescaped values. The double quote (") character is for the wire representation. You can't just put the wire representation into the Request.Cookies map values.

Are you saying that this bug is that Go should add double quotes around cookie values in more cases?

ghost · 2017-01-13T02:22:19Z

@bradfitz

Are you saying that this bug is that Go should add double quotes around cookie values in more cases?

No. I'm okay with the current behaviour, where one can send double quoted cookie values by adding them directly to the request headers. But at least nodejs (as the reporter says) behaves differently. How about other languages and platforms?

ghost · 2017-01-13T02:29:55Z

Perhaps Go could recognize and do not remove double quotes around the cookie values in Request.Cookies? net/http server doesn't seem to unescape values like %22 or \", so it should be okay, shouldn't it.

sinylei · 2017-01-13T04:16:30Z

@opennota You provide a really good workaround, that put such quoted string into Header portion. Header related function does not do such value validation. Thank you very much!

bradfitz · 2017-01-13T06:08:39Z

Previous discussion was in #7243.

The Go code is currently:

// http://tools.ietf.org/html/rfc6265#section-4.1.1                                                                                             
// cookie-value      = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )                                                                          
// cookie-octet      = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E                                                                             
//           ; US-ASCII characters excluding CTLs,                                                                                              
//           ; whitespace DQUOTE, comma, semicolon,                                                                                             
//           ; and backslash                                                                                                                    
// We loosen this as spaces and commas are common in cookie values                                                                              
// but we produce a quoted cookie-value in when value starts or ends                                                                            
// with a comma or space.                                                                                                                       
// See https://golang.org/issue/7243 for the discussion.                                                                                        
func sanitizeCookieValue(v string) string {
        v = sanitizeOrWarn("Cookie.Value", validCookieValueByte, v)
        if len(v) == 0 {
                return v
        }
        if v[0] == ' ' || v[0] == ',' || v[len(v)-1] == ' ' || v[len(v)-1] == ',' {
                return `"` + v + `"`
        }
        return v
}

It sounds like this bug is suggesting that we need to double-quote the Set-Cookie value on the wire in more cases. Perhaps if there's a comma anywhere in the value string.

sinylei · 2017-02-03T06:03:07Z

It is really a good workaround, put quoted cookies in Header since Header related function does not do such cookie value validation. Cookie is part of HTTP Header. @opennota Many many thanks.

…

在 2017年1月13日，上午9:30，opennota ***@***.*** ***@***.***>> 写道： The cookies with quoted values added via AddCookie are unquoted, those added via Header.Add are not. package main import ( "fmt" "log" "net/http" "net/http/httptest" "net/http/httputil" ) func main() { s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { data, _ := httputil.DumpRequest(r, false) fmt.Println(string(data)) })) c := http.Cookie{ Name: "name1", Value: `"quoted"`, Path: "/", } r, _ := http.NewRequest("GET", s.URL, nil) r.AddCookie(&c) r.Header.Add("Cookie", `name2="quoted"`) _, err := http.DefaultClient.Do(r) if err != nil { log.Fatal(err) } } Output: 2017/01/13 08:28:38 net/http: invalid byte '"' in Cookie.Value; dropping invalid bytes GET / HTTP/1.1 Host: 127.0.0.1:42767 Accept-Encoding: gzip Cookie: name1=quoted Cookie: name2="quoted" User-Agent: Go-http-client/1.1 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18627 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AX6WMmjDYRPt1P78ljowPwtVjneJtZ5qks5rRtOYgaJpZM4LhcVu>.

bradfitz · 2017-02-03T17:53:34Z

@odeke-em, you interested in this one?

odeke-em · 2017-02-03T22:14:36Z

Say no more fam, I got it.

bradfitz · 2017-02-09T00:06:09Z

@odeke-em, still got it? :)

odeke-em · 2017-02-09T00:09:10Z

Yap yap, I'll work on it tonight after this lab exercise ie in about 4 hours and send in a CL, otherwise I'll ping you for help.

odeke-em · 2017-02-09T06:28:10Z

So am interpreting this issue differently, and we have 2 different options:

Skip validation for '"' entirely, as the reporter requests, regardless of the number of double quotes.
Skip validation of '"' only if they are in pairs because we have the definition of a cookie value
to correctly follow http://tools.ietf.org/html/rfc6265#section-4.1.1 which defines a cookie-value as:

// cookie-value      = *cookie-octet / ( dQuote *cookie-octet dQuote )
// cookie-octet      = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
//           ; US-ASCII characters excluding CTLs,
//           ; whitespace dQuote, comma, semicolon,
//           ; and backslash

That is: if a cookie value has double quotes, they must be in pairs surrounding a cookie octet.

What do y'all think? 1. Is trivial. 2. requires a bit more of cleverness but I've implemented it as well.

EDIT: Just to be clear, with 2. The qualm code in this issue will be accepted

odeke-em · 2017-02-09T07:25:05Z

I've mailed https://go-review.googlesource.com/c/36642.

gopherbot · 2017-02-09T07:25:33Z

CL https://golang.org/cl/36642 mentions this issue.

ghost · 2017-02-09T07:50:55Z

@odeke-em

I've mailed https://go-review.googlesource.com/c/36642.

Am I mistaken, or you code assumes that cookie-octet can be a double quote (it cannot)?

odeke-em · 2017-02-09T07:54:23Z

@opennota let's comment on the CL :) but cookie-octet in my CL cannot be a double quote, it can be the empty string surrounded by a pair of double quotes e.g "" but not """.

ghost · 2017-02-09T08:02:08Z

I just don't get this parse " octet " in pairs business. (And can't log in to comment on the CL at the moment, sorry.)
Isn't it that cookie-value either surrounded by dquotes (so there are only two of them) or doesn't contain dquotes at all?

odeke-em · 2017-02-09T08:09:31Z

Yes you are right, after doing a second reading of the definition, I see that I might have misinterpreted the definition: I read cookie-value instead of cookie-octet so just only 2 of DQUOTE as you mention. Thank you @opennota, I'll comment that on the CL. For now, I've got a crap ton of assignments to work on due for Friday afternoon but I'll update/fix the CL on Friday afternoon when free, or anyone can feel free to drop in.

vdobler · 2017-02-09T09:47:47Z

The cookie-value in question is

2543ebb4-1213-47f8-b31a-a17a31ec93d3,Fy4oMdFmUw5wt1vD/gm0jDBV3hje6omX1VDusKRcf/Bj5Qenv4uTaPE2pRWvt7JNXyO0bFdrQfd0w4TYzxIXGQ==

which consists of valid cookie-octets and an invalid internal comma ",". This comma is illegal according to RFC 6265 in a cookie value, be it unquoted or quoted.

RFC 6265 never requires double quoting cookie values.
We use double quoted cookie values to allow malformed cookie values which are common in the wild, especially cookie values containing spaces and commas because browsers do handle these cookie values properly and lots of systems produce formally illegal cookie values (as no browser complains) and we should deal with them. It turned out (see issue #7243) that all browser accept:

unquoted cookie values with internal spaces or commas
quoted cookie values with leading or trailing spaces and commas

We decided to allow formally malformed cookie values with spaces and commas and send them in quoted form iff required by browsers (i.e. leading or trailing), but send internal spaces/commas unquoted.

It seems as if there are clients out in the wild which require the cookie-values to be quoted also for internal commas (and probably spaces).

Formally there is no bug as commas are disallowed in cookie-values by RFC 6265.

Alternatives are:

We always quote all cookie values, even wellformed ones. This increases network traffic a tiny bit. This should have no drawback on RFC 6265 compliant agents/servers. This is dangerous as lots of cookie values in the wild are wellformed and unquoted and compliance with RFC 6265 seems not too strong.
We decide this is a non-bug, especially as there is a workaround for non compliant third party systems: Add the Cookie or Set-Cookie header manually.
Send cookie values in quoted form also if the cookie-value contains internal (non-leading, non-trailing) spaces or commas.

ddo · 2017-02-21T23:12:22Z

@bradfitz

The double quote (") character is for the wire representation. You can't just put the wire representation into the Request.Cookies map values.

Sorry but why not?

bradfitz · 2017-02-21T23:55:38Z

@vdobler, option (3) sounds good to me.

ddo · 2017-02-22T12:37:32Z

we should handle cookies like what browsers do not RFCs

#validCookieValueByte should allow " \ and ,
cookie name and value must be trim space, #readSetCookies only trim parts that split from ;
#parseCookieValue no need allowDoubleQuote since we allow " in cookie name and value
(optional) not sure is it faster if we use strings.SplitN instead of strings.Index(name, "=")

vdobler · 2017-02-22T13:06:27Z

@ddo

Different browsers behave differently. Doing "like what browser do" is not well defined.
Most browser do not allows " in cookie values (because that interferes with the wire format).
Trimming spaces from cookie values is more than strange. Why would you like to do that?

ddo · 2017-02-22T13:29:47Z

i tested some cases with chrome and firefox

0=qweqweqw
1=
11=    (some spaces in value)
2=""
22="      "
3="
4=,
44=","
55="   w   "
555=     w    (some spaces prefix and suffix value)
5555=      w   w     (some spaces prefix and suffix value)
55555="   w   w    "
6="\"
66=\
7=haha
      7  7    =haha   (some spaces prefix and suffix name)
"  7  7"=haha
9="
99="""
999="   "   "
9999=""   "
99999="      ""
"""=1
"=1
""=1
,=1
/=1

Google Chrome

Firefox

Note: <space><space>w<space><space><space>w<space> gonna be w<space><space><space>w, both 2 browsers show like w<space>w but when you send the next request the cookie in headers is correct one w<space><space><space>w. like strings#TrimSpace

gopherbot · 2017-02-22T14:05:52Z

CL https://golang.org/cl/37328 mentions this issue.

vdobler · 2017-02-22T14:16:18Z

@ddo Chrome is very liberal in some regards, Firefox in others (e.g. UTF-8 values). If only Chrome had to be supported it would be easy.

odeke-em · 2017-02-23T01:34:55Z

@vdobler I'll let you take over the CL, as I don't seem to fully understand the deeper problem. Thanks for looking into it.

JamesCullum · 2017-10-19T06:39:32Z

Still having that issue, Microsoft is using JSON in a cookie value as well, using double quotes. While I did code a workaround (extracting the raw value from the header), is there some way to supress the error message?

davecheney · 2017-10-19T07:16:54Z

@JamesCullum please open a new issue, this one is closed.

bradfitz added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jan 13, 2017

bradfitz added this to the Unplanned milestone Jan 13, 2017

bradfitz modified the milestones: Go1.9Maybe, Unplanned Jan 13, 2017

bradfitz added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Jan 13, 2017

odeke-em self-assigned this Feb 3, 2017

gopherbot closed this as completed in 8f6d68e May 22, 2017

lucas-clemente mentioned this issue Jun 28, 2017

net/http: Document change in cookie sanitization in the release notes? #20834

Closed

na-- mentioned this issue Jul 16, 2018

Error when cookies contain double quotes grafana/k6#711

Open

golang locked and limited conversation to collaborators Oct 19, 2018

gopherbot added the FrozenDueToAge label Oct 19, 2018

rsc unassigned odeke-em Jun 23, 2022

net/http: invalid byte '"' in Cookie.Value; dropping invalid bytes #18627

net/http: invalid byte '"' in Cookie.Value; dropping invalid bytes #18627

Comments

sinylei commented Jan 12, 2017 • edited Loading

What version of Go are you using (go version)?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

davecheney commented Jan 12, 2017 via email

kennygrant commented Jan 12, 2017 • edited Loading

sinylei commented Jan 12, 2017 • edited by bradfitz Loading

davecheney commented Jan 12, 2017

sinylei commented Jan 12, 2017 • edited Loading

ghost commented Jan 13, 2017

bradfitz commented Jan 13, 2017

ghost commented Jan 13, 2017 • edited by ghost Loading

bradfitz commented Jan 13, 2017

ghost commented Jan 13, 2017

ghost commented Jan 13, 2017

sinylei commented Jan 13, 2017 • edited Loading

bradfitz commented Jan 13, 2017

sinylei commented Feb 3, 2017 via email

bradfitz commented Feb 3, 2017

odeke-em commented Feb 3, 2017

bradfitz commented Feb 9, 2017

odeke-em commented Feb 9, 2017

odeke-em commented Feb 9, 2017 • edited Loading

odeke-em commented Feb 9, 2017

gopherbot commented Feb 9, 2017

ghost commented Feb 9, 2017

odeke-em commented Feb 9, 2017

ghost commented Feb 9, 2017

odeke-em commented Feb 9, 2017

vdobler commented Feb 9, 2017

ddo commented Feb 21, 2017

bradfitz commented Feb 21, 2017

ddo commented Feb 22, 2017 • edited Loading

vdobler commented Feb 22, 2017

ddo commented Feb 22, 2017 • edited Loading

gopherbot commented Feb 22, 2017

vdobler commented Feb 22, 2017

odeke-em commented Feb 23, 2017

JamesCullum commented Oct 19, 2017

davecheney commented Oct 19, 2017

sinylei commented Jan 12, 2017 •

edited

Loading

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

kennygrant commented Jan 12, 2017 •

edited

Loading

sinylei commented Jan 12, 2017 •

edited by bradfitz

Loading

sinylei commented Jan 12, 2017 •

edited

Loading

ghost commented Jan 13, 2017 •

edited by ghost

Loading

sinylei commented Jan 13, 2017 •

edited

Loading

odeke-em commented Feb 9, 2017 •

edited

Loading

ddo commented Feb 22, 2017 •

edited

Loading

ddo commented Feb 22, 2017 •

edited

Loading