net/http: RoundTrip unexpectedly changes Request

[neild]

https://go-review.googlesource.com/c/go/+/234894 causes (*http.Transport).RoundTrip to return a different *http.Request in the *http.Response than was passed to it, under some circumstances.

func main() {
        req, err := http.NewRequest("GET", "http://google.com/", bytes.NewBuffer([]byte{0, 1, 2}))
        if err != nil {
                log.Fatal(err)
        }
        resp, err := http.DefaultTransport.RoundTrip(req)
        if err != nil {
                log.Fatal(err)
        }
        fmt.Println(req == resp.Request) // false, where it used to be true.
}

The documentation for (http.Response).Request says, "Request is the request that was sent to obtain this Response." While this doesn't precisely specify that it's the exact *Request passed to RoundTrip, that's probably close enough to consider this an incorrect API change.

Discovered because this breaks code which keeps a map of *http.Requests.

[neild]

Even prior to this change, RoundTrip would change the Request, but only when calling GetBody to rewind the body:
https://go.googlesource.com/go/+/0e617d3d5c7e89b1ad1b0285fc77314b8d056211/src/net/http/transport.go#588

Tracking the history of this behavior, I believe it originates with https://golang.org/cl/33971, which causes the HTTP2 transport to replace the request when retrying.

My first thought was that it seems odd to replace the entire *Request rather than just updating its Body field, but the http.RoundTripper documentation states:

RoundTrip should not modify the request, except for consuming and closing the Request's Body.

Replacing the *Request in RoundTrip only under unusual circumstances (when calling GetBody to reset the body) is error-prone; users may depend on the *Request passing through unchanged, and only encounter infrequent errors. I think the *Request should be replaced consistently, or never.

My inclination is to say that "never" is the right answer, since it causes the least disruption to existing users.

If that's the case, we could:

• Change the RoundTripper contract to permit changing the request Body when GetBody is called. I think this is reasonable behavior, but I'm not certain if this would be consistent with our compatibility guarantees.
• Temporarily replace the request Body, but put the original one back before returning.
• Create a new Request when retrying, but return the original in the Response.

[rsc]

Requests are read-only data, immutable. I believe it is allowed today to call RoundTrip from multiple goroutines using the same request (provided Body = NoBody or nil). Mutating the Request would break that, introducing races.

The main time that Requests change and the reason Response records the final Request is when http.Client.Do follows redirects. That's the time when you most often need to look at the Response - to find out what the final fetched URL even was (if you care). Since they can change in Do, I'm not convinced it's important they can't change in RoundTrip.

Maybe we can say that the Response returned by RoundTrip always returns the original Request? I'm not sure. It's either that or decide the tests are broken. You can't mutate in place.

/cc @bradfitz

[gopherbot]

Change https://golang.org/cl/237560 mentions this issue: net/http: make Transport.RoundTrip preserve Requests

[neild]

We'd only need to mutate the Request if the Body is neither NoBody nor nil, which are the cases where you aren't allowed to call RoundTrip from multiple goroutines. If immutable requests are an invariant, though, then it is what it is.

[dmitshur]

@gopherbot Please backport to Go 1.14.

This is a fixup to a previous fix (CL 242117) for an approved backport issue #39279. We can't submit that CL without this one. This needs to be backported to Go 1.14 only, since the fix is already included in Go 1.1​5.

[gopherbot]

Backport issue(s) opened: #40973 (for 1.14).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[gopherbot]

Change https://golang.org/cl/249880 mentions this issue: [release-branch.go1.14] net/http: make Transport.RoundTrip preserve Requests