net: add support for TCP Fast Open

[bradfitz]

TCP listeners and TCP client connections should both be able to conditionally enable
TCP_FASTOPEN.

See https://lwn.net/Articles/508865/

This is now available in Linux 3.7 (both client & server).

This would be nice to conditionally use in http.DefaultTransport, for GET requests.

See also: issue #3097 (adding Timeouts to DialTCP)
See also: issue #3610 (DialPlan)

[alexbrainman]

Comment 2:

Probably unrelated, but FYI. ConnectEx and AcceptEx we use to dial / accept tcp
connections allow to send / receive some data as part of the same api call.
Alex

[bradfitz]

Comment 3:

Re #2: IIUC, that's related, but against spec. TCP permits such a packet, but also says
that the recipient isn't supposed to act on it until the connection is verified, since
the initial SYN packet could've had its source IP/port spoofed. TCP_FASTOPEN is a
formalized version of that which isn't susceptible to spoofing.
I never knew that AcceptEx makes that optional. I suppose a Windows webserver could use
AcceptEx w/ data and verify first it's a GET request to a public resource and send the
response immediately.  I remember IE / Microsoft doing this years ago (to win HTTP
benchmarks?), but I didn't remember how. Interesting.

[bradfitz]

Comment 4:

Now that DialOpt is in (https://code.google.com/p/go/source/detail?r=54731addb542 /
docs: http://tip.golang.org/pkg/net/#DialOpt), this is easier.
There are two API possibilities.
a) lazy.
b) explicit.
In the lazy API, the connect would be like:
  conn, err := net.DialOpt("google.com:80", net.FastOpen())
... and a fake net.Conn is returned immediately.  No possibility for timeouts or errors.
 Not until the first Write does the actual sendto system call (on Linux) get called. 
This is what Chrome does.
With the explicit API alternative (my preference), the API will probably be like: 
  conn, err := net.DialOpt("google.com:80", net.InitialData("GET / HTTP/1.0\r\n\r\n"))
Where net.InitialData returns a DialOption (http://tip.golang.org/pkg/net/#DialOption)
that enables TFO if the client supports it (on Linux: checking the low bit of
/proc/sys/net/ipv6/tcp_fastopen) and otherwise just does a Write after connect.
Speaking with Yuchung Cheng, it seems there's enough momentum beh that something like it
will be formalized one way or another, it will be formalized one way or another.  And
Linux 3.8 supporting it (client & server) is promising (and Windows in an alternate
form).  Thus I'd like to keep "TFO" or "TCPFastOpen" out of the Go API name and make the
DialOption be just net.InitialData(...).  Up for debate whether that's a []byte, string,
or io.Reader.  Probably a []byte for consistency with Write.

[bradfitz]

Comment 5:

This issue was updated by revision abf5700.

R=golang-dev, r
CC=golang-dev
https://golang.org/cl/7369060

[rsc]

Comment 6:

[The time for maybe has passed.]

Labels changed: removed go1.1maybe.

[mikioh]

Comment 7:

Labels changed: added go1.3maybe.

[rsc]

Comment 9:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 10:

Labels changed: added repo-main.

[mikioh]

Comment 11:

Status changed to Thinking.

[mikioh]

Comment 12:

Status changed to New.

[mikioh]

Published, at last: http://tools.ietf.org/html/rfc7413

[JunfengJia]

how to make tfo work with http package? http's persist cache depend on established connections.

[bronze1man]

@JunfengJia you can use syscall to dial a new connection in http.Transport to work with http package.
Here is the tfo library I used:
http://godoc.org/github.com/bronze1man/kmg/kmgNet#TfoLazyDial

[mikioh]

@JunfengJia,

Still thinking. We need to handle the entire stack, TCP through HTTP 1.x, HTTP/2, or WebSocket w/ or w/o TLS, effectively. Once HTTP/2 support lands, I will revisit this issue.

[JunfengJia]

@bronze1man,

http will make a read call first, then the connect will fall into 3whs.

[bronze1man]

@JunfengJia

Thanks.It looks like my library will not work with net/http library right now.
Read or write call first is a complex stuff.I have waited 1ms to make sure write call come first from client with my own defined protocol.

[mikioh]

FWIW, old changelist 27150044 has moved to https://go-review.googlesource.com/#/c/7121/.

[bradfitz]

In addition to TFO, we need to figure out APIs for crypto/tls TLS 1.3 0-RTT stuff. And then the net/http client can use them both.

/cc @agl

[mikioh]

The change golang.org/cl/7121 is ready to land. However I'm not sure whether crypto/tls and net/http packages really need this feature. Please ping golang.org/cl/7121 when the time comes.

[bradfitz]

The change golang.org/cl/7121 is ready to land.

It says "DO NOT REVIEW" so I stopped reviewing.

[mikioh]

Thanks for the notice, just did s/DO NOT REVIEW/FOR DISCUSSION/

[gopherbot]

CL https://golang.org/cl/7121 mentions this issue.

[riobard]

Are we going to get this in GO 1.11?

[bradfitz]

@riobard, no. This bug is still open.

[AudriusButkevicius]

Isn't fastopen just a socket option that should be possible in go 1.11 atleast on linux?
For this to be closed, does it need to be solved on all platforms?

[bradfitz]

Yeah, this bug was about adding nice, portable API for it (ioctls aren't great API), and using it where it makes sense (the top comment mentions net/http for instance)

But it's true that in Go 1.11 you could use https://tip.golang.org/pkg/net/#ListenConfig

[rumpelsepp]

@AudriusButkevicius

For the server yes. The client needs to send the initial data with sendto(2), continuing with read(2)/write(2). connect(2) triggers a three way handshake instead.

UPDATE: I missed this one: torvalds/linux@19f6d3f

[xqdoo00o]

Are we going to get this in GO 2?

[bradfitz]

This has little if anything to do with Go 2.

[qJkee]

Any progress?
Or it's freezed for a long time?

[acln0]

Hello. Is there still interest in this?

Given the flow described in torvalds/linux@19f6d3f, assuming support for TFO only for Linux 4.11+, the code could be simplified greatly, compared to what CL 7121 does. Adoption in other places in the standard library would also be easier, since caller code would not need to change.

@mikioh, @bradfitz: would it be okay if I tried this from scratch in the upcoming cycle? It looks like https://go-review.googlesource.com/c/go/+/7121/ would need a pretty complicated rebase to bring it into the present.

[bradfitz]

@acln0, oh, nice! I hadn't seen that. Given that new API, this is basically trivial. In fact, people can do it already today with Go 1.11 and its new:

https://tip.golang.org/pkg/net/#Dialer.Control

Then you can just do the setsockopt(s, IPPROTO_TCP, TCP_FASTOPEN_CONNECT ...) and give that net.Dialer to net/http, etc.

(Of course, if you do so, you should only use that http.Client for GET/HEAD/idempotent requests.)

But sure, you could work on it.

[mikioh]

@acln0,

If you are talking about stuff in the net package, probably still not the time. If net/http, crypto/tls, or net/http over crypto/tls, it should be another issue and probably it's fine unless you try to introduce a) a quick hack for message boundary handling; especially without considering byte-seq.+partial read/writes vs. datagram/chunks+full reads/writes, b) a quick hack for multipath transport functionality.

[acln0]

@bradfitz, I had the net/http use case for idempotent requests in mind. I don't know the intricacies of how crypto/tls might make use of TFO in the larger context of TLS 1.3 etc., so I cannot comment on that.

Yes, people can do it at the net.Dialer level themselves today, but I thought there might be some value in adding options to the net package as well. If the standard library is going to use TFO, my thought was the following: instead of making net/http and crypto/tls set socket options themselves, we could try to introduce FastOpen knobs to net.Dialer and net.Listener, to consolidate that code in one well defined place.

Once the new API existed in the net package, both the standard library and other clients could start adopting it, at whichever pace they like. I wrote my initial message in this bug because I was interested in giving the ground work in package net another try.

@mikioh, can you please clarify what you mean about "stuff in the net package, probably still not the time"? Do you mean the possibility of FastOpen knobs on net.Dialer and net.Listener? It seems to me like Linux and FreeBSD support both passive and active sides pretty easily. And OS X supports active side too. Do you think this level of support across operating systems is not wide enough to warrant an option in package net?

[mikioh]

@acln0,

Do you mean the possibility of FastOpen knobs on net.Dialer and net.Listener?

Yes.

FreeBSD support both passive and active sides pretty easily

As far as I can see, on FreeBSD, a few GENERIC conf files include TCP_RFC7413 option by default from 12-STABLE, but not all, also unlike passive-open side operation, enabling active-open operation still needs sysctl tweaking.

find freebsd/sys/conf -name GENERIC | xargs grep TCP_RFC7413
amd64/conf/GENERIC:options            TCP_RFC7413             # TCP Fast Open
powerpc/conf/GENERIC:options  TCP_RFC7413             # TCP Fast Open
arm64/conf/GENERIC:options            TCP_RFC7413             # TCP Fast Open

And OS X supports active side too.

As far as I can see, it still supports active-open operation only and requires connectx thru syscall.Syscall9.

this level of support across operating systems is not wide enough to warrant an option in package net?

Still a bit early to me. If it's pretty fine within the real world, it's better to be enabled by default (and we should avoid introducing a control knob each time we need to fix something user uncomfortable on the operation of TCP Fast Open option, including configuring cookie-related life-duration/invalidation/capacity control.)

[qJkee]

@brain

@acln0, oh, nice! I hadn't seen that. Given that new API, this is basically trivial. In fact, people can do it already today with Go 1.11 and its new:

https://tip.golang.org/pkg/net/#Dialer.Control

Then you can just do the setsockopt(s, IPPROTO_TCP, TCP_FASTOPEN_CONNECT ...) and give that net.Dialer to net/http, etc.

(Of course, if you do so, you should only use that http.Client for GET/HEAD/idempotent requests.)

But sure, you could work on it.

Any more complete example?
all setsockoptByte(Inet|ICMP) require fd, but we got only syscall.RawConn, or it can be casted to fd(int)?

[acln0]

@qJkee, call the Control method on the syscall.RawConn. Its argument is a function of the integer file descriptor. You can call setsockopt in that function.

[qJkee]

@acln0 thank you! I will try to implement it asap, and maybe post some bech.

[bradfitz]

I'm going to close this. No longer desired.

See:
https://squeeze.isobar.com/2019/04/11/the-sad-story-of-tcp-fast-open/