SPDY is deprecated. Switch to HTTP/2.

[lavalamp]

Filing for tracking purposes.

See #7392 where the spdy package we use doesn't even exist any more. (And btw it has bugs anyway, possibly security bugs-- see the trophies here: https://github.com/dvyukov/go-fuzz)

Blocked on github.com/docker/spdystream switching to HTTP/2.

(Note: HTTP/2 is based on SPDY, with just a few differences-- hopefully this transition will be easy.)

@thockin @ncdc

[ncdc]

Waiting on HTTP/2 support for Go that includes low level client/server ability to create and handle streams (https://github.com/bradfitz/http2).

moby/spdystream#53 pulls the upstream spdy code into spdystream.

[lavalamp]

Ah, after moby/spdystream#53 is merged, we can at least make godep happy again. :)

[ncdc]

@lavalamp spdystream PR is merged, FYI

[ncdc]

@thockin @lavalamp it looks like the Go HTTP/2 issue was just closed (golang/go#6891 (comment)). I glanced quickly through the code, but I don't see a way to hijack the connection and create our own streams, like we do with spdystream. Hopefully we can get this functionality in there at some point...

[ncdc]

cc @smarterclayton

[ncdc]

I created golang/go#13444 as an RFE for end-user stream support

[ddysher]

What's the plan for the deprecation? We are using nginx in front of apiserver which drops support for SPDY since version 1.9.5. Our workaround is to use an older nginx, but would like to see http/2 support go upstream :)

[ncdc]

I prototyped this with a prerelease version of go1.6. It worked somewhat, but there were some rough spots (probably stuff I was doing incorrectly). We can't move to http/2 until go1.6 at the earliest, and even then we'll have to do R&D to see if it's technically feasible. I had been hoping that the http/2 support in go1.6 would allow me to create streams just like we can do in spdystream, but that isn't exposed via the API. For my prototype, I ended up writing a simple multiplexer just like @smarterclayton wrote for the websocket support we have in Kubernetes now.

[ddysher]

Thanks for the heads up. Looking at the linked comments, there is no intention to add the support in go1.6, so I guess it's probably still months away.

[JensRantil]

We can't move to http/2 until go1.6 at the earliest

Golang 1.7 is out now. Does that mean we can make progress on this issue now? Anything else stopping us?

[smarterclayton]

A usable HTTP/2 framer in Go that allows us hijack control and the ability
to stand up the connection. Last I saw we were still blocked without
explicitly forking the Go implementation.

In the meantime, websockets continue to be an option (with portforward
being the last bit remaining).

On Oct 29, 2016, at 2:22 PM, Jens Rantil [email protected] wrote:

We can't move to http/2 until go1.6 at the earliest

Golang 1.7 is out now. Does that mean we can make progress on this issue
now? Anything else stopping us?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7452 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p-fe3BrfKe4VjAQyYLOgK7AjvhLkks5q447ngaJpZM4EK7sh
.

[ncdc]

We should be able to use HTTP/2 without any changes to the Go implementation. But we would have to write our own code to do multiplexing of multiple data streams over a single request/response body pair, which we'd rather not do if we can avoid it.

[bradfitz]

@ncdc, if it helps, https://godoc.org/golang.org/x/build/revdial is used by the Go build system to multiplex multiple net.Conns and a net.Listener all over a single net.Conn.

[fraenkel]

@bradfitz Thanks for the pointer, but it doesn't fix the underlying issue. While SPDY is being used, it is not following HTTP request/response semantics. A logical connection is created and then data is sent in either direction as needed. It would be equivalent to the client sending an HTTP request and the server doing a Push whenever.

[ncdc]

@fraenkel I don't think there's any technical limitation in the facilities provided by the Go HTTP/2 implementation. As I wrote above, we would just need to write our own muxing/framing code, which we'd rather avoid (or find a library that can do it?).

[fraenkel]

@ncdc It might be easier to just build on top of gRPC which provides bi-directional streaming or copy what they have done.

[smarterclayton]

There's definitely benefit to implementing a gRPC endpoint vs continuing to have to maintain a franken-protocol. That is tied up with the CRI changes though.

…

On Mon, Nov 28, 2016 at 11:25 AM, Michael Fraenkel ***@***.*** > wrote: @ncdc <https://github.com/ncdc> It might be easier to just build on top of gRPC which provides bi-directional streaming or copy what they have done. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7452 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p4YovgyT-NPE9MilgvO0kf_CaSksks5rCwB4gaJpZM4EK7sh> .

[luxas]

Ping, any update on this?

[wlan0]

We at Rancher Labs are also interested in this change.

[smarterclayton]

We can actually just move to web sockets in the near term. The server has supported it for a while and HOL blocking is not an issue

[lavalamp]

Thanks -- at least some of our protocols are already available via websockets and that's a good reason to go all in on that.

Note that there is also an apiserver <-> kubelet / container runtime leg of some calls (exec, attach), and this also needs to not use SPDY any more, and it doesn't need to go through a proxy. So that needn't use websockets (although it could).

[sfxworks]

I believe that would work out of the box with cloudflare

[slandelle]

some calls (exec, attach) ... and it doesn't need to go through a proxy

@lavalamp Why is that?

[danopia]

I'm a user of Kubernetes so the below is my own viewpoint as an API consumer, not a Kubernetes contributor.

some calls (exec, attach) ... and it doesn't need to go through a proxy

Why is that?

When the API server makes HTTP calls to a particular kubelet, the connection is generally made without an HTTP proxy in the way; the kubelets each have their own TLS certificates and the apiserver checks this encryption, so installing an HTTP proxy in between would introduce MITM-like security concerns.

This connectivity of course may need to be relayed e.g. across the internet and this is done at the TCP level. For example GKE runs the apiserver->kubelet traffic through long-lived SSH tunnels, and the new system to replace those SSH tunnels, Konnectivity, is also a TCP-level proxy.

---

The client -> apiserver traffic is definitely more important for compatibility and this is where Websockets are already offered. Unfortunately the websocket interop isn't perfect:

1. The new WebSocket() constructor in browsers don't allow attaching an Authorization header, so token auth isn't possible from an actual web client.
2. Each WebSocket is historically its own TCP connection, so kubectl port-forward functionality would need to open a new TCP connection to the API Server for each new tunnel client if it were using WebSockets. RFC 8441 addresses this, but similarly, nginx seems to have no plans to implement websocket over HTTP2.

I've been working on a WebSocket client to Kubernetes and you can get things done with it. If the websocket subprotocol used by Kubernetes was extended to support functions like dynamically opening and closing port-forward streams, I could see it replacing SPDY. I just don't know if creating a more special-cased subprotocol in Websocket would be worth it vs. leveraging the http2 standard.

[lavalamp]

Thanks for the websocket feedback. I don't know if anyone is working on this right now, though. It may be a good idea to file a separate issue about that for visibility.

[pires]

@danopia doesn't it make sense for the Authorization header go side-by-side with an Upgrade header in the same HTTP request to kube-apiserver and then, provided the requestor is authorized to proxy, upgrade to Websocket (not needing the Authorization header)? nevermind, it seems I misunderstood what you said.

[aeriksson]

The new WebSocket() constructor in browsers don't allow attaching an Authorization header, so token auth isn't possible from an actual web client.

FYI it seems like there's a way to do this using the websocket subprotocol header: #47740

[danopia]

FYI it seems like there's a way to do this using the websocket subprotocol header: #47740

That hack looks questionable at best, despicable or insecure at worst, and I'm sure not at all documented. (If the Kubernetes websocket subprotocols are documented somewhere I haven't seen it)

I've verified that the hack works for my usecase. Thanks for the link. 😄

[maradwan]

I solved the issue, see my article
https://dev.to/aws-builders/kubectl-execport-forward-with-aws-alb-and-nginx-ingress-controller-4fn0

[anders-swanson]

Hi, is this issue actively being worked on?

[lavalamp]

No --

We know the user facing path should be swapped out for websockets, but I don't know if there is any consensus on what should happen for the apiserver -> kubelet path.

[GreenCappuccino]

kubernetes/enhancements#3401 was closed, noting issues with implementing an application-layer multiplexing system for a websocket stream.

I'm really unsure as to how well this is ready for the k8s ecosystem, but it might be worth keeping an eye on WebTransport (W3C), as it seems to natively support many of the desired features missing from WebSockets (multiplexing, bidirectional+unidirectional streams). However, afaik it's still being standardized, and depends on HTTP/3 (QUIC), which is (newer?) to k8s.

Notably, WebTransport should support the HTTP/3 (QUIC) STOP_SENDING and RESET_STREAM which (might?) implement what the KEP was looking for.

[dims]

xref: #89163

[lavalamp]

BTW, the title of this issue scans to "Rudolph the red nosed reindeer".

[sftim]

Although WebTransport is not yet standardized, we do have both WebSocket and WebRTC Datachannels as (fairly) stable things.

WebSockets over HTTP/3 over QUIC is standardized albeit with few browser implementations. That could nonetheless work fine for communication between code we write (eg kubectl talking to the Kubernetes API server), with a fallback to an earlier HTTP version and TCP.

[aojea]

I'm going to close this, since KEP https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/4006-transition-spdy-to-websockets moved to websockets that is widely available I don't think we are going to do another protocol change

/close

[k8s-ci-robot]

@aojea: Closing this issue.

In response to this:

I'm going to close this, since KEP https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/4006-transition-spdy-to-websockets moved to websockets that is widely available I don't think we are going to do another protocol change

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.