gRPC apps cannot be invoked without setting host explicitly

[ahmetb]

Problem: I have a set of gRPC microservices (written in Go, Python, Ruby, Node.js, .NET and Java) that work fine on Kubernetes. However if I migrate them to Knative, I have to make "code changes" to ALL of these services explicitly set the host header, ideally I wasn't hoping to do that.

When you deploy a gRPC app to Knative, you have to explicitly specify its hostname.

For example, if an app is calling into a gRPC service on Knative, it must do:

conn, err := grpc.Dial("grpc-ping.default.svc.cluster.local:80", 
    grpc.WithInsecure(),
    grpc.WithAuthority("grpc-ping.default.svc.cluster.local"))
...

Without specifying the WithAuthority dial option, the response always gives error:

rpc error: code = Unimplemented desc = :

On the other hand, if I wasn't deploying the gRPC app to Knative, I could just connect to it by saying:

conn, err := grpc.Dial("grpc-ping.default.svc.cluster.local:80", grpc.WithInsecure())

It would be great if running gRPC apps on Knative didn't come with this cost of explicitly re-specifying the host header. This is costly if I'm doing service discovery through a variable like OTHER_SERVICE_ADDR=grpc-ping.default.svc.cluster.local:80 as now I have to split the host/port. And I also have to refactor my code to work on Knative.

I recommend we:

• find a way to do this without requiring users to set Authority explicitly
• figure out who's sending gRPC unimplemented response and add an explicit error message to gRPC status like request doesn't specify an authority (Host header), knative cannot route this request so others don't spend multiple days debugging.

/area networking
/kind bug

[ahmetb]

/cc @mattmoor
/cc @tcnghia

[tanzeeb]

This is happening because the VirtualService created by cluster ingress explicitly only allows the hostnames:

$ kubectl describe virtualservices -n knative-serving
...
    Match:
      Authority:
        Exact:  grpc-ping.default
      Authority:
        Exact:  grpc-ping.default.example.com
      Authority:
        Exact:  grpc-ping.default.svc
      Authority:
        Exact:  grpc-ping.default.svc.cluster.local
...

In order to work with the ports, we'll need to expand this list:

    Match:
      Authority:
        Exact:  grpc-ping.default
      Authority:
        Exact:  grpc-ping.default.example.com
      Authority:
        Exact:  grpc-ping.default.svc
      Authority:
        Exact:  grpc-ping.default.svc.cluster.local
    Match:
      Authority:
        Exact:  grpc-ping.default:80
      Authority:
        Exact:  grpc-ping.default.example.com:80
      Authority:
        Exact:  grpc-ping.default.svc:80
      Authority:
        Exact:  grpc-ping.default.svc.cluster.local:80

The only complication here is that the port shouldn't be hard coded. It can be whatever is specified in the ingress gateway.

[ahmetb]

Are you positive that grpc implementations already send an Authority header (is it a thing, or is it just Host header) with a port name in the request? I'm actually not sure hostname info gets sent by default (although I'd expect it to be) in grpc client implementations.

[tcnghia]

@tanzeeb if grpc already sent this header we could probably add a RegExp match here to allow arbitrary port name.

[ahmetb]

Looks like indeed adding :80 to Authority is working. We just cloned the Istio VirtualService into a different one, prefixed hostnames with x- (to prevent collisions) and added :80 to all hostnames in authority field. It worked!

Looks like gRPC clients are passing hostname with :80 (this is visible in the istio-ingressgateway logs):

[2019-03-07T23:10:11.046Z] "POST /ping.PingService/PingHTTP/2" 200 NR 0 0 0 - "10.128.0.41" "grpc-go/1.17.0" "2eaa7ca1-1fcc-473e-b491-ab1fbd56334b" "grpc-ping.default.35.192.110.183.xip.io:80" "-" - - 10.20.3.3:80 10.128.0.41:35579

[tanzeeb]

Cool, I'll take a look

/assign

[tanzeeb]

Are you positive that grpc implementations already send an Authority header (is it a thing, or is it just Host header) with a port name in the request? I'm actually not sure hostname info gets sent by default (although I'd expect it to be) in grpc client implementations.

Yes. In HTTP 2.0, the :authority pseudo-header is used instead of the Host header, and this applies to gRPC as well. Most clients will just use your host as the authority if you don't specify the authority separately.

conn, err := grpc.Dial("grpc-ping.default.svc.cluster.local:80", grpc.WithInsecure())

Since your client call used grpc-ping.default.svc.cluster.local:80 as the host, that's the authority it passed.

[ahmetb]

@tcnghia @tanzeeb so while this app was working on kubernetes, I was doing service discovery like

            env:
              - name: PRODUCT_CATALOG_SERVICE_ADDR
                value: "productcatalogservice:80"
              - name: CURRENCY_SERVICE_ADDR
                value: "currencyservice:80"
              - name: CART_SERVICE_ADDR
                value: "cartservice:80"

however I'm realizing these hostnames have to be fully-qualified like cartservice.default.cluster.local on Knative.

Is there a reason why we're not adding ALL OF these?

• servicename.ns.cluster.local
• servicename.ns.cluster.local:80
• servicename.ns.cluster
• servicename.ns.cluster:80
• servicename.ns
• servicename.ns:80
• servicename
• servicename:80

as the authority?

I can see that [servicename] would cause collisions if two services named the same on multiple namespaces on Istio-Ingressgateway, but I guess this is just something that creates friction from migrating to Kubernetes -> Knative.

[tcnghia]

@ahmetb I don't think servicename.ns.svc.cluster would work in K8s either? If you can confirm that such work we can certainly add it to the list.

Other than that the only thing that left is servicename which would depend on which namespace the traffic comes from. We currently can't support this yet (we used to but have to undo the change because it cause collision and broke one or both services, in the case of something like helloworld-go deploying to two namespaces)