Configure OutlierDetectionLoadBalancer for GrpcClient

[aditya-32]

Hi i'm doing a POC for outlierDetection using grpc with xds for a grpc client, so wanted to use the xds OutlierDetection feature (link) for the client , but cannot find any resource on how to use the OutlierDetectionLoadBalancer present in io.grpc.util. Is there a way on how to use it and provide it as loadBalancerPolicy in Managed Channel, Also how to provide the service_config including loadbalancer config to the client while creating channels.

[ejona86]

Cross-linking: https://stackoverflow.com/questions/77458419/configure-outlierdetectionloadbalancer-for-grpc-client

[temawi]

Thanks for your question @aditya-32. If you are working on a client that is driven by a configuration provided by a control plane (seems like Istio in your case), the outlier detection configuration would be included in the xDS data from the control plane. You should not need to do anything in your client code. The service_config you see mentioned is generated based on the xDS configuration received from the control plane.

As long as Istio provides outlier detection configuration in the way specified in the gRFC you linked to (it should), you can just add your settings on the Istio side. I have not tried Istio with outlier detection, but I did find this page with details on how to configure it. Maybe that is helpful.

Let us know if you get it to work!

[aditya-32]

@temawi i'm using proxyless grpc(no envoy), i tried to configure using DestinationRule but it was unable to trigger the outlierDetection for the server even when i was deliberately returning 5xx and the fields mentioned in gRFC like the failure_percentage_ejection. cannot be configured in DestinationRule right right?. The gRFC also mentioned xds with grpc supports only 2 Algorithm failure_percentage_ejection or success_rate_ejection how to configure these two?

used this DestinationRule but could not trigger OutlierDetection

kind: DestinationRule
metadata:
  name: grpc-server-destination
  namespace: istio-poc
spec:
  host: grpc-server.istio-poc.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1
      http:
        http2MaxRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      baseEjectionTime: 3m
      consecutive5xxErrors: 1
      interval: 1s
      maxEjectionPercent: 100

[ejona86]

Triggering via 5xx is not supported. Taking a look at the overall xds feature tracker (which is generally easier to follow than the gRFC), success rate and failure percentage are the only triggers. Looking at Istio, I don't see an overlap between gRPC and Istio's support. @temawi, do you see anything?

[aditya-32]

@ejona86, how should I configure the outlierDetection in grpc with xds? since the DestinationRule doesn't understand success rate and failure percentage algo.

[temawi]

My apologies, for some reason I thought I saw success rate/failure percentage in the Istio doc, but indeed it doesn't look like Istio allows specifying those outlier detection algorithms.

@aditya-32 since this seems to be a gap between the xDS spec and what Istio has implemented, short of changing your control plane, you probably should pursue getting this added in Istio.

[ejona86]

I think it may be possible to configure Istio using an EnvoyFilter. I have little clue what I'm doing (I've only tried once before), but something in the shape of:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: yolo
spec:
  workloadSelector:
    labels:
      app: myapp
  configPatches:
  - applyTo: CLUSTER
    match:
      cluster:
        # See https://istio.io/latest/docs/reference/config/networking/envoy-filter/#EnvoyFilter-ClusterMatch
        service: foo
    patch:
      operation: MERGE
      value:
        outlier_detection:
          enforcing_success_rate: 50 # 50%

[anuragagarwal561994]

@ejona86 we have tried different ways but the issue is we are getting eds updates and we are not able to see outlier_detection config in the same, while with envoy proxy we are able to see the configuration properly getting to the cluster.

In the proto message when I tried to check the data I am not seeing the outlier_detection config, any clue on what should we check further?

[anuragagarwal561994]

@ejona86 I think I understood what is happening

https://github.com/istio/istio/blob/b2338d2c38b0d42ce1604b14a98bf0286c1c48a9/pilot/pkg/networking/grpcgen/cds.go

Here we are getting only the eds updates which is filtering out the outlier_detection config

[anuragagarwal561994]

@ejona86 I think we can close this issue, since the support needs to be added via istio