[EKS/fargate] : Support for ISTIO service mesh sidecars

[studees]

We are very keen to migrate our EKS workloads over to the recently released Fargate integration.

Unfortunately it appears as though it is not currently possible to deploy pods that use ISTIO injected sidecars, which seems to significantly limit the potential use of fargate.

The istio-proxy sidecar itself can be injected successfully but we've not managed to apply the subsequent iptables configuration to direct all traffic via the envoy proxy within the sidecar.

Istio supports two mechanisms for this:

1. istio-init container - this fails as under fargate the pod cannot be deployed within the required NET_ADMIN permission
2. istio-cni - this fails as we cannot schedule daemonsets on fargate, and even if we were to manually reconfigure as a sidecar then we cannot mount the required /opt/cni/bin host volume.

Does the migration to firecracker and the associated move of the fargate agent to outside the VM change the security posture for granting pods NET_ADMIN access? This appears to be the most flexible option, but we'd like to request support for network sidecar proxies be provided by whatever means...

Thank you!

[ofiliz]

Please note that Fargate already supports App Mesh as a managed solution for both ECS and EKS. Customers don't need to run any init containers or additional CNI plugins to setup any iptables rules, which also means that no privileges (NET_ADMIN etc.) are required.

P.S. in 2024: My comment above was in response to the initial issue which requested supporting init containers with NET_ADMIN privileges to run sidecar proxies for any mesh. The issue has since then been updated to ask specifically about ISTIO, hence the answer no longer matches the request.

[dracmic]

Hello @ofiliz ,

I am new in Fargate stuff ... So ask you directly maybe a stupid question: Can I use istio sidecar injector with Fargate?
Best regards.

[eightnoteight]

Hello @ofiliz ,

I am new in Fargate stuff ... So ask you directly maybe a stupid question: Can I use istio sidecar injector with Fargate?
Best regards.

hey @dracmic ,
yes, all components of istio which doesn't use istio-proxy are possible to run in fargate, only istio-proxy container needs NET_ADMIN to add the iptable rules

Please note that Fargate already supports App Mesh as a managed solution for both ECS and EKS. Customers don't need to run any init containers or additional CNI plugins to setup any iptables rules, which also means that no privileges (NET_ADMIN etc.) are required.

@ofiliz
it's awesome that app mesh as a managed solution is supported in EKS, but I think the support for istio is very much needed because app mesh doesn't offer as much flexibility as istio does in terms of custom protocols, adding filters and many more things.

fargate as a managed solution solves many of management and reliability problems of self-managing ec2 instances but restrictions like being unable to use istio with it is a huge blocker from adopting fargate in production workloads

[vorishirne]

the NET_ADMIN is a priviliged user, whom even I wouldnt allow if fargate does. Hence, Istio now provides ISTIO-CNI, we need to use it. https://github.com/istio/istio/tree/master/cni

[studees]

the NET_ADMIN is a priviliged user, whom even I wouldnt allow if fargate does. Hence, Istio now provides ISTIO-CNI, we need to use it. https://github.com/istio/istio/tree/master/cni

As far as I'm aware Fargate does not allow us to deploy CNI, or any other daemonset either.

Aside from that, relying on daemonsets for single pod worker nodes will be very inefficient, and exarcebates the existing race conditions of the CNI pod being scheduled & starting before the application pod is scheduled

[vorishirne]

@studees can we use this cni plugin from fargate aws/aws-app-mesh-roadmap#3
I can raise a pr in istio, but I am not aware of the aws side permissions.

[wmorgan]

This isn't Istio-specific, the same applies to @linkerd or to any service mesh that does injection via iptables.

[vorishirne]

Ya we can make atleast istio support locked cloud services

[wmorgan]

@Dhua-v it's not about Istio adding support for Fargate, it's about Fargate adding support for this type of pod network programming, which will allow any service mesh (or other tool) to do proxy sidecar injection.

[vorishirne]

@wmorgan aws seems to have provided this
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ProxyConfiguration.html

But I am not aws guy, cant say how we can utilize that.