Private link for API server internal load balancer

[nprokopic]

/kind feature

Describe the solution you'd like
[A clear and concise description of what you want to happen.]

In order to access private workload clusters, i.e. workload clusters deployed to private networks where API server is exposed via internal load balancer and accessible only inside of the VNet, some form of private and secure connectivity must me established to the workload cluster.

Currently CAPZ supports VNet peering and e.g. a managment cluster VNet and workload cluster VNets can be peered, which enables the connectivity between the two. While probably the most used and th easiest to setup, VNet peering is not the only way to get connectivity to the private workload cluster.

Even more private, more secure and more precise (more narrowed down access) way can be to use Private Endpoint and Private Link setup. This would be achieved in a following way:

• Private Link Service is deployed in the workload cluster. It exposes access to the API server internal load balancer.
• Consumers of the workload cluster API server then deploy Private Endpoints which can connect to the workload cluster API server's Private Link.
  • One example for this can be a management cluster that has a Private Endpoint (alrady supported by CAPZ) that is connecting to the workload cluster Private Link which exposes API server. This way, in addition to VNet peering, CAPZ would have an additional and more secure feature that enables complete deployment of all Azure network resources that are needed for private clusters to be deployed in private VNets and for them to be accessible from management clusters.

Intro to Private Links from Azure docs:

Azure Private Link enables you to access Azure PaaS Services (for example, Azure Storage and SQL Database) and Azure hosted customer-owned/partner services over a private endpoint in your virtual network.

Traffic between your virtual network and the service travels the Microsoft backbone network. Exposing your service to the public internet is no longer necessary. You can create your own private link service in your virtual network and deliver it to your customers. Setup and consumption using Azure Private Link is consistent across Azure PaaS, customer-owned, and shared partner services.

...
Extend to your own services: Enable the same experience and functionality to render your service privately to consumers in Azure. By placing your service behind a standard Azure Load Balancer, you can enable it for Private Link. The consumer can then connect directly to your service using a private endpoint in their own virtual network. You can manage the connection requests using an approval call flow. Azure Private Link works for consumers and services belonging to different Azure Active Directory tenants.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

CAPZ already has support for Private Endpoints, implemented in this PR by @adriananeci.

In that PR, there was #3044 (comment) from @CecileRobertMichon:

wondering if this is something we should look into as a potential improvement for "private" clusters

With Private Links support in CAPZ we would get that improvement and both sides of the Private Endpoint and Private Links setup.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[nprokopic]

/remove-lifecycle stale

[nawazkh]

Thank you, @nprokopic, for posting such a detailed pitch on adding Private Link support in the Workload cluster. The links you posted were helpful; I appreciate your thoroughness. 👍🏼

I have a few questions for my clarity. While I understand the benefits of having Azure Private Link added to CAPZ, I am still trying to understand the business scope of this addition.
Will this (Azure Private Link) addition to CAPZ give users a better way to access Azure Services?
How is VNet Peering lacking in this scenario?

Your doc snippet does help in answering my questions, but I think I would appreciate another take on my doubts.

Intro to Private Links from Azure docs:

Azure Private Link enables you to access Azure PaaS Services (for example, Azure Storage and SQL Database) and Azure hosted customer-owned/partner services over a private endpoint in your virtual network.

Traffic between your virtual network and the service travels the Microsoft backbone network. Exposing your service to the public internet is no longer necessary. You can create your own private link service in your virtual network and deliver it to your customers. Setup and consumption using Azure Private Link is consistent across Azure PaaS, customer-owned, and shared partner services.

...
Extend to your own services: Enable the same experience and functionality to render your service privately to consumers in Azure. By placing your service behind a standard Azure Load Balancer, you can enable it for Private Link. The consumer can then connect directly to your service using a private endpoint in their own virtual network. You can manage the connection requests using an approval call flow. Azure Private Link works for consumers and services belonging to different Azure Active Directory tenants.

[nprokopic]

Will this (Azure Private Link) addition to CAPZ give users a better way to access Azure Services?

This is about a better way to access private workload cluster API server (behind Internal Load Balancer with private IP).

Private endpoints (already implemented in CAPZ) are giving the better way to access Azure services (by connecting to Private Links of those services, which is an Azure feature). From the cluster's point of view that's about outbound traffic, accessing something from the workload cluster and the workload cluster being the consumer of some service, while this issue is about the other side of the coin, it's about inbound traffic and the workload cluster being the service provider (where the service provided is the workload cluster API server and Azure Private Link enables that).

Currently, if users want to deploy private clusters with Internal Load Balancer for the API server, they have limited options to access that workload cluster then, e.g.:

• Management cluster must be in the same or in a peered VNet (documented here)
• Any other Azure workload (e.g. another CAPZ cluster) that wants to access the private workload cluster API server also must be in the same or in a peered VNet.
• In order to access the private workload cluster from on-prem, you need:
  • VPN gateway in every workload cluster, which really does not scale (too slow to deploy, too expensive), or
  • VPN gateway in a central (Hub) VNet, and then peering with gateway transit to workload cluster VNets (spokes). Here then there are other limitations, e.g. workload clusters cannot have overlapping VNets, as they are all peered to the same Hub VNet, then workload cluster VNets cannot be peered to other VNets that also have VPN gateway, etc.

Azure Private Link will give CAPZ users a better, more scaleable, more private and secure way to access a private CAPZ workload cluster (when Internal Load Balancer is used for API server) by exposing API server Internal Load Balancer via Azure Private Link. Once the Private Link for the Internal Load Balancer is deployed, the workload cluster API server consumers can connect to it via a Private Endpoint that connects to the Private Link.

With Azure Internal Load Balancer being exposed and accessible via Azure Private Link, the workload cluster API server can be accessed more securely because the traffic from the private endpoint to the private link goes via Azure backbone network, and in this scenario VNet peering is not needed to access workload cluster API server.

This article explains some benefits of Azure Private Link compared to VNet peering, e.g.:

Overlapping address space: Traditionally, customers use VNet peering as the mechanism to connect multiple VNets. VNet peering requires the VNets to have non-overlapping address space. In enterprise use cases, its often common to find networks with an overlapping IP address space. Azure Private Link provides an alternative way to privately connect applications in different VNets that have an overlapping IP address space.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[nawazkh]

/remove-lifecycle stale