Skip to content

Add Private Link support #6274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
robinkb wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add Private Link support #6274

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions api/v1beta1/consts.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,8 @@ const (
NetworkInterfaceReadyCondition clusterv1beta1.ConditionType = "NetworkInterfacesReady"
// PrivateEndpointsReadyCondition means the private endpoints exist and are ready to be used.
PrivateEndpointsReadyCondition clusterv1beta1.ConditionType = "PrivateEndpointsReady"
// PrivateLinksReadyCondition means the private links exist and are ready to be used.
PrivateLinksReadyCondition clusterv1beta1.ConditionType = "PrivateLinksReady"
// FleetReadyCondition means the Fleet exists and is ready to be used.
FleetReadyCondition clusterv1beta1.ConditionType = "FleetReady"
// AKSExtensionsReadyCondition means the AKS Extensions exist and are ready to be used.
Expand Down
54 changes: 54 additions & 0 deletions api/v1beta1/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,9 @@ type LoadBalancerSpec struct {
// FrontendIPsCount specifies the number of frontend IP addresses for the load balancer.
// +optional
FrontendIPsCount *int32 `json:"frontendIPsCount,omitempty"`
// PrivateLinks to the load balancer (max 8 private links).
// +optional
PrivateLinks []PrivateLink `json:"privateLinks,omitempty"`
Copy link
Copy Markdown
Contributor

@willie-yao willie-yao Jun 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since PrivateLinks is only for internal LBs, I think it would help to have the webhook reject this field being set for node/control plane outbound lbs. You can maybe add something like this to validateNodeOutboundLB and validateControlPlaneOutboundLB:

if len(lb.PrivateLinks) > 0 {
     allErrs = append(allErrs, field.Forbidden(
         fldPath.Child("privateLinks"),
         "privateLinks are only supported on the API server load balancer",
     ))
 }

// BackendPool describes the backend pool of the load balancer.
// +optional
BackendPool BackendPool `json:"backendPool,omitempty"`
Expand Down Expand Up @@ -415,6 +418,57 @@ type IPTag struct {
Tag string `json:"tag"`
}

// PrivateLink configures an Azure private link.
type PrivateLink struct {
// Name of the private link.
// +optional
Name string `json:"name,omitempty"`

// NATIPConfigurations specify up to 8 NAT IP configurations for the private link.
NATIPConfigurations []PrivateLinkNATIPConfiguration `json:"natIPConfigurations"`

// LBFrontendIPConfigNames are the names of the load balancer FrontendIP to which the private link will forward
// requests. The specified frontend IP configs must have the private IP set.
LBFrontendIPConfigNames []string `json:"lbFrontendIPConfigNames"`

// AllowedSubscriptions is a list of subscriptions from which the private link can be accessed.
// +optional
AllowedSubscriptions []*string `json:"allowedSubscriptions,omitempty"`

// AutoApprovedSubscriptions is a list of subscription for which the connections to private link are automatically
// approved.
// +optional
AutoApprovedSubscriptions []*string `json:"autoApprovedSubscriptions,omitempty"`

// EnableProxyProtocol indicates whether the private link service is enabled for proxy protocol or not.
// +optional
EnableProxyProtocol *bool `json:"enableProxyProtocol,omitempty"`
}

// PrivateLinkNATIPConfiguration specifies NAT IP configuration for the private link.
type PrivateLinkNATIPConfiguration struct {
// AllocationMethod specifies how the private link NAT IPs are allocated: "Static" or "Dynamic".
AllocationMethod string `json:"allocationMethod"`
Copy link
Copy Markdown
Contributor

@willie-yao willie-yao Jun 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this use the PrivateLinkNATIPAllocationMethod type defined below instead of a string?


// Subnet from which the IP is allocated.
Subnet string `json:"subnet"`

// +optional
PrivateIPAddress string `json:"privateIPAddress,omitempty"`
}

// PrivateLinkNATIPAllocationMethod specifies whether the private link NAT IPs are allocated statically or dynamically.
// +kubebuilder:validation:Enum=Static;Dynamic
type PrivateLinkNATIPAllocationMethod string

const (
// NATIPAllocationMethodStatic specifies that NAT IP for private link is allocated statically (manually set by user).
NATIPAllocationMethodStatic PrivateLinkNATIPAllocationMethod = "Static"

// NATIPAllocationMethodDynamic specifies that NAT IP for private link is allocated dynamically (by Azure).
NATIPAllocationMethodDynamic PrivateLinkNATIPAllocationMethod = "Dynamic"
)

// VMState describes the state of an Azure virtual machine.
//
// Deprecated: use ProvisioningState.
Expand Down
74 changes: 74 additions & 0 deletions api/v1beta1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

57 changes: 55 additions & 2 deletions azure/scope/cluster.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ import (
"sigs.k8s.io/cluster-api-provider-azure/azure/services/natgateways"
"sigs.k8s.io/cluster-api-provider-azure/azure/services/privatedns"
"sigs.k8s.io/cluster-api-provider-azure/azure/services/privateendpoints"
"sigs.k8s.io/cluster-api-provider-azure/azure/services/privatelinks"
"sigs.k8s.io/cluster-api-provider-azure/azure/services/publicips"
"sigs.k8s.io/cluster-api-provider-azure/azure/services/routetables"
"sigs.k8s.io/cluster-api-provider-azure/azure/services/securitygroups"
Expand Down Expand Up @@ -452,8 +453,16 @@ func (s *ClusterScope) SubnetSpecs() []azure.ASOResourceSpecGetter[*asonetworkv1
numberOfSubnets++
}

subnetSpecs := make([]azure.ASOResourceSpecGetter[*asonetworkv1api20201101.VirtualNetworksSubnet], 0, numberOfSubnets)
ipConfigSubnetNames := make(map[string]struct{})
if apiServerLB := s.AzureCluster.Spec.NetworkSpec.APIServerLB; apiServerLB != nil {
for _, privateLink := range s.AzureCluster.Spec.NetworkSpec.APIServerLB.PrivateLinks {
for _, ipConfig := range privateLink.NATIPConfigurations {
ipConfigSubnetNames[ipConfig.Subnet] = struct{}{}
}
}
}

subnetSpecs := make([]azure.ASOResourceSpecGetter[*asonetworkv1api20201101.VirtualNetworksSubnet], 0, numberOfSubnets)
for _, subnet := range s.AzureCluster.Spec.NetworkSpec.Subnets {
subnetSpec := &subnets.SubnetSpec{
Name: subnet.Name,
Expand All @@ -468,6 +477,11 @@ func (s *ClusterScope) SubnetSpecs() []azure.ASOResourceSpecGetter[*asonetworkv1
NatGatewayName: subnet.NatGateway.Name,
ServiceEndpoints: subnet.ServiceEndpoints,
}
// Check if subnet is used for the private link NAT IP
if _, ok := ipConfigSubnetNames[subnet.Name]; ok {
subnetSpec.UsedForPrivateLinkNATIP = true
}

subnetSpecs = append(subnetSpecs, subnetSpec)
}

Expand Down Expand Up @@ -1117,6 +1131,11 @@ func (s *ClusterScope) GetLongRunningOperationState(name, service, futureType st
return futures.Get(s.AzureCluster, name, service, futureType)
}

// GetLongRunningOperationStates will get the specified futures on the AzureCluster status.
func (s *ClusterScope) GetLongRunningOperationStates(service, futureType string) infrav1.Futures {
return futures.GetByServiceAndType(s.AzureCluster, service, futureType)
}

// DeleteLongRunningOperationState will delete the future from the AzureCluster status.
func (s *ClusterScope) DeleteLongRunningOperationState(name, service, futureType string) {
futures.Delete(s.AzureCluster, name, service, futureType)
Expand Down Expand Up @@ -1237,7 +1256,41 @@ func (s *ClusterScope) PrivateEndpointSpecs() []azure.ASOResourceSpecGetter[*aso
return privateEndpointSpecs
}

func (s *ClusterScope) getLastAppliedSecurityRules(nsgName string) map[string]any {
// PrivateLinkSpecs returns the private link specs.
func (s *ClusterScope) PrivateLinkSpecs() []azure.ResourceSpecGetter {
// First we get all private links to API server load balancer.
// Other load balancers (ControlPlaneOutboundLB and NodeOutboundLB) are outbound, so we cannot create private links
// for those.
privateLinks := s.AzureCluster.Spec.NetworkSpec.APIServerLB.PrivateLinks
Copy link
Copy Markdown
Contributor

@willie-yao willie-yao Jun 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you'll need to add a nil check for Spec.NetworkSpec.APIServerLB here as well. For externally managed control planes, Spec.NetworkSpec.APIServerLB will be set to nil as seen here" https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/internal/api/v1beta1/azurecluster_default.go#L74

You can do something like what you had on line 457:

if apiServerLB := s.AzureCluster.Spec.NetworkSpec.APIServerLB; apiServerLB != nil { ... }

privateLinksSpecs := make([]azure.ResourceSpecGetter, 0, len(privateLinks))

for _, privateLink := range privateLinks {
privateLinkSpec := privatelinks.PrivateLinkSpec{
Name: privateLink.Name,
ResourceGroup: s.ResourceGroup(),
SubscriptionID: s.SubscriptionID(),
Location: s.Location(),
VNetResourceGroup: s.Vnet().ResourceGroup,
VNet: s.Vnet().Name,
LoadBalancerName: s.APIServerLBName(),
LBFrontendIPConfigNames: privateLink.LBFrontendIPConfigNames,
AllowedSubscriptions: privateLink.AllowedSubscriptions,
AutoApprovedSubscriptions: privateLink.AutoApprovedSubscriptions,
EnableProxyProtocol: privateLink.EnableProxyProtocol,
ClusterName: s.ClusterName(),
AdditionalTags: s.AdditionalTags(),
}
// Set NAT IP configuration
for _, natIPConfiguration := range privateLink.NATIPConfigurations {
privateLinkSpec.NATIPConfiguration = append(privateLinkSpec.NATIPConfiguration, privatelinks.NATIPConfiguration(natIPConfiguration))
}
privateLinksSpecs = append(privateLinksSpecs, &privateLinkSpec)
}

return privateLinksSpecs
}

func (s *ClusterScope) getLastAppliedSecurityRules(nsgName string) map[string]interface{} {
// Retrieve the last applied security rules for all NSGs.
lastAppliedSecurityRulesAll, err := s.AnnotationJSON(azure.SecurityRuleLastAppliedAnnotation)
if err != nil {
Expand Down
Loading
Loading