Implement private network workload clusters configuration

[Rotfuks]

Motivation

During the exploration in https://github.com/giantswarm/giantswarm/issues/24762 and https://github.com/giantswarm/giantswarm/issues/25773 we learned how to create a private network configuration for workload clusters in CAPZ based on the prototype - now we need to implement this in our CAPZ clusters.

Todo

• Implement the learnings from the Investigation Prototype in our CAPZ Clusters
• Make sure at creation time of the MC you can select if it is creating private or public workload clusters
• Test the implementation by making Glippy private

Outcome

• We can select for any new installation if they can create private or public workload clusters.
• Private workload clusters can only be accessed over the VPN and do not expose public IPs.
• Glippy Installation creates private network workload clusters.

[nprokopic]

I would limit the scope here to just enabling private workload clusters created from the public MC (glippy), as that is the first step required in order to make glippy itself private.

And then we make glippy private in the next step (and separate issue).

The following are concrete todos here.

• Merge cluster-azure pull request for reducing the default WC network range Reduce default network range to 10.0.0.0/16 cluster-azure#29
• Redeploy glippy with reduced and updated network range (see below what range to use).

The selected network range for glippy (and planned network range for WCs) must not overlap with any other MC (across all providers). We also need bastion to be deployed here (which we get with newer versions of cluster-azure).

• Deploy Azure VPN gateway to glippy

If possible, configure it the same way as in Azure vintage product.

To begin with, we can do this manually (which will be done as a part of the SPIKE which comes before this issue), as there is just a handful of Azure CLI commands that achieve this.

Then we can also add this to MC bootstrap, so VPN gets recreated, or even better reused, at least the public IP in Azure, so we don't recreate it every time, which would mean that we have to reconfigure our VPN every time.

• Add glippy connection to our VPN configuration.

Here we configure shared network range to include glippy MC network range as well as a larger network range that will accommodate all WCs created in glippy.

The suggestion is to use 10.x.y.0/20 range for glippy shared IP network range via the VPN connection (MC and all WCs).

This will enable us to deploy, for example, 16 clusters in total (counting all MCs and WCs) with 10.x.y.0/24 network range (256 addresses in total), where cluster network is split in the following way:

• 10.x.y.0/25 for workers subnet. This is 128 addresses in total, where 5 are reserved by Azure, which gives us 123 addresses for worker nodes.
• 10.x.y.128/25 divided into:
  • 10.x.y.128/26 for control plane subnet
  • 10.x.y.192/26 for bastion and other subnets that we will possibly need.

The smallest possible WC that we can create with the current WC setup would be 10.x.y.z/27, where:

• 10.x.y.z/28 is for workers subnet, which gives us 16 addresses (11 for nodes, 5 reserved by Azure)
• 10.x.y.z/29 is for control plane, which gives us 8 addresses, so 3 usable, where 1 is required for the internal load balancer, so we get 2 for control plane nodes, meaning we can have only 1 control plane node.

To expand a bit there, by using 10.x.y.0/20 network range for an MC (and its WCs), we get the following possibilities and constraints (max addresses, workers and CP nodes):

Total WCs	WC mask	Addresses	Workers	CP nodes
8	/23	512	251	123
16	/24	256	123	59
32	/25	128	59	27
64	/26	64	27	11
128	/27	32	11	1

The above examples show that this should be more than enough for testing purposes.

Then a bit of coding:

• Update VNet peering implementation in CAPZ to enable modification of VNet peering properties.
  • We should bring this up to upstream before implementing it in our fork.
  • In case upstream is not interested for this, we can have it in our fork temporarily, since we will probably move from VNet peering to private endpoints in not too far future.
  • We can also implement VNet peering in a custom operator, but if we will be working on a custom operator, we might as well implement access via private endpoints.

Finally:

• Update and merge cluster-azure pull requests for adding private clusters support.
• Deploy private clusters 🎉

[primeroz]

To begin with, we can do this manually (which will be done as a part of the SPIKE which comes before this issue), as there is just a handful of Azure CLI commands that achieve this.

sounds good but let's document the commands in case we need to do it again / makes it easier to to add to mc-bootstrap later

10.x.y.0/25 for workers subnet. This is 128 addresses in total, where 5 are reserved by Azure, which gives us 123 addresses for worker nodes.
10.x.y.128/25 divided into:
10.x.y.128/26 for control plane subnet
10.x.y.192/26 for bastion and other subnets that we will possibly need.

sound good but while the code supports putting the bastion in another subnet i have not tested it , i only run it in the CP one so far.

The smallest possible WC that we can create with the current WC setup would be 10.x.y.z/27, where:

WC also get a bastion by default, but it can be disabled. I don't see it listed in there but is just one more ip

[bavarianbidi]

Then we can also add this to MC bootstrap, so VPN gets recreated, or even better reused, at least the public IP in Azure, so we don't recreate it every time, which would mean that we have to reconfigure our VPN every time.

i personally wouldn't care to much about this. As the VPN config in general is all defined with ansible it should be safe to rollout each time a change is needed. (But that's my high-level assumption how the VPN config is working - maybe it's a snowflake and everyone is happy to not touch it - then i would agree with your proposal.)

• Update VNet peering implementation in CAPZ to enable modification of VNet peering properties.
  * We should bring this up to upstream before implementing it in our fork.
  * [...] since we will probably move from VNet peering to private endpoints in not too far future.

If the endgoal will be "private endpoints" and i assume a migration from "VNet peering" to "private endpoints" is technically possible i would propose the following steps:

1. create upstream issue for VNet peering properties and open an upstream PR from our fork
2. implement VNet peering properties in CAPZ (in our fork)
3. create upstream issue for "private endpoints" and discuss with upstream is this could be a possible feature for CAPZ or if this is something which should go into a dedicated operator

1. and 3. should be done immediately to get early feedback from upstream (especially for 3. - as sooner as we get feedback, as sooner we could start with an implementation).
For a smooth customer experience we need 2. anyways (if VNet peering is the technical solution in our first version of private clusters). Depending on the result of 1. we can bring our own fork to upstream or drop it once we have clearance and a technical solution for 3.. As 2. is "only" adding some new properties and from what i know the code it shouldn't be a big deal to keep this change in sync with upstream changes (one good commit + git rebase shouldn't be a big deal IMO)

[Rotfuks]

Because of too big of a scope for this Issue the Ticket will be split in the following tickets:

• Azure VPN Gateway for Workload Clusters #2039
• VPN Configuration for Clippy #2040
• Improve CAPZ upstream VNet Peering #2041
• Private Link for upstream CAPZ #2042