gcve: Add tags for failure domain testing, reprovision for vsphere 8 and add inline comments

infra/gcp/terraform/k8s-infra-gcp-gcve/README.md

@@ -2,7 +2,7 @@
 
 The code in `k8s-infra-gcp-gcve` sets up the infra required to allow prow jobs to create VMs on vSphere, e.g. to allow testing  of the [Cluster API provider vSphere (CAPV)](https://github.com/kubernetes-sigs/cluster-api-provider-vsphere).
 
-![Overview](./docs/images/overview.jpg)
+![Overview](./docs/images/GVCE.drawio.png)
 
 Prow container settings are managed outside of this folder, but understanding high level components could
 help to understand how the `k8s-infra-gcp-gcve` is set up and consumed.
@@ -17,9 +17,10 @@ More specifically, to allow prow jobs to create VM on vSphere, a few resources a
 Also, the network of the prow container is going to be paired to the VMware engine network, thus
 allowing access to both the GCVE management network and the NSX-T network where all the VM are running.
 
-The `k8s-infra-gcp-gcve` project sets up the infrastructure that actually runs the VMs created from the prow container. There are ther main components of this infrastracture:
+The `k8s-infra-gcp-gcve` project sets up the infrastructure that actually runs the VMs created from the prow container. 
+These are the main components of this infrastructure:
 
-The terraform manifest in this folder, which is applied by test-infra automation (Atlantis), uses the GCP terraform provider for creating.
+The terraform manifest in this folder uses the GCP terraform provider for creating.
 - A VMware Engine instance
 - The network infrastructure required for vSphere and for allowing communication between vSphere and Prow container.
     -  The network used is `192.168.0.32/21`
@@ -28,12 +29,21 @@ The terraform manifest in this folder, which is applied by test-infra automation
             - IPPool for 40 Projects having 16 IPs each: `192.168.35.0 - 192.168.37.127`
 - The network infrastructure used for maintenance.
 
+See [terraform](../docs/terraform.md) for prerequisites.
+
+When ready:
+
+```sh
+terraform init
+terraform plan # Check diff
+terraform apply
+```
+
 See inline comments for more details.
 
-The terraform manifest in the `/maintenance-jumphost` uses the GCP terraform provider to setup a jumphost VM to be used to set up vSphere or for maintenance pourposes. See
+The terraform manifest in the `/maintenance-jumphost` uses the GCP terraform provider to setup a jumphost VM to be used to set up vSphere or for maintenance purposes. See
 - [maintenance-jumphost](./maintenance-jumphost/README.md)
 
-The terraform manifest in the `/vsphere` folder uses the vSphere and the NSX terraform providers to setup e.g. content libraries, templetes, folders, 
+The terraform manifest in the `/vsphere` folder uses the vSphere and the NSX terraform providers to setup e.g. content libraries, templates, folders, 
 resource pools and other vSphere components required when running tests. See:
-- [vsphere](./vsphere/README.md) 
-
+- [vsphere](./vsphere/README.md)

infra/gcp/terraform/k8s-infra-gcp-gcve/docs/boskos.md

@@ -1,6 +1,6 @@
 # Boskos
 
-Boskos support resources of type `gcve-vsphere-project` to allow each test run to use a subset of vSphere resources.
+Boskos resources of type `gcve-vsphere-project` allow each test run to use a subset of vSphere resources.
 
 Boskos configuration is split in three parts:
 

infra/gcp/terraform/k8s-infra-gcp-gcve/docs/terraform.md

@@ -2,7 +2,7 @@
 
 See [README.md](https://github.com/kubernetes/k8s.io/tree/main/infra/gcp/terraform) for a general intro about using terraform in k8s.io.
 
-In order to apply terraform manifests you  must be enabled to use the "broadcom-451918" project, please reach to [owners](../OWNERS) in case of need.
+In order to apply terraform manifests you  must be enabled to use the "broadcom-451918" project, please reach out to [owners](../OWNERS) in case of need.
 
 Quick reference:
 
@@ -19,7 +19,7 @@ You can use terraform from your local workstation or via a docker container prov
 docker run -it --rm -v $(pwd):/workspace --entrypoint=/bin/bash gcr.io/k8s-staging-infra-tools/k8s-infra:v20241217-f8b07a049
 ```
 
-From your local workstatin / from inside the terraform container:
+From your local workstation / from inside the terraform container:
 
 Login to GCP to get an authentication token to use with terraform.
 
@@ -30,15 +30,21 @@ gcloud auth application-default login
 Ensure all the env variables expected by the terraform manifest you are planning to run are set:
 - [vsphere](../vsphere/)
 
+Ensure the right terraform version expected by the terraform manifest you are planning to run is installed (Note: this requires `tfswitch` which is pre-installed in the docker image. In case of version mismatches, terraform will make you know):
+
+```bash
+tfswich
+```
+
 Additionally, if applying the vsphere terraform manifest, use the following script to generate `/etc/hosts` entries for vSphere and NSX.
 
 ```sh
-gcloud vmware private-clouds describe k8s-gcp-gcve-pc --location us-central1-a --format='json' | jq -r '.vcenter.internalIp + " " + .vcenter.fqdn +"\n" + .nsx.internalIp + " " + .nsx.fqdn'
+gcloud vmware private-clouds describe k8s-gcp-gcve --location us-central1-a --format='json' | jq -r '.vcenter.internalIp + " " + .vcenter.fqdn +"\n" + .nsx.internalIp + " " + .nsx.fqdn'
 ```
 
-Add those entry to `/etc/hosts`.
+Add those entries to `/etc/hosts`.
 
-At this point you are ready to start using `terraform init`, `terraform plan`, `terraform apply` etc. 
+At this point you are ready to start using `terraform init`, `terraform plan`, `terraform apply` etc.
 
 Notes:
 - Terraform state is stored in a gcs bucket with name `k8s-infra-tf-gcp-gcve`, with a folder for each one of the terraform scripts managed in the `k8s-infra-gcp-gcve` folder (gcve, gcve-vcenter, maintenance-jumphost).

infra/gcp/terraform/k8s-infra-gcp-gcve/docs/wiregard.md → infra/gcp/terraform/k8s-infra-gcp-gcve/docs/wireguard.md

@@ -1,11 +1,11 @@
-# Wiregard
+# Wireguard
 
-Wiregard is used to get a secure and convenient access through the maintenace jump host VM.
+Wireguard is used to get a secure and convenient access through the maintenace jump host VM.
 
-In order to use wiregard you must be enabled to use the "broadcom-451918" project, please reach to [owners](../OWNERS) in case of need.
+In order to use wireguard you must be enabled to use the "broadcom-451918" project, please reach out to [owners](../OWNERS) in case of need.
 
 It is also required to first setup things both on on your local machine and on the GCP side
-following instruction below.
+following the instruction below.
 
 Install wireguard following one of the methods described in https://www.wireguard.com/install/.
 
@@ -58,7 +58,7 @@ EOF
 
 Then create new version of the `maintenance-vm-wireguard-config` by appending this entry at the end of the current value [here](https://console.cloud.google.com/security/secret-manager/secret/maintenance-vm-wireguard-config/versions?project=broadcom-451918).
 
-Additionally, if the jumphost VM is up, you might want to add it to the wiregard configuration  in the current VM (it is also possible to recreate the jumphost VM, but this is going to change the wireguard enpoint also for other users).
+Additionally, if the jumphost VM is up, you might want to add it to the wireguard configuration  in the current VM (it is also possible to recreate the jumphost VM, but this is going to change the wireguard enpoint also for other users).
 
 ```sh
 gcloud compute ssh maintenance-jumphost --zone us-central1-f
@@ -79,7 +79,7 @@ MTU = 1360
 
 [Peer]
 PublicKey = $(gcloud secrets versions access --secret maintenance-vm-wireguard-pubkey latest)
-AllowedIPs = 192.168.30.0/24, 192.168.32.0/21
+AllowedIPs = 192.168.31.0/24, 192.168.32.0/21
 Endpoint = $(gcloud compute instances list --format='get(networkInterfaces[0].accessConfigs[0].natIP)' --filter='name=maintenance-jumphost'):51820
 PersistentKeepalive = 25
 EOF
@@ -94,7 +94,7 @@ You can then either
 Generate `/etc/hosts` entries for vSphere and NSX; this is required to run the vSphere terraform scripts and it will also make the vSphere and NSX UI to work smootly.
 
 ```sh
-gcloud vmware private-clouds describe k8s-gcp-gcve-pc --location us-central1-a --format='json' | jq -r '.vcenter.internalIp + " " + .vcenter.fqdn +"\n" + .nsx.internalIp + " " + .nsx.fqdn'
+gcloud vmware private-clouds describe k8s-gcp-gcve --location us-central1-a --format='json' | jq -r '.vcenter.internalIp + " " + .vcenter.fqdn +"\n" + .nsx.internalIp + " " + .nsx.fqdn'
 ```
 
-Add those entry to `/etc/hosts`.
+Add those entries to `/etc/hosts`.

infra/gcp/terraform/k8s-infra-gcp-gcve/iam.tf

@@ -14,6 +14,7 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
+# Ensures admin access for groups and secret access for prow.
 module "iam" {
   source  = "terraform-google-modules/iam/google//modules/projects_iam"
   version = "~> 8.1"

infra/gcp/terraform/k8s-infra-gcp-gcve/main.tf

@@ -14,19 +14,17 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-locals {
-  project_id = "broadcom-451918"
-}
-
 data "google_project" "project" {
-  project_id = local.project_id
+  project_id = var.project_id
 }
 
+# Enables all required APIs for this project.
 resource "google_project_service" "project" {
   project = data.google_project.project.id
 
   for_each = toset([
     "compute.googleapis.com",
+    "essentialcontacts.googleapis.com",
     "secretmanager.googleapis.com",
     "vmwareengine.googleapis.com"
   ])

infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/README.md

@@ -3,8 +3,8 @@
 The maintenance jump host is a VM hosting a wireguard instance for secure and convenient access 
 to vSphere and NSX from local machines.
 
-Before using wiregard it is required to first setup things both on on your local machine and on the GCP side.
-see [wireguard](../docs/wiregard.md)
+Before using wireguard it is required to first setup things both on on your local machine and on the GCP side.
+see [wireguard](../docs/wireguard.md)
 
 The maintenance jump host VM is not required to be always up & running and it can also be recreated if necessary; however, by doing so the IP address of the VM will change and all the 
 local machine config have to be updated accordingly.

infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/jumphost.tf

@@ -14,12 +14,21 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-locals {
-  project_id = "broadcom-451918"
+variable "project_id" {
+  description = "The project ID to use for the gcve cluster."
+  default     = "broadcom-451918"
+  type        = string
 }
 
+# Read the secret from Secret Manager which contains the wireguard server configuration. 
+data "google_secret_manager_secret_version_access" "wireguard-config" {
+  project      = var.project_id
+  secret = "maintenance-vm-wireguard-config"
+}
+
+# Create the maintenance jumphost which runs SSH and a wireguard server.
 resource "google_compute_instance" "jumphost" {
-  project      = local.project_id
+  project      = var.project_id
   name         = "maintenance-jumphost"
   machine_type = "f1-micro"
   zone         = "us-central1-f"
@@ -33,7 +42,7 @@ resource "google_compute_instance" "jumphost" {
   network_interface {
     network = "maintenance-vpc-network"
     subnetwork = "maintenance-subnet"
-    subnetwork_project = local.project_id
+    subnetwork_project = var.project_id
     access_config {
       network_tier = "STANDARD"
     }
@@ -43,8 +52,3 @@ resource "google_compute_instance" "jumphost" {
     user-data = templatefile("${path.module}/cloud-config.yaml.tftpl", { wg0 = base64encode(data.google_secret_manager_secret_version_access.wireguard-config.secret_data) })
   }
 }
-
-data "google_secret_manager_secret_version_access" "wireguard-config" {
-  project      = local.project_id
-  secret = "maintenance-vm-wireguard-config"
-}

infra/gcp/terraform/k8s-infra-gcp-gcve/maintenance-jumphost/provider.tf

@@ -21,6 +21,7 @@ This file defines:
 */
 
 terraform {
+  required_version = "1.10.5"
 
   backend "gcs" {
     bucket = "k8s-infra-tf-gcp-gcve"

infra/gcp/terraform/k8s-infra-gcp-gcve/provider.tf

@@ -22,12 +22,12 @@ This file defines:
 
 terraform {
   required_version = "1.10.5"
+
   backend "gcs" {
     bucket = "k8s-infra-tf-gcp-gcve"
     prefix = "k8s-infra-gcp-gcve"
   }
 
-
   required_providers {
     google = {
       source  = "hashicorp/google"

infra/gcp/terraform/k8s-infra-gcp-gcve/variables.tf

@@ -17,4 +17,5 @@ limitations under the License.
 variable "project_id" {
   description = "The project ID to use for the gcve cluster."
   default     = "broadcom-451918"
+  type        = string
 }

infra/gcp/terraform/k8s-infra-gcp-gcve/vmware-engine.tf

@@ -14,15 +14,16 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
+# Creates the VMware Engine Private Cloud which is a vSphere Cluster including NSX-T.
 resource "google_vmwareengine_private_cloud" "vsphere-cluster" {
   location    = "us-central1-a"
-  name        = "k8s-gcp-gcve-pc"
+  name        = "k8s-gcp-gcve"
   project     = var.project_id
   description = "k8s Community vSphere Cluster for CI."
   # TODO(chrischdi): figure out discount and switch to STANDARD
   type = "TIME_LIMITED"
   network_config {
-    management_cidr       = "192.168.30.0/24"
+    management_cidr       = "192.168.31.0/24"
     vmware_engine_network = google_vmwareengine_network.vsphere-network.id
   }
 
@@ -36,24 +37,27 @@ resource "google_vmwareengine_private_cloud" "vsphere-cluster" {
   }
 }
 
+# Creates the VMware Engine Network for the Private Cloud.
 resource "google_vmwareengine_network" "vsphere-network" {
   name     = "k8s-gcp-gcve-network"
   project  = var.project_id
   type     = "STANDARD"
   location = "global"
 }
 
+# Creates the Network Policy to allow created virtual machines to reach out to the internet.
 resource "google_vmwareengine_network_policy" "external-access-rule-np" {
   name                  = "k8s-gcp-gcve-network-policy"
   project               = var.project_id
   location              = "us-central1"
-  edge_services_cidr    = "192.168.31.0/26"
+  edge_services_cidr    = "192.168.27.0/26"
   vmware_engine_network = google_vmwareengine_network.vsphere-network.id
   internet_access {
     enabled = true
   }
 }
 
+# Creates the Peering to the prow cluster to allow Pods running in Prow to access vCenter and created VMs in vSphere.
 resource "google_vmwareengine_network_peering" "prow_peering" {
   name                                = "peer-with-k8s-infra-prow-build"
   project                             = var.project_id
@@ -64,12 +68,14 @@ resource "google_vmwareengine_network_peering" "prow_peering" {
   import_custom_routes_with_public_ip = false
 }
 
+# Creates a maintenance network used for creating Google Compute VM(s) for setup or debugging purposes via ssh or wireguard VPN.
 resource "google_compute_network" "maintenance-vpc" {
   name                    = "maintenance-vpc-network"
   project                 = var.project_id
   auto_create_subnetworks = false
 }
 
+# Creates the Subnet for the above maintenance network.
 resource "google_compute_subnetwork" "maintenance-subnet" {
   name          = "maintenance-subnet"
   project       = var.project_id
@@ -78,6 +84,7 @@ resource "google_compute_subnetwork" "maintenance-subnet" {
   network       = google_compute_network.maintenance-vpc.id
 }
 
+# Creates the Peering to the maintenance network to maintenance VMs to access vCenter and created VMs in vSphere.
 resource "google_vmwareengine_network_peering" "maintenance_peering" {
   name                  = "peer-with-maintenance-vpc-network"
   description           = "Peering with maintenance vpc network"
@@ -87,6 +94,8 @@ resource "google_vmwareengine_network_peering" "maintenance_peering" {
   vmware_engine_network = google_vmwareengine_network.vsphere-network.id
 }
 
+# Creates the firewall rules for VMs running in the maintenance network so they can be accessed
+# via SSH or to expose wireguard VPN.
 resource "google_compute_firewall" "maintenance-firewall-internet" {
   name    = "maintenance-firewall-internet"
   project = var.project_id
@@ -105,6 +114,8 @@ resource "google_compute_firewall" "maintenance-firewall-internet" {
   }
 }
 
+# Creates the firewall rule to allow any traffic from the maintenance subnet to
+# the VMware Engine network or the internet.
 resource "google_compute_firewall" "maintenance-firewall-internal" {
   name    = "maintenance-firewall-internal"
   project = var.project_id