Testing of all providers and distros

[AverageMarcus]

Is your feature request related to a problem? Please describe.

We've had a few instances now where changes introduced issues into existing targets, e.g.

• Ensure all targets are tested #881
• Raw build target is broken #879
• QEMU Flatcar build is broken #828

and most recently #1596 caused an issue for all OS's that made use of init data that wasn't caught during manual testing due to focussing on Flatcar.

Describe the solution you'd like

Where possible we should have automated tests in place to ensure that all currently supported providers and operating systems can build successfully with default values (this might be tricky in situations where there are required values).

This issue is aimed at tracking the progress of this effort and should be updated as progress is made.

Ideally, we should implement provider-specific tests that then build all OSs for that provider (e.g. using the make build-raw-all that should trigger all supported OSs).

Progress

Provider	Implemented	Comments
ami	❌	I think the CAPA project had something that they (used to?) use for testing image builds. We should reach out to them and see if its something we can adopt.
azure	Partial	We currently have two Prow jobs for Azure - pull-azure-vhds & pull-azure-sigs. These make use of ci-azure-e2e.sh but uses a pre-defined list of target (azure_targets.sh). Ideally we should try to update this to dynamically load all targets we support.
digitalocean	❌
gce	Partial	We currently have the Prow job pull-image-builder-gcp-all that uses ci-gce.sh but this is currently configured as an optional test and not automatically run on any PRs. We should update this to at least trigger when changes to GCE files are made, similar to the Azure ones. (see #1601)
hcloud	❌
nutanix	❌
oci	❌
openstack	❌
outscale	❌
ova	partial	We currently have the Prow job pull-ova-all that uses ci-ova.sh which currently has a static list of targets defined. We should update this to at least match our currently supported OSs
powervs	❌
proxmox	❌
qemu	❌	Theoretically, this should be possible to test in our current infrastructure as its emulated. The problem might be the time it takes to run the tests though.
raw	❌	Theoretically, this should be possible to test in our current infrastructure as its emulated. The problem might be the time it takes to run the tests though.
scaleway	❌
vultr	❌

Describe alternatives you've considered

Manual testing but this, as we have seen, isn't enough to catch all the issues.

Additional context

Many of the providers will require actual cloud infrastructure to be able to run the build process. This is likely to be difficult to get unless the respective cloud providers are willing to donate resources for this purpose. (Please reach out to us if you are able to provide these resources)

---

/kind feature
/lifecycle frozen
/help
/priority important-longterm
/triage accepted

[k8s-ci-robot]

@AverageMarcus:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

Details

In response to this:

Is your feature request related to a problem? Please describe.

We've had a few instances now where changes introduced issues into existing targets, e.g.

• Ensure all targets are tested #881
• Raw build target is broken #879
• QEMU Flatcar build is broken #828

and most recently #1596 caused an issue for all OS's that made use of init data that wasn't caught during manual testing due to focussing on Flatcar.

Describe the solution you'd like

Where possible we should have automated tests in place to ensure that all currently supported providers and operating systems can build successfully with default values (this might be tricky in situations where there are required values).

This issue is aimed at tracking the progress of this effort and should be updated as progress is made.

Ideally, we should implement provider-specific tests that then build all OSs for that provider (e.g. using the make build-raw-all that should trigger all supported OSs).

Progress

Provider	Implemented	Comments
ami	❌	I think the CAPA project had something that they (used to?) use for testing image builds. We should reach out to them and see if its something we can adopt.
azure	Partial	We currently have two Prow jobs for Azure - pull-azure-vhds & pull-azure-sigs. These make use of ci-azure-e2e.sh but uses a pre-defined list of target (azure_targets.sh). Ideally we should try to update this to dynamically load all targets we support.
digitalocean	❌
gce	Partial	We currently have the Prow job pull-image-builder-gcp-all that uses ci-gce.sh but this is currently configured as an optional test and not automatically run on any PRs. We should update this to at least trigger when changes to GCE files are made, similar to the Azure ones.
hcloud	❌
nutanix	❌
oci	❌
openstack	❌
outscale	❌
ova	❌	We recently had to remove this and need a new environment to run the tests in, see #1508
powervs	❌
proxmox	❌
qemu	❌	Theoretically, this should be possible to test in our current infrastructure as its emulated. The problem might be the time it takes to run the tests though.
raw	❌	Theoretically, this should be possible to test in our current infrastructure as its emulated. The problem might be the time it takes to run the tests though.
vultr	❌

Describe alternatives you've considered

Manual testing but this, as we have seen, isn't enough to catch all the issues.

Additional context

Many of the providers will require actual cloud infrastructure to be able to run the build process. This is likely to be difficult to get unless the respective cloud providers are willing to donate resources for this purpose. (Please reach out to us if you are able to provide these resources)

---

/kind feature
/lifecycle frozen
/help
/priority important-longterm
/triage accepted

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.