Migrate away from google.com gcp project k8s-testimages

[spiffxp]

Part of umbrella issue to migrate away from google.com gcp projects: #1469

At least some of this is part of the umbrella to migrate kubernetes e2e test images/registries to community-owned infrastructure: #1458

We should migrate away from the google.com-owned gcr.io/k8s-testimages repository and instead use a community-owned repository.

k8s-testimages hosts a variety of images that are built from source in kubernetes/test-infra. They fall broadly into two classes:

• images used to run prowjobs (e.g. kubekins)
• images for sundry components, experiments, services and tools that live in test-infra

/wg k8s-infra
/sig release
/sig testing
/area release-eng

---

EDIT(spiffxp): Went through and exhaustively identified images that need to be migrated from the repo, or can be left behind.

Images that are used and need migration, have already migrated, or are unused and need source deleted:

• gcr.io/k8s-testimages/alpine-bash - unused (PR to remove source: images: remove unused images test-infra#23617)
• gcr.io/k8s-testimages/aws-janitor - moved to k8s-staging-boskos
• gcr.io/k8s-testimages/bazel-krte - unused (PR to remove source: images: remove unused images test-infra#23617)
• gcr.io/k8s-testimages/bazelbuild - Migrate away from google.com gcp project k8s-testimages #1523 (comment)
• gcr.io/k8s-testimages/benchmarkjunit - Migrate away from google.com gcp project k8s-testimages #1523 (comment)
• gcr.io/k8s-testimages/bigquery - Migrate away from google.com gcp project k8s-testimages #1523 (comment)
• gcr.io/k8s-testimages/bootstrap - Migrate away from google.com gcp project k8s-testimages #1523 (comment)
• gcr.io/k8s-testimages/boskos - moved to k8s-staging-boskos
• gcr.io/k8s-testimages/branchprotector - moved to k8s-prow (used for dev only)
• gcr.io/k8s-testimages/cluster-api - unused (PR to remove source: config/jobs: use community hosted images for more jobs test-infra#23587)
• gcr.io/k8s-testimages/commenter - moved to k8s-prow
• gcr.io/k8s-testimages/e2e-kubemci - images: rm kubemci test-infra#27266
• gcr.io/k8s-testimages/gcb-docker-gcloud - Migrate away from google.com gcp project k8s-testimages #1523 (comment)
• gcr.io/k8s-testimages/gcloud-bazel - images: drop gcloud-bazel test-infra#27265
• gcr.io/k8s-testimages/gcloud-in-go - Migrate away from google.com gcp project k8s-testimages #1523 (comment)
• gcr.io/k8s-testimages/gcloud-terraform - used by GoogleCloudPlatform/oss-test-infra
• gcr.io/k8s-testimages/gencred - more of a prow tool
• gcr.io/k8s-testimages/ghproxy - moved to k8s-prow (used in this repo for dev only)
• gcr.io/k8s-testimages/grafana - used by kubernetes/perf-tests/clusterloader2/pkg/prometheus/manifests
• gcr.io/k8s-testimages/greenhouse - used by kubernetes/test-infra/greenhouse, kubernetes/k8s.io prow, etc.
• gcr.io/k8s-testimages/gubernator - used in a job config in kubernetes/test-infra
• gcr.io/k8s-testimages/image-builder - Migrate away from google.com gcp project k8s-testimages #1523 (comment)
• gcr.io/k8s-testimages/issue-creator - moved to k8s-prow (used in this repo for dev only)
• gcr.io/k8s-testimages/janitor - moved to k8s-staging-boskos
• gcr.io/k8s-testimages/kettle - used by kubernetes/test-infra/kettle
• gcr.io/k8s-testimages/krte - Migrate away from google.com gcp project k8s-testimages #1523 (comment)
• gcr.io/k8s-testimages/kubekins-e2e - Migrate away from google.com gcp project k8s-testimages #1523 (comment)
• gcr.io/k8s-testimages/kubekins-e2e-prow - used by kubernetes/test-infra/pod.yaml
• gcr.io/k8s-testimages/kubekins-test - used by kubernetes/test-infra/scenarios/kubernetes_verify.py
• gcr.io/k8s-testimages/label_sync - moved to k8s-prow (used in this repo for dev only)
• gcr.io/k8s-testimages/launcher.gcr.io - multiple images here used by kubernetes/test-infra jobs
• gcr.io/k8s-testimages/logexporter - used in kubernetes/kubernetes, kubernetes/cloud-provider-gcp, and kubernetes/test-infra
• gcr.io/k8s-testimages/metrics - done (TODO: link)
• gcr.io/k8s-testimages/netperfbenchmark - used by kubernetes/perf-tests/clusterloader2
• gcr.io/k8s-testimages/perf-tests-util - used by kubernetes/perf-tests, kubernetes-sigs/cloud-provider-azure
• gcr.io/k8s-testimages/perfdash - shows up in kubernetes/perf-tests
• gcr.io/k8s-testimages/probes - used by kubernetes/perf-tests
• gcr.io/k8s-testimages/quay.io - used by kubernetes/perf-tests
• gcr.io/k8s-testimages/reaper - moved to k8s-staging-boskos
• gcr.io/k8s-testimages/triage - done (TODO: link)

Images that appear to be unused:

• gcr.io/k8s-testimages/bootstrap-dind - unused
• gcr.io/k8s-testimages/boskosctl-base - unused
• gcr.io/k8s-testimages/cherrypick - unused
• gcr.io/k8s-testimages/dind-test-base-amd64 - unused
• gcr.io/k8s-testimages/e2e-kubeadm - unused
• gcr.io/k8s-testimages/experiment - unused
• gcr.io/k8s-testimages/fake-server - unused
• gcr.io/k8s-testimages/fedtidy - unused
• gcr.io/k8s-testimages/gcp-controller-manager - unused
• gcr.io/k8s-testimages/gcr.io - unused
• gcr.io/k8s-testimages/generic_autobump - unused
• gcr.io/k8s-testimages/github-fetcher - unused
• gcr.io/k8s-testimages/github-token-counter - unused
• gcr.io/k8s-testimages/github-transform - unused
• gcr.io/k8s-testimages/gob-test - unused
• gcr.io/k8s-testimages/heapster-test - unused
• gcr.io/k8s-testimages/janitor-aws - unused
• gcr.io/k8s-testimages/kube-deploy - unused
• gcr.io/k8s-testimages/kubekins-node - unused
• gcr.io/k8s-testimages/kubekins-senlu-hack - unused
• gcr.io/k8s-testimages/kubemark - unused
• gcr.io/k8s-testimages/kubernetes-node - unused
• gcr.io/k8s-testimages/leaker - unused
• gcr.io/k8s-testimages/levee-test - unused
• gcr.io/k8s-testimages/misc-mungers - unused
• gcr.io/k8s-testimages/nursery - unused
• gcr.io/k8s-testimages/planter - unused
• gcr.io/k8s-testimages/pohly - unused
• gcr.io/k8s-testimages/prow-test-image - unused
• gcr.io/k8s-testimages/queue-health-base - unused
• gcr.io/k8s-testimages/queue-health-graph - unused
• gcr.io/k8s-testimages/queue-health-poll - unused
• gcr.io/k8s-testimages/redis - unused
• gcr.io/k8s-testimages/resultstore - unused
• gcr.io/k8s-testimages/submit-queue - unused
• gcr.io/k8s-testimages/test-infra-go-test - unused
• gcr.io/k8s-testimages/test-infra-py-test - unused

[spiffxp]

/milestone v1.21

[spiffxp]

I am open to suggestions on where we should move these images.

I was thinking instead of a straight rename, take the time to break up the two classes:

• k8s-testimages/kettle -> k8s.gcr.io/test-infra/kettle (k8s-staging-test-infra)
• k8s-testimages/kubekins -> k8s.gcr.io/job-images/kubekins (k8s-staging-job-images)

Or, if we're just doing a lift-and-shift... then k8s-staging-test-infra

[justaugustus]

@spiffxp -- Maybe a few different ownership levels to consider here:

• common test-infra images
• restricted access test-infra images (this may not be a required category)
• shared access

In the shared access "tier" would be kubekins-e2e and maybe krte.
https://github.com/kubernetes/test-infra/blob/master/images/kubekins-e2e/OWNERS

I have approver on kubekins-e2e and I think this IAM group covers that case:

k8s.io/groups/sig-release/groups.yaml

Lines 20 to 34 in 35c5108

	- email-id: k8s-infra-google-build-admins@kubernetes.io
	name: k8s-infra-google-build-admins
	description: |-
	ACL for Google Build Admins (edit access to Docker Hub mirror, view
	access to Release GCP projects)
	https://git.k8s.io/sig-release/release-managers.md#build-admins
	settings:
	ReconcileMembers: "true"
	members:
	- k8s-infra-release-editors@kubernetes.io
	- amwat@google.com
	- bentheelder@google.com
	- mushuee@google.com
	- spiffxp@google.com

I think the closest existing staging project would be k8s-infra-staging-build-image and I'd be happy to have y'all have access to that one.

[spiffxp]

@spiffxp -- Maybe a few different ownership levels to consider here:

• common test-infra images

• restricted access test-infra images (this may not be a required category)

• shared access

In the shared access "tier" would be kubekins-e2e and maybe krte.

https://github.com/kubernetes/test-infra/blob/master/images/kubekins-e2e/OWNERS

I have approver on kubekins-e2e and I think this IAM group covers that case:

k8s.io/groups/sig-release/groups.yaml

Lines 20 to 34 in 35c5108

	- email-id: k8s-infra-google-build-admins@kubernetes.io
	name: k8s-infra-google-build-admins
	description: |-
	ACL for Google Build Admins (edit access to Docker Hub mirror, view
	access to Release GCP projects)
	https://git.k8s.io/sig-release/release-managers.md#build-admins
	settings:
	ReconcileMembers: "true"
	members:
	- k8s-infra-release-editors@kubernetes.io
	- amwat@google.com
	- bentheelder@google.com
	- mushuee@google.com
	- spiffxp@google.com

I think the closest existing staging project would be k8s-infra-staging-build-image and I'd be happy to have y'all have access to that one.

I agree with putting kubekins-e2e somewhere shared / under releng purview.

I don't think it should be build-images though. Thinking toward applying policies for build provenance and security audit for the build chain of k8s releases. The kubekins-e2e image is an organically evolved mess, easier to keep it out of that repo than attempt to filter policy on certain image names.

I'm thinking of:

• gcr.io/k8s-staging-test-infra/images for all images built from kubernetes/test-infra/images
• give releng access to test-infra for now
• choose somewhere else for eventual destination for kubekins-e2e and its ilk
  • releng? Is for "releng tooling" but these are more about ci for kubernetes
  • ci-images? Is for ci releases of kubernetes components, still not the best fit, but prow can push directly to it (admittedly a special case for now that could be generalized later)
• for each image
  • push to both old k8s-testimages and new destination repo (default to test-infra/images for everything but kubekins-e2e)
  • backfill for at least kubekins-e2e
  • migrate job configs to use new repo
  • stop pushing to k8s-testimages
• revoke releng push access to test-infra when we've migrated remaining shared images out

My preference for shared repo would be releng, you've started moving some kubernetes job images there already IIRC

But ci-images also sounds like a better name, so not a strong preference

Hold on migrating kubekins until after code freeze but move on everything else, and see what else makes sense to move to shared from there

WDYT?

[spiffxp]

Ping @BenTheElder and @kubernetes/release-managers for comment

[spiffxp]

Another bit of followup to consider, setup auto bumping of images used in jobs: kubernetes/test-infra#21137