Root kubernetes images are signed with wrong `docker-reference`

[saschagrunert]

What happened:

We always prefix the .critical.identity.docker-reference signature entry for our root images with /kubernetes:

> cosign verify registry.k8s.io/kube-apiserver-amd64:v1.28.0-alpha.3 \
        --certificate-identity krel-trust@k8s-releng-prod.iam.gserviceaccount.com \
        --certificate-oidc-issuer https://accounts.google.com \
        | jq -r '.[0].critical.identity."docker-reference"'
…

registry.k8s.io/kubernetes/kube-apiserver-amd64

Notice that registry.k8s.io/kubernetes should be registry.k8s.io.

It's fine for our prefixed images, though:

> cosign verify registry.k8s.io/kubernetes/kube-apiserver-amd64:v1.28.0-alpha.3 \
        --certificate-identity krel-trust@k8s-releng-prod.iam.gserviceaccount.com \
        --certificate-oidc-issuer https://accounts.google.com \
        | jq -r '.[0].critical.identity."docker-reference"'
…

registry.k8s.io/kubernetes/kube-apiserver-amd64

[saschagrunert]

Note that it seems to be wrong the other way around for a different image:

> cosign verify registry.k8s.io/kubernetes/kube-controller-manager-amd64:v1.28.0-alpha.3 \
        --certificate-identity krel-trust@k8s-releng-prod.iam.gserviceaccount.com \
        --certificate-oidc-issuer https://accounts.google.com \
        | jq -r '.[0].critical.identity."docker-reference"'
…

registry.k8s.io/kube-controller-manager-amd64

> cosign verify registry.k8s.io/kube-controller-manager-amd64:v1.28.0-alpha.3 \
        --certificate-identity krel-trust@k8s-releng-prod.iam.gserviceaccount.com \
        --certificate-oidc-issuer https://accounts.google.com \
        | jq -r '.[0].critical.identity."docker-reference"'
…

registry.k8s.io/kube-controller-manager-amd64

[saschagrunert]

I think that's an issue with how we treat the prefix /kubernetes, because the rewrite function for the registry receives mixed input:

time="16:26:36.787" level=info msg="Using new production registry reference for image signature: registry.k8s.io/kubernetes/kube-proxy" diff=65ms
time="16:26:36.787" level=info msg="Using new production registry reference for image signature: registry.k8s.io/kube-controller-manager-amd64" diff=0s

from: https://storage.googleapis.com/kubernetes-jenkins/logs/post-k8sio-image-promo/1671186250071019520/build-log.txt

Source code part:

promo-tools/internal/promoter/image/sign.go

Line 224 in d5fbed8

logrus.Infof("Using new production registry reference for image signature: %v", newRef)

[saschagrunert]

We only sign 30 images, I think it's because it's sorted by digest and that overrides each other (which has been done on purpose):

promo-tools/internal/promoter/image/sign.go

Lines 124 to 127 in d5fbed8

	// We only sign the first image of each edge. If there are more
	// than one destination registries for an image, we copy the
	// signature to avoid varying valid signatures in each registry.
	sortedEdges := map[image.Digest][]reg.PromotionEdge{}