[image-builder] image build did not produce a workspace image

[csweichel]

Bug description

Sometimes the image builder mk3 does not produce an image, but errors with image build did not produce a workspace image

Steps to reproduce

Find a base image that exhibits this behaviour and run bob to build the workspace image

Expected behavior

No response

Example repository

No response

Anything else?

No response

[csweichel]

/schedule

[roboquat]

@csweichel: Issue scheduled in the workspace team (WIP: 0)

In response to this:

/schedule

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[iQQBot]

There are many reasons for this problem

1. ws-daemon is not to delete the mount file for imagebuilder which will end soon, resulting in a residual directory, which is not a big problem for workspace because each time the instance is started it will be a new random name, but imagebuilder is different, it is a calculated hash value, and if the mount folder is not yet freed, workspacekit can not prepare for user namespaces

// error from ws-daemon
{
	"@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",
	"error": "rpc error: code = AlreadyExists desc = workspace exists already",
	"instanceId": "a71217a8ee1b7416-c90b191be9db45bb-aa128829",
	"level": "error",
	"message": "InitWorkspace failed",
	"serviceContext": {
		"service": "ws-daemon",
		"version": ""
	},
	"severity": "ERROR",
	"time": "2021-10-03T10:02:32Z",
	"userId": "",
	"workspaceId": ""
}
//error from workspace
{
	"@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",
	"error": "rpc error: code = ResourceExhausted desc = can be used only once",
	"level": "fatal",
	"message": "cannot prepare for user namespaces",
	"ring": 0,
	"serviceContext": {
		"service": "workspacekit",
		"version": ""
	},
	"severity": "CRITICAL",
	"time": "2021-10-03T10:17:14Z"
}

2. imagebuilder uses crane for mirror replication, but this does not seem to be a good solution either. crane cannot distinguish between base and target auth, so if base is a mirror like gcr.io, even it no need auth, it will fail.
   

   gitpod/components/image-builder-bob/pkg/builder/builder.go

   Lines 207 to 218 in 48e5919

   	user, pass, err := authorizer.Authorize(tref.Hostname())
   	if err != nil {
   	return err
   	}
   	craneOpts = append(craneOpts, crane.WithAuth(authn.FromConfig(authn.AuthConfig{
   	Username: user,
   	Password: pass,
   	})))
   	}
   	return crane.Copy(b.Config.BaseRef, b.Config.TargetRef, craneOpts...)

   
   I reattempted to push using buildkit and was unable to reproduce the error you mentioned "cannot reuse body, request must be retried" on ctr images push containerd/containerd#5978 (comment), is there a more detailed way to reproduce it? Maybe we can try again, if containerd has this problem then docker pushing images to gcr.io should also fail, which should be a big problem

[iQQBot]

like this

source image is gcr.io , dest image is builtin registry