[ws-daemon] Retry git clone

[svenefftinge]

Bug description

We regularly (multiple times per day) see random workspace init failures with an error message like:

cannot initialize workspace:
    github.com/gitpod-io/gitpod/content-service/pkg/initializer.InitializeWorkspace
        github.com/gitpod-io/gitpod/[email protected]/pkg/initializer/initializer.go:344
  - git initializer:
    github.com/gitpod-io/gitpod/content-service/pkg/initializer.(*GitInitializer).Run
        github.com/gitpod-io/gitpod/[email protected]/pkg/initializer/git.go:69
  - git clone https://github.com/midwayjs/midway.git . failed (exit status 128): Cloning into '.'...
fatal: Unable to create temporary file '/dst/midway/.git/objects/pack/tmp_idx_XXXXXX': No such file or directory
fatal: index-pack failed

Search for jsonPayload.message="content init failed"

A quick search didn't reveal any insights. The error seems to happen occasionally across the internet.
We might mitigate this with multiple attempts until we call it a failure.

[jankeromnes]

Something similar just bit me as well in #4167 (comment): in my io-dev environment, one specific incremental prebuild was apparently successful, but caused one specific commit context URL to always with:

cannot initialize workspace: cannot initialize workspace: content initializer failed

Checking Google Cloud as described above revealed this complete error message:

cannot initialize workspace:
    github.com/gitpod-io/gitpod/content-service/pkg/initializer.InitializeWorkspace
        github.com/gitpod-io/gitpod/[email protected]/pkg/initializer/initializer.go:344
  - prebuild initializer:
    github.com/gitpod-io/gitpod/content-service/pkg/initializer.(*PrebuildInitializer).Run
        github.com/gitpod-io/gitpod/[email protected]/pkg/initializer/prebuild.go:96
  - git stash push -u failed (exit status 1): components/supervisor/go.sum: needs merge
components/workspacekit/go.sum: needs merge
components/ws-daemon/go.sum: needs merge
components/ws-manager/go.sum: needs merge

EDIT: Seems unrelated to this issue though (because retrying git clone wouldn't help)