[slo] start prebuild error budget is burning

[kylos101]

Bug description

We're recently burning through our start prebuild error budget too quickly.

Steps to reproduce

n/a

Workspace affected

No response

Expected behavior

The error budget should not be going down, unless we expect it to.

Example repository

No response

Anything else?

Reassign the issue to @Furisto (on-call now) during his normal working hours on Friday, but, while he sleeps, please research why it is burning, reach out to the on-call team as needed.

I assume our metric is flawed in gen82, or, there was a change (IDE or webapp) that is causing an unusually high # of prebuilds to fail starting.

[jenting]

Start workspace success ratio is dropping as well.

[jenting]

The GCP log for workspace failed and workspace pod never ready. Mostly errors related to

• No Git binary. -> this one requires user to update the base image with git binary inside. Related PR.
• cannot initialize workspace: cannot find snapshot.

[jenting]

cannot initialize workspace: cannot find snapshot.

The problem started on Jan 8, 6 PM.

I did an investigation, according to the Jaeger tracing [1], [2], and [3]. I can't find the related snapshot on the object storage.

[kylos101]

Start workspace success ratio is dropping as well.

Yes @jenting , however, burn rate is so low that I think it is okay? At least given it's current "burn".

[kylos101]

@jenting given the traces that you shared (thank you), I suspect what is happening, is that the URL to a snapshot was being used to start a new workspace, however the underlying snapshot is no longer valid (was probably deleted).

Why do I suspect a snapshot that has had its source worksapce deleted? The workspace ID (like fuchsia-fly-xkuqh3dh5fh) in all three cases was random, instead of org-repo-random.

When I check this trace, the initializer looks like this:

"initializer": {
        "snapshot": {
            "snapshot": "workspaces/<redacted>/snapshot-1670002948719429142.tar@gitpod-prod-user-<redacted>"
        }
    },

I created this snapshot from an empty workspace, which can be used to start a workspace. In my case, I just created harlequin-cat-uifimy1moab, which you can see a trace for here. The initializer is structured similarly as above:

  "initializer": {
        "snapshot": {
            "snapshot": "workspaces/<redacted>/snapshot-1673630874408709453.tar@gitpod-prod-user-<redacted>"
        }
    },

I just deleted the source workspace, but, I suspect server hasn't done the actual garbage collection (because I can still create new workspaces from the snapshot URL). So, that GC will likely take two weeks from now.

May I ask you to inbox an issue to webapp team? Ideally server would check a snapshot is valid, prior to trying to start a workspace. It might delegate that decision to content-service, but, should check before asking ws-manager to start a workspace.

cc: @geropl @svenefftinge

[kylos101]

@jenting please close this issue, after inboxing an issue to the webapp team? 🙏

[jenting]

@jenting please close this issue, after inboxing an issue to the webapp team? 🙏

Create two issues as follow up:

• [supervisor] workspace start fails if git is not available #15732 for injecting the git binary if not present in the base image
• [server] check the prebuild snapshot is valid before start the workspace #15746 for checking the prebuild snapshot is valid before sending request to the ws-manager

closing this issue.