[server] check the prebuild snapshot is valid before start the workspace

[jenting]

Bug description

We observed that starting from Jan 8, 2023, the workspace start prebuild error rate is increasing.
We checked the Jaeger tracing, the ws-manager can't find the related snapshot on the object storage, causing the prebuild error rate starts burning.

The Jarger tracing [1], [2], and [3].

Steps to reproduce

We don't know yet.

Workspace affected

fuchsia-fly-xkuqh3dh5fh

Expected behavior

The server should check the snapshot is valid in the DB before sending the request to the ws-manager.

Example repository

No response

Anything else?

Reference to #15721 (comment)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.