Shavar instance takes too long to start-up

[say-yawn]

About this Issue

As mentioned in here this issue happens only on prod. Look for ways to log and/or reduce the speed on the instance starting.

[smarnach]

We see a few more issues that only happen in prod that are potentially related:

• Sometimes instances get stuck on an old version of the data, while continuing to serve traffic just fine.
• Sometimes all instances get into a mysterious crash-loop state. This is less likely to happen when we run more instances, but it's not related to any traffic spikes. For this reason, we heavily overprovision the service. Currently we run about four times as many servers as we need, just to keep the service running reliably.

The only reason I think these issues may be related is that they also only happen in prod, and that they are also kind of mysterious and defied any attempts of debugging them so far.