RwLock poisoned

[rillian]

Our current test deployment is rejecting queries:

$ curl https://star-randsrv.bsg.brave.software/info
{"message":"Couldn't lock state: RwLock poisoned"}

This likely means the epoch update loop panicked while holding the write lock.

Also, it seems like we need to take steps to terminate the process on panic. I expected the various panic calls in that function to halt the server so k8s would restart, but tokio-runtime-worker catches it at the thread boundary and continues serving new connections.

[rillian]

Suggestions how the reproduce this would be welcome. I've never seen it in testing!

[rillian]

I think this is probably a double-puncture after the epoch counter overflows. A debug build with panic on the overflow itself. So, consequence of a known issue. Testing is hard, even in Rust.

[rillian]

Looks like the ppoprf key is rotating smoothly as of 2adf203.