Asynchronous APIs

[smklein]

Note: async is overloaded. Usage here refers to an "asynchronous endpoint", as described in RFD 4, not the async Rust keyword.

As discussed in #385 (comment) , Omicron is growing a reasonably sized API surface. As @davepacheco mentioned, "most things that use sagas... probably ought to be async".

At the time of writing this bug, none of our endpoints are async. We would benefit from:

• Identifying long-running operations, and defining a set which are good candidates to be async (at minimum, "create instance" seems like a good candidate).
• Prototype a couple endpoints as async, returning an ID, and providing a wait-for-completion API.
• Iterate on ergonomics, providing utilities (or possibly changes to Dropshot) making these operations as easy to write as possible.

[zephraph]

Things discussed in the huddle today (with a bit of my own bias):

• We want a clear way to differentiate endpoints that are sync and endpoints that are async. This should definitely be encoded into the openapi spec. I'm not sure if it should be encoded into the route.
• When making a request that starts an async process, the request should result in a 202. My preference would be to adhere to a HATEOAS-lite approach and provide follow-up urls in the response of the capacities that we provide. i.e. status, long polling, webhook, websockets, etc. In essence a successful acknowledgement of async work should also describe how the status can be checked and where.
• If possible it would be nice to be able to get the status of an async process from the resource as well as having a generic list endpoint that details all the current tasks in progress. The latter would be useful in console for showing a user all current resource creation they have pending.
• Things that can happen synchronously should. It's preferable we fail fast in the case of something like a resource name collision rather than immediately queuing up the work only to fail shortly later.

[davepacheco]

Relatedly, a bunch of this is already described at least a bit in RFD 4, here and here. This proposes:

• a naming convention for identifying async operations. I imagine we'd also uses HTTP status code 202 Accepted, which means this will also be visible in the OpenAPI spec (the endpoint will return 202 and not 200 or 204).
• a convention for returning an operationId for subsequent operations
• API endpoints for fetching the status of an operation by operationId and waiting for them to finish

(To be clear, there's plenty of detail left out there, too, but it gives us a direction.)

In terms of which operations, my thought is that any Nexus request that reaches out to some other service aside from the database ought to be asynchronous and use a saga. Anything that uses a saga would be asynchronous and vice versa. There's related discussion in RFD 192.

In terms of prototyping some: I believe these endpoints are essentially already async:

• create instance
• start instance
• stop instance
• attach disk
• detach disk

But they do not do any of the RFD 4 things yet -- they don't produce a 202 or return an operation ID and we don't have the endpoints for waiting on those.

[andrewjstone]

When making a request that starts an async process, the request should result in a 202. My preference would be to adhere to a HATEOAS-lite approach and provide follow-up urls in the response of the capacities that we provide. i.e. status, long polling, webhook, websockets, etc. In essence a successful acknowledgement of async work should also describe how the status can be checked and where.

I think this is a great idea. I have used HATEOAS with a homegrown JSON schema in a prior life and it worked very well.