Disk imports slow down and high coachroach usage #11673

While uploading disk images each transaction is chunked into 512KB chunks, perhaps to match the crucible/pantry backend.

The issue: doing some tracing it seems every 512KB chunk opens a new nexus transaction, leading to ~48 SQL queries. If we could increase the size at least the frontend accepts and then auth once, we could see greatly better speeds. With users wanting to upload 8GB or 100GB images, the current system can be slow and hits the DB very hard. Tests show around 6,300 SQL transactions a second at 64MB/s upload.

Currently there is a --parallelism flag in oxide.rs, which to the user speeds up the upload, but also just multiplies this issue.

There is a commend from several years ago that doing uploads in 512kb would be too expensive,

A trace on a dev box shows for a 512KB upload: