today in Propolis we always have the NVMe devices claim there is a volatile write cache: https://github.com/oxidecomputer/propolis/blob/979b72896bc39a876c98fb32fe706da37a4fc408/lib/propolis/src/hw/nvme/mod.rs#L851-L852
I'd like to make this configurable in Propolis: oxidecomputer/propolis#1170 . at this point, Nexus will need to be able to tell Propolis if the storage backing a disk does or doesn't need to set the bit. the NVMe spec describes this in spec-ese:
If the controller is able to guarantee that data present in a write cache is written to non-volatile storage media on loss of power, then that write cache is considered non-volatile
which is to say you can have caches just fine (Crucible could, even!), but you must set the VWC bit if those caches lose data across loss of power. of course, the moral equivalent for us is power loss, crash, etc.
in particular, for local disks backed by devices we know do not have write caches, we can direct Propolis to not report write caches in those emulated controllers. this nudges guests into not even bother sending us flushes, because we're telling the guest they're not necessary. in flush-oriented workloads this can be meaningful (and I think this is relevant in cases like closing a file you've written to, where you would send a flush to get data sync'd out). everywhere else and by default we should still set the bit.
anyhow, the determinations by Nexus here still need to be scrutinized: we must not direct Propolis to a disk with VWC semantics to not report it; that path leads to data loss. it's an outright invalid configuration we can't let happen.
today in Propolis we always have the NVMe devices claim there is a volatile write cache: https://github.com/oxidecomputer/propolis/blob/979b72896bc39a876c98fb32fe706da37a4fc408/lib/propolis/src/hw/nvme/mod.rs#L851-L852
I'd like to make this configurable in Propolis: oxidecomputer/propolis#1170 . at this point, Nexus will need to be able to tell Propolis if the storage backing a disk does or doesn't need to set the bit. the NVMe spec describes this in spec-ese:
which is to say you can have caches just fine (Crucible could, even!), but you must set the VWC bit if those caches lose data across loss of power. of course, the moral equivalent for us is power loss, crash, etc.
in particular, for local disks backed by devices we know do not have write caches, we can direct Propolis to not report write caches in those emulated controllers. this nudges guests into not even bother sending us flushes, because we're telling the guest they're not necessary. in flush-oriented workloads this can be meaningful (and I think this is relevant in cases like closing a file you've written to, where you would send a flush to get data sync'd out). everywhere else and by default we should still set the bit.
anyhow, the determinations by Nexus here still need to be scrutinized: we must not direct Propolis to a disk with VWC semantics to not report it; that path leads to data loss. it's an outright invalid configuration we can't let happen.