Tracking issue for RFD 739 (Instance restart during live update) #11169

Activity

1. karencfv

   self-assigned this

   on Aug 26, 2026
2. karencfv

   added a commit that references this issue on Oct 7, 2026

   [reconfigurator] background task for marking VMMs to be stopped for u…

   ...

   d43c79e
3. karencfv commented on Oct 13, 2026

   

   karencfv

   on Oct 13, 2026

   Last edited by karencfv

   ContributorAuthor

   More actions

   I have a few thoughts about the 4th task in the list here (Modify the instance update saga to account for VMMs that stopped due to an update).

   The first idea was to have a marker on the Instance struct to record that it is eligible for a restart that is similar to the stop_for_update_disposition_generation field on the Vmm one.

   This approach has several problems:

   • If we set a check constraint on the table where this marker can only be set if the state is NoVmm, we can have db errors if the background task for stopping the VMMs races with sled-agent notifying Nexus to update the instance with the NoVmm state. We could integrate the logic for updating this field within the notify_vmm_updated method, but that may overcomplicate the logic there, and we also have the instance watcher background task and sled agent trying publish the new state to Nexus. There are just too many possibilities of a race here.
   • If we don't set the check constraint we have the same risks of adding the marker when the instance is not in the correct state.
   • Unlike the Vmm marker, which just stays marked once the VMM is gone, we would also have to add logic to remove that marker. And what happens if the task to "un-mark" that field fails? We can end up with all sorts of unexpected behaviour due to a borked state/marker mix.
   • NoVmm is part of InstanceState::NOT_INCARNATED_STATES. This constant is used to allow users to update cpu/memory and change the boot disk when the instance is in one of these states. We may want to allow users to update these parameters during an update in the future, but for now we should take small steps. During an update there are a lot of things going on, we should be conservative and disallow these changes to start with. Once we see the process running smoothly we can relax these constraints and allow users to update cpu/memory/boot disk during an update.
   • Other actions like disk attach/detach, deletion, NIC changes and external IP attaches allow changes when the Instance is in a NoVmm state. for the same reasons above, we want to disallow those actions during an update for now.

   My second idea was to modify the NoVmm state to contain the marker within it, turn it into NoVmm { pending_reincarnation: Option<UpdateDispositionGeneration> }. With this option we avoid all of the racy implications of the previous option, but it leaves one. We would have to handle the case of making a change to the same state when a user actually stops the instance after the stopping background task has used the sled-agent/nexus machinery to set pending_reincarnation to Some(_).

   The more I think about it this pending_reincarnation field feels more and more like a distinct state. We would certainly be treating it as one.

   My proposal here is to create a new InstanceState::NoVmmPendingReincarnation (or some other better name) state.

   Thoughts? @hawkw @jgallagher @davepacheco @sunshowers
4. hawkw commented on Oct 13, 2026

   

   hawkw

   on Oct 13, 2026

   Member

   More actions

   The more I think about it this pending_reincarnation field feels more and more like a distinct state. We would certainly be treating it as one.
   My proposal here is to create a new InstanceState::NoVmmPendingReincarnation (or some other better name) state.

   @karencfv at a glance, I think the idea of a new state makes sense to me, especially because I agree that we probably want to assess what operations can be performed on instances in this state separately from those that are Stopped, as you discussed. I also think the idea of a whole new InstanceState makes sense here because I think these instances should show up in the external API as Rebooting, rather than Stopped: I think we should be communicating to the user that the instance is not running at this time but that we are going to restart it for you automatically, and I'd like it if we preserved the semantics of Stopped instances not spontaneously restarting themselves. A separate internal InstanceState variant makes mapping it to a different external API state than NoVmm easy.

   Personally, I'd probably call the new internal state StoppedForEvacuation, or even just Evacuated. I think the name of the state should probably not say that it was for an update, since I imagine we might want to reuse the same behavior if we were to add an "evacuate a sled and restart its instances elsewhere in order to perform maintenance on that sled" operator action, which it sounds like some customers would like.

   👍React with 👍1Reacted by Karen Cárcamo
5. karencfv

   added 2 commits that reference this issue on Oct 27, 2026

   [reconfigurator] background task for stopping marked VMMs (#11380)

   ...

   af0a32e

   [live-update] adapt instance update saga to account for VMMs stopped …

   ...

   b1a0d50