Skip to content

[reconfigurator] background task for stopping marked VMMs #11380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
karencfv merged 46 commits into from
Oct 27, 2026
Merged

[reconfigurator] background task for stopping marked VMMs #11380

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
310551c
[reconfigurator] background task for stopping marked VMMs
karencfv Sep 30, 2026
e1f14c3
let's do batches
karencfv Oct 2, 2026
99e577d
fix some comments
karencfv Oct 5, 2026
cb3ea34
use IdOrdMap everywhere
karencfv Oct 5, 2026
b1a9a19
have nicer return types
karencfv Oct 5, 2026
055b6b0
pondering...
karencfv Oct 5, 2026
81b1f62
merge main
karencfv Oct 6, 2026
a8f3e4e
Do the stopping and add a bunch of random thoughts
karencfv Oct 6, 2026
787853b
merge main
karencfv Oct 7, 2026
f4fea86
integrate with vmm marking bg task
karencfv Oct 7, 2026
a91d7d1
handle all possible scenarios attempting to stop a VMM
karencfv Oct 7, 2026
56d116c
Do not change the intended state if we're stopping for an update
karencfv Oct 7, 2026
267ed82
better logging
karencfv Oct 7, 2026
5f3b05e
expectorate
karencfv Oct 7, 2026
bf73666
add test for task
karencfv Oct 8, 2026
3a62920
organise test
karencfv Oct 8, 2026
8c2fdae
clean up
karencfv Oct 8, 2026
6196cb9
add db-query test
karencfv Oct 8, 2026
9ef8b8e
clean up
karencfv Oct 8, 2026
97e2afc
formatting
karencfv Oct 12, 2026
ce41e5e
call notify_instance_updated
karencfv Oct 12, 2026
f076ccb
add comment
karencfv Oct 13, 2026
1a63679
change interval for running task
karencfv Oct 15, 2026
26cb9ac
remove nexus
karencfv Oct 15, 2026
26779a3
one reqwest client
karencfv Oct 15, 2026
bc3cb0f
typos and small fixes
karencfv Oct 19, 2026
14a620f
use paginated_multicolumn
karencfv Oct 19, 2026
c0154d4
put the client where it should go
karencfv Oct 19, 2026
9f418e3
parallel tasks
karencfv Oct 19, 2026
6d41c21
use BTreeSet and leave TODOs for tomorrow
karencfv Oct 19, 2026
e42049b
omfg
karencfv Oct 19, 2026
179a441
log the instance id
karencfv Oct 19, 2026
29db8ef
handle vmm_gone
karencfv Oct 20, 2026
f56309c
optimise the optimisation
karencfv Oct 20, 2026
a603bfa
clean up
karencfv Oct 20, 2026
80eafe6
improve documentation
karencfv Oct 20, 2026
a30797d
make field more explicit
karencfv Oct 20, 2026
2533d11
cache the reqwest client
karencfv Oct 20, 2026
73529c9
fix logs
karencfv Oct 21, 2026
1e77105
fix typos
karencfv Oct 21, 2026
7355d30
do not stop Creating VMMs
karencfv Oct 22, 2026
a0b22ca
only retrieve vmms in in-service sleds
karencfv Oct 22, 2026
6e18f90
adapt tests
karencfv Oct 22, 2026
63d3678
fix typos before John finds them
karencfv Oct 22, 2026
54357d1
address comments
karencfv Oct 27, 2026
304c2f8
improve testing on failures
karencfv Oct 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions dev-tools/omdb/src/bin/omdb/nexus.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1296,6 +1296,9 @@ fn print_task_details(bgtask: &BackgroundTask, details: &serde_json::Value) {
"abandoned_vmm_reaper" => {
print_task_abandoned_vmm_reaper(details);
}
"vmm_stop_for_update" => {
print_task_vmm_stop_for_update(details);
}
"attached_subnet_manager" => {
print_task_attached_subnet_manager_status(details);
}
Expand Down Expand Up @@ -1510,6 +1513,58 @@ fn print_task_abandoned_vmm_reaper(details: &serde_json::Value) {
};
}

fn print_task_vmm_stop_for_update(details: &serde_json::Value) {
use nexus_types::internal_api::background::VmmStopForUpdateStatus;

match serde_json::from_value::<VmmStopForUpdateStatus>(details.clone()) {
Err(error) => eprintln!(
"warning: failed to interpret task details: {:?}: {:?}",
error, details
),
Ok(VmmStopForUpdateStatus {
vmms_stopped_by_sled,
vmms_failed_to_stop_by_sled,
error_messages,
}) => {
let total: usize =
vmms_stopped_by_sled.iter().map(|s| s.vmm_ids.len()).sum();
println!(" VMMs stopped for sled evacuation: {total} total");
for sled in vmms_stopped_by_sled {
println!(
" sled_id={} n_stopped={}",
sled.sled_id,
sled.vmm_ids.len()
);
}
if !vmms_failed_to_stop_by_sled.is_empty() {
let total_failed: usize = vmms_failed_to_stop_by_sled
.iter()
.map(|s| s.vmm_ids.len())
.sum();
println!(" VMMs that failed to stop: {total_failed} total");
for sled in vmms_failed_to_stop_by_sled {
println!(
" sled_id={} n_failed_to_stop={}",
sled.sled_id,
sled.vmm_ids.len()
);
for vmm in sled.vmm_ids {
println!(" vmm_id={}", vmm);
}
}
}

if !error_messages.is_empty() {
let total_err = error_messages.len();
println!(" Error messages: {total_err} total");
for error in error_messages {
println!(" {error}");
}
}
}
};
}

fn print_task_blueprint_planner(details: &serde_json::Value) {
let status =
match serde_json::from_value::<BlueprintPlannerStatus>(details.clone())
Expand Down
12 changes: 12 additions & 0 deletions dev-tools/omdb/tests/env.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,10 @@ task: "vmm_mark_stop_for_update"
marks VMMs on evacuating sleds as needing to be stopped for an update


task: "vmm_stop_for_update"
stops VMMs that are marked to be stopped for a sled update


task: "vpc_route_manager"
propagates updated VPC routes to all OPTE ports

Expand Down Expand Up @@ -547,6 +551,10 @@ task: "vmm_mark_stop_for_update"
marks VMMs on evacuating sleds as needing to be stopped for an update


task: "vmm_stop_for_update"
stops VMMs that are marked to be stopped for a sled update


task: "vpc_route_manager"
propagates updated VPC routes to all OPTE ports

Expand Down Expand Up @@ -810,6 +818,10 @@ task: "vmm_mark_stop_for_update"
marks VMMs on evacuating sleds as needing to be stopped for an update


task: "vmm_stop_for_update"
stops VMMs that are marked to be stopped for a sled update


task: "vpc_route_manager"
propagates updated VPC routes to all OPTE ports

Expand Down
16 changes: 16 additions & 0 deletions dev-tools/omdb/tests/successes.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -601,6 +601,10 @@ task: "vmm_mark_stop_for_update"
marks VMMs on evacuating sleds as needing to be stopped for an update


task: "vmm_stop_for_update"
stops VMMs that are marked to be stopped for a sled update


task: "vpc_route_manager"
propagates updated VPC routes to all OPTE ports

Expand Down Expand Up @@ -1209,6 +1213,12 @@ task: "vmm_mark_stop_for_update"
batch size: 1000
VMMs marked to be stopped for an update: 0

task: "vmm_stop_for_update"
configured period: every <REDACTED_DURATION>m
last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
VMMs stopped for sled evacuation: 0 total

task: "vpc_route_manager"
configured period: every <REDACTED_DURATION>s
last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
Expand Down Expand Up @@ -1946,6 +1956,12 @@ task: "vmm_mark_stop_for_update"
batch size: 1000
VMMs marked to be stopped for an update: 0

task: "vmm_stop_for_update"
configured period: every <REDACTED_DURATION>m
last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
VMMs stopped for sled evacuation: 0 total

task: "vpc_route_manager"
configured period: every <REDACTED_DURATION>s
last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
Expand Down
16 changes: 16 additions & 0 deletions nexus-config/src/nexus_config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -457,6 +457,9 @@ pub struct BackgroundTaskConfig {
pub v2p_mapping_propagation: V2PMappingPropagationConfig,
/// configuration for abandoned VMM reaper task
pub abandoned_vmm_reaper: AbandonedVmmReaperConfig,
/// configuration for the task that stops VMMs marked to be stopped for an
/// update
pub vmm_stop_for_update: VmmStopForUpdateConfig,
/// configuration for saga recovery task
pub saga_recovery: SagaRecoveryConfig,
/// configuration for lookup region port task
Expand Down Expand Up @@ -825,6 +828,14 @@ pub struct AbandonedVmmReaperConfig {
pub period_secs: Duration,
}

#[serde_as]
#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)]
pub struct VmmStopForUpdateConfig {
/// period (in seconds) for periodic activations of this background task
#[serde_as(as = "DurationSeconds<u64>")]
pub period_secs: Duration,
}

#[serde_as]
#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)]
pub struct SagaRecoveryConfig {
Expand Down Expand Up @@ -1387,6 +1398,7 @@ mod test {
service_firewall_propagation.period_secs = 300
v2p_mapping_propagation.period_secs = 30
abandoned_vmm_reaper.period_secs = 60
vmm_stop_for_update.period_secs = 300
saga_recovery.period_secs = 60
lookup_region_port.period_secs = 60
region_snapshot_replacement_start.period_secs = 30
Expand Down Expand Up @@ -1627,6 +1639,9 @@ mod test {
abandoned_vmm_reaper: AbandonedVmmReaperConfig {
period_secs: Duration::from_secs(60),
},
vmm_stop_for_update: VmmStopForUpdateConfig {
period_secs: Duration::from_secs(300),
},
saga_recovery: SagaRecoveryConfig {
period_secs: Duration::from_secs(60),
},
Expand Down Expand Up @@ -1803,6 +1818,7 @@ mod test {
service_firewall_propagation.period_secs = 300
v2p_mapping_propagation.period_secs = 30
abandoned_vmm_reaper.period_secs = 60
vmm_stop_for_update.period_secs = 300
saga_recovery.period_secs = 60
lookup_region_port.period_secs = 60
region_snapshot_replacement_start.period_secs = 30
Expand Down
1 change: 1 addition & 0 deletions nexus/background-task-interface/src/init.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ pub struct BackgroundTasks {
pub task_instance_reincarnation: Activator,
pub task_service_firewall_propagation: Activator,
pub task_abandoned_vmm_reaper: Activator,
pub task_vmm_stop_for_update: Activator,
pub task_audit_log_cleanup: Activator,
pub task_audit_log_timeout_incomplete: Activator,
pub task_vpc_route_manager: Activator,
Expand Down
61 changes: 57 additions & 4 deletions nexus/db-model/src/vmm_state.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,16 +107,29 @@ impl VmmState {
self.to_nexus_state().exists_on_sled()
}

/// The states in which a VMM should be marked to be stopped during sled
/// evacuation.
pub const SHOULD_MARK_TO_STOP_FOR_EVACUATION: &'static [Self] = &[
// A VMM in one of these states is on its way, or is already running,
// and can be marked to be stopped.
VmmState::Creating,
VmmState::Starting,
VmmState::Running,
VmmState::Rebooting,
// If it is not in one of these states, it is already stopping/stopped,
// migrating, or has terminated, so it does not need to be marked.
];

/// The states in which a VMM should be stopped during sled evacuation.
pub const SHOULD_STOP_FOR_EVACUATION: &'static [Self] = &[
// A VMM in one of these states is on its way, or is already running,
// and can be stopped.
VmmState::Creating,
// is registered with a sled agent, and can be stopped.
VmmState::Starting,
VmmState::Running,
VmmState::Rebooting,
// If it is not in one of these states, it is already stopping/stopped,
// migrating, or has terminated, so it does not need to be stopped.
// migrating, is not registered with a sled agent yet, or has
// terminated, so it does not need to be stopped.
];
}

Expand Down Expand Up @@ -451,7 +464,7 @@ mod tests {
}

#[test]
fn test_should_stop_for_update_states_consistent() {
fn test_should_mark_to_stop_for_update_states_consistent() {
// This test exists to make sure that when a new state is added, we
// consider whether it is a state where reconfigurator should mark the
// VMM to be stopped for sled evacuation. More detailed information about
Expand All @@ -469,12 +482,52 @@ mod tests {
VmmState::Creating
| VmmState::Starting
| VmmState::Running
| VmmState::Rebooting => {
assert!(
VmmState::SHOULD_MARK_TO_STOP_FOR_EVACUATION
.contains(state)
);
}
VmmState::Failed
| VmmState::Stopping
| VmmState::Stopped
| VmmState::Migrating
| VmmState::Destroyed
| VmmState::SagaUnwound => {
assert!(
!VmmState::SHOULD_MARK_TO_STOP_FOR_EVACUATION
.contains(state)
);
}
}
}
}

#[test]
fn test_should_stop_for_update_states_consistent() {
// This test exists to make sure that when a new state is added, we
// consider whether it is a state where reconfigurator should stop a VMM
// for sled evacuation. More detailed information about the process of
// restarting instances during a live update can be found in RFD 739.
//
// The use of an exhaustive `match state` here ensures that the addition
// of a new `VmmState` variant will result in a compiler error until it
// is added to this test. If you have added a new variant to VmmState,
// consider whether or not that state should be added to the list of
// states in which VMMs are stopped. Do *not* change the match
// here to be non-exhaustive!
for state in VmmState::ALL_STATES.iter() {
match state {
VmmState::Starting
| VmmState::Running
| VmmState::Rebooting => {
assert!(
VmmState::SHOULD_STOP_FOR_EVACUATION.contains(state)
);
}

VmmState::Failed
| VmmState::Creating
| VmmState::Stopping
| VmmState::Stopped
| VmmState::Migrating
Expand Down
Loading
Loading