Repository navigation
[nexus] fail instances if the sled they're on isn't...on - #10334
Conversation
60803f3 to
99f6974
Compare
|
Proof that the eliza@castle ~ $ pilot tp login any
The illumos Project helios-2.0.23916 April 2026
root@oxz_switch1:~# omdb db instances
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd9f:32e6:2c2f:104::3]:32221,[fd9f:32e6:2c2f:101::3]:32221,[fd9f:32e6:2c2f:104::4]:32221,[fd9f:32e6:2c2f:103::3]:32221,[fd9f:32e6:2c2f:102::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (253.0.0)
ID STATE INTENT PROPOLIS_ID SLED_ID HOST_SERIAL NAME
6287c3ea-b2dc-4144-b69c-328c54593d0c running running 0d530db0-4919-4043-a0d7-756261bbc1c9 21f08cd2-4330-4f5e-9b39-b5caabdbb4c0 2F8JEXDK my-computer
root@oxz_switch1:~# omdb db instance show -i 6287c3ea-b2dc-4144-b69c-328c54593d0c
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd9f:32e6:2c2f:104::3]:32221,[fd9f:32e6:2c2f:101::3]:32221,[fd9f:32e6:2c2f:104::4]:32221,[fd9f:32e6:2c2f:103::3]:32221,[fd9f:32e6:2c2f:102::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (253.0.0)
== INSTANCE ====================================================================
ID: 6287c3ea-b2dc-4144-b69c-328c54593d0c
project ID: 1bef82ae-6061-4f17-81d5-8280cd7d5013
name: my-computer
description:
created at: 2026-04-29 16:42:58.723930 UTC
last modified at: 2026-04-29 16:42:58.723930 UTC
== CONFIGURATION ===============================================================
vCPUs: 2
memory: 8 GiB
hostname: my-computer
boot disk: Some(cf3e17a9-b642-497a-bb75-013d1e8c4bbe)
auto-restart:
InstanceAutoRestart {
policy: None,
cooldown: None,
}
== RUNTIME STATE ===============================================================
nexus state: Vmm
(i) external API state: Running
intended state: running
last updated at: 2026-04-29T16:42:58.723930Z (generation 3)
needs reincarnation: false
karmic status: saṃsāra (reincarnation enabled)
last reincarnated at: None
active VMM ID: Some(0d530db0-4919-4043-a0d7-756261bbc1c9)
target VMM ID: None
migration ID: None
updater lock: UNLOCKED at generation: 1
== ACTIVE VMM ==================================================================
ID: 0d530db0-4919-4043-a0d7-756261bbc1c9
instance ID: 6287c3ea-b2dc-4144-b69c-328c54593d0c
created at: 2026-04-29 16:43:01.401519 UTC
state: running
updated at: 2026-04-29T16:43:08.532008Z (generation 4)
propolis address: fd9f:32e6:2c2f:104::1:0:12400
sled ID: 21f08cd2-4330-4f5e-9b39-b5caabdbb4c0
CPU platform: AmdMilan
== ATTACHED DISKS ==============================================================
# ID SIZE TYPE STATE NAME
0 cf3e17a9-b642-497a-bb75-013d1e8c4bbe 10 GiB crucible attached my-computer-debian-12-genericcloud-4f0bea
== VMM HISTORY =================================================================
ID STATE GEN SLED_ID TIME_CREATED TIME_DELETED
0d530db0-4919-4043-a0d7-756261bbc1c9 running 4 21f08cd2-4330-4f5e-9b39-b5caabdbb4c0 2026-04-29T16:43:01.401Z -
root@oxz_switch1:~# pilot sp off 2F8JEXDK
14 2F8JEXDK ok
root@oxz_switch1:~# omdb db instances
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd9f:32e6:2c2f:104::3]:32221,[fd9f:32e6:2c2f:101::3]:32221,[fd9f:32e6:2c2f:104::4]:32221,[fd9f:32e6:2c2f:103::3]:32221,[fd9f:32e6:2c2f:102::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (253.0.0)
ID STATE INTENT PROPOLIS_ID SLED_ID HOST_SERIAL NAME
6287c3ea-b2dc-4144-b69c-328c54593d0c running running 0d530db0-4919-4043-a0d7-756261bbc1c9 21f08cd2-4330-4f5e-9b39-b5caabdbb4c0 2F8JEXDK my-computer ^X^C
root@oxz_switch1:~# omdb -w nexus background-tasks activate instance_watcher
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd9f:32e6:2c2f:102::4]:12232
activated background tasks: instance_watcher
root@oxz_switch1:~# omdb -w nexus background-tasks show instance_watcher
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd9f:32e6:2c2f:102::4]:12232
task: "instance_watcher"
configured period: every 30s
currently executing: iter 2088, triggered by a periodic timer firing
started at 2026-04-29T16:48:36.146Z, running for 32456ms
last completed activation: iter 2087, triggered by a periodic timer firing
started at 2026-04-29T16:47:13.830Z (114s ago) and ran for 82311ms
total instances checked: 1
checks completed: 1
successful checks: 0
update sagas queued: 0
failed checks: 1
-> 1 sled_off
checks that could not be completed: 0
stale instance metrics pruned: 0
root@oxz_switch1:~# omdb db instance show -i 6287c3ea-b2dc-4144-b69c-328c54593d0c
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd9f:32e6:2c2f:104::3]:32221,[fd9f:32e6:2c2f:101::3]:32221,[fd9f:32e6:2c2f:104::4]:32221,[fd9f:32e6:2c2f:103::3]:32221,[fd9f:32e6:2c2f:102::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (253.0.0)
== INSTANCE ====================================================================
ID: 6287c3ea-b2dc-4144-b69c-328c54593d0c
project ID: 1bef82ae-6061-4f17-81d5-8280cd7d5013
name: my-computer
description:
created at: 2026-04-29 16:42:58.723930 UTC
last modified at: 2026-04-29 16:42:58.723930 UTC
== CONFIGURATION ===============================================================
vCPUs: 2
memory: 8 GiB
hostname: my-computer
boot disk: Some(cf3e17a9-b642-497a-bb75-013d1e8c4bbe)
auto-restart:
InstanceAutoRestart {
policy: None,
cooldown: None,
}
== RUNTIME STATE ===============================================================
nexus state: Vmm
(i) external API state: Stopping
intended state: running
last updated at: 2026-04-29T16:42:58.723930Z (generation 3)
needs reincarnation: false
karmic status: saṃsāra (reincarnation enabled)
last reincarnated at: None
active VMM ID: Some(0d530db0-4919-4043-a0d7-756261bbc1c9)
target VMM ID: None
migration ID: None
(i) updater lock: LOCKED by 4039e194-3196-4889-823b-fb61d0631354 at generation: 12
== ACTIVE VMM ==================================================================
ID: 0d530db0-4919-4043-a0d7-756261bbc1c9
instance ID: 6287c3ea-b2dc-4144-b69c-328c54593d0c
created at: 2026-04-29 16:43:01.401519 UTC
state: failed
updated at: 2026-04-29T16:48:32.995071Z (generation 5)
propolis address: fd9f:32e6:2c2f:104::1:0:12400
sled ID: 21f08cd2-4330-4f5e-9b39-b5caabdbb4c0
CPU platform: AmdMilan
== ATTACHED DISKS ==============================================================
# ID SIZE TYPE STATE NAME
0 cf3e17a9-b642-497a-bb75-013d1e8c4bbe 10 GiB crucible attached my-computer-debian-12-genericcloud-4f0bea
== VMM HISTORY =================================================================
ID STATE GEN SLED_ID TIME_CREATED TIME_DELETED
0d530db0-4919-4043-a0d7-756261bbc1c9 failed 5 21f08cd2-4330-4f5e-9b39-b5caabdbb4c0 2026-04-29T16:43:01.401Z -
^X^C
root@oxz_switch1:~# |
|
However, I think there is an additional problem here, which is that the instance has been sitting in root@oxz_switch1:~# omdb db saga running
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd9f:32e6:2c2f:104::3]:32221,[fd9f:32e6:2c2f:101::3]:32221,[fd9f:32e6:2c2f:104::4]:32221,[fd9f:32e6:2c2f:103::3]:32221,[fd9f:32e6:2c2f:102::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (253.0.0)
id | current_sec | time_created | name | state
--------------------------------------+--------------------------------------+--------------------------+-----------------------+---------
7b221169-91b6-4df1-a909-6a8e6baf1983 | b3807085-4172-4665-8194-9db8f799448e | 2026-04-29T16:54:55.661Z | instance-update | Running
aae74a5b-8b70-4440-963d-9f362365644d | b3807085-4172-4665-8194-9db8f799448e | 2026-04-29T16:54:55.597Z | start-instance-update | Running
^X^C
root@oxz_switch1:~# omdb db saga show 7b221169-91b6-4df1-a909-6a8e6baf1983
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd9f:32e6:2c2f:104::3]:32221,[fd9f:32e6:2c2f:101::3]:32221,[fd9f:32e6:2c2f:104::4]:32221,[fd9f:32e6:2c2f:103::3]:32221,[fd9f:32e6:2c2f:102::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (253.0.0)
id | current_sec | time_created | name | state
--------------------------------------+--------------------------------------+--------------------------+-----------------+-------
7b221169-91b6-4df1-a909-6a8e6baf1983 | b3807085-4172-4665-8194-9db8f799448e | 2026-04-29T16:54:55.661Z | instance-update | Done
DAG: {"end_node":14,"graph":{"edge_property":"directed","edges":[[0,1,null],[1,2,null],[2,3,null],[3,4,null],[4,5,null],[5,6,null],[6,7,null],[7,8,null],[9,10,null],[8,9,null],[10,11,null],[11,12,null],[13,0,null],[12,14,null]],"node_holes":[],"nodes":[{"Action":{"action_name":"common.uuid_generate","label":"GenerateInstanceLockId","name":"saga_instance_lock_id"}},{"Action":{"action_name":"instance_update.become_updater","label":"BecomeUpdater","name":"updater_lock"}},{"Constant":{"name":"network_config_update","value":"Delete"}},{"Action":{"action_name":"instance_update.update_network_config","label":"UpdateNetworkConfig","name":"update_network_config"}},{"Action":{"action_name":"instance_update.release_virtual_provisioning","label":"ReleaseVirtualProvisioning","name":"release_virtual_provisioning"}},{"Action":{"action_name":"instance_update.unassign_oximeter_producer","label":"UnassignOximeterProducer","name":"unassign_oximeter_producer"}},{"Action":{"action_name":"instance_update.commit_instance_updates","label":"CommitInstanceUpdates","name":"commit_instance_updates"}},{"Constant":{"name":"destroy_active_vmm_params","value":{"instance_id":"6287c3ea-b2dc-4144-b69c-328c54593d0c","serialized_authn":{"kind":{"Authenticated":[{"actor":{"UserBuiltin":{"user_builtin_id":"001de000-05e4-4000-8000-000000000002"}},"credential_id":null,"device_token_expiration":null},null]}},"vmm_id":"0d530db0-4919-4043-a0d7-756261bbc1c9"}}},{"SubsagaStart":{"params_node_name":"destroy_active_vmm_params","saga_name":"destroy_active_vmm"}},{"Action":{"action_name":"destroy_vmm.release_sled_resources","label":"ReleaseSledResources","name":"release_sled_resources"}},{"Action":{"action_name":"destroy_vmm.mark_vmm_deleted","label":"MarkVmmDeleted","name":"mark_vmm_deleted"}},{"SubsagaEnd":{"name":"active_vmm_destroyed"}},{"Action":{"action_name":"instance_update.chain_successor_saga","label":"ChainSuccessorSaga","name":"chain_successor_saga"}},{"Start":{"params":{"authz_instance":{"key":"6287c3ea-b2dc-4144-b69c-328c54593d0c","lookup_type":{"ById":"6287c3ea-b2dc-4144-b69c-328c54593d0c"},"parent":{"key":"1bef82ae-6061-4f17-81d5-8280cd7d5013","lookup_type":{"ById":"1bef82ae-6061-4f17-81d5-8280cd7d5013"},"parent":{"key":"2d1cf137-3b71-42cf-869a-f455d8edf75a","lookup_type":{"ById":"2d1cf137-3b71-42cf-869a-f455d8edf75a"},"parent":null}}},"orig_lock":{"locked_gen":77,"updater_id":"661341b7-041c-4a6b-a8d2-bad08b2e5a2b"},"serialized_authn":{"kind":{"Authenticated":[{"actor":{"UserBuiltin":{"user_builtin_id":"001de000-05e4-4000-8000-000000000002"}},"credential_id":null,"device_token_expiration":null},null]}},"update":{"deprovision":{"cpus_diff":2,"project_id":"1bef82ae-6061-4f17-81d5-8280cd7d5013","ram_diff":8589934592},"destroy_active_vmm":"0d530db0-4919-4043-a0d7-756261bbc1c9","destroy_target_vmm":null,"network_config":"Delete","new_intent":null,"new_runtime":{"dst_propolis_id":null,"gen":4,"migration_id":null,"nexus_state":"Failed","propolis_id":null,"time_last_auto_restarted":null,"time_updated":"2026-04-29T16:54:55.661627643Z"}}}}},"End"]},"saga_name":"instance-update","start_node":13}
event time | sub saga | node id | event type | data
------------------------ | -------- | ------------------------------------------ | ------------- | ---
2026-04-29T16:54:55.665Z | | 13: start | started |
2026-04-29T16:54:55.668Z | | 13: start | succeeded |
2026-04-29T16:54:55.671Z | | 0: common.uuid_generate | started |
2026-04-29T16:54:55.674Z | | 0: common.uuid_generate | succeeded | "saga_instance_lock_id" => "85b3ed94-6283-489b-b33f-5ff6106b2f07"
2026-04-29T16:54:55.676Z | | 1: instance_update.become_updater | started |
2026-04-29T16:54:55.686Z | | 1: instance_update.become_updater | succeeded | "updater_lock" => {"locked_gen":78,"updater_id":"85b3ed94-6283-489b-b33f-5ff6106b2f07"}
2026-04-29T16:54:55.688Z | | 2: network_config_update | started |
2026-04-29T16:54:55.691Z | | 2: network_config_update | succeeded |
2026-04-29T16:54:55.694Z | | 3: instance_update.update_network_config | started |
2026-04-29T16:55:10.739Z | | 3: instance_update.update_network_config | failed | "update_network_config" => {"ActionFailed":{"source_error":{"InternalError":{"internal_message":"unable to find dendrite client for switch0"}}}}
2026-04-29T16:55:10.750Z | | 2: network_config_update | undo_started |
2026-04-29T16:55:10.754Z | | 2: network_config_update | undo_finished |
2026-04-29T16:55:10.758Z | | 1: instance_update.become_updater | undo_started |
2026-04-29T16:55:10.767Z | | 1: instance_update.become_updater | undo_finished |
2026-04-29T16:55:10.771Z | | 0: common.uuid_generate | undo_started |
2026-04-29T16:55:10.773Z | | 0: common.uuid_generate | undo_finished |
2026-04-29T16:55:10.777Z | | 13: start | undo_started |
2026-04-29T16:55:10.781Z | | 13: start | undo_finished |
To wit: the sled ( I'm gonna give it another go with a sled that is not a scrimlet to make sure that the VM gets restarted immediately if it wasn't on a scrimlet. |
|
Okay, so once we have turned back on the other scrimlet we can see that the instance has restarted, as we would expect: root@oxz_switch1:~# omdb db instance show -i 6287c3ea-b2dc-4144-b69c-328c54593d0c
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd9f:32e6:2c2f:104::3]:32221,[fd9f:32e6:2c2f:101::3]:32221,[fd9f:32e6:2c2f:104::4]:32221,[fd9f:32e6:2c2f:103::3]:32221,[fd9f:32e6:2c2f:102::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (253.0.0)
== INSTANCE ====================================================================
ID: 6287c3ea-b2dc-4144-b69c-328c54593d0c
project ID: 1bef82ae-6061-4f17-81d5-8280cd7d5013
name: my-computer
description:
created at: 2026-04-29 16:42:58.723930 UTC
last modified at: 2026-04-29 16:42:58.723930 UTC
== CONFIGURATION ===============================================================
vCPUs: 2
memory: 8 GiB
hostname: my-computer
boot disk: Some(cf3e17a9-b642-497a-bb75-013d1e8c4bbe)
auto-restart:
InstanceAutoRestart {
policy: None,
cooldown: None,
}
== RUNTIME STATE ===============================================================
nexus state: Vmm
(i) external API state: Running
intended state: running
last updated at: 2026-04-29T17:01:16.014453Z (generation 5)
needs reincarnation: false
karmic status: saṃsāra (reincarnation enabled)
last reincarnated at: Some(2026-04-29T17:01:16.381375Z)
active VMM ID: Some(e2834d81-2e93-4dac-9ef4-420b871e8a14)
target VMM ID: None
migration ID: None
updater lock: UNLOCKED at generation: 628
== ACTIVE VMM ==================================================================
ID: e2834d81-2e93-4dac-9ef4-420b871e8a14
instance ID: 6287c3ea-b2dc-4144-b69c-328c54593d0c
created at: 2026-04-29 17:01:16.301147 UTC
state: running
updated at: 2026-04-29T17:03:40.492980Z (generation 4)
propolis address: fd9f:32e6:2c2f:102::1:0:12400
sled ID: 917dc03f-962c-408e-81ad-1360f67cfe59
CPU platform: AmdMilan
== ATTACHED DISKS ==============================================================
# ID SIZE TYPE STATE NAME
0 cf3e17a9-b642-497a-bb75-013d1e8c4bbe 10 GiB crucible attached my-computer-debian-12-genericcloud-4f0bea
== VMM HISTORY =================================================================
ID STATE GEN SLED_ID TIME_CREATED TIME_DELETED
e2834d81-2e93-4dac-9ef4-420b871e8a14 running 4 917dc03f-962c-408e-81ad-1360f67cfe59 2026-04-29T17:01:16.301Z -
0d530db0-4919-4043-a0d7-756261bbc1c9 failed 13 21f08cd2-4330-4f5e-9b39-b5caabdbb4c0 2026-04-29T16:43:01.401Z 2026-04-29T17:01:16.154Z
root@oxz_switch1:~# |
|
Okay, after restarting the instance a few times to get it onto a sled that is not a scrimlet, when I turn off that sled, it comes back pretty much immediately, even when that sled is not powered on again. So it definitely works! root@oxz_switch1:~# omdb db instance show -i 6287c3ea-b2dc-4144-b69c-328c54593d0c
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd9f:32e6:2c2f:104::3]:32221,[fd9f:32e6:2c2f:101::3]:32221,[fd9f:32e6:2c2f:104::4]:32221,[fd9f:32e6:2c2f:103::3]:32221,[fd9f:32e6:2c2f:102::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (253.0.0)
== INSTANCE ====================================================================
ID: 6287c3ea-b2dc-4144-b69c-328c54593d0c
project ID: 1bef82ae-6061-4f17-81d5-8280cd7d5013
name: my-computer
description:
created at: 2026-04-29 16:42:58.723930 UTC
last modified at: 2026-04-29 17:21:35.895624 UTC
== CONFIGURATION ===============================================================
vCPUs: 2
memory: 8 GiB
hostname: my-computer
boot disk: Some(cf3e17a9-b642-497a-bb75-013d1e8c4bbe)
auto-restart:
InstanceAutoRestart {
policy: None,
cooldown: None,
}
== RUNTIME STATE ===============================================================
nexus state: Vmm
(i) external API state: Running
intended state: running
last updated at: 2026-04-29T17:21:31.036638Z (generation 9)
needs reincarnation: false
karmic status: saṃsāra (reincarnation enabled)
last reincarnated at: Some(2026-04-29T17:01:16.381375Z)
active VMM ID: Some(5a8d62d3-2db9-419c-8a3e-e52d0c40f104)
target VMM ID: None
migration ID: None
updater lock: UNLOCKED at generation: 634
== ACTIVE VMM ==================================================================
ID: 5a8d62d3-2db9-419c-8a3e-e52d0c40f104
instance ID: 6287c3ea-b2dc-4144-b69c-328c54593d0c
created at: 2026-04-29 17:21:36.013644 UTC
state: running
updated at: 2026-04-29T17:21:45.552717Z (generation 4)
propolis address: fd9f:32e6:2c2f:103::1:0:12400
sled ID: dc55061a-a29f-40d2-b42b-5f83a625bab9
CPU platform: AmdMilan
== ATTACHED DISKS ==============================================================
# ID SIZE TYPE STATE NAME
0 cf3e17a9-b642-497a-bb75-013d1e8c4bbe 10 GiB crucible attached my-computer-debian-12-genericcloud-4f0bea
== VMM HISTORY =================================================================
ID STATE GEN SLED_ID TIME_CREATED TIME_DELETED
5a8d62d3-2db9-419c-8a3e-e52d0c40f104 running 4 dc55061a-a29f-40d2-b42b-5f83a625bab9 2026-04-29T17:21:36.013Z -
e81e8398-4853-4928-b61c-5ac1f7377a4d destroyed 7 917dc03f-962c-408e-81ad-1360f67cfe59 2026-04-29T17:20:12.824Z 2026-04-29T17:21:31.277Z
e2834d81-2e93-4dac-9ef4-420b871e8a14 destroyed 7 917dc03f-962c-408e-81ad-1360f67cfe59 2026-04-29T17:01:16.301Z 2026-04-29T17:20:01.275Z
0d530db0-4919-4043-a0d7-756261bbc1c9 failed 13 21f08cd2-4330-4f5e-9b39-b5caabdbb4c0 2026-04-29T16:43:01.401Z 2026-04-29T17:01:16.154Z
root@oxz_switch1:~# omdb db sleds
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd9f:32e6:2c2f:104::3]:32221,[fd9f:32e6:2c2f:101::3]:32221,[fd9f:32e6:2c2f:104::4]:32221,[fd9f:32e6:2c2f:103::3]:32221,[fd9f:32e6:2c2f:102::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (253.0.0)
note: listing all commissioned sleds (use -F to filter, e.g. -F in-service)
SERIAL IP ROLE POLICY STATE ID
BRM27230037 [fd9f:32e6:2c2f:101::1]:12345 - in service active 1b441c4b-9ab4-4af3-98ad-a6cafba5f158
2F8JEXDK [fd9f:32e6:2c2f:104::1]:12345 scrimlet in service active 21f08cd2-4330-4f5e-9b39-b5caabdbb4c0
BRM23230018 [fd9f:32e6:2c2f:102::1]:12345 scrimlet in service active 917dc03f-962c-408e-81ad-1360f67cfe59
BRM23230010 [fd9f:32e6:2c2f:103::1]:12345 - in service active dc55061a-a29f-40d2-b42b-5f83a625bab9
root@oxz_switch1:~# pilot sp off BRM23230010
17 BRM23230010 ok
root@oxz_switch1:~# omdb db instance show -i 6287c3ea-b2dc-4144-b69c-328c54593d0c
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd9f:32e6:2c2f:104::3]:32221,[fd9f:32e6:2c2f:101::3]:32221,[fd9f:32e6:2c2f:104::4]:32221,[fd9f:32e6:2c2f:103::3]:32221,[fd9f:32e6:2c2f:102::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (253.0.0)
== INSTANCE ====================================================================
ID: 6287c3ea-b2dc-4144-b69c-328c54593d0c
project ID: 1bef82ae-6061-4f17-81d5-8280cd7d5013
name: my-computer
description:
created at: 2026-04-29 16:42:58.723930 UTC
last modified at: 2026-04-29 17:21:35.895624 UTC
== CONFIGURATION ===============================================================
vCPUs: 2
memory: 8 GiB
hostname: my-computer
boot disk: Some(cf3e17a9-b642-497a-bb75-013d1e8c4bbe)
auto-restart:
InstanceAutoRestart {
policy: None,
cooldown: None,
}
== RUNTIME STATE ===============================================================
nexus state: Vmm
(i) external API state: Running
intended state: running
last updated at: 2026-04-29T17:21:31.036638Z (generation 9)
needs reincarnation: false
karmic status: saṃsāra (reincarnation enabled)
last reincarnated at: Some(2026-04-29T17:01:16.381375Z)
active VMM ID: Some(5a8d62d3-2db9-419c-8a3e-e52d0c40f104)
target VMM ID: None
migration ID: None
updater lock: UNLOCKED at generation: 634
== ACTIVE VMM ==================================================================
ID: 5a8d62d3-2db9-419c-8a3e-e52d0c40f104
instance ID: 6287c3ea-b2dc-4144-b69c-328c54593d0c
created at: 2026-04-29 17:21:36.013644 UTC
state: running
updated at: 2026-04-29T17:21:45.552717Z (generation 4)
propolis address: fd9f:32e6:2c2f:103::1:0:12400
sled ID: dc55061a-a29f-40d2-b42b-5f83a625bab9
CPU platform: AmdMilan
== ATTACHED DISKS ==============================================================
# ID SIZE TYPE STATE NAME
0 cf3e17a9-b642-497a-bb75-013d1e8c4bbe 10 GiB crucible attached my-computer-debian-12-genericcloud-4f0bea
== VMM HISTORY =================================================================
ID STATE GEN SLED_ID TIME_CREATED TIME_DELETED
5a8d62d3-2db9-419c-8a3e-e52d0c40f104 running 4 dc55061a-a29f-40d2-b42b-5f83a625bab9 2026-04-29T17:21:36.013Z -
e81e8398-4853-4928-b61c-5ac1f7377a4d destroyed 7 917dc03f-962c-408e-81ad-1360f67cfe59 2026-04-29T17:20:12.824Z 2026-04-29T17:21:31.277Z
e2834d81-2e93-4dac-9ef4-420b871e8a14 destroyed 7 917dc03f-962c-408e-81ad-1360f67cfe59 2026-04-29T17:01:16.301Z 2026-04-29T17:20:01.275Z
0d530db0-4919-4043-a0d7-756261bbc1c9 failed 13 21f08cd2-4330-4f5e-9b39-b5caabdbb4c0 2026-04-29T16:43:01.401Z 2026-04-29T17:01:16.154Z
root@oxz_switch1:~# omdb nexus -w background-tasks activate instance_watcher
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd9f:32e6:2c2f:102::4]:12232
activated background tasks: instance_watcher
root@oxz_switch1:~# omdb nexus background-tasks show instance_watcher
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd9f:32e6:2c2f:102::4]:12232
task: "instance_watcher"
configured period: every 30s
currently executing: iter 2143, triggered by a periodic timer firing
started at 2026-04-29T17:23:56.128Z, running for 47002ms
last completed activation: iter 2142, triggered by a periodic timer firing
started at 2026-04-29T17:23:26.125Z (77s ago) and ran for 76ms
total instances checked: 1
checks completed: 1
successful checks: 1
-> 1 instances running
update sagas queued: 0
failed checks: 0
checks that could not be completed: 0
stale instance metrics pruned: 0
root@oxz_switch1:~# omdb nexus background-tasks show instance_watcher
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd9f:32e6:2c2f:102::4]:12232
task: "instance_watcher"
configured period: every 30s
currently executing: no
last completed activation: iter 2146, triggered by a periodic timer firing
started at 2026-04-29T17:25:41.135Z (5s ago) and ran for 26ms
total instances checked: 1
checks completed: 1
successful checks: 1
-> 1 instances starting
update sagas queued: 0
failed checks: 0
checks that could not be completed: 0
stale instance metrics pruned: 0
root@oxz_switch1:~# omdb db instance show -i 6287c3ea-b2dc-4144-b69c-328c54593d0c
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd9f:32e6:2c2f:104::3]:32221,[fd9f:32e6:2c2f:101::3]:32221,[fd9f:32e6:2c2f:104::4]:32221,[fd9f:32e6:2c2f:103::3]:32221,[fd9f:32e6:2c2f:102::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (253.0.0)
== INSTANCE ====================================================================
ID: 6287c3ea-b2dc-4144-b69c-328c54593d0c
project ID: 1bef82ae-6061-4f17-81d5-8280cd7d5013
name: my-computer
description:
created at: 2026-04-29 16:42:58.723930 UTC
last modified at: 2026-04-29 17:21:35.895624 UTC
== CONFIGURATION ===============================================================
vCPUs: 2
memory: 8 GiB
hostname: my-computer
boot disk: Some(cf3e17a9-b642-497a-bb75-013d1e8c4bbe)
auto-restart:
InstanceAutoRestart {
policy: None,
cooldown: None,
}
== RUNTIME STATE ===============================================================
nexus state: Vmm
(i) external API state: Starting
intended state: running
last updated at: 2026-04-29T17:25:08.854970Z (generation 11)
needs reincarnation: false
/!\ karmic status: cooling down (TimeDelta { secs: 240, nanos: 507732327 } remaining)
last reincarnated at: Some(2026-04-29T17:25:09.405973Z)
active VMM ID: Some(6dd43317-6e78-4eab-8530-f1afd5b4db00)
target VMM ID: None
migration ID: None
updater lock: UNLOCKED at generation: 637
== ACTIVE VMM ==================================================================
ID: 6dd43317-6e78-4eab-8530-f1afd5b4db00
instance ID: 6287c3ea-b2dc-4144-b69c-328c54593d0c
created at: 2026-04-29 17:25:09.325159 UTC
state: starting
updated at: 2026-04-29T17:25:18.418588Z (generation 4)
propolis address: fd9f:32e6:2c2f:101::1:0:12400
sled ID: 1b441c4b-9ab4-4af3-98ad-a6cafba5f158
CPU platform: AmdMilan
== ATTACHED DISKS ==============================================================
# ID SIZE TYPE STATE NAME
0 cf3e17a9-b642-497a-bb75-013d1e8c4bbe 10 GiB crucible attached my-computer-debian-12-genericcloud-4f0bea
== VMM HISTORY =================================================================
ID STATE GEN SLED_ID TIME_CREATED TIME_DELETED
6dd43317-6e78-4eab-8530-f1afd5b4db00 starting 4 1b441c4b-9ab4-4af3-98ad-a6cafba5f158 2026-04-29T17:25:09.325Z -
5a8d62d3-2db9-419c-8a3e-e52d0c40f104 failed 5 dc55061a-a29f-40d2-b42b-5f83a625bab9 2026-04-29T17:21:36.013Z 2026-04-29T17:25:09.088Z
e81e8398-4853-4928-b61c-5ac1f7377a4d destroyed 7 917dc03f-962c-408e-81ad-1360f67cfe59 2026-04-29T17:20:12.824Z 2026-04-29T17:21:31.277Z
e2834d81-2e93-4dac-9ef4-420b871e8a14 destroyed 7 917dc03f-962c-408e-81ad-1360f67cfe59 2026-04-29T17:01:16.301Z 2026-04-29T17:20:01.275Z
0d530db0-4919-4043-a0d7-756261bbc1c9 failed 13 21f08cd2-4330-4f5e-9b39-b5caabdbb4c0 2026-04-29T16:43:01.401Z 2026-04-29T17:01:16.154Z
^X^C
root@oxz_switch1:~#Note that we missed the specific |
|
It seems like, but asking to be sure, if a sled is powered on, but the sled agent on that sled is not responding, then we will take no action based on the changes here, correct? |
That is correct. We've been very cautious about taking action in situations where we cannot conclusively determine that the VMM no longer exists, including situations where the sled-agent may be temporarily stuck, or where there may be a network partition between this particular Nexus and that sled-agent but the instance may still exist and its sled-agent may be happily responding to other Nexuses. This change is only intended to move instances to |
f4894e6 to
07c8ba8
Compare
This sounds like a bug, right? If so, do you know if we already have an issue for it? |
I believe that this is similar to the issues described in #8206 and #6896. Looking at #8206, it looks like it was fixed by PR #8914, which changed the I think we should be able to make similar changes to the Update: I've opened #10343 to track this. |
|
The CI failure for That's clearly unrelated, so I'm going to restart that. |
| Err(error) => { | ||
| let error = InlineErrorChain::new(&error); | ||
| slog::warn!( | ||
| &opctx.log, | ||
| "requesting sled SP status from MGS failed"; | ||
| "gateway_addr" => %addr, | ||
| "error" => &error, | ||
| ); | ||
| continue; | ||
| } |
There was a problem hiding this comment.
I think that if we get an error here that indicates MGS also could not talk to the SP in that slot, we may wish to check whether there is a sled present there via Ignition. I think it should then be okay to fail the VMM as well. But I would kind of like to address this separately as I cannot easily test it today (since it would require physically pulling sleds out of a rack...)
| // TODO(eliza): this sucks, i would rather not have to memcpy the strings | ||
| // here...this would be nicer if we iterated over sled-agents from the | ||
| // inventory in the main loop. change that! |
There was a problem hiding this comment.
I had thought that, perhaps, we could have the main loop use the inventory collection to look up sleds rather than doing it in the "VMMs by sled-agent" query that drives this task. However, that's not quite as simple as I had hoped, since the inventory does not tell us if the sled is expunged (AFAIK) and we really want to be querying for all sleds that an active Vmm record points at, even if that sled was not in the last inventory collection, because we want to try to talk to any sled that a Vmm record references, even if the inventory doesn't contain it. In fact, if we changed the main loop to discover health check targets via inventory, we may just ignore any sleds which are not powered up (I think?) if an inventory collection which didn't see the sled-agent has already been loaded when this task runs, and those VMMs would then not move to failed...
iximeow
left a comment
iximeow
left a comment
There was a problem hiding this comment.
seems good to me: I was initially wondering what happens if the list of MGSes changes over the life of the background task but as you pointed out activate() is the thing that is called whenever the task is activated, not like.. once up front when the task is created. so that seems fine.
I'd also wondered what updates the instance record once we've learned about the VMM here, but as we talked about that's process_vmm_update's job to start an instance_update saga if we've learned something that does need an update to happen. I vaguely remember talking with you about instance_update being ambiguous with the API endpoint and why we used the name instance reconfigure, but it's funny to be on the other side of that confusion for a change...
I did some minor thinkposting about Check but I really don't think that it needs looking at here and now, so I'm +1-since-you'll-be-out-but-if-you-update-this-I'll-take-another-look
| async fn run( | ||
| &mut self, | ||
| opctx: &OpContext, | ||
| inv_rx: watch::Receiver<Option<Arc<inventory::Collection>>>, | ||
| sled: &Sled, | ||
| vmm: &Vmm, | ||
| gateways: &[GatewayClient], | ||
| client: &SledAgentClient, | ||
| ) -> Option<SledVmmState> { |
There was a problem hiding this comment.
it seems a bit funky that Check::run really can only return either None or one kind of SledVmmState: mk_failed(). between that and Check::run taking a &mut self I am left with questions like, what happens if you Check::run() twice? "don't do that"?
I think those kinds of questions can be avoided Check::run changing to take target as an additional argument and returning a Check that has the bits we need for completed_check over on the caller side. then CheckOutcome::Unknown kind of stops being necessary, I think; either the check completed with a success or failure, or we didn't actually get to an answer and there's a .result with an error.
and if the result is folded in as a CheckOutcome::Incomplete(Incomplete) we don't really need result anymore anyway. the combination of CheckOutcome::Unknown && result.is_ok() never really meant anything, and an outcome of Success/Failure mixed with a result.is_err() was impossible but bogus in a different way. so folding the two together avoids all those questions too.
probably worth doing since you're in here moving this around BUT it's also not really a correctness thing i'm like 1000% fine with that being another change later.
There was a problem hiding this comment.
it seems a bit funky that
Check::runreally can only return eitherNoneor one kind ofSledVmmState:mk_failed().
This part is not true, it can also return the SledVmmState returned by the sled-agent, here:
omicron/nexus/src/app/background/tasks/instance_watcher.rs
Lines 240 to 250 in 014fe6a
You're right about the bogus states that the current combination of check.result and check.outcome can represent, I would like to tidy this up. I'm on the fence about whether I want to do more refactoring in this PR, or save it for another one, though. We also don't return a structured background task status type in this task, because it predates that pattern, and instead slam some JSON together; I'd like to tidy that up as well. But, I didn't want to rewrite everything in this PR and instead just make the functional change...what do you think?
There was a problem hiding this comment.
This part is not true,
oh ugh, yeah, I'd read that match from bottom to top and just kinda missed the "Ok" arm... sorry about that.
doing the functional change here and cleaning up in a different commit makes sense to me, shipit
As described in #10275, instances on a sled which is powered off will be considered "running" from the control plane's perspective until the sled is powered on again and its sled-agent denies knowledge of any VMMs it was previously running. This means that those instances will not be restarted, and we will (misleadingly) tell the user that they are running, until the sled is restored to service. If the sled abruptly leaves the rack and never comes back at all, any instances that were running on it remain stuck. This is not great.
This is because the
instance_watcherbackground task currently treats all instance health checks in which an error communicating with the sled-agent occurs as "inconclusive". Because we do not wish to spuriously fail instances that are actually fine, we behave very conservatively here. A communication error could indicate a network partition between the particular Nexus performing the health check and the sled-agent, and it would be incorrect to fail the VMM if we are the ones who are isolated from the rest of the rack.However, we can determine conclusively if the sled is off or not present, by asking MGS. When encountering a communication error, we can check whether the sled is present in the slot it was last believed to be in, and ask the SP if the sled is in A0. If it is not in A0, the VMM can safely be moved to
Failedwithout the risk of killing an instance that is actually healthy, since, well...a computer which is not turned on is definitely not capable of running virtual machines.