health check reported hung sagas that aren't actually running

In our dogfood environment, when we first deployed oxidecomputer/console#3226, we noticed the "contact support" bit was set. The reason was a bunch of sagas that have been running for a long time. Investigation of one of these sagas revealed a case we didn't consider with the health checks: a saga that (1) has not finished (it's still unwinding), but (2) is not running and can never be running in Nexus again (because a steno bug is preventing it from being recovered).

Questions here include:

Should Nexus handoff during update block on such sagas (i.e., not proceed while there are sagas that couldn't be recovered)? I'm thinking "no" because these aren't running and won't be running on the other side of the update, either. But it's definitely a dicey situation.
If not, should the health check still identify these? I'm inclined to say "no" because although they are a problem, they're not related to update (again, assuming we decide handoff should continue ignoring these).
What do we do to repair a system in this situation? (This question is not update-related.)

@wfchandler

@wfchandler found the health check log message showing the problem:

17:19:35.457Z WARN 06333a5d-b553-4f3a-b680-c9ee572b3992 (dropshot_external): found problems in the system before or after an update
    authenticated = true
    file = nexus/src/app/update.rs:613
    local_addr = 172.30.2.8:443
    method = GET
    remote_addr = 172.20.16.166:53753
    req_id = 1ce757d3-cb94-4ebf-8142-da5ecb009d90
    silo_id = 7bd7623a-68ed-4636-8ecb-b59e3b068787
    silo_user_id = dcef2ede-a083-4826-805e-b79d05007aec
    stuck_sagas = {StuckSaga { id: SagaId(1d249d2a-156f-43d6-881c-d1ea03c1e356), name: "instance-start" }, StuckSaga { id: SagaId(34f5a000-259d-4abe-ac93-4b3c787a9ea8), name: "instance-start" }, StuckSaga { id: SagaId(3aba3dda-32d7-46dd-8cf7-4e0c1ed07018), name: "instance-start" }, StuckSaga { id: SagaId(52fd4cb7-71ed-42f4-b7d2-32a167a07b53), name: "instance-start" }, StuckSaga { id: SagaId(7edd6cd9-28dc-4c68-8788-0ab57683b58a), name: "instance-start" }, StuckSaga { id: SagaId(8b9bc46d-1b91-485d-8333-8eb502a71f60), name: "instance-start" }, StuckSaga { id: SagaId(8dbd27f3-711d-46d2-9c27-050d18e15fdb), name: "instance-start" }, StuckSaga { id: SagaId(abe40088-e88a-4cff-bf9b-bf16b77991c4), name: "region-replacement-finish" }, StuckSaga { id: SagaId(b9df69fa-73a4-4a2e-be4a-d0832b88d01e), name: "instance-start" }, StuckSaga { id: SagaId(ebc5ae43-1e67-42c7-b3da-1c52e8800ec4), name: "instance-create" }}
    type = silo_user
    uri = https://oxide.sys.rack2.eng.oxide.computer/v1/system/update/status

@jgallagher collected this:

root@[fd00:1122:3344:109::3]:32221/omicron> select id,creator,time_created,now() - time_created as age,saga_state from saga where saga_state in ('running', 'unwinding') order by age;
                   id                  |               creator                |         time_created          |           age            | saga_state
---------------------------------------+--------------------------------------+-------------------------------+--------------------------+-------------
  b9df69fa-73a4-4a2e-be4a-d0832b88d01e | 0b1cc325-3c30-400d-80e4-59918f68cd65 | 2026-05-01 18:16:52.270956+00 | 31 days 23:13:33.774476  | unwinding
  8dbd27f3-711d-46d2-9c27-050d18e15fdb | 0b1cc325-3c30-400d-80e4-59918f68cd65 | 2026-05-01 18:16:51.678499+00 | 31 days 23:13:34.366933  | unwinding
  3aba3dda-32d7-46dd-8cf7-4e0c1ed07018 | 0b1cc325-3c30-400d-80e4-59918f68cd65 | 2026-05-01 18:16:49.807687+00 | 31 days 23:13:36.237745  | unwinding
  8b9bc46d-1b91-485d-8333-8eb502a71f60 | e6810ed6-a7ee-4980-aaa2-cafb40e98299 | 2026-05-01 15:22:36.639842+00 | 32 days 02:07:49.40559   | unwinding
  1d249d2a-156f-43d6-881c-d1ea03c1e356 | e6810ed6-a7ee-4980-aaa2-cafb40e98299 | 2026-05-01 15:22:33.172165+00 | 32 days 02:07:52.873267  | unwinding
  34f5a000-259d-4abe-ac93-4b3c787a9ea8 | e6810ed6-a7ee-4980-aaa2-cafb40e98299 | 2026-05-01 15:22:32.592305+00 | 32 days 02:07:53.453127  | unwinding
  52fd4cb7-71ed-42f4-b7d2-32a167a07b53 | e6810ed6-a7ee-4980-aaa2-cafb40e98299 | 2026-05-01 15:22:32.39906+00  | 32 days 02:07:53.646372  | unwinding
  7edd6cd9-28dc-4c68-8788-0ab57683b58a | 96d03c44-3ca6-420c-948f-64a9429a2eaf | 2026-05-01 00:51:33.820687+00 | 32 days 16:38:52.224745  | unwinding
  abe40088-e88a-4cff-bf9b-bf16b77991c4 | 65a11c18-7f59-41ac-b9e7-680627f996e7 | 2025-07-04 04:56:50.332554+00 | 333 days 12:33:35.712878 | unwinding
  ebc5ae43-1e67-42c7-b3da-1c52e8800ec4 | 65a11c18-7f59-41ac-b9e7-680627f996e7 | 2023-10-09 14:39:15.803991+00 | 967 days 02:51:10.241441 | unwinding
(10 rows)

I focused on the newest of these sagas and the history of upgrades on this system. Here are the potentially relevant entries related to recent updates from omdb reconfigurator history:

42381 2026-04-21T22:47:11.779Z 54741c68-1f86-4e7c-8ba2-c49cb9e506cc  enabled: updated target release minimum generation from 70 to 71
42382 2026-04-21T22:48:02.759Z 6d932086-c277-477d-af35-d725563ab493  enabled: sled 0c7011f7-a4bf-4daf-90cc-1c2410103300: performed all noop measurements updates ...
42383 2026-04-21T23:11:49.874Z f25087aa-9859-41c7-a5ee-1e3c185bdaf6  enabled: sled 0c7011f7-a4bf-4daf-90cc-1c2410103300: performed 11 noop zone image source updates ...
42384 2026-05-05T23:45:47.689Z 8b775768-b829-4f69-a29e-06c797569b94  enabled: update Sled 7 (BRM27230045) host OS to 19.3.0-0.ci+gitd4a126e8cc3
42385 2026-05-05T23:56:55.420Z a07c1ff0-fa3e-4177-ac15-197db1621ad9  enabled: update Sled 11 (BRM42220006) host OS to 19.3.0-0.ci+gitd4a126e8cc3

So there was a MUPdate long before this saga was created, and a self-service update on May 5, but nothing in between. (That rules out a MUPdate during saga execution causing a saga to become unrecoverable.)

The owning Nexus instance for that saga is 0b1cc325-3c30-400d-80e4-59918f68cd65. Here's its history:

root@oxz_switch0:~# omdb db saga show b9df69fa-73a4-4a2e-be4a-d0832b88d01e
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (261.0.0)
 id                                   | current_sec                          | time_created             | name           | state     
--------------------------------------+--------------------------------------+--------------------------+----------------+-----------
 b9df69fa-73a4-4a2e-be4a-d0832b88d01e | 0b1cc325-3c30-400d-80e4-59918f68cd65 | 2026-05-01T18:16:52.270Z | instance-start | Unwinding 

DAG: {"end_node":25,"graph":{"edge_property":"directed","edges":[[0,1,null],[1,2,null],[2,3,null],[3,4,null],[4,5,null],[5,6,null],[5,7,null],[5,8,null],[5,9,null],[5,10,null],[5,11,null],[5,12,null],[5,13,null],[5,14,null],[5,15,null],[5,16,null],[5,17,null],[6,18,null],[7,18,null],[8,18,null],[9,18,null],[10,18,null],[11,18,null],[12,18,null],[13,18,null],[14,18,null],[15,18,null],[16,18,null],[17,18,null],[18,19,null],[19,20,null],[20,21,null],[21,22,null],[22,23,null],[24,0,null],[23,25,null]],"node_holes":[],"nodes":[{"Action":{"action_name":"instance_start.generate_propolis_id","label":"GeneratePropolisId","name":"propolis_id"}},{"Action":{"action_name":"instance_start.alloc_server","label":"AllocServer","name":"sled_id"}},{"Action":{"action_name":"instance_start.alloc_propolis_ip","label":"AllocPropolisIp","name":"propolis_ip"}},{"Action":{"action_name":"instance_start.create_vmm_record","label":"CreateVmmRecord","name":"vmm_record"}},{"Action":{"action_name":"instance_start.mark_as_starting","label":"MarkAsStarting","name":"started_record"}},{"Action":{"action_name":"instance_start.list_local_storage","label":"ListLocalStorage","name":"local_storage_records"}},{"Action":{"action_name":"instance_start.ensure_local_storage_0","label":"EnsureLocalStorage_0","name":"ensure_local_storage_0"}},{"Action":{"action_name":"instance_start.ensure_local_storage_1","label":"EnsureLocalStorage_1","name":"ensure_local_storage_1"}},{"Action":{"action_name":"instance_start.ensure_local_storage_2","label":"EnsureLocalStorage_2","name":"ensure_local_storage_2"}},{"Action":{"action_name":"instance_start.ensure_local_storage_3","label":"EnsureLocalStorage_3","name":"ensure_local_storage_3"}},{"Action":{"action_name":"instance_start.ensure_local_storage_4","label":"EnsureLocalStorage_4","name":"ensure_local_storage_4"}},{"Action":{"action_name":"instance_start.ensure_local_storage_5","label":"EnsureLocalStorage_5","name":"ensure_local_storage_5"}},{"Action":{"action_name":"instance_start.ensure_local_storage_6","label":"EnsureLocalStorage_6","name":"ensure_local_storage_6"}},{"Action":{"action_name":"instance_start.ensure_local_storage_7","label":"EnsureLocalStorage_7","name":"ensure_local_storage_7"}},{"Action":{"action_name":"instance_start.ensure_local_storage_8","label":"EnsureLocalStorage_8","name":"ensure_local_storage_8"}},{"Action":{"action_name":"instance_start.ensure_local_storage_9","label":"EnsureLocalStorage_9","name":"ensure_local_storage_9"}},{"Action":{"action_name":"instance_start.ensure_local_storage_10","label":"EnsureLocalStorage_10","name":"ensure_local_storage_10"}},{"Action":{"action_name":"instance_start.ensure_local_storage_11","label":"EnsureLocalStorage_11","name":"ensure_local_storage_11"}},{"Action":{"action_name":"instance_start.dpd_ensure","label":"DpdEnsure","name":"dpd_ensure"}},{"Action":{"action_name":"instance_start.v2p_ensure","label":"V2PEnsure","name":"v2p_ensure"}},{"Action":{"action_name":"instance_start.ensure_registered","label":"EnsureRegistered","name":"ensure_registered"}},{"Action":{"action_name":"instance_start.update_multicast_sled_id","label":"UpdateMulticastSledId","name":"multicast_sled_id"}},{"Action":{"action_name":"instance_start.add_virtual_resources","label":"AddVirtualResources","name":"virtual_resources"}},{"Action":{"action_name":"instance_start.ensure_running","label":"EnsureRunning","name":"ensure_running"}},{"Start":{"params":{"db_instance":{"auto_restart":{"cooldown":null,"policy":null},"boot_disk_id":null,"cpu_platform":null,"dst_propolis_id":null,"hostname":"aa23","identity":{"description":"aa23 test host","id":"0605b167-842a-4c18-a54d-59373c6d8e28","name":"aa23","time_created":"2026-05-01T18:16:26.984048Z","time_deleted":null,"time_modified":"2026-05-01T18:16:26.984048Z"},"intended_state":"Running","memory":8589934592,"migration_id":null,"ncpus":8,"nexus_state":"NoVmm","project_id":"759beaf2-517d-4d24-bc17-1eed69bc8801","propolis_id":null,"state_generation":2,"time_last_auto_restarted":null,"time_state_updated":"2026-05-01T18:16:26.984048Z","updater_gen":1,"updater_id":null,"user_data":[35,99,108,111,117,100,45,99,111,110,102,105,103,10,109,97,110,97,103,101,95,101,116,99,95,104,111,115,116,115,58,32,116,114,117,101,10]},"reason":"AutoStart","serialized_authn":{"kind":{"Authenticated":[{"actor":{"SiloUser":{"silo_id":"7bd7623a-68ed-4636-8ecb-b59e3b068787","silo_user_id":"93521315-630d-439b-b1fc-43a5e68b7c5b"}},"credential_id":"0bfa2c5b-5325-4f87-8056-8b9d948cec8f","device_token_expiration":null},{"mapped_fleet_roles":{"admin":["admin"]}}]}}}}},"End"]},"saga_name":"instance-start","start_node":24}

              event time | sub saga | node id                                     | event type    | data
------------------------ | -------- | ------------------------------------------- | ------------- | ---
2026-05-01T18:16:52.278Z |          |                                   24: start | started       | 
2026-05-01T18:16:52.290Z |          |                                   24: start | succeeded     | 
2026-05-01T18:16:52.293Z |          |      0: instance_start.generate_propolis_id | started       | 
2026-05-01T18:16:52.295Z |          |      0: instance_start.generate_propolis_id | succeeded     | "propolis_id" => "138613b1-a8d1-4b86-85f4-2a825aaebe56"
2026-05-01T18:16:52.300Z |          |              1: instance_start.alloc_server | started       | 
2026-05-01T18:16:52.648Z |          |              1: instance_start.alloc_server | succeeded     | "sled_id" => "b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88"
2026-05-01T18:16:52.652Z |          |         2: instance_start.alloc_propolis_ip | started       | 
2026-05-01T18:16:52.659Z |          |         2: instance_start.alloc_propolis_ip | succeeded     | "propolis_ip" => "fd00:1122:3344:106::1:15f3"
2026-05-01T18:16:52.663Z |          |         3: instance_start.create_vmm_record | started       | 
2026-05-01T18:16:52.796Z |          |         3: instance_start.create_vmm_record | succeeded     | "vmm_record" => {"cpu_platform":"AmdMilan","gen":1,"id":"138613b1-a8d1-4b86-85f4-2a825aaebe56","instance_id":"0605b167-842a-4c18-a54d-59373c6d8e28","propolis_ip":"fd00:1122:3344:106::1:15f3/128","propolis_port":12400,"sled_id":"b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88","state":"Creating","time_created":"2026-05-01T18:16:52.678147Z","time_deleted":null,"time_state_updated":"2026-05-01T18:16:52.678147Z"}
2026-05-01T18:16:52.800Z |          |          4: instance_start.mark_as_starting | started       | 
2026-05-01T18:16:52.870Z |          |          4: instance_start.mark_as_starting | succeeded     | "started_record" => {"auto_restart":{"cooldown":null,"policy":null},"boot_disk_id":null,"cpu_platform":null,"dst_propolis_id":null,"hostname":"aa23","identity":{"description":"aa23 test host","id":"0605b167-842a-4c18-a54d-59373c6d8e28","name":"aa23","time_created":"2026-05-01T18:16:26.984048Z","time_deleted":null,"time_modified":"2026-05-01T18:16:26.984048Z"},"intended_state":"Running","memory":8589934592,"migration_id":null,"ncpus":8,"nexus_state":"Vmm","project_id":"759beaf2-517d-4d24-bc17-1eed69bc8801","propolis_id":"138613b1-a8d1-4b86-85f4-2a825aaebe56","state_generation":3,"time_last_auto_restarted":null,"time_state_updated":"2026-05-01T18:16:26.984048Z","updater_gen":1,"updater_id":null,"user_data":[35,99,108,111,117,100,45,99,111,110,102,105,103,10,109,97,110,97,103,101,95,101,116,99,95,104,111,115,116,115,58,32,116,114,117,101,10]}
2026-05-01T18:16:52.875Z |          |        5: instance_start.list_local_storage | started       | 
2026-05-01T18:16:52.974Z |          |        5: instance_start.list_local_storage | succeeded     | "local_storage_records" => [{"disk":{"attach_instance_id":"0605b167-842a-4c18-a54d-59373c6d8e28","block_size":"AdvancedFormat","disk_state":"attached","disk_type":"LocalStorage","identity":{"description":"many local 0","id":"27225b7f-e835-4d2f-88c1-8a3d0e408ca7","name":"aa23-ls0","time_created":"2026-05-01T18:16:38.474010Z","time_deleted":null,"time_modified":"2026-05-01T18:16:38.474010Z"},"project_id":"759beaf2-517d-4d24-bc17-1eed69bc8801","rcgen":1,"size":5368709120,"slot":1,"state_generation":2,"time_state_updated":"2026-05-01T18:16:38.635740Z"},"disk_type_local_storage":{"disk_id":"27225b7f-e835-4d2f-88c1-8a3d0e408ca7","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"41244850-96c4-41ba-adbc-6e3795c9c9e2","required_dataset_overhead":367001600},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":5735710720,"id":"41244850-96c4-41ba-adbc-6e3795c9c9e2","local_storage_unencrypted_dataset_id":"d9b53b92-dc9f-4c9a-a798-104b1fd3d408","pool_id":"4366f80d-3902-4b93-8f2d-380008e805fc","sled_id":"b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88","time_created":"2026-05-01T18:16:52.448812Z","time_deleted":null}}},{"disk":{"attach_instance_id":"0605b167-842a-4c18-a54d-59373c6d8e28","block_size":"AdvancedFormat","disk_state":"attached","disk_type":"LocalStorage","identity":{"description":"many local 1","id":"deb4aad5-6ead-4958-be74-04c621310e71","name":"aa23-ls1","time_created":"2026-05-01T18:16:38.823986Z","time_deleted":null,"time_modified":"2026-05-01T18:16:38.823986Z"},"project_id":"759beaf2-517d-4d24-bc17-1eed69bc8801","rcgen":1,"size":5368709120,"slot":2,"state_generation":2,"time_state_updated":"2026-05-01T18:16:39.031689Z"},"disk_type_local_storage":{"disk_id":"deb4aad5-6ead-4958-be74-04c621310e71","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"149ebd6b-cf92-4e9e-8fed-d1a516bb5d98","required_dataset_overhead":367001600},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":5735710720,"id":"149ebd6b-cf92-4e9e-8fed-d1a516bb5d98","local_storage_unencrypted_dataset_id":"e3f59871-5615-4f02-9ba5-7fd6380b14e8","pool_id":"025725fa-9e40-4b46-b018-c420408394ef","sled_id":"b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88","time_created":"2026-05-01T18:16:52.448812Z","time_deleted":null}}},{"disk":{"attach_instance_id":"0605b167-842a-4c18-a54d-59373c6d8e28","block_size":"AdvancedFormat","disk_state":"attached","disk_type":"LocalStorage","identity":{"description":"many local 2","id":"60e15d25-8ba4-4330-b758-56f9814e3d6e","name":"aa23-ls2","time_created":"2026-05-01T18:16:39.253121Z","time_deleted":null,"time_modified":"2026-05-01T18:16:39.253121Z"},"project_id":"759beaf2-517d-4d24-bc17-1eed69bc8801","rcgen":1,"size":5368709120,"slot":3,"state_generation":2,"time_state_updated":"2026-05-01T18:16:39.491699Z"},"disk_type_local_storage":{"disk_id":"60e15d25-8ba4-4330-b758-56f9814e3d6e","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"10768c92-a246-42d2-9447-e1c46f9b88b1","required_dataset_overhead":367001600},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":5735710720,"id":"10768c92-a246-42d2-9447-e1c46f9b88b1","local_storage_unencrypted_dataset_id":"9463cf4e-54dc-4052-9847-6dccf8789aff","pool_id":"cf14d1b9-b4db-4594-b3ab-a9957e770ce9","sled_id":"b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88","time_created":"2026-05-01T18:16:52.448812Z","time_deleted":null}}},{"disk":{"attach_instance_id":"0605b167-842a-4c18-a54d-59373c6d8e28","block_size":"AdvancedFormat","disk_state":"attached","disk_type":"LocalStorage","identity":{"description":"many local 3","id":"60ec8a48-c1e8-4fa3-998d-464712b97de3","name":"aa23-ls3","time_created":"2026-05-01T18:16:39.716734Z","time_deleted":null,"time_modified":"2026-05-01T18:16:39.716734Z"},"project_id":"759beaf2-517d-4d24-bc17-1eed69bc8801","rcgen":1,"size":5368709120,"slot":4,"state_generation":2,"time_state_updated":"2026-05-01T18:16:39.921967Z"},"disk_type_local_storage":{"disk_id":"60ec8a48-c1e8-4fa3-998d-464712b97de3","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"eb0f74c9-45c8-4b7b-89f4-2c80e4f7bacf","required_dataset_overhead":367001600},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":5735710720,"id":"eb0f74c9-45c8-4b7b-89f4-2c80e4f7bacf","local_storage_unencrypted_dataset_id":"7524bde0-734b-4af2-9cb7-5817bd77688e","pool_id":"534bcd4b-502f-4109-af6e-4b28a22c20f1","sled_id":"b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88","time_created":"2026-05-01T18:16:52.448812Z","time_deleted":null}}},{"disk":{"attach_instance_id":"0605b167-842a-4c18-a54d-59373c6d8e28","block_size":"AdvancedFormat","disk_state":"attached","disk_type":"LocalStorage","identity":{"description":"many local 4","id":"ce2cff58-c6fc-442a-a57b-af810ee2d0c5","name":"aa23-ls4","time_created":"2026-05-01T18:16:40.085629Z","time_deleted":null,"time_modified":"2026-05-01T18:16:40.085629Z"},"project_id":"759beaf2-517d-4d24-bc17-1eed69bc8801","rcgen":1,"size":5368709120,"slot":5,"state_generation":2,"time_state_updated":"2026-05-01T18:16:40.288201Z"},"disk_type_local_storage":{"disk_id":"ce2cff58-c6fc-442a-a57b-af810ee2d0c5","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"c0cd17fe-06fc-4903-affa-7506f9c6186c","required_dataset_overhead":367001600},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":5735710720,"id":"c0cd17fe-06fc-4903-affa-7506f9c6186c","local_storage_unencrypted_dataset_id":"64a8d406-a263-4929-8714-f1755d09cfef","pool_id":"834c9aad-c53b-4357-bc3f-f422efa63848","sled_id":"b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88","time_created":"2026-05-01T18:16:52.448812Z","time_deleted":null}}},{"disk":{"attach_instance_id":"0605b167-842a-4c18-a54d-59373c6d8e28","block_size":"AdvancedFormat","disk_state":"attached","disk_type":"LocalStorage","identity":{"description":"many local 5","id":"72c08a3a-e574-4d74-ac56-017c03a72820","name":"aa23-ls5","time_created":"2026-05-01T18:16:40.458571Z","time_deleted":null,"time_modified":"2026-05-01T18:16:40.458571Z"},"project_id":"759beaf2-517d-4d24-bc17-1eed69bc8801","rcgen":1,"size":5368709120,"slot":6,"state_generation":2,"time_state_updated":"2026-05-01T18:16:40.607781Z"},"disk_type_local_storage":{"disk_id":"72c08a3a-e574-4d74-ac56-017c03a72820","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"71ac6498-3c47-49b6-bc63-48a04aa12c83","required_dataset_overhead":367001600},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":5735710720,"id":"71ac6498-3c47-49b6-bc63-48a04aa12c83","local_storage_unencrypted_dataset_id":"ece2e454-a5cf-459b-982e-d262762c41cd","pool_id":"4d2f5aaf-eb14-4b1e-aa99-ae38ec844605","sled_id":"b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88","time_created":"2026-05-01T18:16:52.448812Z","time_deleted":null}}},{"disk":{"attach_instance_id":"0605b167-842a-4c18-a54d-59373c6d8e28","block_size":"AdvancedFormat","disk_state":"attached","disk_type":"LocalStorage","identity":{"description":"many local 6","id":"4d3e44a6-807b-4794-b251-8ee159c84790","name":"aa23-ls6","time_created":"2026-05-01T18:16:40.841793Z","time_deleted":null,"time_modified":"2026-05-01T18:16:40.841793Z"},"project_id":"759beaf2-517d-4d24-bc17-1eed69bc8801","rcgen":1,"size":5368709120,"slot":7,"state_generation":2,"time_state_updated":"2026-05-01T18:16:41.015939Z"},"disk_type_local_storage":{"disk_id":"4d3e44a6-807b-4794-b251-8ee159c84790","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"0515fdda-149f-4961-9008-992876569a06","required_dataset_overhead":367001600},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":5735710720,"id":"0515fdda-149f-4961-9008-992876569a06","local_storage_unencrypted_dataset_id":"c8d50017-0346-42c4-ae7f-13afde0f005a","pool_id":"cf5f8849-0c5a-475b-8683-6d17da88d1d1","sled_id":"b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88","time_created":"2026-05-01T18:16:52.448812Z","time_deleted":null}}},{"disk":{"attach_instance_id":"0605b167-842a-4c18-a54d-59373c6d8e28","block_size":"AdvancedFormat","disk_state":"attached","disk_type":"LocalStorage","identity":{"description":"many local 7","id":"197fefb8-531a-4876-bab5-f217aea00f01","name":"aa23-ls7","time_created":"2026-05-01T18:16:41.239078Z","time_deleted":null,"time_modified":"2026-05-01T18:16:41.239078Z"},"project_id":"759beaf2-517d-4d24-bc17-1eed69bc8801","rcgen":1,"size":5368709120,"slot":8,"state_generation":2,"time_state_updated":"2026-05-01T18:16:41.463415Z"},"disk_type_local_storage":{"disk_id":"197fefb8-531a-4876-bab5-f217aea00f01","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"09cd7dd4-64f9-4c31-9c75-fb167453bf32","required_dataset_overhead":367001600},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":5735710720,"id":"09cd7dd4-64f9-4c31-9c75-fb167453bf32","local_storage_unencrypted_dataset_id":"ead18f63-b2ea-488c-9944-fcdfd702738b","pool_id":"7f778610-7328-4554-98f6-b17f74f551c7","sled_id":"b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88","time_created":"2026-05-01T18:16:52.448812Z","time_deleted":null}}},{"disk":{"attach_instance_id":"0605b167-842a-4c18-a54d-59373c6d8e28","block_size":"AdvancedFormat","disk_state":"attached","disk_type":"LocalStorage","identity":{"description":"many local 8","id":"b34dfb9d-d4fa-4b58-903f-8f5cb94ec45e","name":"aa23-ls8","time_created":"2026-05-01T18:16:41.645019Z","time_deleted":null,"time_modified":"2026-05-01T18:16:41.645019Z"},"project_id":"759beaf2-517d-4d24-bc17-1eed69bc8801","rcgen":1,"size":5368709120,"slot":9,"state_generation":2,"time_state_updated":"2026-05-01T18:16:41.877246Z"},"disk_type_local_storage":{"disk_id":"b34dfb9d-d4fa-4b58-903f-8f5cb94ec45e","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"9fefd86d-b9cc-4a6f-842d-03fb5dd9f722","required_dataset_overhead":367001600},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":5735710720,"id":"9fefd86d-b9cc-4a6f-842d-03fb5dd9f722","local_storage_unencrypted_dataset_id":"66022530-fb9e-468b-8606-6c94f503c720","pool_id":"32b5303f-f667-4345-84d2-c7eec63b91b2","sled_id":"b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88","time_created":"2026-05-01T18:16:52.448812Z","time_deleted":null}}},{"disk":{"attach_instance_id":"0605b167-842a-4c18-a54d-59373c6d8e28","block_size":"AdvancedFormat","disk_state":"attached","disk_type":"LocalStorage","identity":{"description":"many local 9","id":"c1d97655-7563-4c09-8b64-36d6cf6aaf7f","name":"aa23-ls9","time_created":"2026-05-01T18:16:42.017501Z","time_deleted":null,"time_modified":"2026-05-01T18:16:42.017501Z"},"project_id":"759beaf2-517d-4d24-bc17-1eed69bc8801","rcgen":1,"size":5368709120,"slot":10,"state_generation":2,"time_state_updated":"2026-05-01T18:16:42.357249Z"},"disk_type_local_storage":{"disk_id":"c1d97655-7563-4c09-8b64-36d6cf6aaf7f","local_storage_dataset_allocation_id":null,"local_storage_unencrypted_dataset_allocation_id":"d7f76d63-a849-4dc0-be2e-58c4aa1197c7","required_dataset_overhead":367001600},"local_storage_dataset_allocation":{"Unencrypted":{"dataset_size":5735710720,"id":"d7f76d63-a849-4dc0-be2e-58c4aa1197c7","local_storage_unencrypted_dataset_id":"d5520ae8-87d9-43a0-8722-1c87fb4701a6","pool_id":"d7665e0d-9354-4341-a76f-965d7c49f277","sled_id":"b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88","time_created":"2026-05-01T18:16:52.448812Z","time_deleted":null}}}]
2026-05-01T18:16:52.978Z |          |    7: instance_start.ensure_local_storage_1 | started       | 
2026-05-01T18:16:52.981Z |          |  17: instance_start.ensure_local_storage_11 | started       | 
2026-05-01T18:16:52.984Z |          |   15: instance_start.ensure_local_storage_9 | started       | 
2026-05-01T18:16:52.988Z |          |  17: instance_start.ensure_local_storage_11 | succeeded     | 
2026-05-01T18:16:52.990Z |          |  16: instance_start.ensure_local_storage_10 | started       | 
2026-05-01T18:16:52.994Z |          |   13: instance_start.ensure_local_storage_7 | started       | 
2026-05-01T18:16:52.998Z |          |  16: instance_start.ensure_local_storage_10 | succeeded     | 
2026-05-01T18:16:53.000Z |          |   14: instance_start.ensure_local_storage_8 | started       | 
2026-05-01T18:16:53.003Z |          |   11: instance_start.ensure_local_storage_5 | started       | 
2026-05-01T18:16:53.006Z |          |   10: instance_start.ensure_local_storage_4 | started       | 
2026-05-01T18:16:53.011Z |          |   12: instance_start.ensure_local_storage_6 | started       | 
2026-05-01T18:16:53.014Z |          |    6: instance_start.ensure_local_storage_0 | started       | 
2026-05-01T18:16:53.017Z |          |    8: instance_start.ensure_local_storage_2 | started       | 
2026-05-01T18:16:53.022Z |          |    9: instance_start.ensure_local_storage_3 | started       | 
2026-05-01T18:16:53.668Z |          |   12: instance_start.ensure_local_storage_6 | failed        | "ensure_local_storage_6" => {"ActionFailed":{"source_error":{"InternalError":{"internal_message":"failed to ensure local storage: failed at attempt 2: progenitor API operation failed: Error Response: status: 500 Internal Server Error; headers: {\"content-type\": \"application/json\", \"x-request-id\": \"0113ddf2-f724-4bc8-a488-50701227030f\", \"content-length\": \"124\", \"date\": \"Fri, 01 May 2026 18:16:53 GMT\"}; value: Error { error_code: Some(\"Internal\"), message: \"Internal Server Error\", request_id: \"0113ddf2-f724-4bc8-a488-50701227030f\" }"}}}}
2026-05-01T18:16:53.735Z |          |  17: instance_start.ensure_local_storage_11 | undo_started  | 
2026-05-01T18:16:53.739Z |          |  17: instance_start.ensure_local_storage_11 | undo_finished | 
2026-05-01T18:16:53.741Z |          |  16: instance_start.ensure_local_storage_10 | undo_started  | 
2026-05-01T18:16:53.745Z |          |  16: instance_start.ensure_local_storage_10 | undo_finished | 
2026-05-01T18:16:54.280Z |          |   10: instance_start.ensure_local_storage_4 | succeeded     | 
2026-05-01T18:16:54.283Z |          |   10: instance_start.ensure_local_storage_4 | undo_started  | 
2026-05-01T18:16:54.287Z |          |   10: instance_start.ensure_local_storage_4 | undo_finished | 
2026-05-01T18:16:54.320Z |          |    8: instance_start.ensure_local_storage_2 | succeeded     | 
2026-05-01T18:16:54.324Z |          |    8: instance_start.ensure_local_storage_2 | undo_started  | 
2026-05-01T18:16:54.327Z |          |    8: instance_start.ensure_local_storage_2 | undo_finished | 
2026-05-01T18:16:54.365Z |          |   15: instance_start.ensure_local_storage_9 | succeeded     | 
2026-05-01T18:16:54.368Z |          |   15: instance_start.ensure_local_storage_9 | undo_started  | 
2026-05-01T18:16:54.372Z |          |   15: instance_start.ensure_local_storage_9 | undo_finished | 
2026-05-01T18:16:54.428Z |          |    7: instance_start.ensure_local_storage_1 | succeeded     | 
2026-05-01T18:16:54.432Z |          |    7: instance_start.ensure_local_storage_1 | undo_started  | 
2026-05-01T18:16:54.435Z |          |    7: instance_start.ensure_local_storage_1 | undo_finished | 
2026-05-01T18:16:54.472Z |          |   14: instance_start.ensure_local_storage_8 | succeeded     | 
2026-05-01T18:16:54.474Z |          |   14: instance_start.ensure_local_storage_8 | undo_started  | 
2026-05-01T18:16:54.477Z |          |   14: instance_start.ensure_local_storage_8 | undo_finished | 
2026-05-01T18:16:54.496Z |          |    6: instance_start.ensure_local_storage_0 | succeeded     | 
2026-05-01T18:16:54.499Z |          |    6: instance_start.ensure_local_storage_0 | undo_started  | 
2026-05-01T18:16:54.503Z |          |    6: instance_start.ensure_local_storage_0 | undo_finished | 
2026-05-01T18:16:54.508Z |          |   13: instance_start.ensure_local_storage_7 | succeeded     | 
2026-05-01T18:16:54.511Z |          |   13: instance_start.ensure_local_storage_7 | undo_started  | 
2026-05-01T18:16:54.514Z |          |   13: instance_start.ensure_local_storage_7 | undo_finished |

I went to the log files containing entries for this saga. The earliest one I found was /pool/ext/eb81728c-3b83-42fb-8133-ac32a0bdf70f/crypt/debug/oxz_nexus_0b1cc325-3c30-400d-80e4-59918f68cd65/oxide-nexus:default.log.1777660201 on BRM42220014 (sled 16). Of note, that shows this crash due to oxidecomputer/steno#17:

BRM42220014 # grep -v ^{ /pool/ext/eb81728c-3b83-42fb-8133-ac32a0bdf70f/crypt/debug/oxz_nexus_0b1cc325-3c30-400d-80e4-59918f68cd65/oxide-nexus:default.log.1777660201
thread 'tokio-rt-worker' (83) panicked at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/steno-0.4.1/src/saga_exec.rs:1587:13:
assertion failed: self.node_errors.contains_key(&node_id)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[ May  1 18:17:10 Stopping because all processes in service exited. ]
[ May  1 18:17:10 Executing stop method (:kill). ]
[ May  1 18:17:10 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/omicron-nexus/bin/nexus /var/svc/manifest/site/nexus/config.toml &"). ]
[ May  1 18:17:10 Method "start" exited with status 0. ]
note: configured to log to "/dev/stdout"

Here's are some relevant records:

BRM42220014 # grep b9df69fa-73a4-4a2e-be4a-d0832b88d01e /pool/ext/eb81728c-3b83-42fb-8133-ac32a0bdf70f/crypt/debug/oxz_nexus_0b1cc325-3c30-400d-80e4-59918f68cd65/oxide-nexus:default.log.1777660201 | looker -o long
2026-05-01 18:16:52.270Z INFO 0b1cc325-3c30-400d-80e4-59918f68cd65/19699 (ServerContext) on oxz_nexus_0b1cc325-3c30-400d-80e4-59918f68cd65: tracking newly created saga
    file = nexus/types/src/quiesce.rs:587
    saga_id = b9df69fa-73a4-4a2e-be4a-d0832b88d01e
...
2026-05-01 18:16:52.270Z INFO 0b1cc325-3c30-400d-80e4-59918f68cd65/19699 (ServerContext) on oxz_nexus_0b1cc325-3c30-400d-80e4-59918f68cd65: saga create
...
2026-05-01 18:16:52.278Z INFO 0b1cc325-3c30-400d-80e4-59918f68cd65/19699 (ServerContext) on oxz_nexus_0b1cc325-3c30-400d-80e4-59918f68cd65: starting saga
    file = nexus/src/app/saga.rs:383
    saga_id = b9df69fa-73a4-4a2e-be4a-d0832b88d01e
    saga_name = instance-start
...
2026-05-01 18:16:53.231Z WARN 0b1cc325-3c30-400d-80e4-59918f68cd65/19699 (ServerContext) on oxz_nexus_0b1cc325-3c30-400d-80e4-59918f68cd65: failed to ensure local storage dataset, retrying in 224.316821ms
    error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "9e203ac8-2d4c-427b-9bde-512b04146960", "content-length": "94", "date": "Fri, 01 May 2026 18:16:53 GMT"}; value: Error { error_code: None, message: "Service Unavailable", request_id: "9e203ac8-2d4c-427b-9bde-512b04146960" }
    file = nexus/src/app/sagas/instance_start.rs:712
    saga_id = b9df69fa-73a4-4a2e-be4a-d0832b88d01e
...
2026-05-01 18:16:53.668Z DEBG 0b1cc325-3c30-400d-80e4-59918f68cd65/19699 (ServerContext) on oxz_nexus_0b1cc325-3c30-400d-80e4-59918f68cd65: recording saga event
    event_type = Failed(ActionFailed { source_error: Object {"InternalError": Object {"internal_message": String("failed to ensure local storage: failed at attempt 2: progenitor API operation failed: Error Response: status: 500 Internal Server Error; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"0113ddf2-f724-4bc8-a488-50701227030f\\", \\"content-length\\": \\"124\\", \\"date\\": \\"Fri, 01 May 2026 18:16:53 GMT\\"}; value: Error { error_code: Some(\\"Internal\\"), message: \\"Internal Server Error\\", request_id: \\"0113ddf2-f724-4bc8-a488-50701227030f\\" }")}} })
    node_id = 12
    saga_id = b9df69fa-73a4-4a2e-be4a-d0832b88d01e
...
2026-05-01 18:16:55.976Z WARN 0b1cc325-3c30-400d-80e4-59918f68cd65/19699 (ServerContext) on oxz_nexus_0b1cc325-3c30-400d-80e4-59918f68cd65: failed to ensure local storage dataset, retrying in 2.485759139s
    error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "d5c57f2f-b19d-4e37-8e8b-40a0f0d8a812", "content-length": "94", "date": "Fri, 01 May 2026 18:16:55 GMT"}; value: Error { error_code: None, message: "Service Unavailable", request_id: "d5c57f2f-b19d-4e37-8e8b-40a0f0d8a812" }
    file = nexus/src/app/sagas/instance_start.rs:712
    saga_id = b9df69fa-73a4-4a2e-be4a-d0832b88d01e
    saga_name = instance-start

Then we have the Nexus crash on oxidecomputer/steno#17. Then:

2026-05-01 18:17:12.388Z INFO 0b1cc325-3c30-400d-80e4-59918f68cd65/12339 (ServerContext) on oxz_nexus_0b1cc325-3c30-400d-80e4-59918f68cd65: found saga that needs to be recovered
    background_task = saga_recovery
    file = nexus/saga-recovery/src/recovery.rs:400
    saga_id = b9df69fa-73a4-4a2e-be4a-d0832b88d01e
2026-05-01 18:17:12.791Z INFO 0b1cc325-3c30-400d-80e4-59918f68cd65/12339 (ServerContext) on oxz_nexus_0b1cc325-3c30-400d-80e4-59918f68cd65: recovering saga: start
    file = nexus/saga-recovery/src/recovery.rs:466
    saga_id = b9df69fa-73a4-4a2e-be4a-d0832b88d01e
    saga_name = instance-start
...
2026-05-01 18:17:12.798Z ERRO 0b1cc325-3c30-400d-80e4-59918f68cd65/12339 (ServerContext) on oxz_nexus_0b1cc325-3c30-400d-80e4-59918f68cd65: failed to recover saga
    error = Internal Error: failed to resume saga: recovery for saga b9df69fa-73a4-4a2e-be4a-d0832b88d01e: node NodeIndex(18): load status is "NeverStarted", which is illegal for parent load status "UndoFinished"
    file = nexus/saga-recovery/src/recovery.rs:486
    saga_id = b9df69fa-73a4-4a2e-be4a-d0832b88d01e
    saga_name = instance-start

This goes on for a while.

All of this is consistent with:

We hit panic in node_exec_state steno#17, which appears to have affected not just in-memory state but persistent state as well.
This saga is currently unrecoverable.
Handoff proceeded anyway. This makes sense, at least based on the current implementation, because it waits for sagas running in the Nexus process to finish (this one isn't running) or for at least one complete recovery pass (which did happen).
The health check reported it because it uses the database, not what's running in Nexus. I think we didn't consider that these could be distinct sets if some sagas were permanently unrecoverable.

which gets us to the questions I asked in the description.

Determinations from today's update sync:

Handoff should not block on these sagas. (The reason handoff blocks on sagas is because they can't be run by the new version of Nexus. But these sagas won't be run by the new version of Nexus anyway if they're unrecoverable.)
The health check should not include these sagas to avoid false positives.

Basically, these sagas are low-level problems that should be surfaced (see #10530) but they don't need to be coupled to the update process.

[update-status] temporarily disable stuck saga reporting (#10539)

Did some investigation here. There are two separate bugs.

The first is a race in in-memory state -- see the series of three comments starting at oxidecomputer/steno#17 (comment) for detailed discussion. This is fixed by:

The second is a separate bug in the recovery path after Nexus crashes and restarts, tripped by some of the same conditions that the first bug happens with. Note that durable state is not corrupt -- rather, steno's recovery validation was too strict. This is fixed by:

[2/n] relax saga recovery requirement for NeverStarted steno#462

[fm] Add saga diagnosis engine (#10592)

I believe we fixed the underlying issues here: the two steno bugs, plus the fact that these sagas were staying running (#10581), which was causing health check to report these sagas.

Navigation Menu

Repository navigation

health check reported hung sagas that aren't actually running #10531

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

health check reported hung sagas that aren't actually running #10531

Description

Activity

davepacheco commented on Jun 2, 2026

davepacheco commented on Jun 3, 2026

sunshowers commented on Jun 4, 2026

davepacheco commented on Oct 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions