Instance stuck in 'Creating' due to failure to create region ('tcp connect error: Connection refused (os error 146))

Rack: rack2 (Dogfood)

While running sanity tests on Dogfood, we got an instance stuck in 'Creating'.
Upon checking further, looks like the saga got stuck while creating a region:

= INSTANCE ====================================================================
                        ID: 8719ab23-4949-4841-89f4-99e4401a7415
                project ID: 290be4ae-3f36-40b0-9fc7-6d49277c9bf4
                      name: alldisks-instance-main-862916
               description: test-instance
                created at: 2026-09-22 18:54:16.575461 UTC
          last modified at: 2026-09-22 18:54:16.575461 UTC

== CONFIGURATION ===============================================================
                     vCPUs: 4
                    memory: 16 GiB
                  hostname: test-host
                 boot disk: None
              auto-restart:
                  InstanceAutoRestart {
                      policy: None,
                      cooldown: None,
                  }

== RUNTIME STATE ===============================================================
               nexus state: Creating
(i)     external API state: Creating
            intended state: running
           last updated at: 2026-09-22T18:54:16.575461Z (generation 1)
       needs reincarnation: false
             karmic status: saṃsāra (reincarnation enabled)
      last reincarnated at: None
             active VMM ID: None
             target VMM ID: None
              migration ID: None
              updater lock: UNLOCKED at generation: 1

18:54:17.011Z DEBG ba00935f-ad63-4bc2-a61e-f459809f986e (ServerContext): authorize result
    action = CreateChild
    actor = Some(Actor::SiloUser { silo_user_id: 9ceeb971-95af-4cbe-8b25-0c9ad57c925b (silo_user), silo_id: 7bd7623a-68ed-4636-8ecb-b59e3b068787, .. })
    authenticated = true
    resource = Instance { parent: Project { parent: Silo { parent: Fleet, key: 7bd7623a-68ed-4636-8ecb-b59e3b068787, lookup_type: ById(7bd7623a-68ed-4636-8ecb-b59e3b068787) }, key: 290be4ae-3f36-40b0-9fc7-6d49277c9bf4, lookup_type: ById(290be4ae-3f36-40b0-9fc7-6d49277c9bf4) }, key: 8719ab23-4949-4841-89f4-99e4401a7415, lookup_type: ById(8719ab23-4949-4841-89f4-99e4401a7415) }
    result = Ok(())
    saga_id = 17156bc3-8b9a-47b0-bcee-a41ceca46211
    saga_name = instance-create
    saga_node = CreateNetworkInterface0
    silo_id = 7bd7623a-68ed-4636-8ecb-b59e3b068787
    silo_user_id = 9ceeb971-95af-4cbe-8b25-0c9ad57c925b
    type = silo_user

18:56:19.123Z WARN ba00935f-ad63-4bc2-a61e-f459809f986e (ServerContext): failed to create region 7117425b-7aaa-4696-a700-b10665b52f7f, retrying in 118.445287179s
    error = Communication Error: error sending request for url (http://[fd00:1122:3344:10b::7]:32345/crucible/0/regions): error sending request for url (http://[fd00:1122:3344:10b::7]:32345/crucible/0/regions): client error (Connect): tcp connect error: Connection refused (os error 146)
    file = nexus/src/app/crucible.rs:209
    saga_id = 17156bc3-8b9a-47b0-bcee-a41ceca46211
    saga_name = instance-create
18:58:01.845Z INFO ba00935f-ad63-4bc2-a61e-f459809f986e (ServerContext): observed saga start
    background_task = saga_recovery
    file = nexus/saga-recovery/src/recovery.rs:78
    saga_id = 17156bc3-8b9a-47b0-bcee-a41ceca46211
18:58:01.846Z DEBG ba00935f-ad63-4bc2-a61e-f459809f986e (ServerContext): found saga that can be ignored (already running)
    background_task = saga_recovery
    saga_id = 17156bc3-8b9a-47b0-bcee-a41ceca46211
18:58:17.583Z WARN ba00935f-ad63-4bc2-a61e-f459809f986e (ServerContext): failed to create region 7117425b-7aaa-4696-a700-b10665b52f7f, retrying in 283.290336713s
    error = Communication Error: error sending request for url (http://[fd00:1122:3344:10b::7]:32345/crucible/0/regions): error sending request for url (http://[fd00:1122:3344:10b::7]:32345/crucible/0/regions): client error (Connect): tcp connect error: Connection refused (os error 146)
    file = nexus/src/app/crucible.rs:209
    saga_id = 17156bc3-8b9a-47b0-bcee-a41ceca46211
    saga_name = instance-create

note: database schema version matches expected (301.0.0)
 id                                   | current_sec                          | time_created             | name            | state
--------------------------------------+--------------------------------------+--------------------------+-----------------+---------
 17156bc3-8b9a-47b0-bcee-a41ceca46211 | ba00935f-ad63-4bc2-a61e-f459809f986e | 2026-09-22T18:54:16.415Z | instance-create | Running

DAG: {"end_node":32,"graph":{"edge_property":"directed","edges":[[0,1,null],[1,2,null],[2,3,null],[3,4,null],[4,5,null],[5,6,null],[6,7,null],[7,8,null],[8,9,null],[9,10,null],[10,11,null],[11,12,null],[12,13,null],[13,14,null],[15,16,null],[16,17,null],[17,18,null],[18,19,null],[19,20,null],[20,21,null],[21,22,null],[22,23,null],[14,15,null],[23,24,null],[24,25,null],[25,26,null],[26,27,null],[27,28,null],[28,29,null],[29,30,null],[31,0,null],[30,32,null]],"node_holes":[],"nodes":[{"Constant":{"name":"instance_id","value":"8719ab23-4949-4841-89f4-99e4401a7415"}},{"Action":{"action_name":"instance_create.create_instance_record","label":"CreateInstanceRecord","name":"instance_record"}},{"Action":{"action_name":"instance_create.associate_ssh_keys","label":"AssociateSshKeys","name":"output"}},{"Constant":{"name":"network_interface_params0","value":{"instance_id":"8719ab23-4949-4841-89f4-99e4401a7415","instance_name":"alldisks-instance-main-862916","interface_id":"3ee128c8-db02-43b3-8925-5c181b061360","nic_spec":{"Default":"Ipv4"},"project_id":"290be4ae-3f36-40b0-9fc7-6d49277c9bf4","serialized_authn":{"kind":{"Authenticated":[{"actor":{"SiloUser":{"silo_id":"7bd7623a-68ed-4636-8ecb-b59e3b068787","silo_user_id":"9ceeb971-95af-4cbe-8b25-0c9ad57c925b"}},"credential_id":"f0f22621-bd0e-40b9-bb35-2566c1c2b89a","device_token_expiration":null},{"mapped_fleet_roles":{"admin":["admin"]}}]}}}}},{"SubsagaStart":{"params_node_name":"network_interface_params0","saga_name":"instance-create-nic0"}},{"Action":{"action_name":"instance_create.create_network_interface","label":"CreateNetworkInterface0","name":"output"}},{"SubsagaEnd":{"name":"network_interface0"}},{"Action":{"action_name":"common.uuid_generate","label":"CreateSnatIpv4Id","name":"snat_ipv4_id"}},{"Action":{"action_name":"instance_create.create_snat_ipv4","label":"CreateSnatIpv4","name":"snat_ipv4"}},{"Constant":{"name":"external_ip_params0","value":{"eip_spec":{"pool_selector":{"ip_version":"v4","type":"auto"},"type":"ephemeral"},"instance_id":"8719ab23-4949-4841-89f4-99e4401a7415","ip_index":0,"new_eip_id":"b12f05c8-bc9c-439d-98cf-20e20af5b901","project_id":"290be4ae-3f36-40b0-9fc7-6d49277c9bf4","serialized_authn":{"kind":{"Authenticated":[{"actor":{"SiloUser":{"silo_id":"7bd7623a-68ed-4636-8ecb-b59e3b068787","silo_user_id":"9ceeb971-95af-4cbe-8b25-0c9ad57c925b"}},"credential_id":"f0f22621-bd0e-40b9-bb35-2566c1c2b89a","device_token_expiration":null},{"mapped_fleet_roles":{"admin":["admin"]}}]}}}}},{"SubsagaStart":{"params_node_name":"external_ip_params0","saga_name":"instance-create-external-ip0"}},{"Action":{"action_name":"instance_create.create_external_ip","label":"CreateExternalIp0","name":"external-ip-0"}},{"SubsagaEnd":{"name":"external_ip0"}},{"Constant":{"name":"create_disk_params0","value":{"create_params":{"description":"alldisks-instance-main-862916 disk","disk_backend":{"disk_source":{"image_id":"856488cf-adef-4171-a00e-2cf2ca2fc53f","read_only":false,"type":"image"},"type":"distributed"},"name":"alldisks-instance-main-862916-disk","size":32212254720},"project_id":"290be4ae-3f36-40b0-9fc7-6d49277c9bf4","serialized_authn":{"kind":{"Authenticated":[{"actor":{"SiloUser":{"silo_id":"7bd7623a-68ed-4636-8ecb-b59e3b068787","silo_user_id":"9ceeb971-95af-4cbe-8b25-0c9ad57c925b"}},"credential_id":"f0f22621-bd0e-40b9-bb35-2566c1c2b89a","device_token_expiration":null},{"mapped_fleet_roles":{"admin":["admin"]}}]}}}}},{"SubsagaStart":{"params_node_name":"create_disk_params0","saga_name":"instance-create-disk-0"}},{"Action":{"action_name":"common.uuid_generate","label":"GenerateDiskId","name":"disk_id"}},{"Action":{"action_name":"common.uuid_generate","label":"GenerateVolumeId","name":"volume_id"}},{"Action":{"action_name":"disk_create.space_account","label":"SpaceAccount","name":"no_result"}},{"Action":{"action_name":"disk_create.create_crucible_disk_record","label":"CreateCrucibleDiskRecord","name":"crucible_disk"}},{"Action":{"action_name":"disk_create.regions_alloc","label":"RegionsAlloc","name":"datasets_and_regions"}},{"Action":{"action_name":"disk_create.regions_ensure_undo","label":"RegionsEnsureUndo","name":"regions_ensure_undo"}},{"Action":{"action_name":"disk_create.regions_ensure","label":"RegionsEnsure","name":"regions_ensure"}},{"Action":{"action_name":"disk_create.create_volume_record","label":"CreateVolumeRecord","name":"created_volume"}},{"Action":{"action_name":"disk_create.finalize_disk_record","label":"FinalizeDiskRecord","name":"created_disk"}},{"SubsagaEnd":{"name":"create_disk0"}},{"Constant":{"name":"attach_disk_params0","value":{"attach_params":{"description":"alldisks-instance-main-862916 disk","disk_backend":{"disk_source":{"image_id":"856488cf-adef-4171-a00e-2cf2ca2fc53f","read_only":false,"type":"image"},"type":"distributed"},"name":"alldisks-instance-main-862916-disk","size":32212254720,"type":"create"},"instance_id":"8719ab23-4949-4841-89f4-99e4401a7415","project_id":"290be4ae-3f36-40b0-9fc7-6d49277c9bf4","serialized_authn":{"kind":{"Authenticated":[{"actor":{"SiloUser":{"silo_id":"7bd7623a-68ed-4636-8ecb-b59e3b068787","silo_user_id":"9ceeb971-95af-4cbe-8b25-0c9ad57c925b"}},"credential_id":"f0f22621-bd0e-40b9-bb35-2566c1c2b89a","device_token_expiration":null},{"mapped_fleet_roles":{"admin":["admin"]}}]}}}}},{"SubsagaStart":{"params_node_name":"attach_disk_params0","saga_name":"instance-attach-disk-0"}},{"Action":{"action_name":"instance_create.attach_disks_to_instance","label":"AttachDisksToInstance-0","name":"attach_disk_output"}},{"SubsagaEnd":{"name":"attach_disk0"}},{"Action":{"action_name":"instance_create.set_boot_disk","label":"SetBootDisk","name":"set_boot_disk"}},{"Action":{"action_name":"instance_create.move_to_stopped","label":"MoveToStopped","name":"stopped_instance"}},{"Start":{"params":{"boundary_switches":["switch1","switch0"],"create_params":{"anti_affinity_groups":[],"auto_restart_policy":null,"boot_disk":{"description":"alldisks-instance-main-862916 disk","disk_backend":{"disk_source":{"image_id":"856488cf-adef-4171-a00e-2cf2ca2fc53f","read_only":false,"type":"image"},"type":"distributed"},"name":"alldisks-instance-main-862916-disk","size":32212254720,"type":"create"},"cpu_platform":null,"description":"test-instance","disks":[],"enable_jumbo_frames":false,"external_ips":[{"pool_selector":{"ip_version":"v4","type":"auto"},"type":"ephemeral"}],"hostname":"test-host","memory":17179869184,"multicast_groups":[],"name":"alldisks-instance-main-862916","ncpus":4,"network_interfaces":{"type":"default_ipv4"},"ssh_public_keys":null,"start":true,"user_data":""},"project_id":"290be4ae-3f36-40b0-9fc7-6d49277c9bf4","serialized_authn":{"kind":{"Authenticated":[{"actor":{"SiloUser":{"silo_id":"7bd7623a-68ed-4636-8ecb-b59e3b068787","silo_user_id":"9ceeb971-95af-4cbe-8b25-0c9ad57c925b"}},"credential_id":"f0f22621-bd0e-40b9-bb35-2566c1c2b89a","device_token_expiration":null},{"mapped_fleet_roles":{"admin":["admin"]}}]}}}}},"End"]},"saga_name":"instance-create","start_node":31}

              event time | sub saga | node id                                                               | event type | data
------------------------ | -------- | --------------------------------------------------------------------- | ---------- | ---
2026-09-22T18:54:16.531Z |          |                                                             31: start | started    |
2026-09-22T18:54:16.551Z |          |                                                             31: start | succeeded  |
2026-09-22T18:54:16.559Z |          |                                                        0: instance_id | started    |
2026-09-22T18:54:16.564Z |          |                                                        0: instance_id | succeeded  |
2026-09-22T18:54:16.570Z |          |                             1: instance_create.create_instance_record | started    |
2026-09-22T18:54:16.784Z |          |                             1: instance_create.create_instance_record | succeeded  | "instance_record" => {"auto_restart":{"cooldown":null,"policy":null},"boot_disk_id":null,"cpu_platform":null,"dst_propolis_id":null,"enable_jumbo_frames":false,"hostname":"test-host","identity":{"description":"test-instance","id":"8719ab23-4949-4841-89f4-99e4401a7415","name":"alldisks-instance-main-862916","time_created":"2026-09-22T18:54:16.575461Z","time_deleted":null,"time_modified":"2026-09-22T18:54:16.575461Z"},"intended_state":"Running","memory":17179869184,"migration_id":null,"ncpus":4,"nexus_state":"Creating","project_id":"290be4ae-3f36-40b0-9fc7-6d49277c9bf4","propolis_id":null,"state_generation":1,"time_last_auto_restarted":null,"time_state_updated":"2026-09-22T18:54:16.575461Z","updater_gen":1,"updater_id":null,"user_data":[]}
2026-09-22T18:54:16.790Z |          |                                 2: instance_create.associate_ssh_keys | started    |
2026-09-22T18:54:16.866Z |          |                                 2: instance_create.associate_ssh_keys | succeeded  |
2026-09-22T18:54:16.871Z |          |                                          3: network_interface_params0 | started    |
2026-09-22T18:54:16.875Z |          |                                          3: network_interface_params0 | succeeded  |
2026-09-22T18:54:16.882Z | 0        |     4: subsaga start instance-create-nic0 (network_interface_params0) | started    |
2026-09-22T18:54:16.889Z | 0        |     4: subsaga start instance-create-nic0 (network_interface_params0) | succeeded  |
2026-09-22T18:54:16.917Z | 0        |                           5: instance_create.create_network_interface | started    |
2026-09-22T18:54:17.156Z | 0        |                           5: instance_create.create_network_interface | succeeded  |
2026-09-22T18:54:17.162Z | 0        |                                     6: subsaga end network_interface0 | started    |
2026-09-22T18:54:17.167Z | 0        |                                     6: subsaga end network_interface0 | succeeded  |
2026-09-22T18:54:17.176Z |          |                                               7: common.uuid_generate | started    |
2026-09-22T18:54:17.182Z |          |                                               7: common.uuid_generate | succeeded  | "snat_ipv4_id" => "878284df-e9c0-478c-be91-457dffba9283"
2026-09-22T18:54:17.187Z |          |                                   8: instance_create.create_snat_ipv4 | started    |
2026-09-22T18:54:17.856Z |          |                                   8: instance_create.create_snat_ipv4 | succeeded  |
2026-09-22T18:54:17.861Z |          |                                                9: external_ip_params0 | started    |
2026-09-22T18:54:17.867Z |          |                                                9: external_ip_params0 | succeeded  |
2026-09-22T18:54:17.872Z | 1        |  10: subsaga start instance-create-external-ip0 (external_ip_params0) | started    |
2026-09-22T18:54:17.877Z | 1        |  10: subsaga start instance-create-external-ip0 (external_ip_params0) | succeeded  |
2026-09-22T18:54:17.882Z | 1        |                                11: instance_create.create_external_ip | started    |
2026-09-22T18:54:18.330Z | 1        |                                11: instance_create.create_external_ip | succeeded  | "external-ip-0" => {"description":null,"first_port":0,"id":"b12f05c8-bc9c-439d-98cf-20e20af5b901","ip":"172.20.52.91/32","ip_pool_id":"cadb8535-e32f-4be2-9259-e45dec9fa3c7","ip_pool_range_id":"1ceb9b68-321d-4fa2-80d6-202b172b04b3","is_probe":false,"is_service":false,"kind":"Ephemeral","last_port":65535,"name":null,"parent_id":"8719ab23-4949-4841-89f4-99e4401a7415","project_id":null,"state":"Attaching","time_created":"2026-09-22T18:54:18.073413Z","time_deleted":null,"time_modified":"2026-09-22T18:54:18.124014Z"}
2026-09-22T18:54:18.336Z | 1        |                                          12: subsaga end external_ip0 | started    |
2026-09-22T18:54:18.340Z | 1        |                                          12: subsaga end external_ip0 | succeeded  |
2026-09-22T18:54:18.344Z |          |                                               13: create_disk_params0 | started    |
2026-09-22T18:54:18.349Z |          |                                               13: create_disk_params0 | succeeded  |
2026-09-22T18:54:18.354Z | 2        |        14: subsaga start instance-create-disk-0 (create_disk_params0) | started    |
2026-09-22T18:54:18.359Z | 2        |        14: subsaga start instance-create-disk-0 (create_disk_params0) | succeeded  |
2026-09-22T18:54:18.363Z | 2        |                                              15: common.uuid_generate | started    |
2026-09-22T18:54:18.367Z | 2        |                                              15: common.uuid_generate | succeeded  | "disk_id" => "94d897ab-478a-4b94-85e5-735df71d4e21"
2026-09-22T18:54:18.372Z | 2        |                                              16: common.uuid_generate | started    |
2026-09-22T18:54:18.376Z | 2        |                                              16: common.uuid_generate | succeeded  | "volume_id" => "322ff401-8647-4221-b63a-c0aedaab6b31"
2026-09-22T18:54:18.381Z | 2        |                                         17: disk_create.space_account | started    |
2026-09-22T18:54:18.413Z | 2        |                                         17: disk_create.space_account | succeeded  |
2026-09-22T18:54:18.418Z | 2        |                           18: disk_create.create_crucible_disk_record | started    |
2026-09-22T18:54:18.574Z | 2        |                           18: disk_create.create_crucible_disk_record | succeeded  | "crucible_disk" => {"disk":{"attach_instance_id":null,"block_size":"Traditional","disk_state":"creating","disk_type":"Crucible","identity":{"description":"alldisks-instance-main-862916 disk","id":"94d897ab-478a-4b94-85e5-735df71d4e21","name":"alldisks-instance-main-862916-disk","time_created":"2026-09-22T18:54:18.448041Z","time_deleted":null,"time_modified":"2026-09-22T18:54:18.448041Z"},"project_id":"290be4ae-3f36-40b0-9fc7-6d49277c9bf4","rcgen":1,"size":32212254720,"slot":null,"state_generation":1,"time_state_updated":"2026-09-22T18:54:18.448040726Z"},"disk_type_crucible":{"create_image_id":"856488cf-adef-4171-a00e-2cf2ca2fc53f","create_snapshot_id":null,"disk_id":"94d897ab-478a-4b94-85e5-735df71d4e21","pantry_address":null,"read_only":false,"volume_id":"322ff401-8647-4221-b63a-c0aedaab6b31"}}
2026-09-22T18:54:18.579Z | 2        |                                         19: disk_create.regions_alloc | started    |
2026-09-22T18:54:20.027Z | 2        |                                         19: disk_create.regions_alloc | succeeded  | "datasets_and_regions" => [[{"identity":{"id":"2ebbac4f-7b0f-43eb-99fd-dd6ff7f9e097","time_created":"2023-08-30T18:59:10.758516Z","time_modified":"2023-08-30T18:59:10.758516Z"},"ip":"fd00:1122:3344:102::6","no_provision":false,"pool_id":"138663ad-a382-4595-baf0-08f6b0276a67","port":32345,"rcgen":1,"size_used":903285309440,"time_deleted":null},{"block_size":512,"blocks_per_extent":131072,"dataset_id":"2ebbac4f-7b0f-43eb-99fd-dd6ff7f9e097","deleting":false,"extent_count":480,"identity":{"id":"6cdab412-3619-492a-84ba-985bfd4f1caf","time_created":"2026-09-22T18:54:18.665018Z","time_modified":"2026-09-22T18:54:18.665018Z"},"port":null,"read_only":false,"reservation_percent":"TwentyFive","volume_id":"322ff401-8647-4221-b63a-c0aedaab6b31"}],[{"identity":{"id":"668a4d4a-96dc-4b45-866b-bed3d64c26ec","time_created":"2023-08-30T18:59:10.758507Z","time_modified":"2023-08-30T18:59:10.758507Z"},"ip":"fd00:1122:3344:10b::7","no_provision":false,"pool_id":"021afd19-2f87-4def-9284-ab7add1dd6ae","port":32345,"rcgen":1,"size_used":2291096616960,"time_deleted":null},{"block_size":512,"blocks_per_extent":131072,"dataset_id":"668a4d4a-96dc-4b45-866b-bed3d64c26ec","deleting":false,"extent_count":480,"identity":{"id":"7117425b-7aaa-4696-a700-b10665b52f7f","time_created":"2026-09-22T18:54:18.665018Z","time_modified":"2026-09-22T18:54:18.665018Z"},"port":null,"read_only":false,"reservation_percent":"TwentyFive","volume_id":"322ff401-8647-4221-b63a-c0aedaab6b31"}],[{"identity":{"id":"180d466d-eb36-4546-8922-e52c4c076823","time_created":"2023-08-30T18:59:10.758514Z","time_modified":"2023-08-30T18:59:10.758514Z"},"ip":"fd00:1122:3344:101::5","no_provision":false,"pool_id":"ac789935-fa42-4d00-8967-df0d96dbb74e","port":32345,"rcgen":1,"size_used":1312649379840,"time_deleted":null},{"block_size":512,"blocks_per_extent":131072,"dataset_id":"180d466d-eb36-4546-8922-e52c4c076823","deleting":false,"extent_count":480,"identity":{"id":"ff03ebc0-c3c0-443e-96c1-04016dfb1c9a","time_created":"2026-09-22T18:54:18.665018Z","time_modified":"2026-09-22T18:54:18.665018Z"},"port":null,"read_only":false,"reservation_percent":"TwentyFive","volume_id":"322ff401-8647-4221-b63a-c0aedaab6b31"}]]
2026-09-22T18:54:20.033Z | 2        |                                   20: disk_create.regions_ensure_undo | started    |
2026-09-22T18:54:20.041Z | 2        |                                   20: disk_create.regions_ensure_undo | succeeded  |
2026-09-22T18:54:20.047Z | 2        |                                        21: disk_create.regions_ensure | started    |

It appears the saga is still running --- is it retrying the connection attempt indefinitely?

Looked like that, eventually the saga showed things wrapped up, there's just this 16 h gap in the disk_create.regions_ensure (which eventually failed).

026-09-22T18:54:20.027Z | 2        |                                         19: disk_create.regions_alloc | succeeded     | "datasets_and_regions" => [[{"identity":{"id":"2ebbac4f-7b0f-43eb-99fd-dd6ff7f9e097","time_created":"2023-08-30T18:59:10.758516Z","time_modified":"2023-08-30T18:59:10.758516Z"},"ip":"fd00:1122:3344:102::6","no_provision":false,"pool_id":"138663ad-a382-4595-baf0-08f6b0276a67","port":32345,"rcgen":1,"size_used":903285309440,"time_deleted":null},{"block_size":512,"blocks_per_extent":131072,"dataset_id":"2ebbac4f-7b0f-43eb-99fd-dd6ff7f9e097","deleting":false,"extent_count":480,"identity":{"id":"6cdab412-3619-492a-84ba-985bfd4f1caf","time_created":"2026-09-22T18:54:18.665018Z","time_modified":"2026-09-22T18:54:18.665018Z"},"port":null,"read_only":false,"reservation_percent":"TwentyFive","volume_id":"322ff401-8647-4221-b63a-c0aedaab6b31"}],[{"identity":{"id":"668a4d4a-96dc-4b45-866b-bed3d64c26ec","time_created":"2023-08-30T18:59:10.758507Z","time_modified":"2023-08-30T18:59:10.758507Z"},"ip":"fd00:1122:3344:10b::7","no_provision":false,"pool_id":"021afd19-2f87-4def-9284-ab7add1dd6ae","port":32345,"rcgen":1,"size_used":2291096616960,"time_deleted":null},{"block_size":512,"blocks_per_extent":131072,"dataset_id":"668a4d4a-96dc-4b45-866b-bed3d64c26ec","deleting":false,"extent_count":480,"identity":{"id":"7117425b-7aaa-4696-a700-b10665b52f7f","time_created":"2026-09-22T18:54:18.665018Z","time_modified":"2026-09-22T18:54:18.665018Z"},"port":null,"read_only":false,"reservation_percent":"TwentyFive","volume_id":"322ff401-8647-4221-b63a-c0aedaab6b31"}],[{"identity":{"id":"180d466d-eb36-4546-8922-e52c4c076823","time_created":"2023-08-30T18:59:10.758514Z","time_modified":"2023-08-30T18:59:10.758514Z"},"ip":"fd00:1122:3344:101::5","no_provision":false,"pool_id":"ac789935-fa42-4d00-8967-df0d96dbb74e","port":32345,"rcgen":1,"size_used":1312649379840,"time_deleted":null},{"block_size":512,"blocks_per_extent":131072,"dataset_id":"180d466d-eb36-4546-8922-e52c4c076823","deleting":false,"extent_count":480,"identity":{"id":"ff03ebc0-c3c0-443e-96c1-04016dfb1c9a","time_created":"2026-09-22T18:54:18.665018Z","time_modified":"2026-09-22T18:54:18.665018Z"},"port":null,"read_only":false,"reservation_percent":"TwentyFive","volume_id":"322ff401-8647-4221-b63a-c0aedaab6b31"}]]
2026-09-22T18:54:20.033Z | 2        |                                   20: disk_create.regions_ensure_undo | started       |
2026-09-22T18:54:20.041Z | 2        |                                   20: disk_create.regions_ensure_undo | succeeded     |
----------------------------------------- Time gap here -----------------------------------------
2026-09-22T18:54:20.047Z | 2        |                                        21: disk_create.regions_ensure | started       |
2026-09-23T11:24:08.674Z | 2        |                                        21: disk_create.regions_ensure | failed        | "regions_ensure" => {"ActionFailed":{"source_error":{"InternalError":{"internal_message":"Failed to create region, unexpected state: Failed"}}}}

there's just this 16 h gap in the disk_create.regions_ensure (which eventually failed).

wow! that's a while!

I'll close because it did get resolved but the '16 h waiting period' is still a mystery.

So the connection-refusing crucible-agent (http://[fd00:1122:3344:10b::7]:32345) was indeed the problem one on sled 14 that crashed repeatedly because the root dataset ran out of space. The stuck instance creation timestamps fall into the duration when crucible-agent stopped responding:

17:41:51.162Z INFO crucible-agent (worker): Region size:274877906944 reservation:343597383680 quota:824633720832
17:41:51.191Z INFO crucible-agent (worker): zfs set reservation of 343597383680 for oxp_021afd19-2f87-4def-9284-ab7add1dd6ae/crucible/regions/2aa35e0f-7cf5-40ed-a80f-2f1b5b92665c
17:41:51.191Z INFO crucible-agent (worker): zfs set quota of 824633720832 for oxp_021afd19-2f87-4def-9284-ab7add1dd6ae/crucible/regions/2aa35e0f-7cf5-40ed-a80f-2f1b5b92665c
17:41:51.421Z ERRO crucible-agent (worker): Dataset 2aa35e0f-7cf5-40ed-a80f-2f1b5b92665c creation failed: zfs create failed! out: err:cannot create 'oxp_021afd19-2f87-4def-9284-ab7add1dd6ae/crucible/regions/2aa35e0f-7cf5-40ed-a80f-2f1b5b92665c': out of space
17:41:51.421Z INFO crucible-agent (datafile): region 2aa35e0f-7cf5-40ed-a80f-2f1b5b92665c state: Requested -> Failed
17:41:51.467Z INFO crucible-agent (dropshot): request completed
    latency_us = 81
    local_addr = [fd00:1122:3344:10b::7]:32345
    method = POST
    remote_addr = [fd00:1122:3344:10b::92]:34047
    req_id = d1858cce-b887-4e00-8c13-58bfb1e19a29
    response_code = 200
    uri = /crucible/0/regions
17:41:59.442Z INFO crucible-agent (dropshot): request completed
    latency_us = 80
    local_addr = [fd00:1122:3344:10b::7]:32345
    method = GET
    remote_addr = [fd00:1122:3344:10b::92]:34047
    req_id = bcdd130a-d6fb-4237-ae59-bf945b1c4a45
    response_code = 200
    uri = /crucible/0/regions/2aa35e0f-7cf5-40ed-a80f-2f1b5b92665c
17:41:59.449Z INFO crucible-agent (datafile): region 2aa35e0f-7cf5-40ed-a80f-2f1b5b92665c state: Failed -> Tombstoned
17:41:59.468Z INFO crucible-agent (dropshot): request completed
    latency_us = 19274
    local_addr = [fd00:1122:3344:10b::7]:32345
    method = DELETE
    remote_addr = [fd00:1122:3344:10b::92]:34047
    req_id = 89cf5c6e-c5ad-4b2c-8cdf-bfaa7d7e4df4
    response_code = 204
    uri = /crucible/0/regions/2aa35e0f-7cf5-40ed-a80f-2f1b5b92665c
17:41:59.468Z INFO crucible-agent (worker): applying SMF actions before removal...
17:41:59.475Z INFO crucible-agent (dropshot): request completed
    latency_us = 60
    local_addr = [fd00:1122:3344:10b::7]:32345
    method = GET
    remote_addr = [fd00:1122:3344:10b::92]:34047
    req_id = 51358111-daa8-4715-999b-71654d5e33ee
    response_code = 200
    uri = /crucible/0/regions/2aa35e0f-7cf5-40ed-a80f-2f1b5b92665c
17:41:59.476Z INFO crucible-agent (worker): "downstairs-087d379b-b735-4a72-92af-97e0f9511744" marking clone as read_only
17:41:59.516Z INFO crucible-agent (worker): SMF ok!
17:41:59.524Z ERRO crucible-agent (worker): Cannot find region "2aa35e0f-7cf5-40ed-a80f-2f1b5b92665c" to remove: Dataset does not exist!
17:41:59.524Z INFO crucible-agent (datafile): region 2aa35e0f-7cf5-40ed-a80f-2f1b5b92665c state: Tombstoned -> Destroyed
17:41:59.560Z INFO crucible-agent (dropshot): request completed
    latency_us = 78
    local_addr = [fd00:1122:3344:10b::7]:32345
    method = GET
    remote_addr = [fd00:1122:3344:10b::92]:34047
    req_id = 51176b9e-086a-4803-8692-08dfce6db5c9
    response_code = 200
    uri = /crucible/0/regions/2aa35e0f-7cf5-40ed-a80f-2f1b5b92665c
[ Sep 22 21:42:58 Stopping because all processes in service exited. ]
[ Sep 22 21:42:58 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_021afd19-2f87-4def-9284-ab7add1dd6ae/crucible -l [fd00:1122:3344:10b::7]:32345 -P 19000 -p downstairs -s snapshot"). ]
note: configured to log to "/dev/stdout"
21:42:58.624Z INFO crucible-agent: Crucible Version: Crucible Version: 0.0.1
    Commit SHA: ad8a31742adc45e925e63443a5b43c8e30604022
    Commit timestamp: 2026-07-28T19:41:59.000000000Z  branch: main
    rustc: 1.90.0 stable x86_64-unknown-illumos
    Cargo: x86_64-unknown-illumos  Debug: false Opt level: 3
...

Crucible-agent recovered after I cleaned up the thousands of core files on that sled the next day.

Navigation Menu

Repository navigation

Instance stuck in 'Creating' due to failure to create region ('tcp connect error: Connection refused (os error 146)) #11351

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Instance stuck in 'Creating' due to failure to create region ('tcp connect error: Connection refused (os error 146)) #11351

Description

Activity

paudmir commented on Sep 22, 2026

hawkw commented on Sep 22, 2026

paudmir commented on Sep 23, 2026

hawkw commented on Sep 23, 2026

paudmir commented on Sep 28, 2026

askfongjojo commented on Oct 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions