Nexus didn't notice second MUPdate while resolving first one

[davepacheco]

Following up on #10796: in https://github.com/oxidecomputer/customer-support/issues/1372, we found a sequence where:

• Due to an operational issue, TUF artifact replication was not working. This wasn't caught because nothing was using it -- the system had only been MUPdated for updates.
• The rack was MUPdated to release N.
• The release N TUF repo was uploaded.
• Nexus began trying to resolve the MUPdate and its artifacts. Requests to send new sled configs to sled agents were failing because it didn't recognize many of the artifacts.
• The rack was MUPdated to release N + 1.
• The planner generated a new blueprint that modified sled configs only to add a new measurement file. I assume this was the measurements need for release N + 1.

At this point:

• at least one sled agent is reporting a local MUPdate override in its inventory
• Nexus is not generating any new blueprints

I would have expected Nexus to notice from the sled inventory that sleds had been MUPdated a second time. It should have generated new sled configs that set "remove_mupdate_override" and set all the artifacts in the sled config to "install dataset" or equivalent. It didn't.

[davepacheco]

I think this might be a result of #10392.

I have this reconfigurator-cli script that demonstrates the problem:

Details

# This test verifies that a new MUPdate is recognized while the system is in
# the middle of resolution of a previous MUPdate.

# Initial state: a system that has never had a target release set.
load-example --nsleds 3 --ndisks-per-sled 1

# Create a TUF repo that we can use to resolve artifacts.
tuf-assemble ../../update-common/manifests/fake.toml

# Simulate a first MUPdate.  Collect inventory and see the planner clear it.
sled-set serial0 mupdate-override 6123eac1-ec5b-42ba-b73f-9845105a9971
sled-update-install-dataset serial0 --from-repo repo-1.0.0.zip
inventory-generate
blueprint-plan target latest
blueprint-diff latest

# Have the sled acknowledge the clearing of the MUPdate override and go through
# another planner run.  (We also need to have it update its sled config
# generation to the latest so that the planner knows the sled has seen this
# latest update.)  The planner still doesn't have information needed to resolve
# "install-dataset" to specific artifacts.
sled-set serial0 mupdate-override unset
sled-set serial0 omicron-config latest
inventory-generate
blueprint-plan latest latest
blueprint-diff latest

# The next step will be to set the target release and have the planner resolve
# artifacts.
# NOTE: Due to what seems like the same bug, the following two steps seem
# to be necessary first.  That doesn't seem right. The most recent blueprint
# changed the sled config in a way that removed the MUPdate override flag.
# There's no reason the sled should need to acknowledge that before the
# planner can resolve artifacts on the sled.
# When fixed, I think we should be able to comment out these two lines
# and get the right result.
sled-set serial0 omicron-config latest
inventory-generate

# Now, set the target release.  This allows the planner to resolve the
# artifacts.
set target-release repo-1.0.0.zip
blueprint-plan latest latest
blueprint-diff latest

# NOTE: if you uncomment these lines, that works around the bug.
# The bug is that the planner won't make some changes related to MUPdate resolution
# until the generation reported by the sled's inventory matches what's in the parent
# blueprint.  But because of the MUPdate, that can't happen: Nexus needs to first
# process the mupdate_override reported by the sled.  It needs to do this even
# if the sled's inventory isn't up-to-date with the parent blueprint.
#sled-set serial0 omicron-config latest
#inventory-generate

# But suppose that the sled gets MUPdated again before it receives that request.
# The planner needs to process that as a new MUPdate, generating a new blueprint
# with the MUPdate override set and all the artifacts back at "install-dataset".
sled-set serial0 mupdate-override 306c7aee-0d06-4fc1-baeb-c18af4f78757
inventory-generate
blueprint-plan latest latest
blueprint-diff latest
blueprint-show latest

and here's the output:

Details

generated RNG seed: covetously-quality-admiral
> # This test verifies that a new MUPdate is recognized while the system is in
> # the middle of resolution of a previous MUPdate.

> # Initial state: a system that has never had a target release set.
> load-example --nsleds 3 --ndisks-per-sled 1
loaded example system with:
- collection: 715683b8-e4ac-414c-83a4-814a1035ec58
- blueprint: 65781fd8-5ed2-4075-bdb6-8e67a0fec18b


> # Create a TUF repo that we can use to resolve artifacts.
> tuf-assemble ../../update-common/manifests/fake.toml
error: output path `repo-1.0.0.zip` already exists


> # Simulate a first MUPdate.  Collect inventory and see the planner clear it.
> sled-set serial0 mupdate-override 6123eac1-ec5b-42ba-b73f-9845105a9971
set sled 60c217c8-0c65-412c-b680-4e5e545368ab mupdate override: unset -> 6123eac1-ec5b-42ba-b73f-9845105a9971

> sled-update-install-dataset serial0 --from-repo repo-1.0.0.zip
INFO extracting uploaded archive to /var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gp/T/.tmpuxAjaa
INFO created directory to store extracted artifacts, path: /var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gp/T/update-artifacts.VOxgFj
INFO added artifact, name: fake-gimlet-sp, kind: gimlet_sp, version: 1.0.0, hash: 68465b8e3f808f475510b525cfd62086d37ddd57688bd854184fdafb2b2198a4, length: 732
INFO added artifact, name: fake-rot, kind: gimlet_rot_image_a, version: 1.0.0, hash: d11e65f934bf0de51df2e5b484f61ee72072417b43ac87f33e958008428e7b02, length: 783
INFO added artifact, name: fake-rot, kind: gimlet_rot_image_b, version: 1.0.0, hash: d11e65f934bf0de51df2e5b484f61ee72072417b43ac87f33e958008428e7b02, length: 783
INFO added artifact, name: fake-rot-bootloader, kind: gimlet_rot_bootloader, version: 1.0.0, hash: 5b0f601b1fbb8674db9c751a02f8b14f8e6d4e8470f4f7b686fecb2c49ec11f9, length: 797
INFO added artifact, name: fake-host, kind: gimlet_host_phase_1, version: 1.0.0, hash: b99d5273ba1418bebb19d74b701d716896409566d41de76ada71bded4c9b166b, length: 524288
INFO added artifact, name: fake-host, kind: cosmo_host_phase_1, version: 1.0.0, hash: 9525f567106549a3fc32df870a74803d77e51dcc44190b218e227a2c5d444f58, length: 524288
INFO added artifact, name: fake-host, kind: host_phase_2, version: 1.0.0, hash: d944ae205b61ccf4322448f7d0311a819c53d9844769de066c5307c1682abb47, length: 1048576
INFO added artifact, name: fake-trampoline, kind: gimlet_trampoline_phase_1, version: 1.0.0, hash: bcb27520ee5a56e19f6df9662c66d69ac681fbd873a97547be5f6a5ae3d250c4, length: 524288
INFO added artifact, name: fake-trampoline, kind: cosmo_trampoline_phase_1, version: 1.0.0, hash: e235b8afb58ee69d966853bd5efe7c7e904da84b9035a332b3e691dc1d5cdbd0, length: 524288
INFO added artifact, name: fake-trampoline, kind: trampoline_phase_2, version: 1.0.0, hash: a5dfcc4bc69b791f1c509df499e9e72cce844cb2b53a56d8bb357b264bdf13b6, length: 1048576
INFO added artifact, name: clickhouse, kind: zone, version: 1.0.0, hash: 52b1eb4daff6f9140491d547b11248392920230db3db0eef5f5fa5333fe9e659, length: 1686
INFO added artifact, name: clickhouse_keeper, kind: zone, version: 1.0.0, hash: cda702919449d86663be97295043aeca0ead69ae5db3bbdb20053972254a27a3, length: 1690
INFO added artifact, name: clickhouse_server, kind: zone, version: 1.0.0, hash: 5f9ae6a9821bbe8ff0bf60feddf8b167902fe5f3e2c98bd21edd1ec9d969a001, length: 1690
INFO added artifact, name: cockroachdb, kind: zone, version: 1.0.0, hash: f3a1a3c0b3469367b005ee78665d982059d5e14e93a479412426bf941c4ed291, length: 1689
INFO added artifact, name: crucible-zone, kind: zone, version: 1.0.0, hash: 6f17cf65fb5a5bec5542dd07c03cd0acc01e59130f02c532c8d848ecae810047, length: 1690
INFO added artifact, name: crucible-pantry-zone, kind: zone, version: 1.0.0, hash: 21f0ada306859c23917361f2e0b9235806c32607ec689c7e8cf16bb898bc5a02, length: 1695
INFO added artifact, name: external-dns, kind: zone, version: 1.0.0, hash: ccca13ed19b8731f9adaf0d6203b02ea3b9ede4fa426b9fac0a07ce95440046d, length: 1689
INFO added artifact, name: internal-dns, kind: zone, version: 1.0.0, hash: ffbf1373f7ee08dddd74c53ed2a94e7c4c572a982d3a9bc94000c6956b700c6a, length: 1689
INFO added artifact, name: ntp, kind: zone, version: 1.0.0, hash: 67593d686ed04a1709f93972b71f4ebc148a9362120f65d239943e814a9a7439, length: 1681
INFO added artifact, name: nexus, kind: zone, version: 1.0.0, hash: 0e32b4a3e5d3668bb1d6a16fb06b74dc60b973fa479dcee0aae3adbb52bf1388, length: 1682
INFO added artifact, name: oximeter, kind: zone, version: 1.0.0, hash: 048d8fe8cdef5b175aad714d0f148aa80ce36c9114ac15ce9d02ed3d37877a77, length: 1682
INFO added artifact, name: fake-corpus, kind: measurement_corpus, version: 1.0.0, hash: 8a0e23157bae655fceec7376926c9758efee6511c7b7ff8355bbb49545a2257f, length: 1048576
INFO added artifact, name: fake-psc-sp, kind: psc_sp, version: 1.0.0, hash: 89245fe2ac7e6a2ac8dfa4e7d6891a6e6df95e4141395c07c64026778f6d76d7, length: 721
INFO added artifact, name: fake-psc-rot, kind: psc_rot_image_a, version: 1.0.0, hash: 5d8ea834dd6d42d386f1eb8a2c5f6e99b697c9958bb4ab8edf63e56003e25d8d, length: 775
INFO added artifact, name: fake-psc-rot, kind: psc_rot_image_b, version: 1.0.0, hash: 5d8ea834dd6d42d386f1eb8a2c5f6e99b697c9958bb4ab8edf63e56003e25d8d, length: 775
INFO added artifact, name: fake-psc-rot-bootloader, kind: psc_rot_bootloader, version: 1.0.0, hash: 18c9c774bfe4bb086e869509dcccacee8476fd87670692b765aee216f2c7f003, length: 805
INFO added artifact, name: fake-switch-sp, kind: switch_sp, version: 1.0.0, hash: bf1bc1da5059f76182c3007c3049941f8898abede2f3765b106c6e7f7c42d44c, length: 740
INFO added artifact, name: fake-switch-rot, kind: switch_rot_image_a, version: 1.0.0, hash: 32307d6d75c9707e8499ba4a4d379f99c0358237b6e190ff6a8024b470f62342, length: 774
INFO added artifact, name: fake-switch-rot, kind: switch_rot_image_b, version: 1.0.0, hash: 32307d6d75c9707e8499ba4a4d379f99c0358237b6e190ff6a8024b470f62342, length: 774
INFO added artifact, name: fake-switch-rot-bootloader, kind: switch_rot_bootloader, version: 1.0.0, hash: 70836d170abd5621f95bb4225987b27b3d3dd6168e73cd60e44309bdfeb94e98, length: 804
INFO added artifact, name: installinator_document, kind: installinator_document, version: 1.0.0, hash: 16717c0caa420c7811666e720936c7ca913cd59a4d376331cc79a90544e5e2e8, length: 526
sled 60c217c8-0c65-412c-b680-4e5e545368ab: install dataset updated: from repo at repo-1.0.0.zip (system version 1.0.0)

> inventory-generate
generated inventory collection 568902f7-6068-4764-be4f-5f51b6f929b9 from configured sleds

> blueprint-plan target latest
INFO blueprint mupdate override updated to match inventory, phase: do_plan_mupdate_override, sled_id: 60c217c8-0c65-412c-b680-4e5e545368ab, new_bp_override: 6123eac1-ec5b-42ba-b73f-9845105a9971, prev_bp_override: None, zones: 
  - zone 4aab364a-390e-4aae-a5c9-c3f82f2726c6 (ExternalDns) left unchanged, image source: install dataset
  - zone 4c37c87c-62de-4813-b963-2b21e57c6199 (InternalDns) left unchanged, image source: install dataset
  - zone 5b721761-1186-495b-b166-ed177f2feb74 (CruciblePantry) left unchanged, image source: install dataset
  - zone 661d1b79-ac2a-42bc-8c26-457bb6b86fae (Nexus) left unchanged, image source: install dataset
  - zone 7e37ada2-dce7-41de-8ef2-55ce445a2476 (InternalNtp) left unchanged, image source: install dataset
  - zone bf1c77a1-8bdb-4f2b-b0a1-ed5e682468bd (Clickhouse) left unchanged, image source: install dataset
  - zone e49e5412-d8e9-47cc-b715-edfb9b70b43c (Crucible) left unchanged, image source: install dataset
, host_phase_2: 
  - host phase 2 slot A: current contents (unchanged)
  - host phase 2 slot B: current contents (unchanged)
, measurements: (unknown)
INFO no previous MGS update found as part of updating blueprint mupdate override to match inventory, phase: do_plan_mupdate_override, sled_id: 60c217c8-0c65-412c-b680-4e5e545368ab
INFO updating target release minimum generation based on new set-override actions, phase: do_plan_mupdate_override, current_generation: 1, new_generation: 2
INFO skipping noop image source check for all sleds, reason: no target release is currently set
generated blueprint e4c4632f-f71a-4ee9-b6bd-b5cbf60d5fb5 based on parent blueprint 65781fd8-5ed2-4075-bdb6-8e67a0fec18b
blueprint source: planner with report:
planning report:
* zone adds and updates are blocked:
  - current target release generation (1) is lower than minimum required by blueprint (2)
  - sleds have remove mupdate override set in blueprint: 60c217c8-0c65-412c-b680-4e5e545368ab
  - sleds have deployment units with image sources not set to Artifact:
    - sled 60c217c8-0c65-412c-b680-4e5e545368ab: 7 zones
    - sled 678f5c63-c633-40e3-8e4c-b9b1ec5e8668: 6 zones
    - sled 6aa99dbd-8aa4-47df-bae8-276c27f98d85: 6 zones

  - sleds have measurements with image sources not set to Artifact:
    - sled 60c217c8-0c65-412c-b680-4e5e545368ab set to install dataset

* adding zones despite being blocked, because: target release generation is 1
* zone updates waiting on zone add blockers
Measurement updates:
Waiting on zone add/update blockers



> blueprint-diff latest
from: blueprint 65781fd8-5ed2-4075-bdb6-8e67a0fec18b
to:   blueprint e4c4632f-f71a-4ee9-b6bd-b5cbf60d5fb5

 MODIFIED SLEDS:

  sled 60c217c8-0c65-412c-b680-4e5e545368ab (active, config generation 2 -> 3):
+   will remove mupdate override:   (none) -> 6123eac1-ec5b-42ba-b73f-9845105a9971

    measurements:
    -----------------------------------
    hash              version          
    -----------------------------------
-   unknown           (unknown version)
+   install dataset   (no version)     


    host phase 2 contents:
    ------------------------
    slot   boot image source
    ------------------------
    A      current contents 
    B      current contents 


    physical disks:
    ------------------------------------------------------------------------------------
    vendor        model        serial                                        disposition
    ------------------------------------------------------------------------------------
    fake-vendor   fake-model   serial-77e738de-f45f-4074-9e16-60d6a52686f1   in service 


    datasets:
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    dataset name                                                                                                   dataset id                             disposition   quota     reservation   compression
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crucible                                                              85653e99-3c9e-4818-9b41-02da58cfea1c   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/clickhouse                                                      5a2820a5-0634-4cf9-b48d-5439511f518f   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/external_dns                                                    51b3b6b9-b33e-4233-ba39-c71c463a8377   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/internal_dns                                                    9d915b65-ef5b-4fa8-987b-bc4aee96bbb2   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone                                                            6f503fce-5c25-4f3f-b64c-16a60271a782   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_clickhouse_bf1c77a1-8bdb-4f2b-b0a1-ed5e682468bd        3fc0f387-fb75-4979-9964-3f607e87bed5   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_crucible_e49e5412-d8e9-47cc-b715-edfb9b70b43c          9309eb85-7ad9-421a-be51-9eebca3c24bd   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_crucible_pantry_5b721761-1186-495b-b166-ed177f2feb74   b1ce5354-fe3f-4dab-bc02-3ed783400f49   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_external_dns_4aab364a-390e-4aae-a5c9-c3f82f2726c6      02be96e7-da73-4545-be5f-7473f9c706d0   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_internal_dns_4c37c87c-62de-4813-b963-2b21e57c6199      0803f364-0eff-4573-9edb-8651f5893a52   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_nexus_661d1b79-ac2a-42bc-8c26-457bb6b86fae             78e58ac9-99cc-4f47-9d18-af9d77c72ac1   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_ntp_7e37ada2-dce7-41de-8ef2-55ce445a2476               6794611f-981e-4e63-a320-32f5eb97f284   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/debug                                                           21e3b5bf-0a0e-42d4-969d-f86a2c7021b2   in service    100 GiB   none          gzip-9     
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/local_storage                                                   280abe43-b702-4741-ae5d-ccbbe46ccee0   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/local_storage_unencrypted                                             fc570888-5e69-4063-8b7f-43c2dbde9725   in service    none      none          off        


    omicron zones:
    --------------------------------------------------------------------------------------------------------------
    zone type         zone id                                image source      disposition   underlay IP          
    --------------------------------------------------------------------------------------------------------------
    clickhouse        bf1c77a1-8bdb-4f2b-b0a1-ed5e682468bd   install dataset   in service    fd00:1122:3344:101::5
    crucible          e49e5412-d8e9-47cc-b715-edfb9b70b43c   install dataset   in service    fd00:1122:3344:101::8
    crucible_pantry   5b721761-1186-495b-b166-ed177f2feb74   install dataset   in service    fd00:1122:3344:101::7
    external_dns      4aab364a-390e-4aae-a5c9-c3f82f2726c6   install dataset   in service    fd00:1122:3344:101::6
    internal_dns      4c37c87c-62de-4813-b963-2b21e57c6199   install dataset   in service    fd00:1122:3344:1::1  
    internal_ntp      7e37ada2-dce7-41de-8ef2-55ce445a2476   install dataset   in service    fd00:1122:3344:101::3
    nexus             661d1b79-ac2a-42bc-8c26-457bb6b86fae   install dataset   in service    fd00:1122:3344:101::4


 COCKROACHDB SETTINGS:
    state fingerprint:::::::::::::::::   (none) (unchanged)
    cluster.preserve_downgrade_option:   (do not modify) (unchanged)

 METADATA:
    internal DNS version:::   1 (unchanged)
    external DNS version:::   1 (unchanged)
*   target release min gen:   1 -> 2
    nexus gen::::::::::::::   1 (unchanged)

 OXIMETER SETTINGS:
    generation:   1 (unchanged)
    read from::   SingleNode (unchanged)


internal DNS:
  DNS zone: "control-plane.oxide.internal" (unchanged)
    unchanged names: 40 (records: 56)

external DNS:
  DNS zone: "oxide.example" (unchanged)
    unchanged names: 5 (records: 9)




> # Have the sled acknowledge the clearing of the MUPdate override and go through
> # another planner run.  (We also need to have it update its sled config
> # generation to the latest so that the planner knows the sled has seen this
> # latest update.)  The planner still doesn't have information needed to resolve
> # "install-dataset" to specific artifacts.
> sled-set serial0 mupdate-override unset
set sled 60c217c8-0c65-412c-b680-4e5e545368ab mupdate override: 6123eac1-ec5b-42ba-b73f-9845105a9971 -> unset

> sled-set serial0 omicron-config latest
set sled 60c217c8-0c65-412c-b680-4e5e545368ab omicron config from latest blueprint (e4c4632f-f71a-4ee9-b6bd-b5cbf60d5fb5)

> inventory-generate
generated inventory collection 99d0a505-4bb6-4905-89ed-bbcb27fc03f8 from configured sleds

> blueprint-plan latest latest
INFO inventory override no longer exists, blueprint override cleared, phase: do_plan_mupdate_override, sled_id: 60c217c8-0c65-412c-b680-4e5e545368ab, prev_bp_override: 6123eac1-ec5b-42ba-b73f-9845105a9971
INFO skipping noop image source check for all sleds, reason: no target release is currently set
generated blueprint 4b72edc7-1983-4684-8e6a-da5ba5490c3d based on parent blueprint e4c4632f-f71a-4ee9-b6bd-b5cbf60d5fb5
blueprint source: planner with report:
planning report:
* zone adds and updates are blocked:
  - current target release generation (1) is lower than minimum required by blueprint (2)
  - sleds have deployment units with image sources not set to Artifact:
    - sled 60c217c8-0c65-412c-b680-4e5e545368ab: 7 zones
    - sled 678f5c63-c633-40e3-8e4c-b9b1ec5e8668: 6 zones
    - sled 6aa99dbd-8aa4-47df-bae8-276c27f98d85: 6 zones

  - sleds have measurements with image sources not set to Artifact:
    - sled 60c217c8-0c65-412c-b680-4e5e545368ab set to install dataset

* adding zones despite being blocked, because: target release generation is 1
* zone updates waiting on zone add blockers
Measurement updates:
Waiting on zone add/update blockers



> blueprint-diff latest
from: blueprint e4c4632f-f71a-4ee9-b6bd-b5cbf60d5fb5
to:   blueprint 4b72edc7-1983-4684-8e6a-da5ba5490c3d

 MODIFIED SLEDS:

  sled 60c217c8-0c65-412c-b680-4e5e545368ab (active, config generation 3 -> 4):
-   will remove mupdate override:   6123eac1-ec5b-42ba-b73f-9845105a9971 -> (none)

    measurements:
    ------------------------------
    hash              version     
    ------------------------------
    install dataset   (no version)


    host phase 2 contents:
    ------------------------
    slot   boot image source
    ------------------------
    A      current contents 
    B      current contents 


    physical disks:
    ------------------------------------------------------------------------------------
    vendor        model        serial                                        disposition
    ------------------------------------------------------------------------------------
    fake-vendor   fake-model   serial-77e738de-f45f-4074-9e16-60d6a52686f1   in service 


    datasets:
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    dataset name                                                                                                   dataset id                             disposition   quota     reservation   compression
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crucible                                                              85653e99-3c9e-4818-9b41-02da58cfea1c   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/clickhouse                                                      5a2820a5-0634-4cf9-b48d-5439511f518f   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/external_dns                                                    51b3b6b9-b33e-4233-ba39-c71c463a8377   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/internal_dns                                                    9d915b65-ef5b-4fa8-987b-bc4aee96bbb2   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone                                                            6f503fce-5c25-4f3f-b64c-16a60271a782   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_clickhouse_bf1c77a1-8bdb-4f2b-b0a1-ed5e682468bd        3fc0f387-fb75-4979-9964-3f607e87bed5   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_crucible_e49e5412-d8e9-47cc-b715-edfb9b70b43c          9309eb85-7ad9-421a-be51-9eebca3c24bd   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_crucible_pantry_5b721761-1186-495b-b166-ed177f2feb74   b1ce5354-fe3f-4dab-bc02-3ed783400f49   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_external_dns_4aab364a-390e-4aae-a5c9-c3f82f2726c6      02be96e7-da73-4545-be5f-7473f9c706d0   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_internal_dns_4c37c87c-62de-4813-b963-2b21e57c6199      0803f364-0eff-4573-9edb-8651f5893a52   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_nexus_661d1b79-ac2a-42bc-8c26-457bb6b86fae             78e58ac9-99cc-4f47-9d18-af9d77c72ac1   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_ntp_7e37ada2-dce7-41de-8ef2-55ce445a2476               6794611f-981e-4e63-a320-32f5eb97f284   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/debug                                                           21e3b5bf-0a0e-42d4-969d-f86a2c7021b2   in service    100 GiB   none          gzip-9     
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/local_storage                                                   280abe43-b702-4741-ae5d-ccbbe46ccee0   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/local_storage_unencrypted                                             fc570888-5e69-4063-8b7f-43c2dbde9725   in service    none      none          off        


    omicron zones:
    --------------------------------------------------------------------------------------------------------------
    zone type         zone id                                image source      disposition   underlay IP          
    --------------------------------------------------------------------------------------------------------------
    clickhouse        bf1c77a1-8bdb-4f2b-b0a1-ed5e682468bd   install dataset   in service    fd00:1122:3344:101::5
    crucible          e49e5412-d8e9-47cc-b715-edfb9b70b43c   install dataset   in service    fd00:1122:3344:101::8
    crucible_pantry   5b721761-1186-495b-b166-ed177f2feb74   install dataset   in service    fd00:1122:3344:101::7
    external_dns      4aab364a-390e-4aae-a5c9-c3f82f2726c6   install dataset   in service    fd00:1122:3344:101::6
    internal_dns      4c37c87c-62de-4813-b963-2b21e57c6199   install dataset   in service    fd00:1122:3344:1::1  
    internal_ntp      7e37ada2-dce7-41de-8ef2-55ce445a2476   install dataset   in service    fd00:1122:3344:101::3
    nexus             661d1b79-ac2a-42bc-8c26-457bb6b86fae   install dataset   in service    fd00:1122:3344:101::4


 COCKROACHDB SETTINGS:
    state fingerprint:::::::::::::::::   (none) (unchanged)
    cluster.preserve_downgrade_option:   (do not modify) (unchanged)

 METADATA:
    internal DNS version:::   1 (unchanged)
    external DNS version:::   1 (unchanged)
    target release min gen:   2 (unchanged)
    nexus gen::::::::::::::   1 (unchanged)

 OXIMETER SETTINGS:
    generation:   1 (unchanged)
    read from::   SingleNode (unchanged)


internal DNS:
  DNS zone: "control-plane.oxide.internal" (unchanged)
    unchanged names: 40 (records: 56)

external DNS:
  DNS zone: "oxide.example" (unchanged)
    unchanged names: 5 (records: 9)




> # The next step will be to set the target release and have the planner resolve
> # artifacts.
> # NOTE: Due to what seems like the same bug, the following two steps seem
> # to be necessary first.  That doesn't seem right. The most recent blueprint
> # changed the sled config in a way that removed the MUPdate override flag.
> # There's no reason the sled should need to acknowledge that before the
> # planner can resolve artifacts on the sled.
> # When fixed, I think we should be able to comment out these two lines
> # and get the right result.
> sled-set serial0 omicron-config latest
set sled 60c217c8-0c65-412c-b680-4e5e545368ab omicron config from latest blueprint (4b72edc7-1983-4684-8e6a-da5ba5490c3d)

> inventory-generate
generated inventory collection 500afa51-d6b2-4436-b38a-dba27f2dad09 from configured sleds


> # Now, set the target release.  This allows the planner to resolve the
> # artifacts.
> set target-release repo-1.0.0.zip
INFO extracting uploaded archive to /var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gp/T/.tmpyCvQhe
INFO created directory to store extracted artifacts, path: /var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gp/T/update-artifacts.X1Okxy
INFO added artifact, name: fake-gimlet-sp, kind: gimlet_sp, version: 1.0.0, hash: 68465b8e3f808f475510b525cfd62086d37ddd57688bd854184fdafb2b2198a4, length: 732
INFO added artifact, name: fake-rot, kind: gimlet_rot_image_a, version: 1.0.0, hash: d11e65f934bf0de51df2e5b484f61ee72072417b43ac87f33e958008428e7b02, length: 783
INFO added artifact, name: fake-rot, kind: gimlet_rot_image_b, version: 1.0.0, hash: d11e65f934bf0de51df2e5b484f61ee72072417b43ac87f33e958008428e7b02, length: 783
INFO added artifact, name: fake-rot-bootloader, kind: gimlet_rot_bootloader, version: 1.0.0, hash: 5b0f601b1fbb8674db9c751a02f8b14f8e6d4e8470f4f7b686fecb2c49ec11f9, length: 797
INFO added artifact, name: fake-host, kind: gimlet_host_phase_1, version: 1.0.0, hash: b99d5273ba1418bebb19d74b701d716896409566d41de76ada71bded4c9b166b, length: 524288
INFO added artifact, name: fake-host, kind: cosmo_host_phase_1, version: 1.0.0, hash: 9525f567106549a3fc32df870a74803d77e51dcc44190b218e227a2c5d444f58, length: 524288
INFO added artifact, name: fake-host, kind: host_phase_2, version: 1.0.0, hash: d944ae205b61ccf4322448f7d0311a819c53d9844769de066c5307c1682abb47, length: 1048576
INFO added artifact, name: fake-trampoline, kind: gimlet_trampoline_phase_1, version: 1.0.0, hash: bcb27520ee5a56e19f6df9662c66d69ac681fbd873a97547be5f6a5ae3d250c4, length: 524288
INFO added artifact, name: fake-trampoline, kind: cosmo_trampoline_phase_1, version: 1.0.0, hash: e235b8afb58ee69d966853bd5efe7c7e904da84b9035a332b3e691dc1d5cdbd0, length: 524288
INFO added artifact, name: fake-trampoline, kind: trampoline_phase_2, version: 1.0.0, hash: a5dfcc4bc69b791f1c509df499e9e72cce844cb2b53a56d8bb357b264bdf13b6, length: 1048576
INFO added artifact, name: clickhouse, kind: zone, version: 1.0.0, hash: 52b1eb4daff6f9140491d547b11248392920230db3db0eef5f5fa5333fe9e659, length: 1686
INFO added artifact, name: clickhouse_keeper, kind: zone, version: 1.0.0, hash: cda702919449d86663be97295043aeca0ead69ae5db3bbdb20053972254a27a3, length: 1690
INFO added artifact, name: clickhouse_server, kind: zone, version: 1.0.0, hash: 5f9ae6a9821bbe8ff0bf60feddf8b167902fe5f3e2c98bd21edd1ec9d969a001, length: 1690
INFO added artifact, name: cockroachdb, kind: zone, version: 1.0.0, hash: f3a1a3c0b3469367b005ee78665d982059d5e14e93a479412426bf941c4ed291, length: 1689
INFO added artifact, name: crucible-zone, kind: zone, version: 1.0.0, hash: 6f17cf65fb5a5bec5542dd07c03cd0acc01e59130f02c532c8d848ecae810047, length: 1690
INFO added artifact, name: crucible-pantry-zone, kind: zone, version: 1.0.0, hash: 21f0ada306859c23917361f2e0b9235806c32607ec689c7e8cf16bb898bc5a02, length: 1695
INFO added artifact, name: external-dns, kind: zone, version: 1.0.0, hash: ccca13ed19b8731f9adaf0d6203b02ea3b9ede4fa426b9fac0a07ce95440046d, length: 1689
INFO added artifact, name: internal-dns, kind: zone, version: 1.0.0, hash: ffbf1373f7ee08dddd74c53ed2a94e7c4c572a982d3a9bc94000c6956b700c6a, length: 1689
INFO added artifact, name: ntp, kind: zone, version: 1.0.0, hash: 67593d686ed04a1709f93972b71f4ebc148a9362120f65d239943e814a9a7439, length: 1681
INFO added artifact, name: nexus, kind: zone, version: 1.0.0, hash: 0e32b4a3e5d3668bb1d6a16fb06b74dc60b973fa479dcee0aae3adbb52bf1388, length: 1682
INFO added artifact, name: oximeter, kind: zone, version: 1.0.0, hash: 048d8fe8cdef5b175aad714d0f148aa80ce36c9114ac15ce9d02ed3d37877a77, length: 1682
INFO added artifact, name: fake-corpus, kind: measurement_corpus, version: 1.0.0, hash: 8a0e23157bae655fceec7376926c9758efee6511c7b7ff8355bbb49545a2257f, length: 1048576
INFO added artifact, name: fake-psc-sp, kind: psc_sp, version: 1.0.0, hash: 89245fe2ac7e6a2ac8dfa4e7d6891a6e6df95e4141395c07c64026778f6d76d7, length: 721
INFO added artifact, name: fake-psc-rot, kind: psc_rot_image_a, version: 1.0.0, hash: 5d8ea834dd6d42d386f1eb8a2c5f6e99b697c9958bb4ab8edf63e56003e25d8d, length: 775
INFO added artifact, name: fake-psc-rot, kind: psc_rot_image_b, version: 1.0.0, hash: 5d8ea834dd6d42d386f1eb8a2c5f6e99b697c9958bb4ab8edf63e56003e25d8d, length: 775
INFO added artifact, name: fake-psc-rot-bootloader, kind: psc_rot_bootloader, version: 1.0.0, hash: 18c9c774bfe4bb086e869509dcccacee8476fd87670692b765aee216f2c7f003, length: 805
INFO added artifact, name: fake-switch-sp, kind: switch_sp, version: 1.0.0, hash: bf1bc1da5059f76182c3007c3049941f8898abede2f3765b106c6e7f7c42d44c, length: 740
INFO added artifact, name: fake-switch-rot, kind: switch_rot_image_a, version: 1.0.0, hash: 32307d6d75c9707e8499ba4a4d379f99c0358237b6e190ff6a8024b470f62342, length: 774
INFO added artifact, name: fake-switch-rot, kind: switch_rot_image_b, version: 1.0.0, hash: 32307d6d75c9707e8499ba4a4d379f99c0358237b6e190ff6a8024b470f62342, length: 774
INFO added artifact, name: fake-switch-rot-bootloader, kind: switch_rot_bootloader, version: 1.0.0, hash: 70836d170abd5621f95bb4225987b27b3d3dd6168e73cd60e44309bdfeb94e98, length: 804
INFO added artifact, name: installinator_document, kind: installinator_document, version: 1.0.0, hash: 16717c0caa420c7811666e720936c7ca913cd59a4d376331cc79a90544e5e2e8, length: 526
set target release based on repo-1.0.0.zip

> blueprint-plan latest latest
INFO performed noop zone image source checks on sled, sled_id: 60c217c8-0c65-412c-b680-4e5e545368ab, num_total: 7, num_already_artifact: 0, num_eligible: 7, num_ineligible: 0
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 60c217c8-0c65-412c-b680-4e5e545368ab, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 60c217c8-0c65-412c-b680-4e5e545368ab, slot: b, expected_hash: 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b
INFO measurement is eligible for noop image source conversion, sled_id: 60c217c8-0c65-412c-b680-4e5e545368ab, new_measurement_source: 8a0e23157bae655fceec7376926c9758efee6511c7b7ff8355bbb49545a2257f version 1.0.0

INFO performed noop zone image source checks on sled, sled_id: 678f5c63-c633-40e3-8e4c-b9b1ec5e8668, num_total: 6, num_already_artifact: 0, num_eligible: 0, num_ineligible: 6
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 678f5c63-c633-40e3-8e4c-b9b1ec5e8668, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 678f5c63-c633-40e3-8e4c-b9b1ec5e8668, slot: b, expected_hash: 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b
INFO current sled measurements are in an unknown state, sled_id: 678f5c63-c633-40e3-8e4c-b9b1ec5e8668
INFO performed noop zone image source checks on sled, sled_id: 6aa99dbd-8aa4-47df-bae8-276c27f98d85, num_total: 6, num_already_artifact: 0, num_eligible: 0, num_ineligible: 6
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 6aa99dbd-8aa4-47df-bae8-276c27f98d85, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 6aa99dbd-8aa4-47df-bae8-276c27f98d85, slot: b, expected_hash: 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b
INFO current sled measurements are in an unknown state, sled_id: 6aa99dbd-8aa4-47df-bae8-276c27f98d85
generated blueprint fb9439c1-60f6-4752-a954-862620dbf59a based on parent blueprint 4b72edc7-1983-4684-8e6a-da5ba5490c3d
blueprint source: planner with report:
planning report:
* noop converting 7/7 install-dataset zones to artifact store on sled 60c217c8-0c65-412c-b680-4e5e545368ab
* noop converting current measurements to Artifact on sled 60c217c8-0c65-412c-b680-4e5e545368ab
* zone adds waiting on blockers
* zone adds and updates are blocked:
  - sleds have deployment units with image sources not set to Artifact:
    - sled 678f5c63-c633-40e3-8e4c-b9b1ec5e8668: 6 zones
    - sled 6aa99dbd-8aa4-47df-bae8-276c27f98d85: 6 zones

* zone updates waiting on zone add blockers
* waiting to update top-level nexus_generation: some non-Nexus zone are not yet updated
Measurement updates:
Waiting on zone add/update blockers



> blueprint-diff latest
from: blueprint 4b72edc7-1983-4684-8e6a-da5ba5490c3d
to:   blueprint fb9439c1-60f6-4752-a954-862620dbf59a

 MODIFIED SLEDS:

  sled 60c217c8-0c65-412c-b680-4e5e545368ab (active, config generation 4 -> 5):

    measurements:
    --------------------------------------------------------------------------------
    hash                                                               version      
    --------------------------------------------------------------------------------
-   install dataset                                                    (no version) 
+   8a0e23157bae655fceec7376926c9758efee6511c7b7ff8355bbb49545a2257f   version 1.0.0


    host phase 2 contents:
    ------------------------
    slot   boot image source
    ------------------------
    A      current contents 
    B      current contents 


    physical disks:
    ------------------------------------------------------------------------------------
    vendor        model        serial                                        disposition
    ------------------------------------------------------------------------------------
    fake-vendor   fake-model   serial-77e738de-f45f-4074-9e16-60d6a52686f1   in service 


    datasets:
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    dataset name                                                                                                   dataset id                             disposition   quota     reservation   compression
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crucible                                                              85653e99-3c9e-4818-9b41-02da58cfea1c   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/clickhouse                                                      5a2820a5-0634-4cf9-b48d-5439511f518f   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/external_dns                                                    51b3b6b9-b33e-4233-ba39-c71c463a8377   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/internal_dns                                                    9d915b65-ef5b-4fa8-987b-bc4aee96bbb2   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone                                                            6f503fce-5c25-4f3f-b64c-16a60271a782   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_clickhouse_bf1c77a1-8bdb-4f2b-b0a1-ed5e682468bd        3fc0f387-fb75-4979-9964-3f607e87bed5   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_crucible_e49e5412-d8e9-47cc-b715-edfb9b70b43c          9309eb85-7ad9-421a-be51-9eebca3c24bd   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_crucible_pantry_5b721761-1186-495b-b166-ed177f2feb74   b1ce5354-fe3f-4dab-bc02-3ed783400f49   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_external_dns_4aab364a-390e-4aae-a5c9-c3f82f2726c6      02be96e7-da73-4545-be5f-7473f9c706d0   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_internal_dns_4c37c87c-62de-4813-b963-2b21e57c6199      0803f364-0eff-4573-9edb-8651f5893a52   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_nexus_661d1b79-ac2a-42bc-8c26-457bb6b86fae             78e58ac9-99cc-4f47-9d18-af9d77c72ac1   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_ntp_7e37ada2-dce7-41de-8ef2-55ce445a2476               6794611f-981e-4e63-a320-32f5eb97f284   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/debug                                                           21e3b5bf-0a0e-42d4-969d-f86a2c7021b2   in service    100 GiB   none          gzip-9     
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/local_storage                                                   280abe43-b702-4741-ae5d-ccbbe46ccee0   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/local_storage_unencrypted                                             fc570888-5e69-4063-8b7f-43c2dbde9725   in service    none      none          off        


    omicron zones:
    ------------------------------------------------------------------------------------------------------------------------
    zone type         zone id                                image source                disposition   underlay IP          
    ------------------------------------------------------------------------------------------------------------------------
*   clickhouse        bf1c77a1-8bdb-4f2b-b0a1-ed5e682468bd   - install dataset           in service    fd00:1122:3344:101::5
     └─                                                      + artifact: version 1.0.0                                      
*   crucible          e49e5412-d8e9-47cc-b715-edfb9b70b43c   - install dataset           in service    fd00:1122:3344:101::8
     └─                                                      + artifact: version 1.0.0                                      
*   crucible_pantry   5b721761-1186-495b-b166-ed177f2feb74   - install dataset           in service    fd00:1122:3344:101::7
     └─                                                      + artifact: version 1.0.0                                      
*   external_dns      4aab364a-390e-4aae-a5c9-c3f82f2726c6   - install dataset           in service    fd00:1122:3344:101::6
     └─                                                      + artifact: version 1.0.0                                      
*   internal_dns      4c37c87c-62de-4813-b963-2b21e57c6199   - install dataset           in service    fd00:1122:3344:1::1  
     └─                                                      + artifact: version 1.0.0                                      
*   internal_ntp      7e37ada2-dce7-41de-8ef2-55ce445a2476   - install dataset           in service    fd00:1122:3344:101::3
     └─                                                      + artifact: version 1.0.0                                      
*   nexus             661d1b79-ac2a-42bc-8c26-457bb6b86fae   - install dataset           in service    fd00:1122:3344:101::4
     └─                                                      + artifact: version 1.0.0                                      


 COCKROACHDB SETTINGS:
    state fingerprint:::::::::::::::::   (none) (unchanged)
    cluster.preserve_downgrade_option:   (do not modify) (unchanged)

 METADATA:
    internal DNS version:::   1 (unchanged)
    external DNS version:::   1 (unchanged)
    target release min gen:   2 (unchanged)
    nexus gen::::::::::::::   1 (unchanged)

 OXIMETER SETTINGS:
    generation:   1 (unchanged)
    read from::   SingleNode (unchanged)


internal DNS:
  DNS zone: "control-plane.oxide.internal" (unchanged)
    unchanged names: 40 (records: 56)

external DNS:
  DNS zone: "oxide.example" (unchanged)
    unchanged names: 5 (records: 9)




> # NOTE: if you uncomment these lines, that works around the bug.
> # The bug is that the planner won't make some changes related to MUPdate resolution
> # until the generation reported by the sled's inventory matches what's in the parent
> # blueprint.  But because of the MUPdate, that can't happen: Nexus needs to first
> # process the mupdate_override reported by the sled.  It needs to do this even
> # if the sled's inventory isn't up-to-date with the parent blueprint.
> #sled-set serial0 omicron-config latest
> #inventory-generate

> # But suppose that the sled gets MUPdated again before it receives that request.
> # The planner needs to process that as a new MUPdate, generating a new blueprint
> # with the MUPdate override set and all the artifacts back at "install-dataset".
> sled-set serial0 mupdate-override 306c7aee-0d06-4fc1-baeb-c18af4f78757
set sled 60c217c8-0c65-412c-b680-4e5e545368ab mupdate override: unset -> 306c7aee-0d06-4fc1-baeb-c18af4f78757

> inventory-generate
generated inventory collection 9e2098ba-b61c-4577-b49c-fceaeac60c73 from configured sleds

> blueprint-plan latest latest
INFO inventory override observed, but blueprint override was not set to match it, phase: do_plan_mupdate_override, sled_id: 60c217c8-0c65-412c-b680-4e5e545368ab, inv_override: 306c7aee-0d06-4fc1-baeb-c18af4f78757, bp_override: None, reason: inventory stale: inventory reported generation (4) that is older than the generation in the parent blueprint (5)
INFO skipped noop image source check on sled, sled_id: 60c217c8-0c65-412c-b680-4e5e545368ab, reason: inventory stale: inventory reported generation (4) that is older than the generation in the parent blueprint (5)
INFO performed noop zone image source checks on sled, sled_id: 678f5c63-c633-40e3-8e4c-b9b1ec5e8668, num_total: 6, num_already_artifact: 0, num_eligible: 0, num_ineligible: 6
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 678f5c63-c633-40e3-8e4c-b9b1ec5e8668, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 678f5c63-c633-40e3-8e4c-b9b1ec5e8668, slot: b, expected_hash: 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b
INFO current sled measurements are in an unknown state, sled_id: 678f5c63-c633-40e3-8e4c-b9b1ec5e8668
INFO performed noop zone image source checks on sled, sled_id: 6aa99dbd-8aa4-47df-bae8-276c27f98d85, num_total: 6, num_already_artifact: 0, num_eligible: 0, num_ineligible: 6
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 6aa99dbd-8aa4-47df-bae8-276c27f98d85, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 6aa99dbd-8aa4-47df-bae8-276c27f98d85, slot: b, expected_hash: 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b
INFO current sled measurements are in an unknown state, sled_id: 6aa99dbd-8aa4-47df-bae8-276c27f98d85
generated blueprint 9cb4b4d2-b854-415e-bdce-049e76f9795e based on parent blueprint fb9439c1-60f6-4752-a954-862620dbf59a
blueprint source: planner with report:
planning report:
* zone adds waiting on blockers
* zone adds and updates are blocked:
  - sleds have deployment units with image sources not set to Artifact:
    - sled 678f5c63-c633-40e3-8e4c-b9b1ec5e8668: 6 zones
    - sled 6aa99dbd-8aa4-47df-bae8-276c27f98d85: 6 zones

* zone updates waiting on zone add blockers
* waiting to update top-level nexus_generation: some non-Nexus zone are not yet updated
Measurement updates:
Waiting on zone add/update blockers



> blueprint-diff latest
from: blueprint fb9439c1-60f6-4752-a954-862620dbf59a
to:   blueprint 9cb4b4d2-b854-415e-bdce-049e76f9795e

 COCKROACHDB SETTINGS:
    state fingerprint:::::::::::::::::   (none) (unchanged)
    cluster.preserve_downgrade_option:   (do not modify) (unchanged)

 METADATA:
    internal DNS version:::   1 (unchanged)
    external DNS version:::   1 (unchanged)
    target release min gen:   2 (unchanged)
    nexus gen::::::::::::::   1 (unchanged)

 OXIMETER SETTINGS:
    generation:   1 (unchanged)
    read from::   SingleNode (unchanged)


internal DNS:
  DNS zone: "control-plane.oxide.internal" (unchanged)
    unchanged names: 40 (records: 56)

external DNS:
  DNS zone: "oxide.example" (unchanged)
    unchanged names: 5 (records: 9)



> blueprint-show latest
blueprint  9cb4b4d2-b854-415e-bdce-049e76f9795e
parent:    fb9439c1-60f6-4752-a954-862620dbf59a

  sled: 60c217c8-0c65-412c-b680-4e5e545368ab
    state:::::::::::::   active
    config generation:   5
    subnet::::::::::::   fd00:1122:3344:101::/64
    last allocated IP:   fd00:1122:3344:101::8

    host phase 2 contents:
    ------------------------
    slot   boot image source
    ------------------------
    A      current contents 
    B      current contents 


    measurements:
    --------------------------------------------------------------------------------
    hash                                                               version      
    --------------------------------------------------------------------------------
    8a0e23157bae655fceec7376926c9758efee6511c7b7ff8355bbb49545a2257f   version 1.0.0


    physical disks:
    ------------------------------------------------------------------------------------
    vendor        model        serial                                        disposition
    ------------------------------------------------------------------------------------
    fake-vendor   fake-model   serial-77e738de-f45f-4074-9e16-60d6a52686f1   in service 


    datasets:
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    dataset name                                                                                                   dataset id                             disposition   quota     reservation   compression
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crucible                                                              85653e99-3c9e-4818-9b41-02da58cfea1c   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/clickhouse                                                      5a2820a5-0634-4cf9-b48d-5439511f518f   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/external_dns                                                    51b3b6b9-b33e-4233-ba39-c71c463a8377   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/internal_dns                                                    9d915b65-ef5b-4fa8-987b-bc4aee96bbb2   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone                                                            6f503fce-5c25-4f3f-b64c-16a60271a782   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_clickhouse_bf1c77a1-8bdb-4f2b-b0a1-ed5e682468bd        3fc0f387-fb75-4979-9964-3f607e87bed5   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_crucible_e49e5412-d8e9-47cc-b715-edfb9b70b43c          9309eb85-7ad9-421a-be51-9eebca3c24bd   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_crucible_pantry_5b721761-1186-495b-b166-ed177f2feb74   b1ce5354-fe3f-4dab-bc02-3ed783400f49   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_external_dns_4aab364a-390e-4aae-a5c9-c3f82f2726c6      02be96e7-da73-4545-be5f-7473f9c706d0   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_internal_dns_4c37c87c-62de-4813-b963-2b21e57c6199      0803f364-0eff-4573-9edb-8651f5893a52   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_nexus_661d1b79-ac2a-42bc-8c26-457bb6b86fae             78e58ac9-99cc-4f47-9d18-af9d77c72ac1   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/zone/oxz_ntp_7e37ada2-dce7-41de-8ef2-55ce445a2476               6794611f-981e-4e63-a320-32f5eb97f284   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/debug                                                           21e3b5bf-0a0e-42d4-969d-f86a2c7021b2   in service    100 GiB   none          gzip-9     
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/crypt/local_storage                                                   280abe43-b702-4741-ae5d-ccbbe46ccee0   in service    none      none          off        
    oxp_77e738de-f45f-4074-9e16-60d6a52686f1/local_storage_unencrypted                                             fc570888-5e69-4063-8b7f-43c2dbde9725   in service    none      none          off        


    omicron zones:
    ----------------------------------------------------------------------------------------------------------------------
    zone type         zone id                                image source              disposition   underlay IP          
    ----------------------------------------------------------------------------------------------------------------------
    clickhouse        bf1c77a1-8bdb-4f2b-b0a1-ed5e682468bd   artifact: version 1.0.0   in service    fd00:1122:3344:101::5
    crucible          e49e5412-d8e9-47cc-b715-edfb9b70b43c   artifact: version 1.0.0   in service    fd00:1122:3344:101::8
    crucible_pantry   5b721761-1186-495b-b166-ed177f2feb74   artifact: version 1.0.0   in service    fd00:1122:3344:101::7
    external_dns      4aab364a-390e-4aae-a5c9-c3f82f2726c6   artifact: version 1.0.0   in service    fd00:1122:3344:101::6
    internal_dns      4c37c87c-62de-4813-b963-2b21e57c6199   artifact: version 1.0.0   in service    fd00:1122:3344:1::1  
    internal_ntp      7e37ada2-dce7-41de-8ef2-55ce445a2476   artifact: version 1.0.0   in service    fd00:1122:3344:101::3
    nexus             661d1b79-ac2a-42bc-8c26-457bb6b86fae   artifact: version 1.0.0   in service    fd00:1122:3344:101::4



  sled: 678f5c63-c633-40e3-8e4c-b9b1ec5e8668
    state:::::::::::::   active
    config generation:   2
    subnet::::::::::::   fd00:1122:3344:102::/64
    last allocated IP:   fd00:1122:3344:102::7

    host phase 2 contents:
    ------------------------
    slot   boot image source
    ------------------------
    A      current contents 
    B      current contents 


    measurements:
    ---------------------------
    hash      version          
    ---------------------------
    unknown   (unknown version)


    physical disks:
    ------------------------------------------------------------------------------------
    vendor        model        serial                                        disposition
    ------------------------------------------------------------------------------------
    fake-vendor   fake-model   serial-8de12b4e-903f-48b3-8d2c-d73c7d1154e3   in service 


    datasets:
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    dataset name                                                                                                   dataset id                             disposition   quota     reservation   compression
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    oxp_8de12b4e-903f-48b3-8d2c-d73c7d1154e3/crucible                                                              cf785b1d-5a23-48d7-8d08-a7f49bda33a4   in service    none      none          off        
    oxp_8de12b4e-903f-48b3-8d2c-d73c7d1154e3/crypt/external_dns                                                    c756267b-8d7d-437e-a926-c5bdd1972118   in service    none      none          off        
    oxp_8de12b4e-903f-48b3-8d2c-d73c7d1154e3/crypt/internal_dns                                                    767c1372-425a-4754-b98f-0f2e6415fb6f   in service    none      none          off        
    oxp_8de12b4e-903f-48b3-8d2c-d73c7d1154e3/crypt/zone                                                            6bcd114c-b508-406c-b7f3-d50f41ec1429   in service    none      none          off        
    oxp_8de12b4e-903f-48b3-8d2c-d73c7d1154e3/crypt/zone/oxz_crucible_f8060215-7f80-4ad5-9ef7-aab48f889d5a          c930966c-3c78-40ae-a832-984cba395fb9   in service    none      none          off        
    oxp_8de12b4e-903f-48b3-8d2c-d73c7d1154e3/crypt/zone/oxz_crucible_pantry_bc4cdc07-07d8-4f0f-a05c-4f17205499f5   8d76b7be-98bb-49be-93e2-a715245a67c9   in service    none      none          off        
    oxp_8de12b4e-903f-48b3-8d2c-d73c7d1154e3/crypt/zone/oxz_external_dns_b01d98c4-cc20-4653-95f8-9b9171ad2445      443716b4-4d0b-465d-a81d-5b25ccf49239   in service    none      none          off        
    oxp_8de12b4e-903f-48b3-8d2c-d73c7d1154e3/crypt/zone/oxz_internal_dns_4162d46c-3179-47aa-94b5-4fcc0631425b      0e8caa5b-465d-4548-a688-88fdf4384728   in service    none      none          off        
    oxp_8de12b4e-903f-48b3-8d2c-d73c7d1154e3/crypt/zone/oxz_nexus_2ea411d8-b831-45e1-8fab-48b0af69974e             d08c8adb-e473-4447-ba03-b3182f4d2be5   in service    none      none          off        
    oxp_8de12b4e-903f-48b3-8d2c-d73c7d1154e3/crypt/zone/oxz_ntp_7ca446a1-2800-467f-ad91-a84ba58d8509               2d370bca-6188-4b9b-a2de-8e7d65233a88   in service    none      none          off        
    oxp_8de12b4e-903f-48b3-8d2c-d73c7d1154e3/crypt/debug                                                           fb4fb0d3-2086-4ef7-9a56-12641114574f   in service    100 GiB   none          gzip-9     
    oxp_8de12b4e-903f-48b3-8d2c-d73c7d1154e3/crypt/local_storage                                                   3f519199-c05c-4673-a1f2-641efb029376   in service    none      none          off        
    oxp_8de12b4e-903f-48b3-8d2c-d73c7d1154e3/local_storage_unencrypted                                             fbb22ff4-7ab5-42ff-a969-e748af6091e1   in service    none      none          off        


    omicron zones:
    --------------------------------------------------------------------------------------------------------------
    zone type         zone id                                image source      disposition   underlay IP          
    --------------------------------------------------------------------------------------------------------------
    crucible          f8060215-7f80-4ad5-9ef7-aab48f889d5a   install dataset   in service    fd00:1122:3344:102::7
    crucible_pantry   bc4cdc07-07d8-4f0f-a05c-4f17205499f5   install dataset   in service    fd00:1122:3344:102::6
    external_dns      b01d98c4-cc20-4653-95f8-9b9171ad2445   install dataset   in service    fd00:1122:3344:102::5
    internal_dns      4162d46c-3179-47aa-94b5-4fcc0631425b   install dataset   in service    fd00:1122:3344:2::1  
    internal_ntp      7ca446a1-2800-467f-ad91-a84ba58d8509   install dataset   in service    fd00:1122:3344:102::3
    nexus             2ea411d8-b831-45e1-8fab-48b0af69974e   install dataset   in service    fd00:1122:3344:102::4



  sled: 6aa99dbd-8aa4-47df-bae8-276c27f98d85
    state:::::::::::::   active
    config generation:   2
    subnet::::::::::::   fd00:1122:3344:103::/64
    last allocated IP:   fd00:1122:3344:103::7

    host phase 2 contents:
    ------------------------
    slot   boot image source
    ------------------------
    A      current contents 
    B      current contents 


    measurements:
    ---------------------------
    hash      version          
    ---------------------------
    unknown   (unknown version)


    physical disks:
    ------------------------------------------------------------------------------------
    vendor        model        serial                                        disposition
    ------------------------------------------------------------------------------------
    fake-vendor   fake-model   serial-198e684a-c90b-46ac-ac9a-945223af876c   in service 


    datasets:
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    dataset name                                                                                                   dataset id                             disposition   quota     reservation   compression
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    oxp_198e684a-c90b-46ac-ac9a-945223af876c/crucible                                                              4ff766a3-d517-4e9b-8905-5b102badfbd1   in service    none      none          off        
    oxp_198e684a-c90b-46ac-ac9a-945223af876c/crypt/external_dns                                                    27137186-f947-45fe-8e66-e071b3679050   in service    none      none          off        
    oxp_198e684a-c90b-46ac-ac9a-945223af876c/crypt/internal_dns                                                    1eb090da-a1a3-475c-aac1-ca30f00fd4dc   in service    none      none          off        
    oxp_198e684a-c90b-46ac-ac9a-945223af876c/crypt/zone                                                            35ca8ee2-92f5-4718-bbc0-ca32296a4a28   in service    none      none          off        
    oxp_198e684a-c90b-46ac-ac9a-945223af876c/crypt/zone/oxz_crucible_4ce62fe1-2e3b-4d9b-b740-7ee8b413bf7d          600c726d-a59f-4408-a92f-a2b6aa112778   in service    none      none          off        
    oxp_198e684a-c90b-46ac-ac9a-945223af876c/crypt/zone/oxz_crucible_pantry_21935d73-5eec-4b74-8b62-dfd30f48a8a6   ae13e456-ae2b-46c1-bcef-dbfb9ba0635b   in service    none      none          off        
    oxp_198e684a-c90b-46ac-ac9a-945223af876c/crypt/zone/oxz_external_dns_0d539549-a3b8-4e35-af06-a8133dba0db8      df4c80fd-a95a-427b-8772-f42bf2d39a86   in service    none      none          off        
    oxp_198e684a-c90b-46ac-ac9a-945223af876c/crypt/zone/oxz_internal_dns_a0a69c92-2644-4fbd-be83-a8b43a4c8d1b      ed892e4e-4280-4458-822c-b44f3354dae8   in service    none      none          off        
    oxp_198e684a-c90b-46ac-ac9a-945223af876c/crypt/zone/oxz_nexus_5a107cd4-8e7e-4234-afd3-d2503e6dc888             5385ae00-c6f0-488d-8c69-a2e126e911cf   in service    none      none          off        
    oxp_198e684a-c90b-46ac-ac9a-945223af876c/crypt/zone/oxz_ntp_76075d32-2e10-4aa4-b931-31acd0edaec5               fc24eb8c-9c9f-49f4-bc4d-4dae7e3a2cec   in service    none      none          off        
    oxp_198e684a-c90b-46ac-ac9a-945223af876c/crypt/debug                                                           8d87737a-052a-45bf-b608-7bdc26b3adbb   in service    100 GiB   none          gzip-9     
    oxp_198e684a-c90b-46ac-ac9a-945223af876c/crypt/local_storage                                                   9a024591-6c13-4c21-ba7c-a87f991e0dd4   in service    none      none          off        
    oxp_198e684a-c90b-46ac-ac9a-945223af876c/local_storage_unencrypted                                             4d453978-0107-4b25-b6cf-e019e30acbf4   in service    none      none          off        


    omicron zones:
    --------------------------------------------------------------------------------------------------------------
    zone type         zone id                                image source      disposition   underlay IP          
    --------------------------------------------------------------------------------------------------------------
    crucible          4ce62fe1-2e3b-4d9b-b740-7ee8b413bf7d   install dataset   in service    fd00:1122:3344:103::7
    crucible_pantry   21935d73-5eec-4b74-8b62-dfd30f48a8a6   install dataset   in service    fd00:1122:3344:103::6
    external_dns      0d539549-a3b8-4e35-af06-a8133dba0db8   install dataset   in service    fd00:1122:3344:103::5
    internal_dns      a0a69c92-2644-4fbd-be83-a8b43a4c8d1b   install dataset   in service    fd00:1122:3344:3::1  
    internal_ntp      76075d32-2e10-4aa4-b931-31acd0edaec5   install dataset   in service    fd00:1122:3344:103::3
    nexus             5a107cd4-8e7e-4234-afd3-d2503e6dc888   install dataset   in service    fd00:1122:3344:103::4


 COCKROACHDB SETTINGS:
    state fingerprint:::::::::::::::::   (none)
    cluster.preserve_downgrade_option:   (do not modify)

 OXIMETER SETTINGS:
    generation:   1
    read from::   SingleNode

 METADATA:
    created by::::::::::::::   reconfigurator-sim
    created at::::::::::::::   2026-07-11T16:56:26.094Z
    comment:::::::::::::::::   (none)
    internal DNS version::::   1
    external DNS version::::   1
    target release min gen::   2
    nexus gen:::::::::::::::   1
    external networking gen:   2

 PENDING MGS-MANAGED UPDATES: 0

blueprint source: planner with report:
planning report:
* zone adds waiting on blockers
* zone adds and updates are blocked:
  - sleds have deployment units with image sources not set to Artifact:
    - sled 678f5c63-c633-40e3-8e4c-b9b1ec5e8668: 6 zones
    - sled 6aa99dbd-8aa4-47df-bae8-276c27f98d85: 6 zones

* zone updates waiting on zone add blockers
* waiting to update top-level nexus_generation: some non-Nexus zone are not yet updated
Measurement updates:
Waiting on zone add/update blockers

If I uncomment out the workaround lines, we get the expected behavior:

Details

generated RNG seed: lazily-enormous-goby > # This test verifies that a new MUPdate is recognized while the system is in > # the middle of resolution of a previous MUPdate.

Initial state: a system that has never had a target release set.

load-example --nsleds 3 --ndisks-per-sled 1
loaded example system with:

• collection: d81c683a-780f-482b-a171-a0da4febc648
• blueprint: 71c65311-8e24-499b-a4ce-1ce0778a8dd0

Create a TUF repo that we can use to resolve artifacts.

tuf-assemble ../../update-common/manifests/fake.toml
error: output path repo-1.0.0.zip already exists

Simulate a first MUPdate. Collect inventory and see the planner clear it.

sled-set serial0 mupdate-override 6123eac1-ec5b-42ba-b73f-9845105a9971
set sled fee647c0-1449-4100-be9d-5cb07da803af mupdate override: unset -> 6123eac1-ec5b-42ba-b73f-9845105a9971

sled-update-install-dataset serial0 --from-repo repo-1.0.0.zip
INFO extracting uploaded archive to /var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gp/T/.tmpkCmUbW
INFO created directory to store extracted artifacts, path: /var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gp/T/update-artifacts.I6dEJl
INFO added artifact, name: fake-gimlet-sp, kind: gimlet_sp, version: 1.0.0, hash: 68465b8e3f808f475510b525cfd62086d37ddd57688bd854184fdafb2b2198a4, length: 732
INFO added artifact, name: fake-rot, kind: gimlet_rot_image_a, version: 1.0.0, hash: d11e65f934bf0de51df2e5b484f61ee72072417b43ac87f33e958008428e7b02, length: 783
INFO added artifact, name: fake-rot, kind: gimlet_rot_image_b, version: 1.0.0, hash: d11e65f934bf0de51df2e5b484f61ee72072417b43ac87f33e958008428e7b02, length: 783
INFO added artifact, name: fake-rot-bootloader, kind: gimlet_rot_bootloader, version: 1.0.0, hash: 5b0f601b1fbb8674db9c751a02f8b14f8e6d4e8470f4f7b686fecb2c49ec11f9, length: 797
INFO added artifact, name: fake-host, kind: gimlet_host_phase_1, version: 1.0.0, hash: b99d5273ba1418bebb19d74b701d716896409566d41de76ada71bded4c9b166b, length: 524288
INFO added artifact, name: fake-host, kind: cosmo_host_phase_1, version: 1.0.0, hash: 9525f567106549a3fc32df870a74803d77e51dcc44190b218e227a2c5d444f58, length: 524288
INFO added artifact, name: fake-host, kind: host_phase_2, version: 1.0.0, hash: d944ae205b61ccf4322448f7d0311a819c53d9844769de066c5307c1682abb47, length: 1048576
INFO added artifact, name: fake-trampoline, kind: gimlet_trampoline_phase_1, version: 1.0.0, hash: bcb27520ee5a56e19f6df9662c66d69ac681fbd873a97547be5f6a5ae3d250c4, length: 524288
INFO added artifact, name: fake-trampoline, kind: cosmo_trampoline_phase_1, version: 1.0.0, hash: e235b8afb58ee69d966853bd5efe7c7e904da84b9035a332b3e691dc1d5cdbd0, length: 524288
INFO added artifact, name: fake-trampoline, kind: trampoline_phase_2, version: 1.0.0, hash: a5dfcc4bc69b791f1c509df499e9e72cce844cb2b53a56d8bb357b264bdf13b6, length: 1048576
INFO added artifact, name: clickhouse, kind: zone, version: 1.0.0, hash: 52b1eb4daff6f9140491d547b11248392920230db3db0eef5f5fa5333fe9e659, length: 1686
INFO added artifact, name: clickhouse_keeper, kind: zone, version: 1.0.0, hash: cda702919449d86663be97295043aeca0ead69ae5db3bbdb20053972254a27a3, length: 1690
INFO added artifact, name: clickhouse_server, kind: zone, version: 1.0.0, hash: 5f9ae6a9821bbe8ff0bf60feddf8b167902fe5f3e2c98bd21edd1ec9d969a001, length: 1690
INFO added artifact, name: cockroachdb, kind: zone, version: 1.0.0, hash: f3a1a3c0b3469367b005ee78665d982059d5e14e93a479412426bf941c4ed291, length: 1689
INFO added artifact, name: crucible-zone, kind: zone, version: 1.0.0, hash: 6f17cf65fb5a5bec5542dd07c03cd0acc01e59130f02c532c8d848ecae810047, length: 1690
INFO added artifact, name: crucible-pantry-zone, kind: zone, version: 1.0.0, hash: 21f0ada306859c23917361f2e0b9235806c32607ec689c7e8cf16bb898bc5a02, length: 1695
INFO added artifact, name: external-dns, kind: zone, version: 1.0.0, hash: ccca13ed19b8731f9adaf0d6203b02ea3b9ede4fa426b9fac0a07ce95440046d, length: 1689
INFO added artifact, name: internal-dns, kind: zone, version: 1.0.0, hash: ffbf1373f7ee08dddd74c53ed2a94e7c4c572a982d3a9bc94000c6956b700c6a, length: 1689
INFO added artifact, name: ntp, kind: zone, version: 1.0.0, hash: 67593d686ed04a1709f93972b71f4ebc148a9362120f65d239943e814a9a7439, length: 1681
INFO added artifact, name: nexus, kind: zone, version: 1.0.0, hash: 0e32b4a3e5d3668bb1d6a16fb06b74dc60b973fa479dcee0aae3adbb52bf1388, length: 1682
INFO added artifact, name: oximeter, kind: zone, version: 1.0.0, hash: 048d8fe8cdef5b175aad714d0f148aa80ce36c9114ac15ce9d02ed3d37877a77, length: 1682
INFO added artifact, name: fake-corpus, kind: measurement_corpus, version: 1.0.0, hash: 8a0e23157bae655fceec7376926c9758efee6511c7b7ff8355bbb49545a2257f, length: 1048576
INFO added artifact, name: fake-psc-sp, kind: psc_sp, version: 1.0.0, hash: 89245fe2ac7e6a2ac8dfa4e7d6891a6e6df95e4141395c07c64026778f6d76d7, length: 721
INFO added artifact, name: fake-psc-rot, kind: psc_rot_image_a, version: 1.0.0, hash: 5d8ea834dd6d42d386f1eb8a2c5f6e99b697c9958bb4ab8edf63e56003e25d8d, length: 775
INFO added artifact, name: fake-psc-rot, kind: psc_rot_image_b, version: 1.0.0, hash: 5d8ea834dd6d42d386f1eb8a2c5f6e99b697c9958bb4ab8edf63e56003e25d8d, length: 775
INFO added artifact, name: fake-psc-rot-bootloader, kind: psc_rot_bootloader, version: 1.0.0, hash: 18c9c774bfe4bb086e869509dcccacee8476fd87670692b765aee216f2c7f003, length: 805
INFO added artifact, name: fake-switch-sp, kind: switch_sp, version: 1.0.0, hash: bf1bc1da5059f76182c3007c3049941f8898abede2f3765b106c6e7f7c42d44c, length: 740
INFO added artifact, name: fake-switch-rot, kind: switch_rot_image_a, version: 1.0.0, hash: 32307d6d75c9707e8499ba4a4d379f99c0358237b6e190ff6a8024b470f62342, length: 774
INFO added artifact, name: fake-switch-rot, kind: switch_rot_image_b, version: 1.0.0, hash: 32307d6d75c9707e8499ba4a4d379f99c0358237b6e190ff6a8024b470f62342, length: 774
INFO added artifact, name: fake-switch-rot-bootloader, kind: switch_rot_bootloader, version: 1.0.0, hash: 70836d170abd5621f95bb4225987b27b3d3dd6168e73cd60e44309bdfeb94e98, length: 804
INFO added artifact, name: installinator_document, kind: installinator_document, version: 1.0.0, hash: 16717c0caa420c7811666e720936c7ca913cd59a4d376331cc79a90544e5e2e8, length: 526
sled fee647c0-1449-4100-be9d-5cb07da803af: install dataset updated: from repo at repo-1.0.0.zip (system version 1.0.0)

inventory-generate
generated inventory collection 312212c0-4c59-41f7-bef2-492a9de424e4 from configured sleds

blueprint-plan target latest
INFO blueprint mupdate override updated to match inventory, phase: do_plan_mupdate_override, sled_id: fee647c0-1449-4100-be9d-5cb07da803af, new_bp_override: 6123eac1-ec5b-42ba-b73f-9845105a9971, prev_bp_override: None, zones:

• zone 14963808-4ef1-477f-af35-e772bb99b93e (Nexus) left unchanged, image source: install dataset
• zone 3a453750-c4d9-4222-99d0-1b45f78baaa1 (ExternalDns) left unchanged, image source: install dataset
• zone 48082366-05f4-4661-b202-67e42f0c19a6 (CruciblePantry) left unchanged, image source: install dataset
• zone 5349f547-14f9-4445-8adf-7f7dd09300ff (InternalNtp) left unchanged, image source: install dataset
• zone 8ccdb840-0913-481e-aba2-58e7df12f6f5 (InternalDns) left unchanged, image source: install dataset
• zone f49bfd40-aa71-4487-abe9-154d2cfb98c6 (Crucible) left unchanged, image source: install dataset
  , host_phase_2:
• host phase 2 slot A: current contents (unchanged)
• host phase 2 slot B: current contents (unchanged)
  , measurements: (unknown)
  INFO no previous MGS update found as part of updating blueprint mupdate override to match inventory, phase: do_plan_mupdate_override, sled_id: fee647c0-1449-4100-be9d-5cb07da803af
  INFO updating target release minimum generation based on new set-override actions, phase: do_plan_mupdate_override, current_generation: 1, new_generation: 2
  INFO skipping noop image source check for all sleds, reason: no target release is currently set
  generated blueprint 081a4db2-882f-48e5-81a3-4f202eb71ccd based on parent blueprint 71c65311-8e24-499b-a4ce-1ce0778a8dd0
  blueprint source: planner with report:
  planning report:

• zone adds and updates are blocked:

  • current target release generation (1) is lower than minimum required by blueprint (2)

  • sleds have remove mupdate override set in blueprint: fee647c0-1449-4100-be9d-5cb07da803af

  • sleds have deployment units with image sources not set to Artifact:

    • sled 2645878e-63a0-4bcd-9e55-45d117e996ca: 7 zones
    • sled b367a3e4-41f8-4028-9087-f882900abf87: 6 zones
    • sled fee647c0-1449-4100-be9d-5cb07da803af: 6 zones

  • sleds have measurements with image sources not set to Artifact:

    • sled fee647c0-1449-4100-be9d-5cb07da803af set to install dataset

• adding zones despite being blocked, because: target release generation is 1

• zone updates waiting on zone add blockers
  Measurement updates:
  Waiting on zone add/update blockers

blueprint-diff latest
from: blueprint 71c65311-8e24-499b-a4ce-1ce0778a8dd0
to: blueprint 081a4db2-882f-48e5-81a3-4f202eb71ccd

MODIFIED SLEDS:

sled fee647c0-1449-4100-be9d-5cb07da803af (active, config generation 2 -> 3):

• will remove mupdate override: (none) -> 6123eac1-ec5b-42ba-b73f-9845105a9971

  measurements:

  hash version

• unknown (unknown version)

• install dataset (no version)

  host phase 2 contents:

  slot boot image source

  A current contents
  B current contents

  physical disks:

  vendor model serial disposition

  fake-vendor fake-model serial-3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602 in service

  datasets:

  dataset name dataset id disposition quota reservation compression

  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crucible e46938a1-7171-4637-9ac2-4f091d736fa5 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/external_dns 51a04e5f-20e8-4a7b-93a1-f734ab907625 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/internal_dns 5e2b3743-6ddb-4b98-b4ee-ee66a35b5bbd in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone 69409f92-341e-46a3-8a64-a41e871c6f5b in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_crucible_f49bfd40-aa71-4487-abe9-154d2cfb98c6 84d44648-1518-4dd7-ad53-00a4f7228a1b in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_crucible_pantry_48082366-05f4-4661-b202-67e42f0c19a6 a8c01f31-3325-4a44-aeba-7c748b6680d3 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_external_dns_3a453750-c4d9-4222-99d0-1b45f78baaa1 bbf725d5-d18f-4372-81e1-b08509209c2b in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_internal_dns_8ccdb840-0913-481e-aba2-58e7df12f6f5 6f0bd291-e2dc-42f3-be71-62e5b483665f in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_nexus_14963808-4ef1-477f-af35-e772bb99b93e 37de332a-5c7f-4043-a0a9-7591a6c3b9c1 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_ntp_5349f547-14f9-4445-8adf-7f7dd09300ff fecd71b4-9fac-4f31-9893-eabf070635e6 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/debug e47ab69c-26c7-4313-af65-548a500dbb13 in service 100 GiB none gzip-9
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/local_storage 80f4a46a-3bc5-4fc7-8809-97220c21c708 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/local_storage_unencrypted 888d06f5-9137-455e-a1a6-175a2e8b6e23 in service none none off

  omicron zones:

  zone type zone id image source disposition underlay IP

  crucible f49bfd40-aa71-4487-abe9-154d2cfb98c6 install dataset in service fd00:1122:3344:101::7
  crucible_pantry 48082366-05f4-4661-b202-67e42f0c19a6 install dataset in service fd00:1122:3344:101::6
  external_dns 3a453750-c4d9-4222-99d0-1b45f78baaa1 install dataset in service fd00:1122:3344:101::5
  internal_dns 8ccdb840-0913-481e-aba2-58e7df12f6f5 install dataset in service fd00:1122:3344:3::1
  internal_ntp 5349f547-14f9-4445-8adf-7f7dd09300ff install dataset in service fd00:1122:3344:101::3
  nexus 14963808-4ef1-477f-af35-e772bb99b93e install dataset in service fd00:1122:3344:101::4

COCKROACHDB SETTINGS:
state fingerprint::::::::::::::::: (none) (unchanged)
cluster.preserve_downgrade_option: (do not modify) (unchanged)

METADATA:
internal DNS version::: 1 (unchanged)
external DNS version::: 1 (unchanged)

• target release min gen: 1 -> 2
  nexus gen:::::::::::::: 1 (unchanged)

OXIMETER SETTINGS:
generation: 1 (unchanged)
read from:: SingleNode (unchanged)

internal DNS:
DNS zone: "control-plane.oxide.internal" (unchanged)
unchanged names: 40 (records: 56)

external DNS:
DNS zone: "oxide.example" (unchanged)
unchanged names: 5 (records: 9)

Have the sled acknowledge the clearing of the MUPdate override and go through

another planner run. (We also need to have it update its sled config

generation to the latest so that the planner knows the sled has seen this

latest update.) The planner still doesn't have information needed to resolve

"install-dataset" to specific artifacts.

sled-set serial0 mupdate-override unset
set sled fee647c0-1449-4100-be9d-5cb07da803af mupdate override: 6123eac1-ec5b-42ba-b73f-9845105a9971 -> unset

sled-set serial0 omicron-config latest
set sled fee647c0-1449-4100-be9d-5cb07da803af omicron config from latest blueprint (081a4db2-882f-48e5-81a3-4f202eb71ccd)

inventory-generate
generated inventory collection b0cf8ca9-2d3d-4b98-815b-b1971e1e67b4 from configured sleds

blueprint-plan latest latest
INFO inventory override no longer exists, blueprint override cleared, phase: do_plan_mupdate_override, sled_id: fee647c0-1449-4100-be9d-5cb07da803af, prev_bp_override: 6123eac1-ec5b-42ba-b73f-9845105a9971
INFO skipping noop image source check for all sleds, reason: no target release is currently set
generated blueprint c4732e19-a1fd-433c-a7cf-719c4627f737 based on parent blueprint 081a4db2-882f-48e5-81a3-4f202eb71ccd
blueprint source: planner with report:
planning report:

• zone adds and updates are blocked:

  • current target release generation (1) is lower than minimum required by blueprint (2)

  • sleds have deployment units with image sources not set to Artifact:

    • sled 2645878e-63a0-4bcd-9e55-45d117e996ca: 7 zones
    • sled b367a3e4-41f8-4028-9087-f882900abf87: 6 zones
    • sled fee647c0-1449-4100-be9d-5cb07da803af: 6 zones

  • sleds have measurements with image sources not set to Artifact:

    • sled fee647c0-1449-4100-be9d-5cb07da803af set to install dataset

• adding zones despite being blocked, because: target release generation is 1

• zone updates waiting on zone add blockers
  Measurement updates:
  Waiting on zone add/update blockers

blueprint-diff latest
from: blueprint 081a4db2-882f-48e5-81a3-4f202eb71ccd
to: blueprint c4732e19-a1fd-433c-a7cf-719c4627f737

MODIFIED SLEDS:

sled fee647c0-1449-4100-be9d-5cb07da803af (active, config generation 3 -> 4):

• will remove mupdate override: 6123eac1-ec5b-42ba-b73f-9845105a9971 -> (none)

  measurements:

  hash version

  install dataset (no version)

  host phase 2 contents:

  slot boot image source

  A current contents
  B current contents

  physical disks:

  vendor model serial disposition

  fake-vendor fake-model serial-3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602 in service

  datasets:

  dataset name dataset id disposition quota reservation compression

  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crucible e46938a1-7171-4637-9ac2-4f091d736fa5 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/external_dns 51a04e5f-20e8-4a7b-93a1-f734ab907625 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/internal_dns 5e2b3743-6ddb-4b98-b4ee-ee66a35b5bbd in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone 69409f92-341e-46a3-8a64-a41e871c6f5b in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_crucible_f49bfd40-aa71-4487-abe9-154d2cfb98c6 84d44648-1518-4dd7-ad53-00a4f7228a1b in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_crucible_pantry_48082366-05f4-4661-b202-67e42f0c19a6 a8c01f31-3325-4a44-aeba-7c748b6680d3 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_external_dns_3a453750-c4d9-4222-99d0-1b45f78baaa1 bbf725d5-d18f-4372-81e1-b08509209c2b in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_internal_dns_8ccdb840-0913-481e-aba2-58e7df12f6f5 6f0bd291-e2dc-42f3-be71-62e5b483665f in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_nexus_14963808-4ef1-477f-af35-e772bb99b93e 37de332a-5c7f-4043-a0a9-7591a6c3b9c1 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_ntp_5349f547-14f9-4445-8adf-7f7dd09300ff fecd71b4-9fac-4f31-9893-eabf070635e6 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/debug e47ab69c-26c7-4313-af65-548a500dbb13 in service 100 GiB none gzip-9
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/local_storage 80f4a46a-3bc5-4fc7-8809-97220c21c708 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/local_storage_unencrypted 888d06f5-9137-455e-a1a6-175a2e8b6e23 in service none none off

  omicron zones:

  zone type zone id image source disposition underlay IP

  crucible f49bfd40-aa71-4487-abe9-154d2cfb98c6 install dataset in service fd00:1122:3344:101::7
  crucible_pantry 48082366-05f4-4661-b202-67e42f0c19a6 install dataset in service fd00:1122:3344:101::6
  external_dns 3a453750-c4d9-4222-99d0-1b45f78baaa1 install dataset in service fd00:1122:3344:101::5
  internal_dns 8ccdb840-0913-481e-aba2-58e7df12f6f5 install dataset in service fd00:1122:3344:3::1
  internal_ntp 5349f547-14f9-4445-8adf-7f7dd09300ff install dataset in service fd00:1122:3344:101::3
  nexus 14963808-4ef1-477f-af35-e772bb99b93e install dataset in service fd00:1122:3344:101::4

COCKROACHDB SETTINGS:
state fingerprint::::::::::::::::: (none) (unchanged)
cluster.preserve_downgrade_option: (do not modify) (unchanged)

METADATA:
internal DNS version::: 1 (unchanged)
external DNS version::: 1 (unchanged)
target release min gen: 2 (unchanged)
nexus gen:::::::::::::: 1 (unchanged)

OXIMETER SETTINGS:
generation: 1 (unchanged)
read from:: SingleNode (unchanged)

internal DNS:
DNS zone: "control-plane.oxide.internal" (unchanged)
unchanged names: 40 (records: 56)

external DNS:
DNS zone: "oxide.example" (unchanged)
unchanged names: 5 (records: 9)

The next step will be to set the target release and have the planner resolve

artifacts.

NOTE: Due to what seems like the same bug, the following two steps seem

to be necessary first. That doesn't seem right. The most recent blueprint

changed the sled config in a way that removed the MUPdate override flag.

There's no reason the sled should need to acknowledge that before the

planner can resolve artifacts on the sled.

When fixed, I think we should be able to comment out these two lines

and get the right result.

sled-set serial0 omicron-config latest
set sled fee647c0-1449-4100-be9d-5cb07da803af omicron config from latest blueprint (c4732e19-a1fd-433c-a7cf-719c4627f737)

inventory-generate
generated inventory collection 822adb0c-1e78-4efc-824e-cad87d01813d from configured sleds

Now, set the target release. This allows the planner to resolve the

artifacts.

set target-release repo-1.0.0.zip
INFO extracting uploaded archive to /var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gp/T/.tmpSU14Mz
INFO created directory to store extracted artifacts, path: /var/folders/x3/6tbrhnss6zz677z4zjsyvkqc0000gp/T/update-artifacts.ICaSnu
INFO added artifact, name: fake-gimlet-sp, kind: gimlet_sp, version: 1.0.0, hash: 68465b8e3f808f475510b525cfd62086d37ddd57688bd854184fdafb2b2198a4, length: 732
INFO added artifact, name: fake-rot, kind: gimlet_rot_image_a, version: 1.0.0, hash: d11e65f934bf0de51df2e5b484f61ee72072417b43ac87f33e958008428e7b02, length: 783
INFO added artifact, name: fake-rot, kind: gimlet_rot_image_b, version: 1.0.0, hash: d11e65f934bf0de51df2e5b484f61ee72072417b43ac87f33e958008428e7b02, length: 783
INFO added artifact, name: fake-rot-bootloader, kind: gimlet_rot_bootloader, version: 1.0.0, hash: 5b0f601b1fbb8674db9c751a02f8b14f8e6d4e8470f4f7b686fecb2c49ec11f9, length: 797
INFO added artifact, name: fake-host, kind: gimlet_host_phase_1, version: 1.0.0, hash: b99d5273ba1418bebb19d74b701d716896409566d41de76ada71bded4c9b166b, length: 524288
INFO added artifact, name: fake-host, kind: cosmo_host_phase_1, version: 1.0.0, hash: 9525f567106549a3fc32df870a74803d77e51dcc44190b218e227a2c5d444f58, length: 524288
INFO added artifact, name: fake-host, kind: host_phase_2, version: 1.0.0, hash: d944ae205b61ccf4322448f7d0311a819c53d9844769de066c5307c1682abb47, length: 1048576
INFO added artifact, name: fake-trampoline, kind: gimlet_trampoline_phase_1, version: 1.0.0, hash: bcb27520ee5a56e19f6df9662c66d69ac681fbd873a97547be5f6a5ae3d250c4, length: 524288
INFO added artifact, name: fake-trampoline, kind: cosmo_trampoline_phase_1, version: 1.0.0, hash: e235b8afb58ee69d966853bd5efe7c7e904da84b9035a332b3e691dc1d5cdbd0, length: 524288
INFO added artifact, name: fake-trampoline, kind: trampoline_phase_2, version: 1.0.0, hash: a5dfcc4bc69b791f1c509df499e9e72cce844cb2b53a56d8bb357b264bdf13b6, length: 1048576
INFO added artifact, name: clickhouse, kind: zone, version: 1.0.0, hash: 52b1eb4daff6f9140491d547b11248392920230db3db0eef5f5fa5333fe9e659, length: 1686
INFO added artifact, name: clickhouse_keeper, kind: zone, version: 1.0.0, hash: cda702919449d86663be97295043aeca0ead69ae5db3bbdb20053972254a27a3, length: 1690
INFO added artifact, name: clickhouse_server, kind: zone, version: 1.0.0, hash: 5f9ae6a9821bbe8ff0bf60feddf8b167902fe5f3e2c98bd21edd1ec9d969a001, length: 1690
INFO added artifact, name: cockroachdb, kind: zone, version: 1.0.0, hash: f3a1a3c0b3469367b005ee78665d982059d5e14e93a479412426bf941c4ed291, length: 1689
INFO added artifact, name: crucible-zone, kind: zone, version: 1.0.0, hash: 6f17cf65fb5a5bec5542dd07c03cd0acc01e59130f02c532c8d848ecae810047, length: 1690
INFO added artifact, name: crucible-pantry-zone, kind: zone, version: 1.0.0, hash: 21f0ada306859c23917361f2e0b9235806c32607ec689c7e8cf16bb898bc5a02, length: 1695
INFO added artifact, name: external-dns, kind: zone, version: 1.0.0, hash: ccca13ed19b8731f9adaf0d6203b02ea3b9ede4fa426b9fac0a07ce95440046d, length: 1689
INFO added artifact, name: internal-dns, kind: zone, version: 1.0.0, hash: ffbf1373f7ee08dddd74c53ed2a94e7c4c572a982d3a9bc94000c6956b700c6a, length: 1689
INFO added artifact, name: ntp, kind: zone, version: 1.0.0, hash: 67593d686ed04a1709f93972b71f4ebc148a9362120f65d239943e814a9a7439, length: 1681
INFO added artifact, name: nexus, kind: zone, version: 1.0.0, hash: 0e32b4a3e5d3668bb1d6a16fb06b74dc60b973fa479dcee0aae3adbb52bf1388, length: 1682
INFO added artifact, name: oximeter, kind: zone, version: 1.0.0, hash: 048d8fe8cdef5b175aad714d0f148aa80ce36c9114ac15ce9d02ed3d37877a77, length: 1682
INFO added artifact, name: fake-corpus, kind: measurement_corpus, version: 1.0.0, hash: 8a0e23157bae655fceec7376926c9758efee6511c7b7ff8355bbb49545a2257f, length: 1048576
INFO added artifact, name: fake-psc-sp, kind: psc_sp, version: 1.0.0, hash: 89245fe2ac7e6a2ac8dfa4e7d6891a6e6df95e4141395c07c64026778f6d76d7, length: 721
INFO added artifact, name: fake-psc-rot, kind: psc_rot_image_a, version: 1.0.0, hash: 5d8ea834dd6d42d386f1eb8a2c5f6e99b697c9958bb4ab8edf63e56003e25d8d, length: 775
INFO added artifact, name: fake-psc-rot, kind: psc_rot_image_b, version: 1.0.0, hash: 5d8ea834dd6d42d386f1eb8a2c5f6e99b697c9958bb4ab8edf63e56003e25d8d, length: 775
INFO added artifact, name: fake-psc-rot-bootloader, kind: psc_rot_bootloader, version: 1.0.0, hash: 18c9c774bfe4bb086e869509dcccacee8476fd87670692b765aee216f2c7f003, length: 805
INFO added artifact, name: fake-switch-sp, kind: switch_sp, version: 1.0.0, hash: bf1bc1da5059f76182c3007c3049941f8898abede2f3765b106c6e7f7c42d44c, length: 740
INFO added artifact, name: fake-switch-rot, kind: switch_rot_image_a, version: 1.0.0, hash: 32307d6d75c9707e8499ba4a4d379f99c0358237b6e190ff6a8024b470f62342, length: 774
INFO added artifact, name: fake-switch-rot, kind: switch_rot_image_b, version: 1.0.0, hash: 32307d6d75c9707e8499ba4a4d379f99c0358237b6e190ff6a8024b470f62342, length: 774
INFO added artifact, name: fake-switch-rot-bootloader, kind: switch_rot_bootloader, version: 1.0.0, hash: 70836d170abd5621f95bb4225987b27b3d3dd6168e73cd60e44309bdfeb94e98, length: 804
INFO added artifact, name: installinator_document, kind: installinator_document, version: 1.0.0, hash: 16717c0caa420c7811666e720936c7ca913cd59a4d376331cc79a90544e5e2e8, length: 526
set target release based on repo-1.0.0.zip

blueprint-plan latest latest
INFO performed noop zone image source checks on sled, sled_id: 2645878e-63a0-4bcd-9e55-45d117e996ca, num_total: 7, num_already_artifact: 0, num_eligible: 0, num_ineligible: 7
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 2645878e-63a0-4bcd-9e55-45d117e996ca, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 2645878e-63a0-4bcd-9e55-45d117e996ca, slot: b, expected_hash: 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b
INFO current sled measurements are in an unknown state, sled_id: 2645878e-63a0-4bcd-9e55-45d117e996ca
INFO performed noop zone image source checks on sled, sled_id: b367a3e4-41f8-4028-9087-f882900abf87, num_total: 6, num_already_artifact: 0, num_eligible: 0, num_ineligible: 6
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: b367a3e4-41f8-4028-9087-f882900abf87, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: b367a3e4-41f8-4028-9087-f882900abf87, slot: b, expected_hash: 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b
INFO current sled measurements are in an unknown state, sled_id: b367a3e4-41f8-4028-9087-f882900abf87
INFO performed noop zone image source checks on sled, sled_id: fee647c0-1449-4100-be9d-5cb07da803af, num_total: 6, num_already_artifact: 0, num_eligible: 6, num_ineligible: 0
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: fee647c0-1449-4100-be9d-5cb07da803af, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: fee647c0-1449-4100-be9d-5cb07da803af, slot: b, expected_hash: 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b
INFO measurement is eligible for noop image source conversion, sled_id: fee647c0-1449-4100-be9d-5cb07da803af, new_measurement_source: 8a0e23157bae655fceec7376926c9758efee6511c7b7ff8355bbb49545a2257f version 1.0.0

generated blueprint c83a5c95-f5d0-486b-b72a-93b5a8a0bab6 based on parent blueprint c4732e19-a1fd-433c-a7cf-719c4627f737
blueprint source: planner with report:
planning report:

• noop converting 6/6 install-dataset zones to artifact store on sled fee647c0-1449-4100-be9d-5cb07da803af

• noop converting current measurements to Artifact on sled fee647c0-1449-4100-be9d-5cb07da803af

• zone adds waiting on blockers

• zone adds and updates are blocked:

  • sleds have deployment units with image sources not set to Artifact:
    • sled 2645878e-63a0-4bcd-9e55-45d117e996ca: 7 zones
    • sled b367a3e4-41f8-4028-9087-f882900abf87: 6 zones

• zone updates waiting on zone add blockers

• waiting to update top-level nexus_generation: some non-Nexus zone are not yet updated
  Measurement updates:
  Waiting on zone add/update blockers

blueprint-diff latest
from: blueprint c4732e19-a1fd-433c-a7cf-719c4627f737
to: blueprint c83a5c95-f5d0-486b-b72a-93b5a8a0bab6

MODIFIED SLEDS:

sled fee647c0-1449-4100-be9d-5cb07da803af (active, config generation 4 -> 5):

measurements:
--------------------------------------------------------------------------------
hash                                                               version      
--------------------------------------------------------------------------------

• install dataset (no version)

• 8a0e23157bae655fceec7376926c9758efee6511c7b7ff8355bbb49545a2257f version 1.0.0

  host phase 2 contents:

  slot boot image source

  A current contents
  B current contents

  physical disks:

  vendor model serial disposition

  fake-vendor fake-model serial-3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602 in service

  datasets:

  dataset name dataset id disposition quota reservation compression

  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crucible e46938a1-7171-4637-9ac2-4f091d736fa5 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/external_dns 51a04e5f-20e8-4a7b-93a1-f734ab907625 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/internal_dns 5e2b3743-6ddb-4b98-b4ee-ee66a35b5bbd in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone 69409f92-341e-46a3-8a64-a41e871c6f5b in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_crucible_f49bfd40-aa71-4487-abe9-154d2cfb98c6 84d44648-1518-4dd7-ad53-00a4f7228a1b in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_crucible_pantry_48082366-05f4-4661-b202-67e42f0c19a6 a8c01f31-3325-4a44-aeba-7c748b6680d3 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_external_dns_3a453750-c4d9-4222-99d0-1b45f78baaa1 bbf725d5-d18f-4372-81e1-b08509209c2b in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_internal_dns_8ccdb840-0913-481e-aba2-58e7df12f6f5 6f0bd291-e2dc-42f3-be71-62e5b483665f in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_nexus_14963808-4ef1-477f-af35-e772bb99b93e 37de332a-5c7f-4043-a0a9-7591a6c3b9c1 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_ntp_5349f547-14f9-4445-8adf-7f7dd09300ff fecd71b4-9fac-4f31-9893-eabf070635e6 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/debug e47ab69c-26c7-4313-af65-548a500dbb13 in service 100 GiB none gzip-9
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/local_storage 80f4a46a-3bc5-4fc7-8809-97220c21c708 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/local_storage_unencrypted 888d06f5-9137-455e-a1a6-175a2e8b6e23 in service none none off

  omicron zones:

  zone type zone id image source disposition underlay IP

• crucible f49bfd40-aa71-4487-abe9-154d2cfb98c6 - install dataset in service fd00:1122:3344:101::7
  └─ + artifact: version 1.0.0
• crucible_pantry 48082366-05f4-4661-b202-67e42f0c19a6 - install dataset in service fd00:1122:3344:101::6
  └─ + artifact: version 1.0.0
• external_dns 3a453750-c4d9-4222-99d0-1b45f78baaa1 - install dataset in service fd00:1122:3344:101::5
  └─ + artifact: version 1.0.0
• internal_dns 8ccdb840-0913-481e-aba2-58e7df12f6f5 - install dataset in service fd00:1122:3344:3::1
  └─ + artifact: version 1.0.0
• internal_ntp 5349f547-14f9-4445-8adf-7f7dd09300ff - install dataset in service fd00:1122:3344:101::3
  └─ + artifact: version 1.0.0
• nexus 14963808-4ef1-477f-af35-e772bb99b93e - install dataset in service fd00:1122:3344:101::4
  └─ + artifact: version 1.0.0

COCKROACHDB SETTINGS:
state fingerprint::::::::::::::::: (none) (unchanged)
cluster.preserve_downgrade_option: (do not modify) (unchanged)

METADATA:
internal DNS version::: 1 (unchanged)
external DNS version::: 1 (unchanged)
target release min gen: 2 (unchanged)
nexus gen:::::::::::::: 1 (unchanged)

OXIMETER SETTINGS:
generation: 1 (unchanged)
read from:: SingleNode (unchanged)

internal DNS:
DNS zone: "control-plane.oxide.internal" (unchanged)
unchanged names: 40 (records: 56)

external DNS:
DNS zone: "oxide.example" (unchanged)
unchanged names: 5 (records: 9)

NOTE: if you uncomment these lines, that works around the bug.

The bug is that the planner won't make some changes related to MUPdate resolution

until the generation reported by the sled's inventory matches what's in the parent

blueprint. But because of the MUPdate, that can't happen: Nexus needs to first

process the mupdate_override reported by the sled. It needs to do this even

if the sled's inventory isn't up-to-date with the parent blueprint.

sled-set serial0 omicron-config latest
set sled fee647c0-1449-4100-be9d-5cb07da803af omicron config from latest blueprint (c83a5c95-f5d0-486b-b72a-93b5a8a0bab6)

inventory-generate
generated inventory collection 65e9fd1c-2b4d-49c1-ac26-3aac1efe88e4 from configured sleds

But suppose that the sled gets MUPdated again before it receives that request.

The planner needs to process that as a new MUPdate, generating a new blueprint

with the MUPdate override set and all the artifacts back at "install-dataset".

sled-set serial0 mupdate-override 306c7aee-0d06-4fc1-baeb-c18af4f78757
set sled fee647c0-1449-4100-be9d-5cb07da803af mupdate override: unset -> 306c7aee-0d06-4fc1-baeb-c18af4f78757

inventory-generate
generated inventory collection 0272221c-9dfb-4566-8d32-0cea20a9ab69 from configured sleds

blueprint-plan latest latest
INFO blueprint mupdate override updated to match inventory, phase: do_plan_mupdate_override, sled_id: fee647c0-1449-4100-be9d-5cb07da803af, new_bp_override: 306c7aee-0d06-4fc1-baeb-c18af4f78757, prev_bp_override: None, zones:

• zone 14963808-4ef1-477f-af35-e772bb99b93e (Nexus) updated from artifact: version 1.0.0 to install dataset
• zone 3a453750-c4d9-4222-99d0-1b45f78baaa1 (ExternalDns) updated from artifact: version 1.0.0 to install dataset
• zone 48082366-05f4-4661-b202-67e42f0c19a6 (CruciblePantry) updated from artifact: version 1.0.0 to install dataset
• zone 5349f547-14f9-4445-8adf-7f7dd09300ff (InternalNtp) updated from artifact: version 1.0.0 to install dataset
• zone 8ccdb840-0913-481e-aba2-58e7df12f6f5 (InternalDns) updated from artifact: version 1.0.0 to install dataset
• zone f49bfd40-aa71-4487-abe9-154d2cfb98c6 (Crucible) updated from artifact: version 1.0.0 to install dataset
  , host_phase_2:
• host phase 2 slot A: current contents (unchanged)
• host phase 2 slot B: current contents (unchanged)
  , measurements:
• 8a0e23157bae655fceec7376926c9758efee6511c7b7ff8355bbb49545a2257f version 1.0.0

INFO no previous MGS update found as part of updating blueprint mupdate override to match inventory, phase: do_plan_mupdate_override, sled_id: fee647c0-1449-4100-be9d-5cb07da803af
INFO updating target release minimum generation based on new set-override actions, phase: do_plan_mupdate_override, current_generation: 2, new_generation: 3
INFO performed noop zone image source checks on sled, sled_id: 2645878e-63a0-4bcd-9e55-45d117e996ca, num_total: 7, num_already_artifact: 0, num_eligible: 0, num_ineligible: 7
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 2645878e-63a0-4bcd-9e55-45d117e996ca, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 2645878e-63a0-4bcd-9e55-45d117e996ca, slot: b, expected_hash: 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b
INFO current sled measurements are in an unknown state, sled_id: 2645878e-63a0-4bcd-9e55-45d117e996ca
INFO performed noop zone image source checks on sled, sled_id: b367a3e4-41f8-4028-9087-f882900abf87, num_total: 6, num_already_artifact: 0, num_eligible: 0, num_ineligible: 6
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: b367a3e4-41f8-4028-9087-f882900abf87, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: b367a3e4-41f8-4028-9087-f882900abf87, slot: b, expected_hash: 0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b
INFO current sled measurements are in an unknown state, sled_id: b367a3e4-41f8-4028-9087-f882900abf87
INFO skipped noop image source check on sled, sled_id: fee647c0-1449-4100-be9d-5cb07da803af, reason: remove_mupdate_override is set in the blueprint (306c7aee-0d06-4fc1-baeb-c18af4f78757)
generated blueprint 4bdd2ed8-e6f3-4d07-8225-e7e1914f6504 based on parent blueprint c83a5c95-f5d0-486b-b72a-93b5a8a0bab6
blueprint source: planner with report:
planning report:

• zone adds waiting on blockers

• zone adds and updates are blocked:

  • current target release generation (2) is lower than minimum required by blueprint (3)

  • sleds have remove mupdate override set in blueprint: fee647c0-1449-4100-be9d-5cb07da803af

  • sleds have deployment units with image sources not set to Artifact:

    • sled 2645878e-63a0-4bcd-9e55-45d117e996ca: 7 zones
    • sled b367a3e4-41f8-4028-9087-f882900abf87: 6 zones
    • sled fee647c0-1449-4100-be9d-5cb07da803af: 6 zones

  • sleds have measurements with image sources not set to Artifact:

    • sled fee647c0-1449-4100-be9d-5cb07da803af set to install dataset

• zone updates waiting on zone add blockers

• waiting to update top-level nexus_generation: some non-Nexus zone are not yet updated
  Measurement updates:
  Waiting on zone add/update blockers

blueprint-diff latest
from: blueprint c83a5c95-f5d0-486b-b72a-93b5a8a0bab6
to: blueprint 4bdd2ed8-e6f3-4d07-8225-e7e1914f6504

MODIFIED SLEDS:

sled fee647c0-1449-4100-be9d-5cb07da803af (active, config generation 5 -> 6):

• will remove mupdate override: (none) -> 306c7aee-0d06-4fc1-baeb-c18af4f78757

  measurements:

  hash version

• 8a0e23157bae655fceec7376926c9758efee6511c7b7ff8355bbb49545a2257f version 1.0.0

• install dataset (no version)

  host phase 2 contents:

  slot boot image source

  A current contents
  B current contents

  physical disks:

  vendor model serial disposition

  fake-vendor fake-model serial-3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602 in service

  datasets:

  dataset name dataset id disposition quota reservation compression

  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crucible e46938a1-7171-4637-9ac2-4f091d736fa5 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/external_dns 51a04e5f-20e8-4a7b-93a1-f734ab907625 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/internal_dns 5e2b3743-6ddb-4b98-b4ee-ee66a35b5bbd in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone 69409f92-341e-46a3-8a64-a41e871c6f5b in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_crucible_f49bfd40-aa71-4487-abe9-154d2cfb98c6 84d44648-1518-4dd7-ad53-00a4f7228a1b in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_crucible_pantry_48082366-05f4-4661-b202-67e42f0c19a6 a8c01f31-3325-4a44-aeba-7c748b6680d3 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_external_dns_3a453750-c4d9-4222-99d0-1b45f78baaa1 bbf725d5-d18f-4372-81e1-b08509209c2b in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_internal_dns_8ccdb840-0913-481e-aba2-58e7df12f6f5 6f0bd291-e2dc-42f3-be71-62e5b483665f in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_nexus_14963808-4ef1-477f-af35-e772bb99b93e 37de332a-5c7f-4043-a0a9-7591a6c3b9c1 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_ntp_5349f547-14f9-4445-8adf-7f7dd09300ff fecd71b4-9fac-4f31-9893-eabf070635e6 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/debug e47ab69c-26c7-4313-af65-548a500dbb13 in service 100 GiB none gzip-9
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/local_storage 80f4a46a-3bc5-4fc7-8809-97220c21c708 in service none none off
  oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/local_storage_unencrypted 888d06f5-9137-455e-a1a6-175a2e8b6e23 in service none none off

  omicron zones:

  zone type zone id image source disposition underlay IP

• crucible f49bfd40-aa71-4487-abe9-154d2cfb98c6 - artifact: version 1.0.0 in service fd00:1122:3344:101::7
  └─ + install dataset
• crucible_pantry 48082366-05f4-4661-b202-67e42f0c19a6 - artifact: version 1.0.0 in service fd00:1122:3344:101::6
  └─ + install dataset
• external_dns 3a453750-c4d9-4222-99d0-1b45f78baaa1 - artifact: version 1.0.0 in service fd00:1122:3344:101::5
  └─ + install dataset
• internal_dns 8ccdb840-0913-481e-aba2-58e7df12f6f5 - artifact: version 1.0.0 in service fd00:1122:3344:3::1
  └─ + install dataset
• internal_ntp 5349f547-14f9-4445-8adf-7f7dd09300ff - artifact: version 1.0.0 in service fd00:1122:3344:101::3
  └─ + install dataset
• nexus 14963808-4ef1-477f-af35-e772bb99b93e - artifact: version 1.0.0 in service fd00:1122:3344:101::4
  └─ + install dataset

COCKROACHDB SETTINGS:
state fingerprint::::::::::::::::: (none) (unchanged)
cluster.preserve_downgrade_option: (do not modify) (unchanged)

METADATA:
internal DNS version::: 1 (unchanged)
external DNS version::: 1 (unchanged)

• target release min gen: 2 -> 3
  nexus gen:::::::::::::: 1 (unchanged)

OXIMETER SETTINGS:
generation: 1 (unchanged)
read from:: SingleNode (unchanged)

internal DNS:
DNS zone: "control-plane.oxide.internal" (unchanged)
unchanged names: 40 (records: 56)

external DNS:
DNS zone: "oxide.example" (unchanged)
unchanged names: 5 (records: 9)

blueprint-show latest
blueprint 4bdd2ed8-e6f3-4d07-8225-e7e1914f6504
parent: c83a5c95-f5d0-486b-b72a-93b5a8a0bab6

sled: 2645878e-63a0-4bcd-9e55-45d117e996ca
state::::::::::::: active
config generation: 2
subnet:::::::::::: fd00:1122:3344:102::/64
last allocated IP: fd00:1122:3344:102::8

host phase 2 contents:
------------------------
slot   boot image source
------------------------
A      current contents 
B      current contents 


measurements:
---------------------------
hash      version          
---------------------------
unknown   (unknown version)


physical disks:
------------------------------------------------------------------------------------
vendor        model        serial                                        disposition
------------------------------------------------------------------------------------
fake-vendor   fake-model   serial-ded12c6d-44ee-49b2-be70-fd3cf16903cc   in service 


datasets:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dataset name                                                                                                   dataset id                             disposition   quota     reservation   compression
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crucible                                                              5c2b063e-df37-4c25-b626-3e110958d261   in service    none      none          off        
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crypt/clickhouse                                                      b6433525-6eea-46fd-a9e3-781e1697f668   in service    none      none          off        
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crypt/external_dns                                                    5bccd14b-92c0-4966-b1bd-2cf6a30b7766   in service    none      none          off        
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crypt/internal_dns                                                    77e09868-0a1c-4c27-a090-93f2c519e1e8   in service    none      none          off        
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crypt/zone                                                            d92f0b8d-a00e-4b83-9d02-2ca525fb9641   in service    none      none          off        
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crypt/zone/oxz_clickhouse_9e502a39-4c54-424e-be0e-131c649ce8c5        1169e922-b559-4a5c-b434-3a14a1abe33d   in service    none      none          off        
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crypt/zone/oxz_crucible_7d45f929-c5e8-4d0b-ab7c-844a5679ddf4          369dfff3-ead1-43c1-84db-29ea018d2c43   in service    none      none          off        
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crypt/zone/oxz_crucible_pantry_9aa00b57-52f0-4ebe-87cc-2eb647757f33   d209e26f-2864-45f3-a3d2-14dce47d22ca   in service    none      none          off        
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crypt/zone/oxz_external_dns_4a4b86a0-bbb5-4c35-9f12-285503d9dad3      a0bb9583-dcdb-4830-9da6-4a6352b9564b   in service    none      none          off        
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crypt/zone/oxz_internal_dns_1361aca9-0b41-4de1-bbfe-f1b024cc54ca      e456b969-47ff-43bc-81be-168e7148979b   in service    none      none          off        
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crypt/zone/oxz_nexus_4fcfb0b0-c269-4db8-b820-f0c3c1c302fe             aba6f56e-a610-4a8d-a00c-26cbd90d03fd   in service    none      none          off        
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crypt/zone/oxz_ntp_8127a691-d2f8-4e4d-91f2-b032aa4be46b               086a1703-c8ba-4410-9a1f-11576c652031   in service    none      none          off        
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crypt/debug                                                           fb0a5111-4349-4097-a410-210f7df6857f   in service    100 GiB   none          gzip-9     
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/crypt/local_storage                                                   0dafc355-6775-4286-b55c-3f2d003cc13d   in service    none      none          off        
oxp_ded12c6d-44ee-49b2-be70-fd3cf16903cc/local_storage_unencrypted                                             9b25a939-b8f8-46f9-a3da-d8fc4ac672c4   in service    none      none          off        


omicron zones:
--------------------------------------------------------------------------------------------------------------
zone type         zone id                                image source      disposition   underlay IP          
--------------------------------------------------------------------------------------------------------------
clickhouse        9e502a39-4c54-424e-be0e-131c649ce8c5   install dataset   in service    fd00:1122:3344:102::5
crucible          7d45f929-c5e8-4d0b-ab7c-844a5679ddf4   install dataset   in service    fd00:1122:3344:102::8
crucible_pantry   9aa00b57-52f0-4ebe-87cc-2eb647757f33   install dataset   in service    fd00:1122:3344:102::7
external_dns      4a4b86a0-bbb5-4c35-9f12-285503d9dad3   install dataset   in service    fd00:1122:3344:102::6
internal_dns      1361aca9-0b41-4de1-bbfe-f1b024cc54ca   install dataset   in service    fd00:1122:3344:1::1  
internal_ntp      8127a691-d2f8-4e4d-91f2-b032aa4be46b   install dataset   in service    fd00:1122:3344:102::3
nexus             4fcfb0b0-c269-4db8-b820-f0c3c1c302fe   install dataset   in service    fd00:1122:3344:102::4

sled: b367a3e4-41f8-4028-9087-f882900abf87
state::::::::::::: active
config generation: 2
subnet:::::::::::: fd00:1122:3344:103::/64
last allocated IP: fd00:1122:3344:103::7

host phase 2 contents:
------------------------
slot   boot image source
------------------------
A      current contents 
B      current contents 


measurements:
---------------------------
hash      version          
---------------------------
unknown   (unknown version)


physical disks:
------------------------------------------------------------------------------------
vendor        model        serial                                        disposition
------------------------------------------------------------------------------------
fake-vendor   fake-model   serial-8d9ef3ce-22de-4412-ac5c-720057ce8a86   in service 


datasets:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dataset name                                                                                                   dataset id                             disposition   quota     reservation   compression
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
oxp_8d9ef3ce-22de-4412-ac5c-720057ce8a86/crucible                                                              79a6be8b-384f-4786-8af7-bf4dd4f1950a   in service    none      none          off        
oxp_8d9ef3ce-22de-4412-ac5c-720057ce8a86/crypt/external_dns                                                    d38b3838-8e15-48b7-ae1d-f433ca54d32c   in service    none      none          off        
oxp_8d9ef3ce-22de-4412-ac5c-720057ce8a86/crypt/internal_dns                                                    0e468ff8-af6b-4c2f-ab0f-3d12556e821f   in service    none      none          off        
oxp_8d9ef3ce-22de-4412-ac5c-720057ce8a86/crypt/zone                                                            f52b057f-6bc3-4bc0-af8b-1f1fe0496614   in service    none      none          off        
oxp_8d9ef3ce-22de-4412-ac5c-720057ce8a86/crypt/zone/oxz_crucible_6e810b73-cc9e-47cc-9baf-e6662ab9ebee          6248b81f-8bc9-4b50-bcfc-39215629058a   in service    none      none          off        
oxp_8d9ef3ce-22de-4412-ac5c-720057ce8a86/crypt/zone/oxz_crucible_pantry_214aa715-d5b9-4b8c-bf1f-649da4d6a655   d4fbfcd1-cc2a-48af-b6e5-5a7e7f0dc335   in service    none      none          off        
oxp_8d9ef3ce-22de-4412-ac5c-720057ce8a86/crypt/zone/oxz_external_dns_101c2e4f-fe46-4f5f-9a03-df02738d48d6      578e8017-e191-4ecd-ae2c-91b5d333cd1a   in service    none      none          off        
oxp_8d9ef3ce-22de-4412-ac5c-720057ce8a86/crypt/zone/oxz_internal_dns_944efe56-f4c1-498f-9a86-6119683719c7      fec35add-cbdf-4311-91ea-09adc24fa53f   in service    none      none          off        
oxp_8d9ef3ce-22de-4412-ac5c-720057ce8a86/crypt/zone/oxz_nexus_70ca149a-fbb4-4612-9327-4e12d956c48f             03492cf4-aaa1-4478-a4eb-b7d6f38d9d71   in service    none      none          off        
oxp_8d9ef3ce-22de-4412-ac5c-720057ce8a86/crypt/zone/oxz_ntp_c20b712e-eecb-4d29-95c4-297660f7fdcd               52a8256a-ce3f-40f5-b8ab-9b09021c301c   in service    none      none          off        
oxp_8d9ef3ce-22de-4412-ac5c-720057ce8a86/crypt/debug                                                           81f8618f-cdbf-4487-a663-f48f0c5f2d03   in service    100 GiB   none          gzip-9     
oxp_8d9ef3ce-22de-4412-ac5c-720057ce8a86/crypt/local_storage                                                   50a18a4c-2c72-46b4-a7db-e160382e2a8b   in service    none      none          off        
oxp_8d9ef3ce-22de-4412-ac5c-720057ce8a86/local_storage_unencrypted                                             aaafb5ad-ee1a-4c1d-b1d4-f09fee3cfc56   in service    none      none          off        


omicron zones:
--------------------------------------------------------------------------------------------------------------
zone type         zone id                                image source      disposition   underlay IP          
--------------------------------------------------------------------------------------------------------------
crucible          6e810b73-cc9e-47cc-9baf-e6662ab9ebee   install dataset   in service    fd00:1122:3344:103::7
crucible_pantry   214aa715-d5b9-4b8c-bf1f-649da4d6a655   install dataset   in service    fd00:1122:3344:103::6
external_dns      101c2e4f-fe46-4f5f-9a03-df02738d48d6   install dataset   in service    fd00:1122:3344:103::5
internal_dns      944efe56-f4c1-498f-9a86-6119683719c7   install dataset   in service    fd00:1122:3344:2::1  
internal_ntp      c20b712e-eecb-4d29-95c4-297660f7fdcd   install dataset   in service    fd00:1122:3344:103::3
nexus             70ca149a-fbb4-4612-9327-4e12d956c48f   install dataset   in service    fd00:1122:3344:103::4

sled: fee647c0-1449-4100-be9d-5cb07da803af
state:::::::::::::::::::::::: active
config generation:::::::::::: 6
subnet::::::::::::::::::::::: fd00:1122:3344:101::/64
last allocated IP:::::::::::: fd00:1122:3344:101::7
will remove mupdate override: 306c7aee-0d06-4fc1-baeb-c18af4f78757

host phase 2 contents:
------------------------
slot   boot image source
------------------------
A      current contents 
B      current contents 


measurements:
------------------------------
hash              version     
------------------------------
install dataset   (no version)


physical disks:
------------------------------------------------------------------------------------
vendor        model        serial                                        disposition
------------------------------------------------------------------------------------
fake-vendor   fake-model   serial-3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602   in service 


datasets:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dataset name                                                                                                   dataset id                             disposition   quota     reservation   compression
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crucible                                                              e46938a1-7171-4637-9ac2-4f091d736fa5   in service    none      none          off        
oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/external_dns                                                    51a04e5f-20e8-4a7b-93a1-f734ab907625   in service    none      none          off        
oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/internal_dns                                                    5e2b3743-6ddb-4b98-b4ee-ee66a35b5bbd   in service    none      none          off        
oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone                                                            69409f92-341e-46a3-8a64-a41e871c6f5b   in service    none      none          off        
oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_crucible_f49bfd40-aa71-4487-abe9-154d2cfb98c6          84d44648-1518-4dd7-ad53-00a4f7228a1b   in service    none      none          off        
oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_crucible_pantry_48082366-05f4-4661-b202-67e42f0c19a6   a8c01f31-3325-4a44-aeba-7c748b6680d3   in service    none      none          off        
oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_external_dns_3a453750-c4d9-4222-99d0-1b45f78baaa1      bbf725d5-d18f-4372-81e1-b08509209c2b   in service    none      none          off        
oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_internal_dns_8ccdb840-0913-481e-aba2-58e7df12f6f5      6f0bd291-e2dc-42f3-be71-62e5b483665f   in service    none      none          off        
oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_nexus_14963808-4ef1-477f-af35-e772bb99b93e             37de332a-5c7f-4043-a0a9-7591a6c3b9c1   in service    none      none          off        
oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/zone/oxz_ntp_5349f547-14f9-4445-8adf-7f7dd09300ff               fecd71b4-9fac-4f31-9893-eabf070635e6   in service    none      none          off        
oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/debug                                                           e47ab69c-26c7-4313-af65-548a500dbb13   in service    100 GiB   none          gzip-9     
oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/crypt/local_storage                                                   80f4a46a-3bc5-4fc7-8809-97220c21c708   in service    none      none          off        
oxp_3dd1c2dd-a89d-4ee7-bb3d-4d99b8268602/local_storage_unencrypted                                             888d06f5-9137-455e-a1a6-175a2e8b6e23   in service    none      none          off        


omicron zones:
--------------------------------------------------------------------------------------------------------------
zone type         zone id                                image source      disposition   underlay IP          
--------------------------------------------------------------------------------------------------------------
crucible          f49bfd40-aa71-4487-abe9-154d2cfb98c6   install dataset   in service    fd00:1122:3344:101::7
crucible_pantry   48082366-05f4-4661-b202-67e42f0c19a6   install dataset   in service    fd00:1122:3344:101::6
external_dns      3a453750-c4d9-4222-99d0-1b45f78baaa1   install dataset   in service    fd00:1122:3344:101::5
internal_dns      8ccdb840-0913-481e-aba2-58e7df12f6f5   install dataset   in service    fd00:1122:3344:3::1  
internal_ntp      5349f547-14f9-4445-8adf-7f7dd09300ff   install dataset   in service    fd00:1122:3344:101::3
nexus             14963808-4ef1-477f-af35-e772bb99b93e   install dataset   in service    fd00:1122:3344:101::4

COCKROACHDB SETTINGS:
state fingerprint::::::::::::::::: (none)
cluster.preserve_downgrade_option: (do not modify)

OXIMETER SETTINGS:
generation: 1
read from:: SingleNode

METADATA:
created by:::::::::::::: reconfigurator-sim
created at:::::::::::::: 2026-07-11T16:58:18.670Z
comment::::::::::::::::: updated target release minimum generation from 2 to 3
internal DNS version:::: 1
external DNS version:::: 1
target release min gen:: 3
nexus gen::::::::::::::: 1
external networking gen: 2

PENDING MGS-MANAGED UPDATES: 0

blueprint source: planner with report:
planning report:

• zone adds waiting on blockers

• zone adds and updates are blocked:

  • current target release generation (2) is lower than minimum required by blueprint (3)

  • sleds have remove mupdate override set in blueprint: fee647c0-1449-4100-be9d-5cb07da803af

  • sleds have deployment units with image sources not set to Artifact:

    • sled 2645878e-63a0-4bcd-9e55-45d117e996ca: 7 zones
    • sled b367a3e4-41f8-4028-9087-f882900abf87: 6 zones
    • sled fee647c0-1449-4100-be9d-5cb07da803af: 6 zones

  • sleds have measurements with image sources not set to Artifact:

    • sled fee647c0-1449-4100-be9d-5cb07da803af set to install dataset

• zone updates waiting on zone add blockers

• waiting to update top-level nexus_generation: some non-Nexus zone are not yet updated
  Measurement updates:
  Waiting on zone add/update blockers