Zpool not marked as deleted in database after disk expungement

[askfongjojo]

I was trying out the new omdb db crucible-dataset command on rack2 but kept hitting this error:

root@oxz_switch0:~# omdb db zpool list
Error: zpool 3af01cc4-1f16-47d9-a489-abafcb91c2db not found in inventory!

This zpool corresponds to an expunged physical disk:

root@[fd00:1122:3344:105::3]:32221/omicron> select * from zpool where id = '3af01cc4-1f16-47d9-a489-abafcb91c2db';
                   id                  |         time_created          |         time_modified         | time_deleted | rcgen |               sled_id                |           physical_disk_id           | control_plane_storage_buffer
---------------------------------------+-------------------------------+-------------------------------+--------------+-------+--------------------------------------+--------------------------------------+-------------------------------
  3af01cc4-1f16-47d9-a489-abafcb91c2db | 2023-08-30 18:57:52.826594+00 | 2024-03-27 21:18:20.842573+00 | NULL         |     4 | 0c7011f7-a4bf-4daf-90cc-1c2410103300 | 0bffef3c-56b1-4b01-b897-161bbf766bea |                 268435456000
(1 row)

root@[fd00:1122:3344:105::3]:32221/omicron> select * from physical_disk where id = '0bffef3c-56b1-4b01-b897-161bbf766bea';
                   id                  |         time_created          |         time_modified         | time_deleted | rcgen | vendor |  serial  |      model      | variant |               sled_id                | disk_policy |   disk_state
---------------------------------------+-------------------------------+-------------------------------+--------------+-------+--------+----------+-----------------+---------+--------------------------------------+-------------+-----------------
  0bffef3c-56b1-4b01-b897-161bbf766bea | 2023-08-30 18:57:23.655377+00 | 2024-03-27 21:18:20.906665+00 | NULL         |     1 | 1b96   | A084A5EE | WUS4C6432DSP3X3 | u2      | 0c7011f7-a4bf-4daf-90cc-1c2410103300 | expunged    | decommissioned
(1 row)

We can probably make omdb more forgiving but the proper fix may be cleaning up the orphan record for a zpool that is permanently gone from the sled.