Skip to content

image: keep libtopo dependencies in the recovery image - #274

Merged
smklein merged 2 commits into
masterfrom
recovery-keep-topo-libs
Nov 2, 2026
Merged

smklein merged 2 commits into
masterfrom
recovery-keep-topo-libs

Conversation

@smklein

@smklein smklein commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #273.

The recovery image removes six shared objects that libtopo and the topo
plugins loaded on Oxide hardware depend on. With omicron's in-progress
change to read disk locations from libtopo, installinator in the recovery
image fails its first snapshot ("empty topology") and is then killed by
ld.so.1 on a later one (librename.so.1: open failed, referenced by
libipcc_inv_persist). This stops elide-ing them from recovery-elide-v3.json.

Library Size Needed by Without it, in the recovery image
librename.so.1 19.7 KiB libipcc (inventory cache write, via oxhc); libdladm (libtopo → libuuid → libdlpi chain) Process killed (librename_atomic_init); libuuid's dlpi_walk binding fails
libfmd_agent.so.1 24.5 KiB pciebus.so, zen.so Both fail to load; hc tree drops from 1226 to 409 nodes
libdiskstatus.so.1 99.1 KiB disk.so Fails to load; NVMe disk nodes missing
libjedec.so.1 309.0 KiB dimm.so Fails to load; DIMM nodes missing
libipmi.so.1 99.4 KiB libtopo (ipmi_open) The first plugin dlopen (pciebus) fails
libidspace.so.1 13.5 KiB libvarpd (libtopo → libuuid → libdlpi chain) The first plugin dlopen (pciebus) fails

Total is about 565 KiB, compared with a 149 MiB recovery zfs.img.

How the set was chosen (to be clear: by claude)

  • Closure: ldd -Lr over libtopo and every 64-bit topo plugin on an
    installed Gimlet (helios-3.0.24348) gives 57 libraries. Nine of them are
    missing from the recovery image: the six above plus libsff, libscsi
    and libses.
  • What loads on Oxide: the Gimlet, Cosmo and Metro maps use only
    oxhc, pciebus and fac_prov_libhp, and oxhc loads ufm, disk,
    dimm and zen. A 64-bit program taking repeated snapshots loaded
    exactly those seven plugins. libsff, libscsi and libses are needed
    only by nic.so and ses.so, which don't load on Oxide (only the
    generic pcibus module loads them), so they stay elided.
  • Leave-one-out in recovery: on a Cosmo booted into the recovery image
    (same helios build), I added the six libraries and removed each one in
    turn, with the libipcc inventory cache absent, as during a mupdate. With
    all six: three full snapshots (1226 hc nodes), and the cache is written.
    Each omission fails as shown in the table.

libipmi and libidspace look optional at first: the hc tree still fills
in without them. But the first RTLD_NOW plugin dlopen fails, and later
ones succeed only because rtld treats the partially relocated libtopo as
already relocated. That leaves an unresolved lazy reference in place, which
is how librename turned into a fatal error rather than a failed dlopen.

@smklein

smklein commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

I am in the process of testing this with oxidecomputer/omicron@58c912e

Basically:

I'll report results when they're ready.

@smklein

smklein commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor Author

Mupdate succeeded with both omicron#11279 (using libtopo) + the omicron branch (oxidecomputer/omicron@58c912e) which adds all these shared libraries to the recovery image. This test was performed on london, mupdating sled 14 + 17 (failed without this change, succeeds with this change).

(If it matters: sled 14 is a gimlet, sled 17 is a cosmo. both seem update-able via installinator)

@smklein
smklein marked this pull request as ready for review October 31, 2026 00:25
@smklein
smklein requested a review from citrus-it October 31, 2026 00:25

@citrus-it citrus-it left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise LGTM, and I test built it and threw it on a bench gimlet out of interest.

@smklein
smklein merged commit 7c0a501 into master Nov 2, 2026
@smklein
smklein deleted the recovery-keep-topo-libs branch November 2, 2026 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recovery image construction prevents us from using libtopo instead of libdevinfo to monitor for disks in omicron

2 participants