Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
d2cfd20
Add libtopo dependency to sled-hardware
smklein Sep 10, 2026
91ff18c
Rename disk "slot" to "pcie_slot" in sled-hardware and Nexus types
smklein Sep 11, 2026
2417ea5
Report each disk's chassis location from the hardware topology
smklein Sep 11, 2026
1f8a88b
Carry disk chassis location through the sled-agent API into inventory
smklein Sep 11, 2026
603073f
Add /usr/lib/fm/amd64 to the illumos runpath for libtopo
smklein Sep 11, 2026
c51a8fa
Give the example inventory disks chassis locations
smklein Sep 11, 2026
62b18e0
Show only the chassis location on the inventory disk line
smklein Sep 11, 2026
e35b60a
Tidy the topo lookup after review
smklein Sep 11, 2026
7ba9b63
Give the planning system builder's disks chassis locations
smklein Sep 11, 2026
a226091
Disk location follow-ups: reconcile late labels, merge topo reads, lo…
smklein Sep 14, 2026
f69fbb6
merge
smklein Sep 16, 2026
7807fec
Use libtopo 0.2.0's hc constants and typed property getter
smklein Sep 16, 2026
901fb2a
Set libtopo's runpath through omicron-rpaths, not .cargo/config.toml
smklein Sep 16, 2026
37dec5e
Read disk locations from topo on every hardware poll
smklein Sep 16, 2026
8eb6538
Rename sled-hardware disk_location.rs to nvme_instance.rs
smklein Sep 16, 2026
831a0ef
Set libtopo's runpath for crates that reach it only via dev-dependencies
smklein Sep 16, 2026
7564f16
Set libtopo's runpath in .cargo/config.toml, not omicron-rpaths
smklein Sep 17, 2026
ce390cc
Give synthetic disks chassis locations that match real hardware
smklein Sep 17, 2026
5219def
Disk location review follow-ups
smklein Sep 17, 2026
15ee23c
Disk location review follow-ups, third round
smklein Sep 17, 2026
8dbe94d
merge
smklein Sep 18, 2026
b253d2e
merge
smklein Oct 8, 2026
ed8b4f3
merge
smklein Oct 16, 2026
d45b283
merge
smklein Oct 20, 2026
6cecf2f
merge
smklein Oct 23, 2026
a1ae16d
merge
smklein Oct 28, 2026
8923b45
merge
smklein Nov 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 33 additions & 24 deletions .cargo/config.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,28 +21,34 @@
rustdocflags = "--document-private-items"
rustflags = "--cfg tokio_unstable"

# On illumos, use `-znocompstrtab` to reduce link time. We also add the Oxide
# specific platform directory to the RPATH where additional libraries can be
# found such as libipcc.
# On illumos, use `-znocompstrtab` to reduce link time. We also add two
# directories to the RPATH where libraries outside the default search path
# live:
#
# Our reasoning for including `-R/usr/platform/oxide/lib/amd64` here:
# - Oxide specific features - This path contains Oxide specific libraries such
# as libipcc and will likely grow over time to include more functionality.
# - Avoid the rpaths crate - The rpaths crate was built to deal with runtime
# paths that are dynamic such as with libraries like libpq which can live in
# different locations based on OS. This path will only ever be found on
# illumos and will be tied directly to the Oxide platform.
# - Less developer burden - Having something like the ipcc crate emit
# `DEP_IPCC_LIBDIRS` means that we end up littering the repo with Cargo.toml
# and build.rs changes whenever ipcc shows up somewhere in the dependency
# tree. While initially exploring that path we ran into a significant number
# of tests failing due to not being able to find libipcc in the runtime path
# - /usr/platform/oxide/lib/amd64 holds Oxide-specific libraries such as
# libipcc, and will likely grow over time to include more functionality.
# - /usr/lib/fm/amd64 holds the illumos fault management libraries such as
# libtopo, which sled-hardware uses to find each disk's chassis location.
#
# Our reasoning for putting these here rather than going through the rpaths
# crate:
# - Static, illumos-only paths - The rpaths crate was built to deal with
# runtime paths that are dynamic such as with libraries like libpq which can
# live in different locations based on OS. These paths will only ever be
# found on illumos and never move.
# - Less developer burden - Having a -sys crate emit `DEP_*_LIBDIRS` means that
# we end up littering the repo with Cargo.toml and build.rs changes in every
# crate whose binaries or tests transitively link the library. For libipcc
# that was already painful; libtopo enters through sled-hardware, which well
# over a hundred workspace crates reach transitively. While initially
# exploring that path for libipcc we ran into a significant number of tests
# failing due to not being able to find the library in the runtime path,
# which can be confusing or surprising to someone doing work on omicron.
#
# We could also update Helios so that a symlink is created from
# /usr/platform/oxide/lib/amd64 into /usr/lib/64 but we opted to not take
# this route forward as it meant waiting for another round of updates on
# shared build machines and to buildomat itself.
# We could also update Helios so that symlinks are created from these
# directories into /usr/lib/64 but we opted to not take this route forward as
# it meant waiting for another round of updates on shared build machines and
# to buildomat itself.
#
# As documented at:
# https://doc.rust-lang.org/cargo/reference/config.html#buildrustflags
Expand All @@ -58,13 +64,16 @@ rustflags = "--cfg tokio_unstable"
# When overriding the defaults in this config by environment variable the user
# may need to manually pass the additional options found below.
#
# Note that other runtime paths should not be added here, but should instead
# reuse the infrastructure found in the `rpaths` crate which can be found in
# this repo. Those paths are usually platform specific and will vary based on a
# variety of things such as host OS.
# Runtime paths for illumos-only libraries with a fixed install location, like
# the two above, belong here. Runtime paths for cross-platform libraries should
# not be added here, but should instead reuse the infrastructure found in the
# `rpaths` crate in this repo. Those paths are usually platform specific and
# will vary based on a variety of things such as host OS. Whatever is added
# here must also be added to the `RUSTFLAGS` override in
# .github/buildomat/build-and-test.sh, which takes precedence over this file.
[target.x86_64-unknown-illumos]
rustflags = [
"-C", "link-arg=-Wl,-znocompstrtab,-R/usr/platform/oxide/lib/amd64",
"-C", "link-arg=-Wl,-znocompstrtab,-R/usr/platform/oxide/lib/amd64:/usr/lib/fm/amd64",
# enable the `tokio-unstable` config for `tokio-dtrace` probes
"--cfg", "tokio_unstable"
]
Expand Down
13 changes: 13 additions & 0 deletions .cargo/xtask.toml
Original file line number Diff line number Diff line change
Expand Up @@ -59,3 +59,16 @@ binary_allow_list = [
"sled-agent",
"sled-agent-sim",
]

# libtopo is the illumos hardware topology library, used by sled-hardware to
# find each disk's chassis location. Like libnvme, it is only needed by
# programs that inspect a sled's hardware from the global zone.
[libraries."libtopo.so.1"]
binary_allow_list = [
"installinator",
"omicron-dev",
"omicron-package",
"services-ledger-check-migrate",
"sled-agent",
"sled-agent-sim",
]
6 changes: 3 additions & 3 deletions .github/buildomat/build-and-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -101,10 +101,10 @@ export RUSTFLAGS="--cfg tokio_unstable -D warnings"
export RUSTDOCFLAGS="--document-private-items -D warnings"
# When running on illumos we need to pass an additional runpath that is
# usually configured via ".cargo/config" but the `RUSTFLAGS` env variable
# takes precedence. This path contains oxide specific libraries such as
# libipcc.
# takes precedence. These paths contain oxide specific libraries such as
# libipcc, and the fault management libraries such as libtopo.
if [[ $target_os == "illumos" ]]; then
RUSTFLAGS="$RUSTFLAGS -C link-arg=-R/usr/platform/oxide/lib/amd64"
RUSTFLAGS="$RUSTFLAGS -C link-arg=-R/usr/platform/oxide/lib/amd64:/usr/lib/fm/amd64"
fi
export TMPDIR="$TEST_TMPDIR"
export RUST_BACKTRACE=1
Expand Down
27 changes: 23 additions & 4 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 4 additions & 3 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -544,8 +544,8 @@ filetime = "0.2.26"
flate2 = "1.1.2"
float-ord = "0.3.2"
flume = "0.11.1"
fmd-adm = { git = "https://github.com/oxidecomputer/fmd-adm", rev = "846361bf0a698a8c7efefd97b2828b9aa74858c4" }
fmd-adm-sys = { git = "https://github.com/oxidecomputer/fmd-adm", rev = "846361bf0a698a8c7efefd97b2828b9aa74858c4" }
fmd-adm = { git = "https://github.com/oxidecomputer/fmd-adm", rev = "5b57d8dd36f96778d0c067ec9647ddde9b58ace7" }
fmd-adm-sys = { git = "https://github.com/oxidecomputer/fmd-adm", rev = "5b57d8dd36f96778d0c067ec9647ddde9b58ace7" }
foreign-types = "0.3.2"
fs-err = "3.1.1"
futures = "0.3.31"
Expand Down Expand Up @@ -600,7 +600,7 @@ hyper-util = "0.1.16"
hyper-rustls = "0.27.7"
hyper-staticfile = "0.10.1"
iddqd = { version = "0.5.0", features = ["daft", "serde", "schemars08"] }
illumos-nvpair = "0.3.0"
illumos-nvpair = "0.4.0"
illumos-nvpair-sys = "0.2.0"
illumos-utils = { path = "illumos-utils" }
iana-time-zone = "0.1.63"
Expand Down Expand Up @@ -630,6 +630,7 @@ libc = "0.2.174"
linkme = "0.3"
libipcc = { git = "https://github.com/oxidecomputer/ipcc-rs", rev = "524eb8f125003dff50b9703900c6b323f00f9e1b" }
libnvme = { git = "https://github.com/oxidecomputer/libnvme", rev = "dd5bb221d327a1bc9287961718c3c10d6bd37da0" }
libtopo = "0.2.0"
linear-map = "1.2.0"
live-tests-macros = { path = "live-tests/macros" }
lldpd_client = { git = "https://github.com/oxidecomputer/lldp", rev = "54b266174d4de9628bca9c97b0db176e16f12154", package = "lldpd-client" }
Expand Down
7 changes: 5 additions & 2 deletions dev-tools/omdb/src/bin/omdb/db.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7598,7 +7598,9 @@ async fn cmd_db_inventory_physical_disks(
struct DiskRow {
inv_collection_id: Uuid,
sled_id: Uuid,
slot: i64,
pcie_slot: i64,
#[tabled(display_with = "display_option_blank")]
location: Option<String>,
vendor: String,
model: String,
serial: String,
Expand Down Expand Up @@ -7651,7 +7653,8 @@ async fn cmd_db_inventory_physical_disks(
DiskRow {
inv_collection_id: disk.inv_collection_id.into_untyped_uuid(),
sled_id: disk.sled_id.into_untyped_uuid(),
slot: disk.slot,
pcie_slot: disk.pcie_slot,
location: disk.location.clone(),
vendor: disk.vendor,
model: disk.model.clone(),
serial: disk.serial.clone(),
Expand Down
18 changes: 9 additions & 9 deletions dev-tools/reconfigurator-cli/tests/output/cmds-example-stdout
Original file line number Diff line number Diff line change
Expand Up @@ -1617,9 +1617,9 @@ sled 2eb69596-f081-4e2d-9425-9994926e0832 (role = Gimlet, serial serial1)
usable memory (GiB): 0
reservoir (GiB): 0
physical disks:
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-64ea76fb-6673-4810-8e53-c2458b75eb01" } in 0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-6d012675-6f54-4b6a-8658-ab0076237569" } in 1
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-9a065406-12a0-4b29-926e-d66bb785d17b" } in 2
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-64ea76fb-6673-4810-8e53-c2458b75eb01" } in N0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-6d012675-6f54-4b6a-8658-ab0076237569" } in N1
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-9a065406-12a0-4b29-926e-d66bb785d17b" } in N2
zpools
64ea76fb-6673-4810-8e53-c2458b75eb01: total size: 100 GiB health: online
6d012675-6f54-4b6a-8658-ab0076237569: total size: 100 GiB health: online
Expand Down Expand Up @@ -1773,9 +1773,9 @@ sled 32d8d836-4d8a-4e54-8fa9-f31d79c42646 (role = Gimlet, serial serial2)
usable memory (GiB): 0
reservoir (GiB): 0
physical disks:
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-2db3c119-38fa-41c0-8dbe-89689b84d655" } in 0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-687e054b-a9b5-4404-8b3f-0ea119c44a6a" } in 1
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-a4e8a5cc-cd1c-4a08-bcf1-31785f7cda28" } in 2
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-2db3c119-38fa-41c0-8dbe-89689b84d655" } in N0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-687e054b-a9b5-4404-8b3f-0ea119c44a6a" } in N1
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-a4e8a5cc-cd1c-4a08-bcf1-31785f7cda28" } in N2
zpools
2db3c119-38fa-41c0-8dbe-89689b84d655: total size: 100 GiB health: online
687e054b-a9b5-4404-8b3f-0ea119c44a6a: total size: 100 GiB health: online
Expand Down Expand Up @@ -1929,9 +1929,9 @@ sled 89d02b1b-478c-401a-8e28-7a26f74fa41b (role = Gimlet, serial serial0)
usable memory (GiB): 0
reservoir (GiB): 0
physical disks:
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-0477165a-a72e-4814-b8d6-74aa02cb2040" } in 0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-ab94a812-86ce-428c-bbbb-6ce1ab0b071b" } in 1
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-f96f5901-2907-4f21-bfeb-772f8a3c4e44" } in 2
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-0477165a-a72e-4814-b8d6-74aa02cb2040" } in N0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-ab94a812-86ce-428c-bbbb-6ce1ab0b071b" } in N1
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-f96f5901-2907-4f21-bfeb-772f8a3c4e44" } in N2
zpools
0477165a-a72e-4814-b8d6-74aa02cb2040: total size: 100 GiB health: online
ab94a812-86ce-428c-bbbb-6ce1ab0b071b: total size: 100 GiB health: online
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ sled 2b8f0cb3-0295-4b3c-bc58-4fe88b57112c (role = Gimlet, serial serial1)
usable memory (GiB): 0
reservoir (GiB): 0
physical disks:
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-72c59873-31ff-4e36-8d76-ff834009349a" } in 0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-72c59873-31ff-4e36-8d76-ff834009349a" } in N0
zpools
72c59873-31ff-4e36-8d76-ff834009349a: total size: 100 GiB health: online
datasets:
Expand Down Expand Up @@ -323,7 +323,7 @@ sled 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6 (role = Gimlet, serial serial0)
usable memory (GiB): 0
reservoir (GiB): 0
physical disks:
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-c6d33b64-fb96-4129-bab1-7878a06a5f9b" } in 0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-c6d33b64-fb96-4129-bab1-7878a06a5f9b" } in N0
zpools
c6d33b64-fb96-4129-bab1-7878a06a5f9b: total size: 100 GiB health: online
datasets:
Expand Down Expand Up @@ -464,7 +464,7 @@ sled d81c6a84-79b8-4958-ae41-ea46c9b19763 (role = Gimlet, serial serial2)
usable memory (GiB): 0
reservoir (GiB): 0
physical disks:
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-4930954e-9ac7-4453-b63f-5ab97c389a99" } in 0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-4930954e-9ac7-4453-b63f-5ab97c389a99" } in N0
zpools
4930954e-9ac7-4453-b63f-5ab97c389a99: total size: 100 GiB health: online
datasets:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -548,9 +548,9 @@ sled 2b8f0cb3-0295-4b3c-bc58-4fe88b57112c (role = Gimlet, serial serial1)
usable memory (GiB): 0
reservoir (GiB): 0
physical disks:
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-727522a7-934f-494d-b5b3-160968e74463" } in 0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-72c59873-31ff-4e36-8d76-ff834009349a" } in 1
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-b5fd5bc1-099e-4e77-8028-a9793c11f43b" } in 2
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-727522a7-934f-494d-b5b3-160968e74463" } in N0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-72c59873-31ff-4e36-8d76-ff834009349a" } in N1
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-b5fd5bc1-099e-4e77-8028-a9793c11f43b" } in N2
zpools
727522a7-934f-494d-b5b3-160968e74463: total size: 100 GiB health: online
72c59873-31ff-4e36-8d76-ff834009349a: total size: 100 GiB health: online
Expand Down Expand Up @@ -757,9 +757,9 @@ sled 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6 (role = Gimlet, serial serial0)
usable memory (GiB): 0
reservoir (GiB): 0
physical disks:
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-073979dd-3248-44a5-9fa1-cc72a140d682" } in 0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-c6d33b64-fb96-4129-bab1-7878a06a5f9b" } in 1
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-e4d937e1-6ddc-4eca-bb08-c1f73791e608" } in 2
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-073979dd-3248-44a5-9fa1-cc72a140d682" } in N0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-c6d33b64-fb96-4129-bab1-7878a06a5f9b" } in N1
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-e4d937e1-6ddc-4eca-bb08-c1f73791e608" } in N2
zpools
073979dd-3248-44a5-9fa1-cc72a140d682: total size: 100 GiB health: online
c6d33b64-fb96-4129-bab1-7878a06a5f9b: total size: 100 GiB health: online
Expand Down Expand Up @@ -957,9 +957,9 @@ sled d81c6a84-79b8-4958-ae41-ea46c9b19763 (role = Gimlet, serial serial2)
usable memory (GiB): 0
reservoir (GiB): 0
physical disks:
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-18b20749-0748-4105-bb10-7b13cfc776e2" } in 0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-30c16fe4-4229-49d0-ab01-3138f2c7dff2" } in 1
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-4930954e-9ac7-4453-b63f-5ab97c389a99" } in 2
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-18b20749-0748-4105-bb10-7b13cfc776e2" } in N0
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-30c16fe4-4229-49d0-ab01-3138f2c7dff2" } in N1
U2: DiskIdentity { vendor: "fake-vendor", model: "fake-model", serial: "serial-4930954e-9ac7-4453-b63f-5ab97c389a99" } in N2
zpools
18b20749-0748-4105-bb10-7b13cfc776e2: total size: 100 GiB health: online
30c16fe4-4229-49d0-ab01-3138f2c7dff2: total size: 100 GiB health: online
Expand Down
Loading
Loading