Add support to discover and assume a hostid, as well as fix arguments passed to a BE (#147)

A persistent problem with ZFS and now with ZFSBootMenu is getting spl.spl_hostid lined up on all pool imports. The contents of an initramfs and the kernel command line can influence this value, making it difficult to determine exactly where and how to change it to get a system to boot.

To help combat this recurring issue, ZBM now accepts the following options to control importing behavior.
 
* zbm.import_policy=strict: If no pools can be imported with the hostid embedded in the initramfs, or set via spl_hostid on the ZBM command line, drop to a recovery shell.

* zbm.import_policy=hostid: If no pools can be imported with the hostid embedded in the initramfs, or set via spl_hostid on the ZBM command line, attempt to scrape the hostid used to import the pool defined in root=zfsbootmenu:POOL=<value> and import the pool. If that pool is missing or can't be imported, any other available pool is scraped for a hostid and attempted to be imported.
 
* zbm.import_policy=force: YOLO mode, just force import a pool. Replaces `zbm.force_import'.

Right now, when undefined, zbm.import_policy is set to strict.

With this option available, ZBM can now in most cases import any pool - regardless of the hostid of the system that built the initramfs or the EFI executable.

Next is to ensure that the boot environment imports the pool correctly. Some ZFS initramfs modules can use spl_hostid to write out /etc/hostid in the initramfs before loading SPL/ZFS modules. Others ... like Arch ... not only do not by default include /etc/hostid in the initramfs, they do not have any methods to line up spl_hostid on the kernel command line with /etc/hostid. However, spl.spl_hostid takes precedence over /etc/hostid, so we can simply set this on the kernel command line. To this end, we now:

    Suppress spl_hostid if it's part of the discovered command line for a boot environment
    Suppress spl.spl_hostid if it's part of the discovered command line for a boot environment
    Insert spl.spl_hostid=0x<hostid> on the command line for each boot environment. (when hostid is 0, insert spl_hostid=0x00000000, to help dracut. spl.spl_hostid=0 is a no-op).

Since we know the hostid used to import a pool - either discovered in zfsbootmenu-parse-commandline.sh, or discovered when zbm.import_policy=hostid is set. Suppression / rewriting can be disabled via zbm.set_hostid=0 when booting ZBM.


Co-authored-by: Zach Dykstra <[email protected]>
Co-authored-by: Andrew J. Hesford <[email protected]>

Author: zdykstra

90zfsbootmenu/module-setup.sh

@@ -82,6 +82,7 @@ install() {
     "ps"
     "env"
     "chmod"
+    "od"
   )
 
   for _exec in "${essential_execs[@]}"; do
@@ -202,6 +203,20 @@ install() {
     type mark_hostonly >/dev/null 2>&1 && mark_hostonly /etc/zfs/vdev_id.conf
   fi
 
+  # Determine platform endianness, defaulting to le
+  ival="$( echo -n 3 | od -tx2 -N2 -An | tr -d '[:space:]' )"
+  if [ "x${ival}" = "x3300" ]; then
+    endian="be"
+  else
+    if [ "x${ival}" != "x0033" ]; then
+      warn "unable to determine platform endianness; assuming little-endian"
+    fi
+    endian="le"
+  fi
+
+  # shellcheck disable=SC2154
+  echo -n "${endian}" > "${initdir}/etc/byte-order"
+
   # Try to synchronize hostid between host and ZFSBootMenu
   #
   # DEPRECATION NOTICE: on musl systems, zfs < 2.0 produced a bad hostid in
@@ -222,7 +237,11 @@ install() {
     if [ -z "${NEWZFS}" ]; then
       # In zfs < 2.0, zgenhostid does not provide necessary behavior
       # shellcheck disable=SC2154
-      echo -ne "\\x${HOSTID:6:2}\\x${HOSTID:4:2}\\x${HOSTID:2:2}\\x${HOSTID:0:2}" > "${initdir}/etc/hostid"
+      if [ "${endian}" = "be" ] ; then
+        echo -ne "\\x${HOSTID:0:2}\\x${HOSTID:2:2}\\x${HOSTID:4:2}\\x${HOSTID:6:2}" > "${initdir}/etc/hostid"
+      else
+        echo -ne "\\x${HOSTID:6:2}\\x${HOSTID:4:2}\\x${HOSTID:2:2}\\x${HOSTID:0:2}" > "${initdir}/etc/hostid"
+      fi
     elif [ "${HOSTID}" != "00000000" ]; then
       # In zfs >= 2.0, zgenhostid writes the output, but only with nonzero hostid
       # shellcheck disable=SC2154

90zfsbootmenu/zfsbootmenu-countdown.sh

@@ -13,36 +13,15 @@ fi
 
 mkdir -p "${BASE}"
 
-# Attempt to import all pools read-only
-read_write='' all_pools=yes import_pool
-
-# Make sure at least one pool can be imported; if not,
-# drop to an emergency shell to allow the user to attempt recovery
-import_success=0
-while true; do
-  while IFS=$'\t' read -r _pool _health; do
-    [ -n "${_pool}" ] || continue
-
-    import_success=1
-    if [ "${_health}" != "ONLINE" ]; then
-      echo "${_pool}" >> "${BASE}/degraded"
-    fi
-  done <<<"$( zpool list -H -o name,health )"
-
-  if [ "${import_success}" -ne 1 ]; then
-    emergency_shell "unable to successfully import a pool"
-  else
-    zdebug "$(
-      echo "zpool list" ; \
-      zpool list
-    )"
-    zdebug "$(
-      echo "zfs list -o name,mountpoint,encroot,keystatus,keylocation,org.zfsbootmenu:keysource" ;\
-      zfs list -o name,mountpoint,encroot,keystatus,keylocation,org.zfsbootmenu:keysource
-    )"
-    break
-  fi
-done
+# Write out a default or overridden hostid
+if [ -n "${spl_hostid}" ] ; then
+  zinfo "ZFSBootMenu: writing /etc/hostid from command line: ${spl_hostid}"
+  write_hostid "${spl_hostid}"
+elif [ ! -e /etc/hostid ]; then
+  zinfo "ZFSBootMenu: no hostid found on kernel command line or /etc/hostid"
+  zinfo "ZFSBootMenu: defaulting hostid to 00000000"
+  write_hostid 0
+fi
 
 # Prefer a specific pool when checking for a bootfs value
 # shellcheck disable=SC2154
@@ -52,11 +31,71 @@ else
   boot_pool="${root}"
 fi
 
-# Make sure the preferred pool was imported
-if [ -n "${boot_pool}" ] && ! zpool list -H -o name "${boot_pool}" >/dev/null 2>&1; then
-  emergency_shell "\nCannot import requested pool '${boot_pool}'\nType 'exit' to try booting anyway"
+# Do a dedicated pass for the preferred pool if one was provided
+if [ -n "${boot_pool}" ]; then
+  first_pass=0
+else
+  first_pass=1
 fi
 
+while true; do
+  if [ "${first_pass}" -eq 0 ]; then
+    # Try the preferred pool, exactly once
+    zdebug "attempting to import preferred pool ${boot_pool}"
+    try_pool="${boot_pool}"
+  else
+    try_pool=""
+  fi
+
+  first_pass=1
+
+  read_write='' import_pool "${try_pool}"
+
+  # shellcheck disable=SC2154
+  if check_for_pools; then
+    if [ -n "${try_pool}" ]; then
+      # If a single pool was requested and imported, try again for the others
+      continue
+    else
+      # Otherwise, all possible pools were imported, nothing more to try
+      break
+    fi
+  elif [ "${import_policy}" == "hostid" ] && poolmatch="$( match_hostid "${try_pool}" )"; then
+    zdebug "match_hostid returned: ${poolmatch}"
+
+    spl_hostid="${poolmatch##*;}"
+
+    export spl_hostid
+
+    # Store the hostid to use for for KCL overrides
+    echo -n "$spl_hostid" > "${BASE}/spl_hostid"
+
+    # Retry the cycle with a matched hostid
+    continue
+  fi
+
+  # Allow the user to attempt recovery
+  emergency_shell "unable to successfully import a pool"
+done
+
+# restrict read-write access to any unhealthy pools
+while IFS=$'\t' read -r _pool _health; do
+  if [ "${_health}" != "ONLINE" ]; then
+    echo "${_pool}" >> "${BASE}/degraded"
+    zerror "prohibiting read/write operations on ${_pool}"
+  fi
+done <<<"$( zpool list -H -o name,health )"
+
+zdebug "$(
+  echo "zpool list" ; \
+  zpool list
+)"
+
+zdebug "$(
+  echo "zfs list -o name,mountpoint,encroot,keystatus,keylocation,org.zfsbootmenu:keysource" ;\
+  zfs list -o name,mountpoint,encroot,keystatus,keylocation,org.zfsbootmenu:keysource
+)"
+
 unsupported=0
 while IFS=$'\t' read -r _pool _property; do
   if [[ "${_property}" =~ "unsupported@" ]]; then

90zfsbootmenu/zfsbootmenu-exec.sh

@@ -1,12 +1,14 @@
 #!/bin/bash
 # vim: softtabstop=2 shiftwidth=2 expandtab
 
+export endian
 export spl_hostid
-export force_import
+export import_policy
 export menu_timeout
 export loglevel
 export root
 export zbm_sort
+export zbm_set_hostid
 
 # Disable all kernel messages to the console
 echo 0 > /proc/sys/kernel/printk