Perform reboot between provision steps

[Sn0wCrack]

Pretty new to use Lima, but currently trying to create a template that has multiple provision steps where after the first step it reboots the Virtual Machine before continuing.

Right now in my first provision step I end it with a reboot and then continue in the next scirpt, however things basically start timing out, failing to connect, etc after the reboot as I imagine Lima doesn't really know what's happening.

My use case here is I want to install the latest Kernel version on AlmaLinux 9 (mimics a production environment we have), but it requires a reboot inbetween to apply the change.

Is this something that's currently supported or do I just need to manually do this once the VM is created.

My current template file:

images:
  # AlmaLinux 9 Cloud Images
  - location: "https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2"
    arch: "x86_64"
  - location: "https://repo.almalinux.org/almalinux/9/cloud/aarch64/images/AlmaLinux-9-GenericCloud-latest.aarch64.qcow2"
    arch: "aarch64"
mounts:
  - location: "~"
    writable: true
portForwards:
  - guestPort: 8443
    hostPort: 8443
provision:
  - mode: system
    script: |
      #!/usr/bin/env bash
      set -eux -o pipefail
      
      # 1. Install prerequisites from AlmaLinux 9 docs
      rpm --import https://www.centos.org/keys/RPM-GPG-KEY-CentOS-SIG-Kmods
      dnf install -y https://mirror.stream.centos.org/SIGs/$(rpm --eval '%{?rhel}/kmods/%{_arch}/repos-main/Packages/c/centos-repos-kmods-%{?rhel}-2.el%{?rhel}.noarch.rpm')

      dnf install -y epel-release dnf-plugins-core centos-release-kmods-kernel
      dnf update -y

      reboot
  - mode: system
    script: |
      #!/usr/bin/env bash
      set -eux -o pipefail

      dnf config-manager --set-enabled crb
      dnf copr enable neelc/incus -y
      
      # Install Incus, Distrobuilder dependencies, Docker dependencies, and socat
      dnf install -y incus incus-tools \
        golang gcc debootstrap rsync gnupg2 squashfs-tools git make hivex genisoimage btrfs-progs btrfs-progs-devel \
        socat yum-utils
      
      # 2. Build Distrobuilder from source
      export GOPATH=/root/go
      git clone https://github.com/lxc/distrobuilder /tmp/distrobuilder
      cd /tmp/distrobuilder
      LATEST_TAG=$(git describe --tags $(git rev-list --tags --max-count=1))
      git checkout $LATEST_TAG
      make
      cp $GOPATH/bin/distrobuilder /usr/local/bin/
      cd -
      
      # Install uv globally
      curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
      
      # 3. System Configuration (sysctls, limits, subuid/subgid)
      # Assuming Lima mounts the current directory to the exact same absolute path as the host.
      # We can dynamically find the mount path by looking for the pyproject.toml
      PROJECT_DIR=$(find /Users /home -maxdepth 5 -name "pyproject.toml" -exec dirname {} \; | head -n 1)
      
      if [ -n "$PROJECT_DIR" ]; then
        cp $PROJECT_DIR/config/limits.d/80-incus.conf /etc/security/limits.d/80-incus.conf
        cp $PROJECT_DIR/config/sysctl.d/80-incus.conf /etc/sysctl.d/80-incus.conf
        sysctl --system
        
        if [ ! -f /etc/subuid ]; then
          cp $PROJECT_DIR/config/subuid /etc/subuid
        fi
        if [ ! -f /etc/subgid ]; then
          cp $PROJECT_DIR/config/subgid /etc/subgid
        fi
      fi
      
      # Enable and start Incus
      systemctl enable --now incus
      
      # 4. Initialize Incus and Docker storage pools
      incus admin init --auto --storage-backend btrfs --storage-create-loop 20
      incus storage create docker btrfs size=20GB
      
      # Disable CoW on storage devices for performance as per docs
      incus storage set default btrfs.mount_options "user_subvol_rm_allowed,nodatacow"
      incus storage set docker btrfs.mount_options "user_subvol_rm_allowed,nodatacow"
      
      # Expose Incus Unix socket over TCP (port 8443) via socat for integration tests
      cat << 'EOF' > /etc/systemd/system/incus-tcp-proxy.service
      [Unit]
      Description=Incus TCP Proxy
      After=incus.service

      [Service]
      ExecStart=/usr/bin/socat TCP-LISTEN:8443,fork,bind=0.0.0.0 UNIX-CONNECT:/run/incus/unix.socket
      Restart=always

      [Install]
      WantedBy=multi-user.target
      EOF
      systemctl daemon-reload
      systemctl enable --now incus-tcp-proxy.service
      
      # Pre-add the required group permissions for Incus and Docker
      usermod -aG incus-admin {{.User}}

[afbjorklund]

The other scripts just do a systemctl reboot, but have to be careful so that you don't end up in a reboot loop...

https://github.com/lima-vm/lima/blob/master/pkg/cidata/cidata.TEMPLATE.d/boot.Linux/00-reboot-if-required.sh

i.e. you need some kind of check, above

[Sn0wCrack]

Thanks for that, I've added in a check around the reboot for it checking the current loaded Kernel version and this seems to work.

if [[ $(uname -r | cut -c1) != "6" ]]; then
  systemctl reboot
fi

Just for my own understanding then, Lima effectively just re-run all provision steps when the VM reboots? Sort of curious how this actually works under the hood.

[afbjorklund]

Each start gets a new instance ID, causing all the scripts to run again

[Sn0wCrack]

Actually... coming back to this it just sort of "appeared" to have worked.

It seems there's a short delay for systemctl reboot to actually reboot as its attempting a graceful shutdown.

In the time from it initiating to it acting on it, it executed few more commands and when my VM was reported as ready, the new kernel was installed successfully.
However most of my install script appears to not run, only a few lines after calling reboot do.

Adding a sleep after the reboot nothing past the reboot runs at all and lima seems to report all good.

INFO[0102] [cloud-init] + systemctl reboot              
INFO[0102] [cloud-init] + sleep 1                       
WARN[0101] [hostagent] guest agent events closed unexpectedly  error=EOF
INFO[0102] [cloud-init]   File "/usr/lib/python3.9/site-packages/cloudinit/helpers.py", line 156, in run 
INFO[0102] [cloud-init]     results = functor(**args)   
INFO[0102] [cloud-init]   File "/usr/lib/python3.9/site-packages/cloudinit/config/cc_scripts_per_boot.py", line 39, in handle 
INFO[0102] [cloud-init]     subp.runparts(runparts_path) 
INFO[0102] [cloud-init]   File "/usr/lib/python3.9/site-packages/cloudinit/subp.py", line 371, in runparts 
INFO[0102] [cloud-init]     subp(prefix + [exe_path], capture=False) 
INFO[0102] [cloud-init]   File "/usr/lib/python3.9/site-packages/cloudinit/subp.py", line 272, in subp 
INFO[0102] [cloud-init]     out, err = sp.communicate(data, timeout=timeout) 
INFO[0102] [cloud-init]   File "/usr/lib64/python3.9/subprocess.py", line 1126, in communicate 
INFO[0102] [cloud-init]     self.wait()                 
INFO[0102] [cloud-init]   File "/usr/lib64/python3.9/subprocess.py", line 1189, in wait 
INFO[0102] [cloud-init]     return self._wait(timeout=timeout) 
INFO[0102] [cloud-init]   File "/usr/lib64/python3.9/subprocess.py", line 1933, in _wait 
INFO[0102] [cloud-init]     (pid, sts) = self._try_wait(0) 
INFO[0102] [cloud-init]   File "/usr/lib64/python3.9/subprocess.py", line 1891, in _try_wait 
INFO[0102] [cloud-init]     (pid, sts) = os.waitpid(self.pid, wait_flags) 
INFO[0102] [cloud-init]   File "/usr/lib/python3.9/site-packages/cloudinit/signal_handler.py", line 48, in _handle_exit 
INFO[0102] [cloud-init]     sys.exit(rc)                
INFO[0102] [cloud-init] SystemExit: 1                   
INFO[0102] [cloud-init] ------------------------------------------------------------ 
INFO[0102] [cloud-init] Cloud-init monitoring completed successfully 
INFO[0102] Cloud-init progress monitoring done.         
INFO[0111] [hostagent] Guest agent is running           
INFO[0111] [hostagent] Not forwarding TCP 0.0.0.0:22    
INFO[0112] [cloud-init] Cloud-init monitoring completed successfully 
INFO[0112] Cloud-init progress monitoring done.         
INFO[0111] [hostagent] Forwarding TCP from 0.0.0.0:111 to 127.0.0.1:111 
INFO[0112] [cloud-init] Cloud-init monitoring completed successfully 
INFO[0112] Cloud-init progress monitoring done.         
INFO[0111] [hostagent] Not forwarding TCP [::]:22       
INFO[0112] [cloud-init] Cloud-init monitoring completed successfully 
INFO[0112] Cloud-init progress monitoring done.         
INFO[0111] [hostagent] Forwarding TCP from [::]:111 to 127.0.0.1:111 
INFO[0112] [cloud-init] Cloud-init monitoring completed successfully 
INFO[0112] Cloud-init progress monitoring done.         
INFO[0111] [hostagent] Forwarding UDP from 0.0.0.0:111 to 127.0.0.1:111 
INFO[0112] [cloud-init] Cloud-init monitoring completed successfully 
INFO[0112] Cloud-init progress monitoring done.         
WARN[0111] [hostagent] failed to listen tcp: 0.0.0.0:111  error="listen tcp 0.0.0.0:111: bind: address already in use"
WARN[0111] [hostagent] failed to listen tcp: 127.0.0.1:111  error="listen tcp 0.0.0.0:111: bind: address already in use"
INFO[0111] [hostagent] Forwarding UDP from 127.0.0.1:323 to 127.0.0.1:323 
INFO[0112] [cloud-init] Cloud-init monitoring completed successfully 
INFO[0112] Cloud-init progress monitoring done.         
INFO[0111] [hostagent] Forwarding UDP from [::]:111 to 127.0.0.1:111 
INFO[0112] [cloud-init] Cloud-init monitoring completed successfully 
INFO[0112] Cloud-init progress monitoring done.         
INFO[0111] [hostagent] Forwarding UDP from [::1]:323 to 127.0.0.1:323 
INFO[0112] [cloud-init] Cloud-init monitoring completed successfully 
INFO[0112] Cloud-init progress monitoring done.         
INFO[0112] [hostagent] Not forwarding UDP 0.0.0.0:323   
INFO[0112] [hostagent] Not forwarding UDP 127.0.0.1:323 
INFO[0114] [hostagent] Time sync: guest clock adjusted (was 658ms off) 
INFO[0124] [hostagent] Time sync: guest clock adjusted (was -142ms off) 
INFO[0147] [hostagent] The final requirement 1 of 1 is satisfied 
INFO[0148] READY. Run `limactl shell almalinux-lxc` to open the shell. 

Updated YAML file:

images:
  # AlmaLinux 9 Cloud Images
  - location: "https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2"
    arch: "x86_64"
  - location: "https://repo.almalinux.org/almalinux/9/cloud/aarch64/images/AlmaLinux-9-GenericCloud-latest.aarch64.qcow2"
    arch: "aarch64"
mounts:
  - location: "~"
    writable: true
portForwards:
  - guestPort: 55221
    hostPort: 55221
containerd:
  system: false
  user: false
provision:
  - mode: system
    script: |
      #!/usr/bin/env bash
      set -eux -o pipefail
      
      # 1. Install prerequisites from AlmaLinux 9 docs
      rpm --import https://www.centos.org/keys/RPM-GPG-KEY-CentOS-SIG-Kmods
      dnf install -y https://mirror.stream.centos.org/SIGs/$(rpm --eval '%{?rhel}/kmods/%{_arch}/repos-main/Packages/c/centos-repos-kmods-%{?rhel}-2.el%{?rhel}.noarch.rpm')
      
      dnf install -y epel-release dnf-plugins-core centos-release-kmods
      dnf update -y
      dnf config-manager --set-enabled crb
      dnf config-manager --set-enabled epel
      dnf copr enable neelc/incus -y
      dnf update -y

      dnf install -y centos-release-kmods-kernel
      dnf update -y

      if [[ $(uname -r | cut -c1) == "5" ]]; then
        systemctl reboot
        sleep 1
      fi
  - mode: system
    script: |
      #!/usr/bin/env bash
      set -eux -o pipefail

      dnf update -y
      
      # Install Incus, Distrobuilder dependencies, Docker dependencies, and socat
      dnf install -y incus incus-tools \
        golang gcc rsync gnupg2 squashfs-tools git make hivex genisoimage btrfs-progs btrfs-progs-devel \
        socat yum-utils
      
      groupadd incus-admin || true

      # 2. Build Distrobuilder from source
      export GOPATH=/root/go
      mkdir -p /tmp/distrobuilder
      git clone https://github.com/lxc/distrobuilder /tmp/distrobuilder
      git config --global --add safe.directory /tmp/distrobuilder
      LATEST_TAG=$(git describe --tags $(git rev-list --tags --max-count=1))
      git checkout $LATEST_TAG
      make
      cp $GOPATH/bin/distrobuilder /usr/local/bin/
      cd -
      
      # Install uv globally
      curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
      
      # 3. System Configuration (sysctls, limits, subuid/subgid)
      # Assuming Lima mounts the current directory to the exact same absolute path as the host.
      # We can dynamically find the mount path by looking for the pyproject.toml
      PROJECT_DIR=$(find /Users /home -maxdepth 5 -name "pyproject.toml" -exec dirname {} \; | head -n 1)
      
      if [ -n "$PROJECT_DIR" ]; then
        cp $PROJECT_DIR/config/limits.d/80-incus.conf /etc/security/limits.d/80-incus.conf
        cp $PROJECT_DIR/config/sysctl.d/80-incus.conf /etc/sysctl.d/80-incus.conf
        sysctl --system
        
        if [ ! -f /etc/subuid ]; then
          cp $PROJECT_DIR/config/subuid /etc/subuid
        fi
        if [ ! -f /etc/subgid ]; then
          cp $PROJECT_DIR/config/subgid /etc/subgid
        fi
      fi
      
      # Enable and start Incus
      systemctl enable --now incus
      
      # 4. Initialize Incus and Docker storage pools
      incus admin init --auto --storage-backend btrfs --storage-create-loop 20
      incus storage create docker btrfs size=20GB
      
      # Disable CoW on storage devices for performance as per docs
      incus storage set default btrfs.mount_options "user_subvol_rm_allowed,nodatacow"
      incus storage set docker btrfs.mount_options "user_subvol_rm_allowed,nodatacow"
      
      # Expose Incus Unix socket over TCP (port 55221) via socat for integration tests
      cat << 'EOF' > /etc/systemd/system/incus-tcp-proxy.service
      [Unit]
      Description=Incus TCP Proxy
      After=incus.service

      [Service]
      ExecStart=/usr/bin/socat TCP-LISTEN:55221,fork,bind=0.0.0.0 UNIX-CONNECT:/run/incus/unix.socket
      Restart=always

      [Install]
      WantedBy=multi-user.target
      EOF
      
      systemctl daemon-reload
      systemctl enable --now incus-tcp-proxy.service
      
      # Pre-add the required group permissions for Incus
      usermod -aG incus-admin {{.User}}

[afbjorklund]

It seems there's a short delay for systemctl reboot to actually reboot as its attempting a graceful shutdown.

That is right, it seems to be asynchronous... Sleep is better than force, but was hoping for something better.