[Ubuntu on M1] Lima VM will not start (`EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary`)

[mateka]

EDIT by @AkihiroSuda

Update (Mar 14, 2022)

⚠️ Running recent Linux guests on M1 macOS needs Homebrew's QEMU 6.2.0_1 or later.

Run brew upgrade to upgrade QEMU.

Intel macOS users are NOT affected by this issue.

For the further information, see #734 and Homebrew/homebrew-core#96743 .

Old information

Ubuntu kernel 5.13.0-35.40 (5.4.0-103.117 for 20.04 LTS) released in March 2022 is not bootable on ARM hosts:

(~/.lima/default/serial.log)
...
EFI stub: Booting Linux Kernel...
EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
SetUefiImageMemoryAttributes - 0x000000013F500000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000013C190000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000013C140000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000013F4C0000 - 0x0000000000030000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000013C0F0000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000013BCB0000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000013BB00000 - 0x0000000000030000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000013BAC0000 - 0x0000000000030000 (0x0000000000000008)
(hangs here)

Until the issue gets resolved, it is HIGHLY RECOMMENDED to run the following command inside the Ubuntu shell to avoid upgrading the kernel:

sudo apt-mark hold linux-image-$(uname -r)

(Make sure to unhold it when we get the fix)

If your kernel was already upgraded and the VM does not boot, try running Lima with the following environment variable to show the video display:

export QEMU_SYSTEM_AARCH64="qemu-system-aarch64 -display cocoa"

After the display is shown, press the Esc key several times to show the GRUB prompt for choosing an older kernel.
You can also mount the disk image from another instance to modify the GRUB config or rescue the data inside the image.
#712 (comment)

See also https://gitlab.com/qemu-project/qemu/-/issues/899

🙏 Help wanted for testing

This QEMU patch may fix the issue: AkihiroSuda/homebrew-core@177d583
(from https://gitlab.com/qemu-project/qemu/-/issues/899)

Download qemu.rb and run brew install --build-from-source ./qemu.rb to apply the patch.

Below is the original post by @mateka

---

Description

Today I was working in lima VM (Ubuntu 20.04; Mac M1). After a while, it has lost connection (as it often does), so I had tried to restart it. Old VM and even brand new will not start and return an error after:

Waiting for the essential requirement 1 of 5: "ssh"

limactl --debug start VM-name prints:

DEBU[0171] [hostagent] executing ssh for script "ssh": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile="/Users/matek/.lima/_config/user" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers="^aes128-gcm@openssh.com,aes256-gcm@openssh.com" -o User=matek -o ControlMaster=auto -o ControlPath="/Users/matek/.lima/test/ssh.sock" -o ControlPersist=5m -p 52070 127.0.0.1 -- /bin/bash]
DEBU[0246] [hostagent] stdout="", stderr="kex_exchange_identification: read: Connection reset by peer\r\nConnection reset by 127.0.0.1 port 52070\r\n", err=failed to execute script "ssh": stdout="", stderr="kex_exchange_identification: read: Connection reset by peer\r\nConnection reset by 127.0.0.1 port 52070\r\n": exit status 255

Rancher Desktop seems to work.

[AkihiroSuda]

Do you see any error in ~/.lima/test/serial.log ?

[mateka]

I am only seeing is:

EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary

Here is full log:

serial.log

[AkihiroSuda]

Could you provide Lima version, QEMU version, and /Users/matek/.lima/test/lima.yaml ?

[mateka]

Both are installed from brew:

> limactl -v                                                                                                                                                                                                
limactl version 0.8.3
> qemu-system-aarch64 --version
QEMU emulator version 6.2.0
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

lima.yaml.txt

[AkihiroSuda]

Thanks

• Does this work? limactl rm -f test && limactl prune && limactl start
• Does Fedora image work? https://github.com/lima-vm/lima/blob/v0.8.3/examples/fedora.yaml

[mateka]

For the clear Ubuntu test, I have used default settings. Still, it is not working:

limactl rm -f test && limactl prune && limactl start
INFO[0000] The QEMU process seems already stopped       
INFO[0000] The host agent process seems already stopped 
INFO[0000] Removing *.pid *.sock under "/Users/matek/.lima/test" 
INFO[0000] Deleted "test" ("/Users/matek/.lima/test")   
INFO[0000] Pruning "/Users/matek/Library/Caches/lima"   
? Creating an instance "default" Proceed with the default configuration
INFO[0001] Attempting to download the image from "~/Downloads/impish-server-cloudimg-arm64.img"  digest=
INFO[0001] Attempting to download the image from "https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-arm64.img"  digest=
539.62 MiB / 539.62 MiB [----------------------------------] 100.00% 40.59 MiB/s
INFO[0016] Downloaded image from "https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-arm64.img" 
INFO[0016] Attempting to download the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v0.17.0/nerdctl-full-0.17.0-linux-arm64.tar.gz"  digest="sha256:b0ae2fc89d362afa18afcc35798d4da613096613d06cae07181b777cd66002e8"
173.82 MiB / 173.82 MiB [----------------------------------] 100.00% 29.51 MiB/s
INFO[0023] Downloaded the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v0.17.0/nerdctl-full-0.17.0-linux-arm64.tar.gz" 
INFO[0024] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/matek/.lima/default/serial.log") 
INFO[0024] SSH Local Port: 60022                        
INFO[0024] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0110] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0195] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0280] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0365] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0450] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0536] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0621] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
FATA[0623] did not receive an event with the "running" status 

serial.log

Fedora seems working:

❯ limactl start
? Creating an instance "default" Open an editor to override the configuration
INFO[0039] Attempting to download the image from "https://download.fedoraproject.org/pub/fedora/linux/releases/35/Cloud/aarch64/images/Fedora-Cloud-Base-35-1.2.aarch64.qcow2"  digest="sha256:c71f2e6ce75b516d565e2c297ea9994c69b946cb3eaa0a4bbea400dbd6f59ae6"
359.94 MiB / 359.94 MiB [----------------------------------] 100.00% 23.11 MiB/s
INFO[0057] Downloaded image from "https://download.fedoraproject.org/pub/fedora/linux/releases/35/Cloud/aarch64/images/Fedora-Cloud-Base-35-1.2.aarch64.qcow2" 
INFO[0057] Attempting to download the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v0.17.0/nerdctl-full-0.17.0-linux-arm64.tar.gz"  digest="sha256:b0ae2fc89d362afa18afcc35798d4da613096613d06cae07181b777cd66002e8"
173.82 MiB / 173.82 MiB [----------------------------------] 100.00% 27.12 MiB/s
INFO[0064] Downloaded the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v0.17.0/nerdctl-full-0.17.0-linux-arm64.tar.gz" 
INFO[0065] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/matek/.lima/default/serial.log") 
INFO[0065] SSH Local Port: 60022                        
INFO[0065] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0085] [hostagent] The essential requirement 1 of 5 is satisfied 
INFO[0085] [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh" 
INFO[0085] [hostagent] The essential requirement 2 of 5 is satisfied 
INFO[0085] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed" 
INFO[0106] [hostagent] The essential requirement 3 of 5 is satisfied 
INFO[0106] [hostagent] Waiting for the essential requirement 4 of 5: "/etc/fuse.conf to contain \"user_allow_other\"" 
INFO[0110] [hostagent] The essential requirement 4 of 5 is satisfied 
INFO[0110] [hostagent] Waiting for the essential requirement 5 of 5: "the guest agent to be running" 
INFO[0110] [hostagent] The essential requirement 5 of 5 is satisfied 
INFO[0110] [hostagent] Mounting "/Users/matek"          
INFO[0110] [hostagent] Mounting "/tmp/lima"             
INFO[0110] [hostagent] Forwarding "/run/lima-guestagent.sock" (guest) to "/Users/matek/.lima/default/ga.sock" (host) 
INFO[0110] [hostagent] Waiting for the optional requirement 1 of 2: "systemd must be available" 
INFO[0110] [hostagent] Forwarding TCP from 0.0.0.0:5355 to 127.0.0.1:5355 
INFO[0110] [hostagent] The optional requirement 1 of 2 is satisfied 
INFO[0110] [hostagent] Waiting for the optional requirement 2 of 2: "containerd binaries to be installed" 
INFO[0110] [hostagent] Not forwarding TCP 127.0.0.53:53 
INFO[0110] [hostagent] Not forwarding TCP 0.0.0.0:22    
INFO[0110] [hostagent] Forwarding TCP from [::]:5355 to 127.0.0.1:5355 
WARN[0110] [hostagent] failed to set up forwarding tcp port 5355 (negligible if already forwarded) 
INFO[0110] [hostagent] Not forwarding TCP [::]:22       
INFO[0110] [hostagent] The optional requirement 2 of 2 is satisfied 
INFO[0110] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished" 
INFO[0126] [hostagent] The final requirement 1 of 1 is satisfied 
INFO[0126] READY. Run `lima` to open the shell. 

serial.log

[AkihiroSuda]

This might be a regression in https://cloud-images.ubuntu.com/impish/20220309/ ?

Does https://cloud-images.ubuntu.com/impish/20220201/ work?