[kernel v6.2 regression, affects Ubuntu 23.04 and Fedora 38] limactl start --set '.vmType = "vz"' crashes: "usernet unable to resolve IP for SSH forwarding"

[AkihiroSuda]

EDIT
This seems to be a regression in kernel 6.2, reported to https://bugzilla.kernel.org/show_bug.cgi?id=217485 .
kernel 6.3 seems to be bootable (via GRUB).

• Report to kernel bugzilla https://bugzilla.kernel.org/show_bug.cgi?id=217485
• Report to LKML https://lkml.org/lkml/2023/5/27/252
• Report to Apple to fix Virtualization.framework (reported via https://www.apple.com/feedback/macos.html , but no bug ticket is available)
• Report to Ubuntu to cherrypick torvalds/linux@8e41e0a into Ubuntu 23.04 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2021403
• Wait for Ubuntu to cherrypick torvalds/linux@8e41e0a
• Report to Fedora to cherrypick torvalds/linux@8e41e0a into Fedora 38 (Fedora usually do not update qcow2 images, so probably we should just wait for Fedora 39, which is likely to use kernel >= 6.3)
• Wait for Fedora to cherrypick torvalds/linux@8e41e0a

---

$ limactl start --set '.vmType = "vz"'
WARN[0000] `--set` is experimental                      
? Creating an instance "default" Proceed with the current configuration
WARN[0003] `vmType: vz` is experimental                 
WARN[0003] Ignoring: vmType vz: [CPUType]               
INFO[0003] Attempting to download the image              arch=x86_64 digest="sha256:13965c84c65cbab0b34326ac34ac0c47a88030f9dff80e6391e56cb9077cadd0" location="https://cloud-images.ubuntu.com/releases/23.04/release-20230502/ubuntu-23.04-server-cloudimg-amd64.img"
INFO[0003] Using cache "/Users/suda/Library/Caches/lima/download/by-url-sha256/74b79a035df1cdfbe81415cae56f2b86377b8d62c75937cdd415e18fbc482a1c/data" 
INFO[0007] Attempting to download the nerdctl archive    arch=x86_64 digest="sha256:3bb5f5358ee2c3bd9097e0bf37649c1775e2449094e75acf629f129ec2c7915f" location="https://github.com/containerd/nerdctl/releases/download/v1.4.0/nerdctl-full-1.4.0-linux-amd64.tar.gz"
INFO[0007] Using cache "/Users/suda/Library/Caches/lima/download/by-url-sha256/7f7d5c10ef2e2b350e26ceca17d2e3a7ce9993c867ce53f0a1114aa0264bfb4b/data" 
INFO[0009] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/suda/.lima/default/serial.log") 
INFO[0009] [hostagent] new connection from  to          
INFO[0009] SSH Local Port: 60022                        
INFO[0009] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0009] [hostagent] [VZ] - vm state change: running  
INFO[0019] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0029] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0039] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0049] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0059] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0069] [hostagent] Driver stopped due to error: "usernet unable to resolve IP for SSH forwarding" 
INFO[0069] [hostagent] Shutting down the host agent     
WARN[0069] [hostagent] failed to exit SSH master        
INFO[0069] [hostagent] Shutting down VZ                 
FATA[0069] exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/suda/.lima/default/ha.stderr.log") 

• Lima: master (ffd57dc)
• macOS: 13.4 (Intel)

template://experimental/vz still works.

---

(Slightly off-topic: a patch for loading kernel >= 6.2 using VZLinuxBootLoader, without GRUB)

• Submitted a kernel patch https://lore.kernel.org/linux-efi/CAG8fp8Te=oT1JJhTpOZvgWJrgcTq2DXan8UOVZ=KYCYNa8cKog@mail.gmail.com/
• Wait for the kernel patch to be merged: torvalds/linux@36e4fc5 (commit message was modified and slightly inaccurate)

[balajiv113]

Checking now, i believe its related to the preset port in default template.

[balajiv113]

Default template itself not loading. I tried this Ubuntu version with vz template it worked there.

So something related to some config set in default. will try to close this by tomorrow.

[AkihiroSuda]

The issue seems specific to Ubuntu 23.04.
22.10 works 🤔

[balajiv113]

Its related to Linux kernel 6.2
utmapp/UTM#5138

[balajiv113]

I could confirm its working in M1 (MacOS 13.3).
Looks like the impact is only with intel mac

[AkihiroSuda]

Seems to be a regression in the merge commit torvalds/linux@888bc86 (Merge branch 'acpica')

$ git log v6.1-rc8..888bc86e --oneline --graph
*   888bc86e7cca (HEAD) Merge branch 'acpica'
|\  
| * 470188b09e92 ACPICA: Fix use-after-free in acpi_ut_copy_ipackage_to_ipackage()
| * 404ec60438ad ACPICA: Fix error code path in acpi_ds_call_control_method()
| * 2b6bab689172 ACPICA: Update version to 20221020
| * 4f4356e6b4f2 ACPICA: Add utcksum.o to the acpidump Makefile
| * f6fc0bf2be79 Revert "LoongArch: Provisionally add ACPICA data structures"
| * 51aad1a6723b ACPICA: Finish support for the CDAT table
| * 3f062a516a63 ACPICA: IORT: Update for revision E.e
| * f350c68e3cd5 ACPICA: Add CXL 3.0 structures (CXIMS & RDPAS) to the CEDT table
| * 183f0a09d32c ACPICA: Improve warning message for "invalid ACPI name"
| * ee64b827a9af ACPICA: Add support for FFH Opregion special context data
| * e92e4a451c0c ACPICA: Add a couple of new UUIDs to the known UUID list
| * 407144ebd445 ACPICA: iASL: Add CCEL table to both compiler/disassembler
| * 8ff2906513f5 ACPICA: Do not touch VGA memory when EBDA < 1ki_b
| * 4fe54f509304 ACPICA: Check that EBDA pointer is in valid memory
| * 5c62d5aab875 ACPICA: Events: Support fixed PCIe wake event
| * 60f2096b59bc ACPICA: MADT: Add loong_arch-specific APICs support
| * 5620fe641620 ACPICA: Make acpi_ex_load_op() match upstream
* 57336224da83 ACPI: thermal: Adjust critical.flags.valid check

Will try to bisect further

[AkihiroSuda]

Turned out to be a regression in torvalds/linux@5c62d5a ACPICA: Events: Support fixed PCIe wake event
(Ported from acpica/acpica@32d8757)

This commit was introduced in v6.2-rc1, and apparently reverted in v6.3 (torvalds/linux@8e41e0a).
However, v6.3 and the latest v6.4-rc3 still don't boot 🤔

[balajiv113]

However, v6.3 and the latest v6.4-rc3 still don't boot

This worked for me. I did the following,

• Boot ubuntu 23.04 via QEMU
• Update kernel to v6.4-rc3 (Steps are download all deb files https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.4-rc3/amd64/ and install using dpkg)
• Reboot
• Boot from vz (Boot was successful here)

Note: I used raw format disk even with QEMU so that it can boot on vz as well

[AkihiroSuda]

@balajiv113 Can you try the vanilla defconfig?

My test steps are:

• Checkout the kernel repo and make defconfig bzImage
• Create initrd:

#!/busybox sh
# Usage: save this script as "init", and run the following commands:
#   wget https://busybox.net/downloads/binaries/1.35.0-x86_64-linux-musl/busybox
#   chmod +x busybox init
#   find . | cpio --quiet -o -H newc | gzip >../initrd.img

set -eux
/busybox mkdir -p /etc /proc /root /bin /sbin /sys /usr/bin /usr/sbin
/busybox mount -t proc proc /proc
/busybox mount -t sysfs sys /sys
/busybox mdev -s
/busybox --install
exec sh

• Run VMLINUZ_PATH=~/tmp/bzImage INITRD_PATH=~/tmp/initrd.img DISKIMG_PATH=/dev/null ./virtualization with https://github.com/Code-Hex/vz/blob/v3.0.6/example/linux/main.go

My host is MacBookPro 2020 (Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz) running macOS 13.4

[AkihiroSuda]

Update kernel to v6.4-rc3 (Steps are download all deb files https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.4-rc3/amd64/ and install using dpkg)

vmlinuz-6.4.0-060400rc3-generic in https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.4-rc3/amd64/linux-image-unsigned-6.4.0-060400rc3-generic_6.4.0-060400rc3.202305212230_amd64.deb doesn't work either for me.
Just tried with https://github.com/Code-Hex/vz/blob/v3.0.6/example/linux/main.go though.

[AkihiroSuda]

Reported to the kernel bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=217485

[AkihiroSuda]

( Cross-posting: https://bugzilla.kernel.org/show_bug.cgi?id=217485#c6 )

Turned out that this is a mixture of an ACPICA issue and an EFISTUB issue.

Kernel v6.2 can boot by reverting the both of the following two commits:

• torvalds/linux@5c62d5a "ACPICA: Events: Support fixed PCIe wake event"
• torvalds/linux@e346beb "efi: libstub: Always enable initrd command line loader and bump version"

Kernel v6.3 can boot by just reverting torvalds/linux@e346bebb, as torvalds/linux@5c62d5a has been already reverted in torvalds/linux@8e41e0a.
The situation is same for v6.4-rc3 too.

Note that in my test I let Virtualization.framework directly load bzImage without GRUB (akin to qemu-system-x86_64 -kernel bzImage).
Apparently, reverting torvalds/linux@e346bebb is not necessary for loading bzImage via GRUB.
( So, Lima can just boot unmodified v6.3 and v6.4-rc3: #1577 (comment) )

[AkihiroSuda]

This seems resolved on macOS 13.5 🎉

[mlavi]

@AkihiroSuda Thanks for your work on this! Could you tell us what changed or was fixed in MacOS 13.5?

[AkihiroSuda]

@AkihiroSuda Thanks for your work on this! Could you tell us what changed or was fixed in MacOS 13.5?

I don't know.
Can't find any reference in https://developer.apple.com/documentation/macos-release-notes/macos-13_5-release-notes
But apparently Apple updated Virtualization.framework in macOS 13.5 to support Linux 6.2 on Intel.