Use lima as a driver

[Patrick0308]

Steps to reproduce the issue:
I want to use lima instead of docker on mac. Can I use lima as a minikube driver?

Run minikube logs --file=logs.txt and drag and drop the log file into this issue

Full output of failed command if not minikube start:

Details

[afbjorklund]

There are some third parties already doing this: https://github.com/abiosoft/colima <-- now using k3s

Ultimately we might want to have a QEMU driver for Mac, or reuse the current libvirt driver called "kvm2"

The main problem is with allocating the network, currrently lima is doing the ssh hacks we rejected earlier:

• Issue [FEATURE] Adding support for additional driver (QEMU/KVM) #2399

---

Note that the default container engine in lima is containerd/buildkitd, which is not yet supported by minikube (or kind)

So you will need to change that for either docker or podman, in order to run. But there are example yaml files of both...

• examples/docker.yaml

• examples/podman.yaml

[afbjorklund]

I think Docker's HyperKit is already deprecated, being replaced with Apple's Virtualization.framework

Unfortunately the needed QEMU patches for arm64 (M1) are not merged yet, but it works on amd64...

There is an -accel hvf flag when starting it, that works the same way as -accel kvm does on Linux.

See upstream lima for how to build a patched qemu version, there is a third-party brew repository for it.

[Patrick0308]

@afbjorklund Ok, I get it.

Unfortunately the needed QEMU patches for arm64 (M1) are not merged yet

Can you link the issue or pr here? I am using the M1 mac now.

[afbjorklund]

They are not available in the QEMU releases yet, so one needs a special qemu branch:

https://github.com/lima-vm/lima#installation

Manual installation steps (required for ARM Mac)

brew install simnalamburt/x/qemu-hvf

[afbjorklund]

This app is also looking promising, as a GUI instead of VirtualBox: https://mac.getutm.app/

There is also a cross-platform version called AQEMU: https://github.com/tobimensch/aqemu

---

Note that with these solutions, you end up with a virtual machine (VM) running on your Mac.

So you still need to configure kubectl on the Mac, to talk to the minikube running on it...

• Either you can use this new virtual desktop to do the Kubernetes development on, instead.
  Both Docker and Podman are supposed to run out-of-the-box, if using for instance Ubuntu.

• Or you can use something like the "ssh" minikube driver together with the Linux VM perhaps.
  Then you don't need a full GUI desktop, but can do a headless server OS installation instead.

[afbjorklund]

Using lima is similar to using WSL2 on Windows, it will handle the VM and network and host mounts for you.

Lima can be considered as a some sort of unofficial "macOS subsystem for Linux", or "containerd for Mac".

Even though intended for macOS, you can run lima also on Linux. This is good for the developers, such as myself.

The mounted filesystems have the same performance problems as the other solutions, and the networking is limited.

[afbjorklund]

If you want to run multi-node or complex networking, or want full control over storage, I would recommend using a VM driver.

But for casual users, these drivers are handy. I call it the "Wizard of Oz mode", pay no attention to the man behind the curtain!

[medyagh]

I am not familiar with lima, but if it is just like docker or podman, and can use standard OCI image, I would accept a contribution that adds it as another minikube driver

[afbjorklund]

I am not familiar with lima, but if it is just like docker or podman

It is similar to WSL. It uses containerd/buildkitd by default, though.

See kubernetes-sigs/kind#2317 for a discussion about nerdctl
(for a "containerd-in-containerd" driver, similar to DinD and CinP)

But Lima can use any distribution with any container runtime...