NixOS guest?

[LATBauerdick]

I would really love to have a NixOS guest -- at the moment I use lima with the nix package manager on an archlinux guest. I think in theory NixOS should work, looking at the stated requirements. I also have a qemu NixOS VM, that works just fine. It does not have the great lima integration features, so I really want it running from lima.

I guess the best way forward would be to just go ahead and do it myself, but I am new to lima and could not (easily) find a thread how to start. Are there some example instructions somewhere, or where should I look to e.g. how it was done for archlinux or Debian?

Any pointers are much appreciated.

[jandubois]

I guess the best way forward would be to just go ahead and do it myself

Yes, it would! 😄

The first step is to find a somewhat stable/official location to download cloud images from.

The second step is implementing support for the package manager in pkg/cidata/cidata.TEMPLATE.d/boot/30-install-packages.sh.

The third step is trying it out, checking if something is broken, and fixing it. 😈

[ghost]

I am also interested in a NixOS guest and may be able to provide some development.

I did try this with the minimal 64-bit iso image I found at NixOS' download.

It did not work correctly and I did not see anything in serial.log.

From what I researched NixOS itself has a way to build isos. We could leverage that to build a pared down image with the appropriate configuration to integrate with lima.
This comment gave me that idea: https://discourse.nixos.org/t/how-to-build-a-standalone-nixos-qemu-vm/5688/3

So we could try step 2 and 3 of what @jandubois notes!

[LATBauerdick]

great! I think I already roughly know how to do the first step, to create a working qemu image and install additional stuff, via a script, starting from the official/current NixOS .iso image (pulling it from the NixOS archive).

I will put this together over the holidays and report back.

I am however pretty hazy about the next steps, and will have to look at the repository in detail to see how it works with alpine or arch linux and go from there.

I am far from an expert though, help will be much appreciated!

[jandubois]

Just a quick note that there are 2 different scenarios: booting from an image vs. booting from an ISO. We latter is only done for Alpine right now, and we build a custom ISO to make sure it includes cloud-init (or our own lima-init replacement). Most ISOs are either install or rescue media and do not support this.

There is special code in Lima to mount a persistent data volume and move some directories to it, so they survive a reboot. This code probably will need some adjustments if you want to boot NixOS from an ISO instead of a regular image.

Let me know if you run into any issues, and I'll try to help out!

[afbjorklund]

Apparently NixOS thinks that cloud-init is too bloated to go into the default image, but it has been packaged:

https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/system/cloud-init.nix

• Issue Support user-data in minimal ISO image (Was: Add cloud-init service to minimal ISO) NixOS/nixpkgs#39076

I don't think it is the python3 itself that is the biggest bloat, but it (ci) does have a truckload of dependencies...

[jandubois]

Apparently NixOS thinks that cloud-init is too bloated to go into the default image, but it has been packaged:

If you build you own image, you could also use the lima-init I wrote for our Alpine ISO instead. I it just a shell script and some awk code, and supports everything needed by Lima from cloud-init. It may need some extra work to make it compatible with nix.

[afbjorklund]

You have to fork cloud-init as well, since the list of distros is hardcoded within... So it might be easier to start with lima-init

[vielmetti]

Also registering interest in this development!

[patryk4815]

Today i created my nixos image https://github.com/patryk4815/ctftools/tree/master/lima-vm
And it is working with limactl :)

[msgilligan]

@patryk4815 wrote:

My "image" does not use cloud-init, I manually rewrote cloud-init to ".nix module" and in runtime/everyboot fetch data from the file "/mnt/lima-cidata/user-data" and "/mnt/lima-cidata/lima.env".

Why did you choose to do it this way rather than directly use cloud-init?

[msgilligan]

Do you want to support this image upstream?

I think this would only be possible if the images were either published by the Nix org itself, or alternatively build by a lima-vm org repo and published by GitHub action to lima-vm GitHub releases. And then it would still require a Lima maintainer to agree to provide ongoing maintenance.

The ability to run and configure NixOS in Lima VMs is important to me. I just got @patryk4815's solution working and need to test it out a little more and learn more about how it works. I would potentially be interested in helping to make this a third-party add-on for Lima (since it is unlikely we'll be able to get it published by Nix.org or by the Lima maintainers.)

The first steps would be giving it a dedicated repository with its own top-level README, adding documentation and making minor improvements. I'm not sure if it makes sense to put something like this in nixpkgs, but that could also be an option.

I've already made one change to the aarch64.yml config file, I added:

ssh:
  forwardAgent: true

which works well. (And is one of my favorite Lima features)

This week I plan on experimenting with the NixOS VM that I build (and perhaps rebuilding it, etc.) by using it to work on some of my projects and seeing if I can use it to build another NixOS/Lima VM, etc.

I would also like to understand the best ways to customize the configuration of NixOS/Lima VMs. There are at least three approaches I can think of:

1. Modify or override the various .nix files in the ctftools/lima-vm directory before the nixos.img is generated.
2. Use some (ideally Nix-based) mechanism to customize the VM during the boot/provisioning phase of Lima
3. Use NixOS generation in the VM after it is started

One example of something I would like to change/customize is to add the "nix-command" and "flakes" features automatically. I would also like to be able to build VMs with particular services installed, etc.

[msgilligan]

@patryk4815 Are you interested in improving or maintaining your solution? I find it useful enough to want to spend at least some time moving it forward. Should I just create a new repository and give you credit, or do you (or anyone else) want to participate?

[sam-goodwin]

The img is 3.2G, is that expected? The official NixOS ISO is only 1GB.

[msgilligan]

The img is 3.2G, is that expected? The official NixOS ISO is only 1GB.

The one I built was 3250585600 bytes.

[msgilligan]

I have found two projects for building Lima-compatible NixOS images on GitHub that are inspired by @patryk4815 's work:

• https://github.com/kasuboski/nixos-lima ("Heavily inspired from patryk4815/ctftools") Blog post: NixOS Dev Environment on Mac
• https://github.com/quinneden/nixos-lima-vm ("Forked from kasuboski/nixos-lima, still mostly identical, slight modification in the user configuration and lima runtime.")

[sam-goodwin]

https://github.com/kasuboski/nixos-lima - this one didn't work for me. It just hung on SSH access.

This fork worked: https://github.com/unidevel/nixos-lima. Minor tweaks, mostly to use qcow2 format (reducing image size to 2.4GB vs >3GB). Not sure why it works and the original doesn't.

[unidevel]

Fixed by https://github.com/unidevel/nixos-lima/blame/main/lima-init.nix#L53, make user-data mounted correctly to make lima's ssh configuration working as expected.

[msgilligan]

Back in July I wrote:

I would potentially be interested in helping to make this a third-party add-on for Lima (since it is unlikely we'll be able to get it published by Nix.org or by the Lima maintainers.)

My nixos Lima VM that I installed with @patryk4815 's build is now broken -- DNS is not working in the guest. I'm not sure what broke it as I have upgraded both NixOS inside the guest as well as Lima (using Nix Home Manager) on the host. I didn't notice the problem until I restarted the guest VM.

There are now at least 4 repositories listed above based on @patryk4815 's original work each with some user-specific customization and various fixes. I found a few more on GitHub. Some have added what look like useful features, but IMHO none of them looks like an ongoing project that other people can use and/or contribute to. Several look like they have made useful enhancements but IMHO none of them meet the criteria I am looking for to build a NixOS LimaVM that I could use for serious work. My criteria are:

1. Multiple contributors via PR
2. Complete User documentation (how to install, customize, upgrade)
3. Issues tab is enabled and issues get responses
4. Developer documentation (explanation of how the modules work, how to contribute, etc)
5. CI (GitHub actions or other)

Now it's easy to be a critic. The people developing these modules clearly know much more about Nix than I and they have made no promises of support and are clearly under no obligation. And I have limited bandwidth so I don't think I would be able to create neat another repository to solve the issues.

I'm just hoping that maybe a few of us can get together and combine our efforts to produce something the community can use and can be maintained long term. Looking at the individual efforts made on the various forks seems to indicate there is enough interest and talent to make this happen.

To this Nix novice, it seems like the best approach would be to create (one or more) flakes/modules that can be used to do the core Lima-related magic necessary to emulate cloud-init. This/these module(s) would need to work in two contexts:

1. In user-defined flakes used to build a Lima-bootable system image (e.g. .qcow)
2. In the /etc/nixos directory of the Lima guest VM so that the system can be updated with nixos-rebuild.

There could then be a separate template repo/flake that shows how to build a starter Lima-compatible image. Decoupling the module(s) from the starter/template repo would make it easier for people to customize to their hearts content, but be able to share the Lima modules and contribute upstream to improve them.

Does this make sense? (I'm going to try tagging various people that have forked @patryk4815 's repo and see if we can get some discussion here...)

[msgilligan]

Most of these seem to be broken with the latest release of Lima because LIMA_CIDATA_COMMENT was added to lima.env and this field typically contains a space. The line:

        # ripped from https://github.com/lima-vm/alpine-lima/blob/main/lima-init.sh
        . "${LIMA_CIDATA_MNT}"/lima.env

seems to be the problem. It was fixed upstream with: lima-vm/alpine-lima@34b03da

[jandubois]

Yes, lima.env is in the same format as /etc/environment. It cannot be sourced directly by the shell, but needs to be read/processed. The same is true for param.env in the latest Lima releases.

[msgilligan]

Add https://github.com/charles-dyfis-net/nixos-lima to the list. @charles-dyfis-net has a series of useful commits from 2 days ago.

[msgilligan]

I have created a fork of @kasuboski 's repository, with several essential patches applied that allow building, running, and rebuilding (using the current nixos-unstable branch). I've created one-fix-per-commit changes in the msgilligan-main branch and updated the README to explain why I created yet-another fork and asking for collaborators/contributors.

https://github.com/msgilligan/nixos-lima

As I say in the README:

If you have input or want to collaborate, please comment on Discussion #430 or open an issue or pull request on https://github.com/msgilligan/nixos-lima.

and:

I'm also happy to archive this project and contribute to another one if other collaborators think that is a better path forward.

[msgilligan]

I created a new repository https://github.com/msgilligan/nixos-lima-config-sample that uses the lima module from the Flake in https://github.com/msgilligan/nixos-lima and allows you to manage the configuration of your NixOS Lima instance separately from the nixos-lima repository.

I hadn't seen this done before, but I believe @kasuboski created the lima module for this reason. This is my first time creating and sharing a NixOS module, so please forgive any rookie mistakes or sloppiness. It does seem to work for me, and I'm looking forward to receiving any feedback.

UPDATE there is now a GitHub organization and I've created a top-level comment to create a new thread about this:
#430 (comment)

[msgilligan]

I've created a nixos-lima organization based upon the work of @patryk4815 @kasuboski and others.

It contains a fork of @kasuboski 's repo that is not a GitHub fork: https://github.com/nixos-lima/nixos-lima

I have also split-out a "sample" of user-specific configuration that re-uses the systemd services defined in the main repository and can be used to customize an instance bootstrapped from the generated "generic" nixos-lima image: https://github.com/nixos-lima/nixos-lima-config-sample

I am hoping some of the creators of the other forks will be willing to submit patches as they all seem to know more about Nix than I do.

[msgilligan]

In Issues I've created a list of desired/possible enhancements: https://github.com/nixos-lima/nixos-lima/issues some of which exist on other forks.

@kasuboski @unidevel @patryk4815

[msgilligan]

It is based upon @kasuboski's last commit to main, but has the following additions:

• Fixes for latest Nixpkgs
• Fixes for latest Lima
• Added LICENSE file (MIT, same as Nixpkgs)
• Removal of user specific configuration
• Separate repository that demonstrates how to reference main flake and add user-specific configuration
• services.lima.enable configuration option to enable the module
• Fix to handle spaces in lima.env
• GitHub action workflow to build the system image
• GitHub action workflow that tests nixos-lima-config-sample
• Experimental Support for Lima's system and user provisioning scripts
• System image format changes to qcow format (for smaller size)

[msgilligan]

We are making progress on VZ support in nixos-lima.

VZ support was literally Issue number 1 for nixos-lima -- see nixos-lima/nixos-lima#1

This morning we received PR nixos-lima/nixos-lima#10 with two changes (in 2 separate commits.) Both are needed to get VZ working. The first has been added to master, but the second (adding --vsock-port 2222 to lima-guestagent daemon) fixes VZ while breaking QEMU, so we need to figure out a mechanism for making it conditional upon the guest being run under VZ. (This should be decided at run-time of course, so the same .qcow image can be used with VZ or QEMU.) Is there a recommended way to do that?

Update: We found variables in lima.env that we can use: nixos-lima/nixos-lima#1 (comment)

[msgilligan]

VZ support is merged to master on nixos-lima!

[msgilligan]

The following should now work:

limactl start https://raw.githubusercontent.com/nixos-lima/nixos-lima/master/nixos.yaml

This is experimental/pre-release, but I have been using this basic configuration as my daily-driver Lima NixOS VM on my aarch64 MacBook Pro for some time now.

[jandubois]

Thanks! Works for me...

I know this is your personal config; if you want to share it, I would prefer that the agent forwarding remains disabled, so users have to explicitly opt in, for security reasons.

I also don't understand the point of the provisioning scripts; they seem to be empty.

[msgilligan]

I know this is your personal config;

It is not meant to be my personal config, see Design Goals -- number 3 and 4 are:

3. User customization of NixOS Lima instance is done separately from initial image creation
4. Keep the base image and Nix services module as generic and reusable by others as possible

I would prefer that the agent forwarding remains disabled, so users have to explicitly opt in, for security reasons.

I opened an issue to remind me to remove it: nixos-lima/nixos-lima#18 -- update fixed/merged.

I also don't understand the point of the provisioning scripts; they seem to be empty.

They are just a placeholder. I agree, they should be removed: nixos-lima/nixos-lima#19 -- update fixed/merged.

[msgilligan]

Update: nixos-lima has had two more releases, usage has been simplified, and the README has been significantly improved. (Note: the URL in #430 (comment) has been updated/shortened)

If you're interested in using or supporting nixos-lima, you can help by:

• Visit the https://github.com/nixos-lima/nixos-lima project and "star" the repo.
• Use or test nixos-lima and provide feedback
• Publish a sample configuration or show how you use Nix deployment tools to deploy or rebuild your images
• Submit a PR with an improvement to the code
• Submit a PR with an improvement to the documentation

[msgilligan]

I have put a substantial amount of effort into having CI for the images, but am having trouble testing aarch64 images on my free/public GitHub project. I just opened a separate "Discussion" for this issue to see if anyone has had any luck or has any suggestions: How to test aarch64 templates / images on GitHub Actions #3934

Update: I got it all working -- see #3934 (comment)

[msgilligan]

I have made a v0.0.4 release of nixos-lima and it now uses the nixos-25.11 stable branch and contains a few other fixes/improvements.

The repository is updated to support the new GitHub Template URLs feature, so spinning up a new NixOS guest is as easy as:

limactl start github:nixos-lima

(Thanks to @jandubois and others for their work on the Template URL feature!)