Flatcar Container Linux support

[louhisuo]

Description

Flatcar Container Linux (https://www.flatcar.org/) is an interesting immutable Linux OS which suits well to run e.g. Kubernetes. Any change support for Flatcar Linux can be added to Lima as it does not seem to work "out-of-the-box" based on quick testing done.

[computeralex92]

A switch to ignition is not needed, as Flatcar is still able to be configured via cloud config:
https://github.com/flatcar/coreos-cloudinit/blob/flatcar-master/Documentation/cloud-config.md

It is only not anymore the main focus of development and only a subset of options is supported; but anyways the support is still there.
I'm using this in production environments and was able to reuse config for the original CoreOS which was more than 4 years old; without any issue.

[afbjorklund]

Are you able to provide a YAML file, for the "stable" image ?

https://www.flatcar.org/docs/latest/installing/vms/qemu/

Typically the cloud-config user-data looks something like:

#cloud-config
# vim:syntax=yaml

growpart:
  mode: auto
  devices: ['/']

users:
  - name: "anders"
    uid: "1000"
    homedir: "/home/anders.linux"
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: true
    ssh-authorized-keys:
      - ... keys go here ...

write_files:
 - content: |
      #!/bin/sh
      set -eux
      LIMA_CIDATA_MNT="/mnt/lima-cidata"
      LIMA_CIDATA_DEV="/dev/disk/by-label/cidata"
      mkdir -p -m 700 "${LIMA_CIDATA_MNT}"
      mount -o ro,mode=0700,dmode=0700,overriderockperm,exec,uid=0 "${LIMA_CIDATA_DEV}" "${LIMA_CIDATA_MNT}"
      export LIMA_CIDATA_MNT
      exec "${LIMA_CIDATA_MNT}"/boot.sh
   owner: root:root
   path: /var/lib/cloud/scripts/per-boot/00-lima.boot.sh
   permissions: '0755'
# This has no effect on systems using systemd-resolved, but is used
# on e.g. Alpine to set up /etc/resolv.conf on first boot.

manage_resolv_conf: true

resolv_conf:
  nameservers:
  - 192.168.5.3


ca-certs:
  remove_defaults: false
  trusted:

Not sure how well the scripts work, on a read-only filesystem ?

[afbjorklund]

Currently there is no support for compressed images in Lima, so the .bz2 needs to be decompressed first.

• Support compressed source images ("http://domain/image.qcow2.gz") #809

https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img.bz2

The image boots, but it doesn't set up the ssh login it seems. From the ssh-authorized-keys in cidata.iso

localhost login: core (automatic login)

Flatcar Container Linux by Kinvolk stable 3374.2.5 for QEMU
core@localhost ~ $ 

---

The /dev/cdrom was found, but never mounted anywhere.

Probably because /var/lib/cloud was never created, either.

It seems like cloudinit only runs for some OEMS, or on-demand ?

Otherwise there is a custom shell script, to set up the qemu cmd:

https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu.sh

[afbjorklund]

The issue was not the OEM, it was that the labels and paths for user-cloudinit are hardcoded...

• The label of the CD is "config-2", not "cidata"

• The mount path of it is "/media/configdrive", not "/mnt/lima-cidata"

• The name of the file is "openstack/latest/user_data", not "user-data"

---

Once Flatcar finds the cloud-init config, the user is created and the ssh keys are setup for lima.

But the rest of the instance configuration is failing.

[AkihiroSuda]

@afbjorklund Thanks for taking a look, it looks like these files need to be modified?
https://github.com/flatcar/coreos-cloudinit/search?q=%2Fmedia%2Fconfigdrive

[afbjorklund]

Maybe they (the locations) can be configured during runtime somehow, env or cmdline ?

But it seems that it only has a partial cloud-init implementation, no /var/lib/cloud etc

[afbjorklund]

Apparently there is a special image for UEFI, noticed when looking at the arm64 variant.

Debug setup:

images:
- location: "https://stable.release.flatcar-linux.net/amd64-usr/3374.2.5/flatcar_production_qemu_uefi_image.img.bz2"
  arch: "x86_64"
  digest: "sha512:3fe0068f838f8055fbdc374e9d59ab6828d79f1d1d3c0b93fe6b89893ff06d1cbc4acdd19c0c6a54e7462cf86cdd29340eaa2ed6ad3fd114c552f43bce526718"
- location: "https://stable.release.flatcar-linux.net/arm64-usr/3374.2.5/flatcar_production_qemu_uefi_image.img.bz2"
  arch: "aarch64"
  digest: "sha512:0f4ee08e9a17ece104d477cbc1ab0eda6f410683aea0067779771587c3864df9f3527c1c647158064424b33918fd5bbabf6384e9d71ef4a98a672565d154aba8"

video:
  display: "default"

[afbjorklund]

Using Fedora CoreOS seemed to be slightly better documented, and it was used by Podman Desktop

It also allows for installing various things in /usr/local, by llinking that directory to /var/usrlocal...

EDIT: Needed feature for installing into /opt instead:

• Add optional prefix parameter to the template #1654

[afbjorklund]

Flatcar doesn't have support for sshfs (or fuse), so it will have to use mountType: "9p" instead.

On a normal operating system you could install it, but that is not possible on an immutable OS.

[afbjorklund]

Not sure that we want to support this (using ignition and butane, instead of cloud-init)

[kouhaidev]

Not sure that we want to support this (using ignition and butane, instead of cloud-init)

Any given reason? This seems interesting.

[afbjorklund]

No interest, duplication of the scripts?

• cloud-init -> ignition.yaml + butane
• boot/* -> ignition.yaml (systemd)

The main user (of CoreOS) is Podman, but they are set on doing their own "machine"...
Automatic updates is a nice feature in theory, but they even disabled that feature eventually.
Since it meant that the VM would spontaneously reboot, in order to "activate" the new install.
https://www.flatcar.org/docs/latest/setup/releases/update-strategies/ (default is reboot in 5m)

[MayCXC]

I'm super interested in this, but I can understand why maintaining ignition.yaml is unappealing.

[afbjorklund]

Most of the cloud-init and ignition/butane is static, so it is actually the OS that is unappealing.

---

I mean it being (mostly) read-only, and requiring a reboot for each (system) installation.
Or that the image is much bigger than the regular OS, or requiring special OCI repositories.

Things like that? I'm sure there is some use case where it fits, I just don't like it much myself.
But some of the new OS (like Talos) even go all the way, and remove ssh and bash completely...