Error when trying to run pihole container on UDM SE

[rpletka]

Describe the bug
When i follow the instructions for pihole I get the following error on step 5
root@UDMSE:/data/podman/cni# podman run -d
--network dns
--restart always
--name pihole
-e TZ="America/Chicago"
--cap-add=NET_ADMIN
-v "/data/etc-pihole/:/etc/pihole/"
-v "/data/pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/"
--dns=127.0.0.1
--dns=1.1.1.1
--dns=8.8.8.8
--hostname pi.hole
-e VIRTUAL_HOST="pi.hole"
-e PROXY_LOCATION="pi.hole"
-e FTLCONF_REPLY_ADDR4="10.0.0.5"
-e IPv6="False"
pihole/pihole:latest
Error: OCI runtime error: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented

UDM Information

• Variant UniFi Dream Machine SE
• Firmware Version: 3.0.18
• Controller Version 7.3.83

[rpletka]

It seems that Ubiquiti has nerfed the ability to run containers via podman. The kernel seems to be compiled without the ability to leverage BPF syscalls.

root@UDMSE:/sys/fs/cgroup# zcat /proc/config.gz | grep -E 'CONFIG_BPF|CONFIG_FREEZER'
CONFIG_BPF=y

CONFIG_BPF_SYSCALL is not set

CONFIG_FREEZER=y

CONFIG_BPFILTER is not set

CONFIG_BPF_JIT=y

[boostchicken]

yeah I have the kernel source and we can make our own, someone was trying with @fabianishere.

@fabianishere any word?

[fabianishere]

@boostchicken Working on porting udm-kernel-tools to firmware v2.x/v3.x at the moment, but haven’t had much time to test the changes.

BPF SysCTL tweaks do not work on 2.4.X and 2.5.X for me either.

[peacey]

Well it's clear now that podman/docker won't work without a custom kernel that enables CONFIG_BPF_SYSCALL, but it's easy enough to create a container yourself without podman by using systemd-nspawn and install whatever you want within it (e.g. pihole). The process is a bit more involved, but it's pretty easy.

Basically, you just use debootstrap to create a directory under /data with a base debian system (this directory is your container), then you can boot and login to that container with systemd-nspawn or machinectl. After that, you can just install whatever you want in that container with apt-get or manually as any debian system (the container storage is persistent). This way your custom system changes don't affect the main Unifi system with regards to dependencies or conflicts with their files, and Unifi can't screw up your configuration with an update.

I've been running a debian unstable container on UDMP 3.x, with pihole, nginx, dnsdist, and some other programs. Works really well. You can even use pacstrap instead of debootstrap to create and run an Arch Linux container, or create a fedora, openSUSE, or other system container (examples).

Also, the debian base system is only like 300MB so the container size doesn't start off so high, but the container size can get to 1GB+ if you start installing a lot of large softwares (this applies to podman too, I guess).

Edit: Here's a guide I wrote on how to create and configure your own custom container on 3.x.

[boostchicken]

So I have not upgraded yet, but on 2.5x is overlayfs working, I expect it will work on 3 as well. So the disk space issue is now fixed @peacey

I am pretty sure I can get podman running on the UDMSE. See BPF is used for security lockdown on syscalls. We are always root anyways so it doesnt matter. I am going to drop seccomp and apparmor from the UDMP-UDMSE build and edit configs accordingly and I think that will get you in business.

Also, they new podman builds come with crun, if you want to use it. It's much better on resources. Uses less memory, and executes faster

Also, netavark is built and in the latest zips as well. This is a replacement for CNI which is now deprecated. It has cool things like macvlan dhcp working, but its not zero effort to migrate your networks, syntax of the files is quite different.

I would move to netavark ASAP. It makes containers much much much faster and much less latent on the network. It is written in Rust instead of Go (much like crun). I also included the dhcp-client-proxy if anyone wants to macvlan dhcp working.

I am not sure when I can move to 3. If someone would volunteer to test my new build for 3 build with the mods above I'll crank it out asap that would be awesome.

[boostchicken]

New builds are running that I think shoudl work on 3

seccomp.json removes permissions for bpf syscall, there is an update to the registries, cni is gone, replaced by netavark. Crun is also available if you want to.

If my IPFS server is being garbage use this link to get at the latest builds. Also you should use this link just to check if the builds are done. Even when they are done, it doesn't mean they are on IPFS. However, they all have versioned names now to make it clear which one you are getting.

https://github.com/unifi-utilities/unifios-utilities/actions

[peacey]

Sorry @boostchicken, I tried your new build and it still doesn't seem to work on 3.x. Still the same issue with BPF syscall with your new build and config files. I tried both crun and runc and also with and without --security-opt=seccomp=unconfined.

With runc I get,

Error: OCI runtime error: runc: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented

And with crun I get,

Error: crun: error opening file /proc/self/setgroups: No such file or directory: OCI runtime attempted to invoke a command that was not found

It seems /proc/self/setgroups is missing from the kernel interface on 3.x for processes.

So I don't think this will work unfortunately. Kernel is too locked down. But systemd-nspawn works well!

[sprocktech]

Just throwing this out there... There are other firewall vendors that embrace the community and actually give you instructions on how to run Pi-hole on their hardware. Why continue to try and fill the feature gaps that Ubiquiti continues to leave in their products when other products already have the features and they let you add your own stuff too? https://help.firewalla.com/hc/en-us/articles/360051625034-Guide-How-to-install-Pi-Hole-on-Gold-Purple-Beta-

[peacey]

@sprocktech I mean the whole point of a modding community is to mod the device. When someone comes here, they don't want to hear it can't be done. They want to hear how it can be done. For all of us that do like to use the UDM-series routers, it's nice to have this modding resource and community for all the extra bells and whistles.

And the UDMP/SE is a very capable, relatively low-cost router with a quad-core CPU! It would be a shame to let it go to waste. IMO, with its Debian architecture, it makes it very easy to mod for a Linux sysadmin.

[sprocktech]

@peacey I'm with you. I'm just saying that Ubiquiti is not with you. Actually, they are against you, as you know already from these recent firmware changes. Frankly, I've never seen anything like it before. It's an uphill battle my friend, and I'd rather put my money and effort elsewhere.