Skip to content

docker-rootful: Increase inotify limits by default #1179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

docker-rootful: Increase inotify limits by default #1179

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions examples/docker-rootful.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,15 @@ provision:
fi
export DEBIAN_FRONTEND=noninteractive
curl -fsSL https://get.docker.com | sh
- mode: system
script: |
#!/bin/bash
# Increase inotify limits to prevent nested Kubernetes control planes
# from crash looping.
set -eux -o pipefail
echo 'fs.inotify.max_user_watches = 524288' >> /etc/sysctl.conf
echo 'fs.inotify.max_user_instances = 512' >> /etc/sysctl.conf
sysctl --system
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we replicate this to docker.yaml, podman*.yaml, k8s.yaml, k3s.yaml too?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea!

probes:
- script: |
#!/bin/bash
Expand Down
9 changes: 9 additions & 0 deletions examples/docker.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,15 @@ provision:
systemctl --user start dbus
dockerd-rootless-setuptool.sh install
docker context use rootless
- mode: system
script: |
#!/bin/bash
# Increase inotify limits to prevent nested Kubernetes control planes
# from crash looping.
set -eux -o pipefail
echo 'fs.inotify.max_user_watches = 524288' >> /etc/sysctl.conf
echo 'fs.inotify.max_user_instances = 512' >> /etc/sysctl.conf
sysctl --system
probes:
- script: |
#!/bin/bash
Expand Down
10 changes: 9 additions & 1 deletion examples/k3s.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,15 @@ provision:
script: |
#!/bin/sh
curl -sfL https://get.k3s.io | sh -

- mode: system
script: |
#!/bin/bash
# Increase inotify limits to prevent nested Kubernetes control planes
# from crash looping.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed for k3s? If so, it should be needed for k8s.yaml too?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know, it is only needed for k3d and kind - not for k3s and k8s

Copy link

@BenTheElder BenTheElder Feb 28, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know, it is only needed for k3d and kind - not for k3s and k8s

Not necessarily, it's used anytime you're using a lot of inotify, which can happen with k3s as well, anything using configmaps will need one per configmap, user workloads of other kinds may also run into this.

Copy link

@BenTheElder BenTheElder Feb 28, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kind usage is a common way to encounter it, because you often start multiple kubelets on the same kernel and some system workloads with configmaps, but that's only one way to run up usage. a single kubelet with many configmaps could hit the same limit.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW on Ubuntu defaults kubernetes's e2e tests created enough pods to exceed it running a Kubernetes worker node on the host (not kind, and not a single node cluster), in particular max_user_instances default seems to be pretty low (128)

kubernetes/kubernetes#130990

I setup my fork the other day and have been meaning to work up a new PR, that didn't happen yet, but leaving this breadcrumb in the meantime. There's also some pointers in the linked issue with example tuning in other cluster tools in the project.

set -eux -o pipefail
echo 'fs.inotify.max_user_watches = 524288' >> /etc/sysctl.conf
echo 'fs.inotify.max_user_instances = 512' >> /etc/sysctl.conf
sysctl --system
probes:
- script: |
#!/bin/bash
Expand Down