Cleanup /var/lib/rancher/k3s/agent

[runningman84]

Version:
k3s version v1.17.4+k3s1 (3eee8ac)

K3s arguments:
na

Describe the bug
I would like to cleanup /var/lib/rancher/k3s/agent
This folder is constantly growing. How can I safeley remove old images, volumes and other old stuff? This folder is more than 30GB on each node:
/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots

To Reproduce

Run a small k3s setup for a few month...

Expected behavior

A way to cleanup stuff like docker system prune

Actual behavior

No corresponding docs available

Additional context / logs

[brandond]

See: #1900 (comment)

[Id2ndR]

Same here: after using the crictl rmi --prune command as suggested in #1900, my /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots folder still is 19GB whereas I've got 27 images in containerd and the biggest one is 80MB. Only few pods are still running and none of them use lots of space.
Before the cleaning, I used much bigger images, but since I cleaned it, it seams that containerd failed to remove all of the corresponding snapshots.

[Id2ndR]

I managed to free up more space with the following hack that I may NOT recommend: for sha in $(ctr snapshot usage | grep ^sha256 | awk '{print $1}'); do ctr snapshot rm $sha ; done
The output begins with lots of ctr: failed to remove "sha256:somethingthatisasha256": cannot remove snapshot with child: failed precondition but at a time I got more and more ctr: failed to remove "sha256:somethingthatisasha256": snapshot sha256:somethingthatisasha256 does not exist: not found because it previously succeeded in removing a unused snapshots.

As a result, my /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots now is 796MB that it more like I expected with my few pods and images.

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[xeruf]

This is still not solved, and I am having this exact issue as well :/

[milosmns]

+1, mine is also around 19g (Apr 2024)

[foxtrot989]

+1 same issue here 30GBs - the cluster has been around for over a year but i need a way clean up

[racerxdl]

+1 200GBs here :|

[brandond]

Answered above in #1905 (comment). Locking to prevent additional me-toos.