-
Notifications
You must be signed in to change notification settings - Fork 5.2k
Description
Title: Unable to read memory stats file at /sys/fs/cgroup/memory.current
Description:
Envoy fail to read cgroups v1 and v2 presumably because of using readlink() for a regular file.
If I remount cgoups to a temporary directory and create symlink for /sys/fs/cgroup/memory.current and /sys/fs/cgroup/memory.max envoy starts reading that files and works without errors.
Repro steps:
take envoy of any version from v1.35.6 to v1.36.2 (latest at the moment) and apply this config in the static (aka bootstrap) configuration
overload_manager:
refresh_interval: 0.25s
resource_monitors:
- name: "envoy.resource_monitors.cgroup_memory"
typed_config:
"@type": type.googleapis.com/envoy.extensions.resource_monitors.cgroup_memory.v3.CgroupMemoryConfig
# max_memory_bytes: 1073741824 # 1GB
actions:
- name: "envoy.overload_actions.stop_accepting_requests"
triggers:
- name: "envoy.resource_monitors.cgroup_memory"
threshold:
value: 0.95 # Trigger at 95% memory utilization#
- name: "envoy.overload_actions.disable_http_keepalive"
triggers:
- name: "envoy.resource_monitors.cgroup_memory"
threshold:
value: 0.92
Envloy logs:
"Failed to update resource envoy.resource_monitors.cgroup_memory: Unable to read memory stats file at /sys/fs/cgroup/memory.max"
What I see at the moment errors happen:
# strace -f -e trace=open,openat,read,write,stat,fstat,lstat,readlink -p 14
readlink("/sys", 0x7ffdeb836810, 1023) = -1 EINVAL (Invalid argument) [pid 14]
readlink("/sys/fs", 0x7ffdeb836810, 1023) = -1 EINVAL (Invalid argument) [pid 14]
readlink("/sys/fs/cgroup", 0x7ffdeb836810, 1023) = -1 EINVAL (Invalid argument) [pid 14]
readlink("/sys/fs/cgroup/memory.current", 0x7ffdeb836810, 1023) = -1 EINVAL (Invalid argument) [pid 14]
write(2, "{"@timestamp": "2025-11-11T11:16"..., 351) = 351 [pid 14]
according to readlink() documentation it will fail if try to open a non symlink file "EINVAL - The named file (i.e., the final filename component of path) is not a symbolic link."
what I see after replacing regular files with symlinks:
[pid 14] readlink("/sys", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid 14] readlink("/sys/fs", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid 14] readlink("/sys/fs/cgroup", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid 14] readlink("/sys/fs/cgroup/memory.current", "/mnt/cgroups/memory.current", 1023) = 27
[pid 14] readlink("/mnt", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid 14] readlink("/mnt/cgroups", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid 14] readlink("/mnt/cgroups/memory.current", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid 14] openat(AT_FDCWD, "/sys/fs/cgroup/memory.current", O_RDONLY) = 29
[pid 14] read(29, "97636352\n", 4096) = 9
[pid 14] read(29, "", 4096) = 0
[pid 14] readlink("/sys", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid 14] readlink("/sys/fs", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid 14] readlink("/sys/fs/cgroup", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid 14] readlink("/sys/fs/cgroup/memory.max", "/mnt/cgroups/memory.max", 1023) = 23
[pid 14] readlink("/mnt", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid 14] readlink("/mnt/cgroups", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid 14] readlink("/mnt/cgroups/memory.max", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid 14] openat(AT_FDCWD, "/sys/fs/cgroup/memory.max", O_RDONLY) = 29
[pid 14] read(29, "max\n", 4096) = 4
[pid 14] read(29, "", 4096) = 0
I.e. finely Envoy reads cgroup files. No errors apear in the logs