Skip to content

Unable to read memory stats file at /sys/fs/cgroup/memory.current #42025

@alekseyastolyarov

Description

@alekseyastolyarov

Title: Unable to read memory stats file at /sys/fs/cgroup/memory.current

Description:
Envoy fail to read cgroups v1 and v2 presumably because of using readlink() for a regular file.
If I remount cgoups to a temporary directory and create symlink for /sys/fs/cgroup/memory.current and /sys/fs/cgroup/memory.max envoy starts reading that files and works without errors.

Repro steps:
take envoy of any version from v1.35.6 to v1.36.2 (latest at the moment) and apply this config in the static (aka bootstrap) configuration

overload_manager:
  refresh_interval: 0.25s
  resource_monitors:
    - name: "envoy.resource_monitors.cgroup_memory"
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.resource_monitors.cgroup_memory.v3.CgroupMemoryConfig
        # max_memory_bytes: 1073741824  # 1GB
  actions:
    - name: "envoy.overload_actions.stop_accepting_requests"
      triggers:
        - name: "envoy.resource_monitors.cgroup_memory"
          threshold:
            value: 0.95  # Trigger at 95% memory utilization#
    - name: "envoy.overload_actions.disable_http_keepalive"
      triggers:
        - name: "envoy.resource_monitors.cgroup_memory"
          threshold:
            value: 0.92

Envloy logs:
"Failed to update resource envoy.resource_monitors.cgroup_memory: Unable to read memory stats file at /sys/fs/cgroup/memory.max"

What I see at the moment errors happen:

# strace -f -e trace=open,openat,read,write,stat,fstat,lstat,readlink -p 14

readlink("/sys", 0x7ffdeb836810, 1023) = -1 EINVAL (Invalid argument) [pid 14]
readlink("/sys/fs", 0x7ffdeb836810, 1023) = -1 EINVAL (Invalid argument) [pid 14]
readlink("/sys/fs/cgroup", 0x7ffdeb836810, 1023) = -1 EINVAL (Invalid argument) [pid 14]
readlink("/sys/fs/cgroup/memory.current", 0x7ffdeb836810, 1023) = -1 EINVAL (Invalid argument) [pid 14]
write(2, "{"@timestamp": "2025-11-11T11:16"..., 351) = 351 [pid 14]

according to readlink() documentation it will fail if try to open a non symlink file "EINVAL - The named file (i.e., the final filename component of path) is not a symbolic link."

what I see after replacing regular files with symlinks:

[pid    14] readlink("/sys", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid    14] readlink("/sys/fs", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid    14] readlink("/sys/fs/cgroup", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid    14] readlink("/sys/fs/cgroup/memory.current", "/mnt/cgroups/memory.current", 1023) = 27
[pid    14] readlink("/mnt", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid    14] readlink("/mnt/cgroups", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid    14] readlink("/mnt/cgroups/memory.current", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)

[pid    14] openat(AT_FDCWD, "/sys/fs/cgroup/memory.current", O_RDONLY) = 29
[pid    14] read(29, "97636352\n", 4096) = 9
[pid    14] read(29, "", 4096)          = 0


[pid    14] readlink("/sys", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid    14] readlink("/sys/fs", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid    14] readlink("/sys/fs/cgroup", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid    14] readlink("/sys/fs/cgroup/memory.max", "/mnt/cgroups/memory.max", 1023) = 23
[pid    14] readlink("/mnt", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid    14] readlink("/mnt/cgroups", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)
[pid    14] readlink("/mnt/cgroups/memory.max", 0x7ffd5bdcca30, 1023) = -1 EINVAL (Invalid argument)

[pid    14] openat(AT_FDCWD, "/sys/fs/cgroup/memory.max", O_RDONLY) = 29
[pid    14] read(29, "max\n", 4096)     = 4
[pid    14] read(29, "", 4096)          = 0

I.e. finely Envoy reads cgroup files. No errors apear in the logs

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions