Skip to content

FileSystemIsReadonly is true, but the node still Ready #1071

@ns-it-safadi

Description

@ns-it-safadi

I have an issue with setting a node in "NotReady", when the FileSystemIsReadOnly.

I tried to illustrate the scenario to test readOnly condition on node.

steps:
NPD: v0.8.20

  • configure node-problem-detector: with extending --config.system-log-monitor=.../config/readonly-monitor.json
    on description I see that the node condtions is set as below:
    Conditions:
    Type Status Reason Message

...
ReadonlyFilesystem False FilesystemIsNotReadOnly Filesystem is not read-only
....
Ready True KubeletReady kubelet is posting ready status

  • on specific node I tried to send an information to /dev/kmsg that "Remounting filesystem read-only",which caused that ReadonlyFilesystem will be true, and it was changed to True

On description this node I got those info:
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message


ReadonlyFilesystem True FilesystemIsReadOnly Remounting filesystem read-only
Ready True KubeletReady kubelet is posting ready status

Warning FilesystemIsReadOnly 24h readonly-monitor Node condition ReadonlyFilesystem is now: True, reason: FilesystemIsReadOnly, message: "Remounting filesystem read-only"

But Node still in status Ready.

when I tried to get
curl localhost:8080/api/v1/nodes/nodeName/status

I got in conditions section:
{
"type": "ReadonlyFilesystem",
"status": "True",
"reason": "FilesystemIsReadOnly",
"message": "Remounting filesystem read-only"
},

Question is: Why Node still Ready, and how does kube-apiserver report this info, when NDP is setting this flag as True on NodeCondition?

Do I miss something in configuration to enforce that node should be NotReady?
and how does the best practice config look like?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions