KEP-2033: KubeletInUserNamespace: promote to beta

Signed-off-by: Akihiro Suda <[email protected]>

Author: AkihiroSuda

keps/prod-readiness/sig-node/2033.yaml

@@ -1,3 +1,5 @@
 kep-number: 2033
 alpha:
   approver: "@ehashman"
+beta:
+  approver: "@soltysh"

keps/sig-node/2033-kubelet-in-userns-aka-rootless/README.md

@@ -148,20 +148,20 @@ checklist items _must_ be updated for the enhancement to be released.
 
 Items marked with (R) are required *prior to targeting to a milestone / release*.
 
-- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
-- [ ] (R) KEP approvers have approved the KEP status as `implementable`
-- [ ] (R) Design details are appropriately documented
-- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
-  - [ ] e2e Tests for all Beta API Operations (endpoints)
-  - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) 
-  - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free
+- [X] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
+- [X] (R) KEP approvers have approved the KEP status as `implementable`
+- [X] (R) Design details are appropriately documented
+- [X] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
+  - [N/A] e2e Tests for all Beta API Operations (endpoints)
+  - [N/A] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) 
+  - [N/A] (R) Minimum Two Week Window for GA e2e tests to prove flake free
 - [ ] (R) Graduation criteria is in place
-  - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) 
+  - [N/A] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) 
 - [ ] (R) Production readiness review completed
 - [ ] (R) Production readiness review approved
-- [ ] "Implementation History" section is up-to-date for milestone
-- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
-- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+- [X] "Implementation History" section is up-to-date for milestone
+- [X] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [X] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
 
 <!--
 **Note:** This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone.
@@ -493,7 +493,11 @@ The patch modifies `kubelet` to ignore errors that happens during setting the fo
 #### kube-proxy
 Patch: ["kube-proxy: allow running in userns"](https://github.com/rootless-containers/usernetes/blob/v20210303.0/src/patches/kubernetes/0002-kube-proxy-allow-running-in-userns.patch)
 
-The patch modifies `kube-proxy` to ignore an error during setting `RLIMIT_NOFILE`.
+The patch modifies `kube-proxy` (`userspace` mode) to ignore an error during setting `RLIMIT_NOFILE`.
+No change is needed for non-userspace mode.
+
+> **Note**
+> `userspace` proxy was removed in v1.26.
 
 ### Test Plan
 
@@ -508,19 +512,19 @@ when drafting this test plan.
 [testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
 -->
 
-[ ] I/we understand the owners of the involved components may require updates to
+[X] I/we understand the owners of the involved components may require updates to
 existing tests to make this code solid enough prior to committing the changes necessary
 to implement this enhancement.
 
-Tests are present in several subproject repos and third party repos:
-- https://github.com/kubernetes-sigs/kind/blob/v0.17.0/.github/workflows/cgroup2.yaml#L24
-- https://github.com/kubernetes/minikube/blob/v1.29.0/.github/workflows/pr.yml#L293-L410
-- https://github.com/k3s-io/k3s/blob/v1.26.1+k3s1/.github/workflows/cgroup.yaml#L92-L99
-- https://github.com/rootless-containers/usernetes/blob/v20221007.0/.cirrus.yml
+See [e2e tests](#e2e-tests) below.
 
-Tests will be added to `kubernetes/test-infra` as well when the [`k8s-infra-prow-build`](https://github.com/kubernetes/k8s.io/blob/a071c4ed0823f193ee29e2f14e191be42dc1a1f0/infra/gcp/terraform/k8s-infra-prow-build/main.tf#L78) cluster
-is upgraded to use cgroup v2.
-This will probably automatically happen when [GKE bumps up their "regular" channel to Kubernetes v1.26 or later](https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config).
+Additional tests are present in several subproject repos and third party repos:
+- https://github.com/kubernetes-sigs/kind/blob/v0.29.0/.github/workflows/vm.yaml#L24
+- https://github.com/kubernetes/minikube/blob/v1.36.0/.github/workflows/pr.yml#L299-L415
+- https://github.com/k3s-io/k3s/blob/v1.33.1%2Bk3s1/.github/workflows/e2e.yaml#L56
+- https://github.com/rootless-containers/usernetes/blob/gen2-v20250501.0/.github/workflows/main.yaml
+  - Covers multi-node clusters with Flannel (VXLAN)
+  - Covers several host distributions (Ubuntu, CentOS Stream, and Fedora)
 
 ##### Prerequisite testing updates
 
@@ -550,7 +554,7 @@ This can inform certain test coverage improvements that we want to do before
 extending the production code to implement this enhancement.
 -->
 
-- `<package>`: `<date>` - `<test coverage>`
+N/A, as unit tests do not make sense here.
 
 ##### Integration tests
 
@@ -576,7 +580,7 @@ This can be done with:
 - a search in the Kubernetes bug triage tool (https://storage.googleapis.com/k8s-triage/index.html)
 -->
 
-- [test name](https://github.com/kubernetes/kubernetes/blob/2334b8469e1983c525c0c6382125710093a25883/test/integration/...): [integration master](https://testgrid.k8s.io/sig-release-master-blocking#integration-master?include-filter-by-regex=MyCoolFeature), [triage search](https://storage.googleapis.com/k8s-triage/index.html?test=MyCoolFeature)
+N/A, as integration tests do not make sense here.
 
 ##### e2e tests
 
@@ -595,7 +599,31 @@ We expect no non-infra related flakes in the last month as a GA graduation crite
 If e2e tests are not necessary or useful, explain why.
 -->
 
-- [test name](https://github.com/kubernetes/kubernetes/blob/2334b8469e1983c525c0c6382125710093a25883/test/e2e/...): [SIG ...](https://testgrid.k8s.io/sig-...?include-filter-by-regex=MyCoolFeature), [triage search](https://storage.googleapis.com/k8s-triage/index.html?test=MyCoolFeature)
+`NodeConformance` tests are executed using [kubetest2-kindinv](https://github.com/rootless-containers/kubetest2-kindinv).
+
+"kindinv" stands for "Kubernetes in (Rootless) Docker in (GCE) VM".
+GCE VM is used for enabling systemd that is required by Rootless Docker to set up cgroup v2.
+
+```bash
+exec kubetest2 kindinv \
+  --boskos-location=http://boskos.test-pods.svc.cluster.local \
+  --gcp-zone=us-central1-b \
+  --instance-image=ubuntu-os-cloud/ubuntu-2204-lts \
+  --instance-type=n2-standard-4 \
+  --kind-rootless \
+  --user=rootless \
+  --build \
+  --up \
+  --down \
+  --test=ginkgo \
+  -- \
+  --focus-regex='\[NodeConformance\]' \
+  --skip-regex='\[Environment:NotInUserNS\]|\[Slow\]' \
+  --parallel=8
+```
+
+- Prow manifest: https://github.com/kubernetes/test-infra/blob/4b7824ff1cfe00c36062035ab6aea3bb6c2e6ba2/config/jobs/kubernetes/sig-testing/kubernetes-kind.yaml#L615-L678
+- Logs: https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-kind-rootless
 
 ### Graduation Criteria
 
@@ -677,9 +705,7 @@ in back-to-back releases.
 
 - Beta: e2e tests coverage.
   Requires [the cgroup v2 KEP](../20191118-cgroups-v2.md ) to reach Beta or GA.
-  To move to beta, we need clarity if we intend to define two separate types of conformance suites:
-  - kubernetes clusters that can run privileged workloads
-  - kubernetes cluster that are restricted to run unprivileged workloads only
+  The tests are covered by `NodeConformance` tests (see above).
 
 - GA: Assuming no negative user feedback based on production experience, promote after >= 2 releases in beta.
   Requires [the cgroup v2 KEP](../20191118-cgroups-v2.md ) to reach GA.
@@ -715,7 +741,8 @@ enhancement:
   CRI or CNI may require updating that component before the kubelet.
 -->
 
-N/A
+N/A.
+This KEP only affects the internal of kubelet, and does not affect any API.
 
 ## Production Readiness Review Questionnaire
 
@@ -761,7 +788,7 @@ well as the [existing list] of feature gates.
 
 - [X] Feature gate (also fill in values in `kep.yaml`)
   - Feature gate name: `KubeletInUserNamespace`
-  - Components depending on the feature gate:
+  - Components depending on the feature gate: kubelet
 - [ ] Other
   - Describe the mechanism:
   - Will enabling / disabling the feature require downtime of the control
@@ -784,7 +811,8 @@ Any change of default behavior may be surprising to users or break existing
 automations, so be extremely careful here.
 -->
 
-During Alpha, we will document what workloads will work and what will not work.
+The limitation is same as Rootless Docker, Podman, etc.
+See <https://rootlesscontaine.rs/caveats/>.
 
 ###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?
 
@@ -799,11 +827,11 @@ feature.
 NOTE: Also set `disable-supported` to `true` or `false` in `kep.yaml`.
 -->
 
-N/A, as switching back rootless to rootful requires redeploying the kubelet, and vice versa.
+Yes, by turning off the feature gate.
 
 ###### What happens if we reenable the feature if it was previously rolled back?
 
-N/A.
+Nothing happens.
 
 ###### Are there any tests for feature enablement/disablement?
 
@@ -820,17 +848,14 @@ You can take a look at one potential example of such test in:
 https://github.com/kubernetes/kubernetes/pull/97058/files#diff-7826f7adbc1996a05ab52e3f5f02429e94b68ce6bce0dc534d1be636154fded3R246-R282
 -->
 
-CI will run `kind` (Kubernetes in Docker) tests with Rootless Docker/Podman.
-Tests with a real cluster will be added later as well.
+Yes. See [Test Plan](#test-plan).
 
 ### Rollout, Upgrade and Rollback Planning
 
 <!--
 This section must be completed when targeting beta to a release.
 -->
 
-This section will be fulfilled when targeting beta graduation to a release.
-
 ###### How can a rollout or rollback fail? Can it impact already running workloads?
 
 <!--
@@ -843,13 +868,22 @@ rollout. Similarly, consider large clusters and how enablement/disablement
 will rollout across nodes.
 -->
 
+Rollout: Rolling out requires recreating a new node instance, in a UserNS.
+Typical failures:
+- [subuids are not allocated](https://rootlesscontaine.rs/getting-started/common/subuid/)
+- [cgroup v2 delegation is not enabled](https://rootlesscontaine.rs/getting-started/common/cgroup2/)
+
+Rollback: this question is not applicable. Rolling back requires recreating a new node instance.
+
 ###### What specific metrics should inform a rollback?
 
 <!--
 What signals should users be paying attention to when the feature is young
 that might indicate a serious problem?
 -->
 
+CrashLoopBackOffs
+
 ###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
 
 <!--
@@ -858,12 +892,16 @@ Longer term, we may want to require automated upgrade/rollback tests, but we
 are missing a bunch of machinery and tooling and can't do that now.
 -->
 
+This question is not applicable. Rolling out and rolling back requires recreating a new node instance.
+
 ###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
 
 <!--
 Even if applying deprecation policies, they may still surprise some users.
 -->
 
+No
+
 ### Monitoring Requirements
 
 <!--
@@ -881,7 +919,7 @@ checking if there are objects with field X set) may be a last resort. Avoid
 logs or events for this purpose.
 -->
 
-N/A
+They can determine if a Pod is running on a node that is running in UserNS.
 
 ###### How can someone using this feature know that it is working for their instance?
 
@@ -894,8 +932,8 @@ and operation of this feature.
 Recall that end users cannot usually observe component logs or access metrics.
 -->
 
-- [ ] Events
-  - Event Reason: 
+- [X] Events
+  - Event Reason:  No CrashLoopBackOff
 - [ ] API .status
   - Condition name: 
   - Other field: 
@@ -919,7 +957,7 @@ These goals will help you determine what you need to measure (SLIs) in the next
 question.
 -->
 
-N/A
+99.9% of /health requests per day finish with 200 code
 
 ###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
 
@@ -941,7 +979,7 @@ Describe the metrics themselves and the reasons why they weren't added (e.g., co
 implementation difficulties, etc.).
 -->
 
-N/A
+No
 
 ### Dependencies
 
@@ -1058,6 +1096,8 @@ Think about adding additional work or introducing new steps in between
 [existing SLIs/SLOs]: https://git.k8s.io/community/sig-scalability/slos/slos.md#kubernetes-slisslos
 -->
 
+No.
+
 ###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
 
 <!--
@@ -1084,6 +1124,8 @@ Are there any tests that were run/should be run to understand performance charac
 and validate the declared limits?
 -->
 
+No
+
 ### Troubleshooting
 
 <!--
@@ -1120,6 +1162,10 @@ Same as traditional rootful Kubernetes.
 
 ###### What steps should be taken if SLOs are not being met to determine the problem?
 
+- Make sure that the supported version of the components are used
+- [Make sure that more than 65536 subuids are allocated](https://rootlesscontaine.rs/getting-started/common/subuid/)
+- [Make sure that cgroup v2 delegation is enabled](https://rootlesscontaine.rs/getting-started/common/cgroup2/)
+
 ## Implementation History
 
 <!--
@@ -1140,6 +1186,8 @@ Major milestones might include:
 - 2019-11-19: @giuseppe submitted [cgroup v2 KEP](https://github.com/kubernetes/enhancements/pull/1370)
 - 2019-11-19: present KEP to SIG-node (cgroup v2 version)
 - 2020-07-07: the cgroup v2 support is in `implementable` status
+- 2021-08-04: Kubernetes v1.22 (Alpha)
+- 2025-12-XX: Kubernetes v1.35 (Beta)
 
 ## Drawbacks
 

keps/sig-node/2033-kubelet-in-userns-aka-rootless/kep.yaml

@@ -15,7 +15,7 @@ reviewers:
   - "@dims"
   - "@sftim"
 approvers:
-  - TBD
+  - "@soltysh"
 see-also:
   # `add KEP for cgroups v2 support`
   - "https://github.com/kubernetes/enhancements/pull/1370"
@@ -24,16 +24,17 @@ replaces:
   - "https://github.com/kubernetes/enhancements/pull/1084"
 
 # The target maturity stage in the current dev cycle for this KEP.
-stage: alpha
+stage: beta
 
 # The most recent milestone for which work toward delivery of this KEP has been
 # done. This can be the current (upcoming) milestone, if it is being actively
 # worked on.
-latest-milestone: "v1.22"
+latest-milestone: "v1.35"
 
 # The milestone at which this feature was, or is targeted to be, at each stage.
 milestone:
   alpha: "v1.22"
+  beta: "v1.35"
 
 # The following PRR answers are required at alpha release
 # List the feature gate name and the components for which it must be enabled
@@ -42,3 +43,6 @@ feature-gates:
     components:
       - kubelet
 disable-supported: true
+
+# The following PRR answers are required at beta release
+metrics: []