Update cleanup job node affinity logic

[dkwon17]

What does this PR do?

Update the cleanup job node affinity logic.

Old logic:

• Schedule the cleanup job node to the node specified in the claim-devworkspace PVC's volume.kubernetes.io/selected-node annotation (if the annotation exists)

This logic can still cause the multi-attach error since there is no guarantee the PVC is mounted to a pod on the same node as what's specified in volume.kubernetes.io/selected-node .

New logic:

• Check if there are devworkspace pods running in the namespace that the to-be-deleted devworkspace is in.
  • If yes, check if any of them mount the claim-devworkspace PVC
    • If yes, schedule the cleanup job pod on the same node as the devworkspace pod found in the previous step
  • If not, don't add a nodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution rule onto the cleanup job pod

What issues does this PR fix or reference?

Fix #1453

Is it tested? How?

To follow these steps, a multi node cluster (where there is more than 1 worker node) is required.

1. Install DWO with this image: quay.io/dkwon17/devworkspace-controller:fix-pvc-node:

export NAMESPACE=openshift-operators
export DWO_IMG="quay.io/dkwon17/devworkspace-controller:fix-pvc-node"
make install

2. Switch to the openshift-operators namespace and create a workspace:

oc project openshift-operators && oc apply -f https://raw.githubusercontent.com/devfile/devworkspace-operator/refs/heads/main/samples/code-latest.yaml

3. After the workspace is running (run oc get dw to check), terminate the first workspace:

oc patch dw code-latest --type merge -p '{"spec": {"started": false}}'

4. Create a second workspace:

curl https://raw.githubusercontent.com/devfile/devworkspace-operator/refs/heads/main/samples/code-latest.yaml | yq '.metadata.name = "code-latest-2"' | oc apply -f -

5. After the workspace is running terminate the second workspace:

oc patch dw code-latest-2 --type merge -p '{"spec": {"started": false}}'

6. Get the node specified from the volume.kubernetes.io/selected-node annotation from the claim-devworkspace PVC:

oc get pvc claim-devworkspace -o jsonpath='{.metadata.annotations.volume\.kubernetes\.io\/selected-node}'

7. Cordon the node, making it unavailable for pod scheduling:

oc adm cordon <output of previous command>

8. Delete code-latest:

oc delete dw code-latest

With changes in the PR, the deletion should be successful after about 10 seconds.

9. Create another workspace:

oc apply -f https://raw.githubusercontent.com/devfile/devworkspace-operator/refs/heads/main/samples/code-latest.yaml

10. Wait until it is running, and delete it:

oc delete dw code-latest

11. Uncordon the node from step 7:

oc adm uncordon <node name>

The deletion should be successful after about 10 seconds.

PR Checklist

• E2E tests pass (when PR is ready, comment /test v8-devworkspace-operator-e2e, v8-che-happy-path to trigger)
  • v8-devworkspace-operator-e2e: DevWorkspace e2e test
  • v8-che-happy-path: Happy path for verification integration with Che

[rohanKanojia]

@dkwon17 : Thanks for fixing this! Would you mind adding some unit tests to cover the changed behavior?

[olkornii]

@dkwon17 Tried with rosa 4.18 and oc 4.18.9.

Step 2 failed with next error:

oc project openshift-operators && oc apply -f https://raw.githubusercontent.com/devfile/devworkspace-operator/refs/heads/main/samples/code-latest.yaml
Now using project "openshift-operators" on server "https://api.mibm8-drrcy-uos.1u73.p3.openshiftapps.com:443".
Error from server (InternalError): error when creating "https://raw.githubusercontent.com/devfile/devworkspace-operator/refs/heads/main/samples/code-latest.yaml": Internal error occurred: failed calling webhook "mutate.devworkspace-controller.svc": failed to call webhook: Post "https://devworkspace-webhookserver.openshift-operators.svc:443/mutate?timeout=10s": no endpoints available for service "devworkspace-webhookserver"

Tried to separate to two commands:

oc project openshift-operators

and

oc apply -f https://raw.githubusercontent.com/devfile/devworkspace-operator/refs/heads/main/samples/code-latest.yaml

passed.

Results. After workaround for step 2 everything works as expected.

[rohanKanojia]

@olkornii : It seems that webhook pod is not running. Could you please check if you have devworkspace-manager devworkspace-controller-manager and devworkspace-webhook-server pods running in openshift-operators namespace:

devworkspace-operator : $ oc get pods -nopenshift-operators
NAME                                               READY   STATUS      RESTARTS   AGE
devworkspace-controller-manager-68fc476665-tzlhj   2/2     Running     0          3h16m
devworkspace-webhook-server-6976b5d46d-c598k       2/2     Running     0          3h12m
devworkspace-webhook-server-6976b5d46d-vsqfw       2/2     Running     0          3h12m

[rohanKanojia]

I tested this PR with the abovementioned steps and it seems to be working ✔️ . Thanks a lot 👍

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dkwon17, ibuziuk, rohanKanojia

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [dkwon17]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[olkornii]

@rohanKanojia

NAME                                               READY   STATUS      RESTARTS   AGE
cleanup-workspaced86095cae49842d3-gx6ks            0/1     Completed   0          119m
devworkspace-controller-manager-68fc476665-qzfrl   2/2     Running     0          154m
devworkspace-webhook-server-67d4bc5fdb-6pv9x       2/2     Running     0          154m
devworkspace-webhook-server-67d4bc5fdb-k5c9r       2/2     Running     0          154m

But as I said, entering those commands one by one works well.

[dkwon17]

Thank you for checking,

@olkornii this error:

Error from server (InternalError): error when creating "https://raw.githubusercontent.com/devfile/devworkspace-
operator/refs/heads/main/samples/code-latest.yaml": Internal error occurred: failed calling webhook 
"mutate.devworkspace-controller.svc": failed to call webhook: Post "https://devworkspace-webhookserver.openshift-
operators.svc:443/mutate?timeout=10s": no endpoints available for service "devworkspace-webhookserver"

is because the devworkspace-webhookserver pods were not running when the devworkspace was created, hence, no endpoints available for service "devworkspace-webhookserver". So it's not an issue for this PR

[openshift-ci]

New changes are detected. LGTM label has been removed.

[dkwon17]

@rohanKanojia I'm still working on the unit tests, I will create a new PR for them.