[JENKINS-75563] First draft on how to kill a windows container

[raul-arabaolaza]

https://issues.jenkins.io/browse/JENKINS-75563

Trying to validate my approach on how we could kill a windows container. The problem I have is that AFAIK there is no way to use CMD in windows to get a result similar to

kubernetes-plugin/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java

Lines 675 to 676 in 48ea4aa

	"kill \\`grep -l '" + COOKIE_VAR + "=" + cookie
	+ "' /proc/*/environ | cut -d / -f 3 \\`")

which identifies the process to kill based on the existence of the JENKINS_SERVER_COOKIE env variable.

Instead what I am trying to do is find the process that was originated by the original command in

kubernetes-plugin/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java

Line 381 in 48ea4aa

private Proc doLaunch(

and kill it. Assuming every command run inside container has its own unique Launcher.

Note I have not tested anything yet as I want first to know if this approach makes sense at all or I should go in another direction.

(edited by jglick to use permalinks)

[raul-arabaolaza]

Not tested yet, provided by ChatGPT. But this should kill any process whose commandline contains launchCmd

[jglick]

So IIRC the difficulty is that we cannot use https://javadoc.jenkins.io/hudson/util/ProcessTree.html#killAll(java.util.Map) since the processes to be killed reside in a different container (and thus process namespace) than the agent.

[jglick]

wmic is not available by default in newer versions of Windows Server I am afraid.

(Anyway you would want by cookie env var, not just doing some fuzzy match on the command line.)

[jglick]

I have not tested anything yet

It is easy to test AFAIK: just run a cluster with a Windows node pool and try unignoring interruptedPodWindows.

#626 (comment) FTR

[jglick]

I think you just need to retest whether .StartInfo.Environment is accessible from Get-Process output in a current Windows version; it has been six years since I last tried it, and things may have changed. (Assume Powershell is available; I do not think we need to deal with batch scripts.)

Also doLaunch probably does need to be replaced by launchWithCookie.

[raul-arabaolaza]

Oh, I was under the impression I was limited to CMD, will test if powershell has improved, thanks!

[jglick]

You would just have to check what is actually available in typical container environments.

[raul-arabaolaza]

I may have some code to share today end of my day.

[raul-arabaolaza]

I have been able to craft (with a bit of AI help) a draft of a script that seems to work on usual usages, testing is yet very limited but something that works sometimes is better that something that never works until we can find something that always works.

Anyway, I have updated the code here in case anyone wants to take a look while I jump into serious testing using #1727.

For the moment tested only with a pipeline like this one which now ends successfully:

pipeline {
  agent {
    kubernetes {
      customWorkspace 'c:/s'
      yaml """
apiVersion: v1
kind: Pod
spec:
  tolerations:
    - key: "node.kubernetes.io/os"
      operator: "Equal"
      value: "windows"
      effect: "NoSchedule"
  containers:
  - name: jnlp
    image: "cloudbees/cloudbees-core-agent:2.492.1.3-windowsservercore-ltsc2019"
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - mountPath: /s
      name: s-volume
    - mountPath: /s@tmp
      name: stmp-volume
    resources:
      cpu:
        request: 1
      memory:
        request: 2Gi
  volumes:
  - emptyDir: {}
    name: s-volume
  - emptyDir: {}
    name: stmp-volume
  nodeSelector:
    dedicated: win
"""
    }
  }

  stages {
    stage('Timeout'){
        options {
        timeout(time: 10, unit: "SECONDS")
    }
        steps{
            container('jnlp') {
                script{
                    try {
                        bat 'ping 127.0.0.1 -n 3601 > test.txt'
                    } catch(Exception ex) {
                        //Catch block
                        println("Exception caught in debugging block")
                        println(ex);
                    } finally {
                        println("Handling final actions in debugging block")
                    }
                }
            }
        }
    }
    stage('Rename'){
        steps{
            container('jnlp') {
                script{
                    bat 'rename test.txt test2.txt'
                }
            }
        }
    } 
  }
}

[raul-arabaolaza]

For development, may move to true or introduce a system property in the future

[jglick]

a pipeline like this one which now ends successfully

I am not sure what that pipeline proves. The version I saw from a plugin user was doing a rename in the catch block, which is the part that fails since the ping was not terminated and thus held a file lock.

[raul-arabaolaza]

a pipeline like this one which now ends successfully

I am not sure what that pipeline proves. The version I saw from a plugin user was doing a rename in the catch block, which is the part that fails since the ping was not terminated and thus held a file lock.

Yeah, same with this one, the rename stage fails as the test.txt file was blocked by the unkilled ping. Now that I have something basic working I will do more in depth testing including a simplified pipeline like the one you mention

[raul-arabaolaza]

Should I remove this? It is not gonna be visible to the end user but may help on local runsa nd debugging

[raul-arabaolaza]

This would only work if a user sends a Ctrl-C to the process, Stop-Process is a hard kill that inmediately kills the process. I have not found a way to do a "soft kill" in windows via powershell and I honestly do not believe is needed.

[jglick]

Well, it would certainly be better for the plugin to emulate the Ctrl-C-like behavior, providing functional parity with the Linux version. If this is simply impossible, then a hard kill is I guess better than the nothing we have now.

What happens if the pod is terminated? (gracefully, e.g. kubectl delete pod) Does that run any Powershell finally code? Tricky to test since you would have to look for some effect outside the pod itself.

[raul-arabaolaza]

Yeah, I thought something similar yesterday while trying to sleep, I have found some examples of C# code that seems to emulate the ctrl+c that I want to try. I will also try to test the pod deletion.