RestartJob deletes failed pods on last try losing logs

[d4l3k]

What happened:

When you specify on PodFailed action RestartJob it will delete all the pods on the last retry even though there will be no more retries. This means that there's no way to access the logs to figure out why the pods failed and debug the jobs.

# success case w/ PodFailed:RestartJob
sh-5tqgv-sh-0-0                               0/1     Completed          0          16s
# failure case without PodFailed:RestartJob
sh-jpfdn-sh-0-0                               0/1     Error              0          16s

# failure case with PodFailed:RestartJob
<nothing>

What you expected to happen:

It should leave the pods for the last retry for debugging purposes and logs.

How to reproduce it (as minimally and precisely as possible):

run job:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  creationTimestamp: "2021-07-30T18:28:15Z"
  generateName: sh-
  generation: 1
  name: sh-kbnmb
  namespace: default
  resourceVersion: "96243182"
  selfLink: /apis/batch.volcano.sh/v1alpha1/namespaces/default/jobs/sh-kbnmb
  uid: 13eaaf6e-66f3-434c-a021-623af82771d7
spec:
  maxRetry: 1
  minAvailable: 1
  plugins:
    env: []
    svc: []
  queue: test
  schedulerName: volcano
  tasks:
  - maxRetry: 1
    minAvailable: 1
    name: sh-0
    policies:
    - action: RestartJob
      event: PodEvicted
    - action: RestartJob
      event: PodFailed
    replicas: 1
    template:
      metadata: {}
      spec:
        containers:
        - command:
          - /bin/sh
          - -c
          - exit 1
          image: alpine:latest
          name: sh-0
          resources: {}
        restartPolicy: Never
status:
  minAvailable: 1
  retryCount: 1
  runningDuration: 6.652806914s
  state:
    lastTransitionTime: "2021-07-30T18:28:21Z"
    phase: Failed
  version: 3

Anything else we need to know?:

Environment:

• Volcano Version:
  image: volcanosh/vc-controller-manager:latest
  imageID: docker-pullable://volcanosh/vc-controller-manager@sha256:897793abe922641b47acc2c872209078346e855a3453a0d35dfb573812ae64b9

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"archive", BuildDate:"2021-07-16T17:16:46Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.16-eks-7737de", GitCommit:"7737de131e58a68dda49cdd0ad821b4cb3665ae8", GitTreeState:"clean", BuildDate:"2021-03-10T21:33:25Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: AWS EKS
• OS (e.g. from /etc/os-release): Arch Linux
• Kernel (e.g. uname -a): Linux tristanr-arch2 5.12.15-arch1-1 #1 SMP PREEMPT Wed, 07 Jul 2021 23:35:29 +0000 x86_64 GNU/Linux
• Install tools:
• Others:

[william-wang]

@d4l3k What you expect is that do not delete the failed pod and create new pod in 'RestartAction'. Is it right?

[wpeng102]

I think it make sense that left the Error pod here for user invetigatation when job is in failed stage.

[d4l3k]

When there's no more retries granted it should just let the pods stay as if they had failed normally with no retry policy. As is it deletes all the pods, there's no more retries so the job is marked as failed and stops. As far as I can tell since the pods are gone there's no way to determine _why_ the job failed since that's done through logs and stack traces which are gone

…

On Fri, Jul 30, 2021, 19:58 william-wang ***@***.***> wrote: @d4l3k <https://github.com/d4l3k> What you expect is that do not delete the failed pod and create new pod in 'RestartAction'. Is it right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1651 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG56MAGLZKEBVKL2CGN3RTT2NRHDANCNFSM5BJAUM2Q> .

[william-wang]

@d4l3k Thanks for your explaination. I had a test with the yaml too. Agree with you to let the error pod stay as it is after the last retry for debug :)

[d4l3k]

appears to be fixed as of #1657 still need to test on master

[d4l3k]

@Thor-wl this seems to have regressed per #1746 (comment)

Is it possible to add unit tests for this behavior?

[aivanou]

Hello. I believe the problem was not resolved.

Steps to repro:
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.4.0/installer/volcano-development.yaml

Job spec:

Name:           sh-vdpmpztfwcw4g-sh-0-0
Namespace:      default
Priority:       0
Node:           ip-192-168-16-165.us-west-2.compute.internal/192.168.16.165
Start Time:     Mon, 11 Oct 2021 14:20:28 -0700
Labels:         torchx.pytorch.org/app-name=sh
                torchx.pytorch.org/replica-id=0
                torchx.pytorch.org/role-index=0
                torchx.pytorch.org/role-name=sh
                torchx.pytorch.org/version=0.1.0rc1
                volcano.sh/job-name=sh-vdpmpztfwcw4g
                volcano.sh/job-namespace=default
                volcano.sh/queue-name=default
Annotations:    kubernetes.io/psp: eks.privileged
                scheduling.k8s.io/group-name: sh-vdpmpztfwcw4g
                sidecar.istio.io/inject: false
                volcano.sh/job-name: sh-vdpmpztfwcw4g
                volcano.sh/job-version: 4
                volcano.sh/queue-name: default
                volcano.sh/task-spec: sh-0
                volcano.sh/template-uid: sh-vdpmpztfwcw4g-sh-0
Status:         Pending
IP:
IPs:            <none>
Controlled By:  Job/sh-vdpmpztfwcw4g
Containers:
  sh-0:
    Container ID:
    Image:         public.ecr.aws/y5x3w0a7/aivanou-tests:latest
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      python my_job.py
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      VC_SH-0_HOSTS:  <set to the key 'VC_SH-0_HOSTS' of config map 'sh-vdpmpztfwcw4g-svc'>  Optional: false
      VC_SH-0_NUM:    <set to the key 'VC_SH-0_NUM' of config map 'sh-vdpmpztfwcw4g-svc'>    Optional: false
      VK_TASK_INDEX:  0
      VC_TASK_INDEX:  0
    Mounts:
      /etc/volcano from sh-vdpmpztfwcw4g-svc (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z8vb9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  sh-vdpmpztfwcw4g-svc:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sh-vdpmpztfwcw4g-svc
    Optional:  false
  default-token-z8vb9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-z8vb9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From     Message
  ----    ------     ----  ----     -------
  Normal  Scheduled  4s    volcano  Successfully assigned default/sh-vdpmpztfwcw4g-sh-0-0 to ip-192-168-16-165.us-west-2.compute.internal
  Normal  Pulling    3s    kubelet  Pulling image "public.ecr.aws/y5x3w0a7/aivanou-tests:latest"

python my_job.py

import time

def main():
   time.sleep(120)
   raise RuntimeError("test")

if __name__=="__main__":
   main()

The jobs just disappears from the kubectl get pods

volcano-scheduler pod:

Name:         volcano-scheduler-7c5d7bbcd4-smqgc
Namespace:    volcano-system
Priority:     0
Node:         ip-192-168-32-51.us-west-2.compute.internal/192.168.32.51
Start Time:   Mon, 11 Oct 2021 12:14:48 -0700
Labels:       app=volcano-scheduler
              pod-template-hash=7c5d7bbcd4
Annotations:  kubernetes.io/psp: eks.privileged
Status:       Running
IP:           192.168.33.60
IPs:
  IP:           192.168.33.60
Controlled By:  ReplicaSet/volcano-scheduler-7c5d7bbcd4
Containers:
  volcano-scheduler:
    Container ID:  docker://44400c3767705e5620735cf60ee4c04c2f189d1b436dba542fd624d09c884484
    Image:         volcanosh/vc-scheduler:latest
    Image ID:      docker-pullable://volcanosh/vc-scheduler@sha256:9353c38d25e14c810e4a07306de74b09208393242297457d3280a087d2ec0bc7
    Port:          <none>
    Host Port:     <none>
    Args:
      --logtostderr
      --scheduler-conf=/volcano.scheduler/volcano-scheduler.conf
      -v=3
      2>&1
    State:          Running
      Started:      Mon, 11 Oct 2021 12:14:50 -0700
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from volcano-scheduler-token-6dfzd (ro)
      /volcano.scheduler from scheduler-config (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  scheduler-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      volcano-scheduler-configmap
    Optional:  false
  volcano-scheduler-token-6dfzd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  volcano-scheduler-token-6dfzd
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

let me know if you need any additional info.