Status of Jobs pods can become "OutOfCpu" when many pods are created and completed in a short time on the same node schedule by batch.volcano.sh/v1alpha1.

[GEEKZX]

What happened?
Status of pods can become "OutOfCpu" when many jobs are created and completed in a short time on the same node schedule by batch.volcano.sh/v1alpha1.

What did you expect to happen?
Status of pods should become "Running", "Pending" or "Completed". When node resources are not satisfied, The jobs pods should not be scheduled.

Anything else we need to know?
We found this issue in volcano v1.7.0 k8s version v1.23.15
Pods QoS is Burstable
Error message:
status:
message: 'Pod Node didn''t have enough resource: cpu, requested: 56033, used: 34036, capacity: 90000'
phase: Failed
reason: OutOfcpu
Environment:

• Volcano Version: v1.7.0
• Kubernetes version (use kubectl version): v1.23.15
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Ubuntu 20.04.3 LTS
• Kernel (e.g. uname -a):Linux k8s-master 5.15.0-43-generic Refresh volcano intro image #46-Ubuntu SMP Tue Jul 12 10:30:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

[wangyang0616]

Looking at the log description, the sum of requested and used exceeds the capacity.

In our system, only the volcano is scheduling tasks, or are both the volcano and kube-scheduler scheduling their own tasks? Is it possible that in a scheduling cycle, the two schedulers At the same time, the task is scheduled to be changed to the node, causing the node to apply for resources exceeding the upper limit.

By the way, do we have a pod on a node in the OutOfcpu state, or are multiple pods in this error state?

[wangyang0616]

It would be better if you can provide the log information of the volcano-scheduler, we will be able to troubleshoot the problem more easily, thanks @GEEKZX

[GEEKZX]

We found no scheduling logs in pod kube-scheduler-k8s-master, all tasks is scheduling by volcano in our Cluster , we have 40 node , when we submit 5000 job at the same time, a small amount pod can become "OutOfCpu" distributed on different nodes

[riverzhang]

+1
Status: Failed
Reason: OutOfcpu
Message: Pod Node didn't have enough resource: cpu, requested: 3000, used: 108524, capacity: 110000

[lowang-bh]

This is usually caused by multiple schedulers, eg, kube-scheduler and volcano, and both of them scheduling a pod to the same node, and kubelet finds that the resource is not enough and send an event message "outOfCpu" or "OutOfMemeory". Usually, we need a coordinate to solve this problem .

[stale]

Hello 👋 Looks like there was no activity on this issue for last 90 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).

[hwdef]

keep activate

[stale]

Closing for now as there was no activity for last 60 days after marked as stale, let us know if you need this to be reopened! 🤗