How can we auto cleanup k8s worker pool's completed and failed job

[schwannden]

I have setup prefect server and worker using helm chart, and was able to run seleral workflows. However I noticed that after flowrun complete, kubectl get pod shows the pods still exist. Here is my prefect.yaml

name: flows
prefect-version: 2.14.2

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.set_working_directory:
    directory: /opt/prefect

# the definitions section allows you to define reusable components for your deployments
definitions:
  tags: &common_tags
    - "sat"
  work_pool: &common_work_pool
    name: "k8s-pool"
    job_variables:
      image: "{{ build-image.image }}"

# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: "flow"
  schedule: null
  entrypoint: "src/sat/sub_flows/flow.py:flow"
  work_pool: *common_work_pool

Is there a flag jere where I can configure auto cleanup setting? Or perhaps in worker pool settings?

[ttq186]

We encountered the same issue before. We set TTL for K8s job and it did help. However, we noticed that half of jobs are not automatically deleted and still take up resources. Finally, we ended up using K8s API for killing the job right before exiting Prefect flow and it's been working well so far.

[mitchnielsen]

Came across this from another thread. Is the Finished Job TTL section in the docs helpful?

This configures the TTL on the worker, which will pass that configuration down to the Job.