PreStop command injection for inference engines

[kerthcet]

What would you like to be added:

Why is this needed:

The inference engine will batch the requests for processing, and maintain a queue inside for best effort scheduling, which means when we kill the pod, there maybe some requests sitting in the queue still, we need to inject the preStop command to detect that no requests there, it looks like:

                while true; do
                  RUNNING=$(curl -s http://localhost:8000/metrics | grep 'vllm:num_requests_running' | grep -v '#' | awk '{print $2}')
                  WAITING=$(curl -s http://localhost:8000/metrics | grep 'vllm:num_requests_waiting' | grep -v '#' | awk '{print $2}')
                  if [ "$RUNNING" = "0.0" ] && [ "$WAITING" = "0.0" ]; then
                    echo "Terminating: No active or waiting requests, safe to terminate" >> /proc/1/fd/1
                    exit 0
                  else
                    echo "Terminating: Running: $RUNNING, Waiting: $WAITING" >> /proc/1/fd/1
                    sleep 5
                  fi
                done

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[googs1025]

So what we want to do is support the prestophook field in the playground or backendruntime and put it in the pod template spec? 🤔

[kerthcet]

Yes, for Playground, we'll put it in the backendRuntime, so people don't need to specify this personally, and we'll pass the command to the Inference Service to make sure the service has the same command as well. But for Inference Service, people have to handle this by hand. Would you like to take it?

[googs1025]

I may take a look next week

[googs1025]

Is this the api-change we expected?

type BackendRuntimeConfig struct {
 // BackendName represents the inference backend under the hood, e.g. vLLM.
 // +kubebuilder:default=vllm
 
 ...
 // Lifecycle defines the hooks executed during the lifecycle of the container.
 // +optional
 Lifecycle *corev1.Lifecycle `json:"lifecycle,omitempty"`
}

apiVersion: inference.llmaz.io/v1alpha1
kind: Playground
metadata:
  name: qwen2-0--5b
spec:
  replicas: 1
  modelClaim:
    modelName: qwen2-0--5b-gguf
  backendRuntimeConfig:
    backendName: llamacpp
    configName: default
    args:
      - -fa # use flash attention
    lifecycle:
      postStart:
        exec:
          command:
            - /bin/sh
            - -c
            - echo "Container started."
      preStop:
        exec:
          command:
            - /bin/sh
            - -c
            - echo "Container is stopping."

[kerthcet]

Let's support this in BackendRuntime only, reduce the cognitive cost for endusers who're not familiar with K8s. Then the administrator will configure this in the backend, reuse for all the backendRuntime. wdyt?

[kerthcet]

/reopen

We need to add the configurations for the in-tree backendRuntimes, for example the vllm.