Support preStop lifecycle for backendRuntimes

[kerthcet]

What would you like to be added:

We finished the work with vllm, see #319, we should support other backendRuntimes as well.

• sglang
• llama.cpp
• ollama

Why is this needed:

Avoid request disruption.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[kerthcet]

/help

[kerthcet]

cc @googs1025

[cr7258]

/assign
I want to take this opportunity to get familiar with the project.

[googs1025]

some info you can get from: #319 😄

[cr7258]

Ollama does not provide metrics for the number of requests being processed, and it currently even lacks a Prometheus metrics endpoint. I have created a feature request issue on the Ollama repo.

[kerthcet]

I have created a ollama/ollama#10419 on the Ollama repo.

Great!

And we can go forward with sglang which I believe has similar metrics:

• sglang:num_queue_reqs: The number of requests in the waiting queue
• sglang:num_running_reqs: The number of running requests

See https://docs.sglang.ai/references/production_metrics.html#production-metrics for more details.