feature: metrics support for controller

[googs1025]

What would you like to be added:

• add some useful metrics for controller.
  metrics are very useful for debugging or troubleshooting. We should add some indicators to intuitively determine whether there is a problem with the application.

and add some example doc like this: https://github.com/kubernetes-sigs/jobset/tree/main/site/static/examples/prometheus-operator

/kind feature

Why is this needed:

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[googs1025]

But I haven’t thought of all the metrics yet, I will come back to this issue again

[kerthcet]

Thanks @googs1025 I'm eager for monitoring system for a long time just have no time. Maybe the first step is to enable the prometheus integration.

[googs1025]

I don't know if my understanding is wrong. Kubebuilder has integrated prometheus by default. We just need to turn on the configuration. 🤔
https://github.com/InftyAI/llmaz/tree/main/config/prometheus

https://github.com/InftyAI/llmaz/blob/main/config/default/kustomization.yaml#L31

[kerthcet]

Yes, but it still needs some document for user to follow. Would you like to pick?

[kerthcet]

The requirement for metric support is urgent, my proposal about lora autoscaling requires the metric as well. It would be great to have the integration with prometheus.
/priority urgent

[kerthcet]

/priority important-critical-urgent

[kerthcet]

/kind feature

[googs1025]

I will look into this issue in this week.