[bugfix] workqueue metrics are mising

[RainbowMango]

What happened:
We received reports that the controller's work queue metrics can not be emitted. (See #5945 for more details, thanks to @CharlesQQ for reporting this).

This issue occurred before and was fixed in release 1.0 in #945, and then we made some subsequent impairments in #2899 and #3012. (Thanks to @Garrybest for doing this)

However after #2998 and #4706, this issue was introduced again in release-1.10.

This issue is used to track any relevant fix to ensure the problem is fully resolved without introducing any potential risk.

Root Cause:
From Go 1.21, the program initialization order was changed, and the metrics from the controller-runtime missed the chance of registration.

Iteration tasks:

• Revist #2998, fix potential risk or twist solutions.
• Clarify the usage of Service Reference in the ResourceInterpreterWebhookConfiguration API. (Explain the interpreter webhook comunication channel #6109, @RainbowMango)
• Clarify the usage documentation of Service Reference in the ResourceInterpreterWebhookConfiguration. (#5969, @XiShanYongYe-Chang)
• Minimum fix on master. (Fix the issue of missing workqueue metrics in the Karmada controller #5972, @XiShanYongYe-Chang )
• Backport to release branches. (release-1.10~release-1.12) (Cherrypick #5972: fix the issue of missing workqueue metrics in the Karmada controller #5979)
• Cleanup workaround made on release-1.0. (remove workaround code for resolving workqueue metrics loss by modifying import order #5945, @chaosi-zju)
• Introduce tests to cover the metrics. (introduce tests to check whether workqueue metrics exist #6003, @chaosi-zju)
• Expose metrics of karmada-metrics-adapter. (expose metrics of karmada-metrics-adapter #6013, @chaosi-zju)

Other Potential risks

• karmada-webhook panic in case of invalid configuration. (#5960, @XiShanYongYe-Chang)
• Cannot access karmada-interpreter-webhook by configuring Service in ResourceInterpreterWebhookConfiguration #5958

References:

• From Go 1.21, the program initialization order was specified more precisely. See Go 1.21 release notes for more details.

Echo from the release notes:

Package initialization order is now specified more precisely. The new algorithm is:

• Sort all packages by import path.
• Repeat until the list of packages is empty:
  • Find the first package in the list for which all imports are already initialized.
  • Initialize that package and remove it from the list.

• Go Program initialization
• Issue in controller-runtime: Add a new metrics to indicates the the current queue length. kubernetes-sigs/controller-runtime#1969
• Issue in Kubernetes: There is a conflict between the metrics of controller-runtime and component-base, the metric workqueue_depth of controller-runtime repository not take effect kubernetes/kubernetes#128326

[chaosi-zju]

the PR #6003 works for the task Introduce tests to cover the metrics

[RainbowMango]

Got it.

[chaosi-zju]

I found another sub issue:

https://github.com/karmada-io/karmada/blob/master/pkg/estimator/server/metrics/metrics.go

you can refer to this file and find that karmada-schedule-estimator has some custom metrics, but they are also missed.

---

so does scheduler's custom metrics:

https://github.com/karmada-io/karmada/blob/master/pkg/estimator/server/metrics/metrics.go

[chaosi-zju]

you can refer to this file and find that karmada-schedule-estimator has some custom metrics, but they are also missed.

The actual situation is that these custom metrics are not available right after installing the two components (not even zero values).

They can only be viewed once actually resources are scheduled.

Besides, ginkgo does not guarantee order of specs. in fact it prefers to enable randomized spec orders to prevent spec pollution, refer to onsi/ginkgo#1165 (comment)

[RainbowMango]

The actual situation is that these custom metrics are not available right after installing the two components (not even zero values).
They can only be viewed once actually resources are scheduled.

Just a guess, this is due to the metrics needing some labels to be initiated. Like result and type in estimating_request_total.

Show an example of the metrics?