✨ Add controller_runtime_reconcile_timeouts_total metric to track ReconciliationTimeout timeouts

[godwinpang]

Add controller_runtime_reconcile_timeouts_total metric

Summary

This PR adds a new metric controller_runtime_reconcile_timeouts_total to track when a controller's reconciliation has reached a ReconciliationTimeout. This provides visibility into when reconcile operations time out due to the controller-runtime wrapper timeout, allowing users to alert / monitor unexpectedly long running controller reconiliations.

Problem

The ReconciliationTimeout feature was added to prevent head-of-line (HOL) blocking by ensuring that a single reconcile cannot block a worker indefinitely. However, when a reconcile exits because ReconciliationTimeout fired, the existing metrics treat it like a normal return. There was no way to distinguish between:

• A reconcile that completed successfully
• A reconcile that timed out due to the ReconciliationTimeout context timeout

Metric

controller_runtime_reconcile_timeouts_total{controller="<name>"}

• Type: Counter
• Labels: controller (controller name)
• Description: Total number of reconciliation timeouts per controller. This metric only increments when the wrapped context with ReconcileTimeout has deadline exceeded, not when user reconciler cancels the context or completes before the timeout.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: godwinpang
Once this PR has been reviewed and has the lgtm label, please assign vincepri for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @godwinpang. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[godwinpang]

sorry this ended up requesting a lot of reviewers - It's been a while since I've worked on c-r and I forgot bout the commit conventions hence the force pushes

[alvaroaleman]

/ok-to-test

[JoelSpeed]

Should this not be part of the before each?