Certwatcher only detects cert changes on leader

[hrak]

What broke? What's expected?

On a controller generated with kubebuilder 4.5.2, we are using secure metrics and a deployment with 2 replicas, and noticed that only the leader notices & acts on cert changes, any non-leaders will not reload the cert.

In main.go the metrics certwatcher is added to the manager as a runnable using mgr.Add. It seems like this runnable is acting like a LeaderElectionRunnable and thus only operates on the leader.

logs from the leader:

2025-04-30T15:10:24Z    DEBUG    controller-runtime.certwatcher    certificate event    {"event": "CHMOD         \"/tmp/k8s-metrics-server/metrics-certs/tls.key\""}
2025-04-30T15:10:24Z    INFO    controller-runtime.certwatcher    Updated current TLS certificate
2025-04-30T15:10:24Z    DEBUG    controller-runtime.certwatcher    certificate event    {"event": "REMOVE        \"/tmp/k8s-metrics-server/metrics-certs/tls.key\""}
2025-04-30T15:10:24Z    INFO    controller-runtime.certwatcher    Updated current TLS certificate
2025-04-30T15:10:24Z    DEBUG    controller-runtime.certwatcher    certificate event    {"event": "CHMOD         \"/tmp/k8s-metrics-server/metrics-certs/tls.crt\""}
2025-04-30T15:10:24Z    INFO    controller-runtime.certwatcher    Updated current TLS certificate
2025-04-30T15:10:24Z    DEBUG    controller-runtime.certwatcher    certificate event    {"event": "REMOVE        \"\""}
2025-04-30T15:10:24Z    INFO    controller-runtime.certwatcher    Updated current TLS certificate

logs from a non leader (instantly starts logging bad certificate after rotation, can only be solved by restarting the pod):

I0425 13:16:25.693120       1 leaderelection.go:250] attempting to acquire leader lease eco-system/etcd-cluster-operator-controller-leader-election-helper...
2025/04/30 15:50:42 http: TLS handshake error from 192.168.2.129:45812: remote error: tls: bad certificate

We were expecting non-leader pods to reload the cert too.

Reproducing this issue

No response

KubeBuilder (CLI) Version

kubebuilder 4.5.2

PROJECT version

3

Plugin versions

go.kubebuilder.io/v4

Other versions

No response

Extra Labels

No response

[camilamacedo86]

Hey, thank you so much for raising this and sharing the detailed logs — really appreciated! 🙌

From what you've described, I think it is the expected behavior of controller-runtime when leader election is enabled.

Only the leader pod runs the manager, which includes components like the metrics server and certwatcher. So when you add the certwatcher with mgr.Add(...), it only runs in the leader, and that’s why only it detects and reloads cert changes.

Non-leader pods stay idle—they don’t start the manager or any runnables, including the certwatcher. I understand that:

✅ When the non-leader becomes leader, it should:

• Start the manager,
• Load the latest certs from disk,
• And start watching for changes again.

🚨 If the new leader fails to pick up valid certs, or continues to throw tls: bad certificate errors, and does not work without intervention then we might need to look on that.

1. Could you confirm whether everything works as expected after the non-leader becomes leader?
2. Could you outline the step-by-step process you're using to reproduce this issue in a fresh Kubebuilder project?

That way we can try to reproduce it on our end and help identify whether there's a gap in the scaffolding or some subtle setup issue.

Thanks again for taking the time to report this — looking forward to your reply! 🙏

[hrak]

Hi @camilamacedo86

If this is expected behavior, for the webhook server that's not an issue, as only the leader's webhook server will be active and handling requests. But for metrics this is a different story. Monitoring metrics just breaks for the non-leader pods after rotation, as Prometheus can't connect anymore due to a bad certificate (192.168.2.129:45812 mentioned in the non-leader log above is our Prometheus instance).

I believe we tested the non-leader becoming a leader scenario yesterday, and it worked (as in: it picked up the new cert after becoming a leader), but we will verify this once more and report back.

[camilamacedo86]

Hi @hrak,

Please let me know if I misunderstood any part of your setup, but I’d like to recap the scenario as I understand it:

✅ Reproduction Steps

• Create a new project using Kubebuilder.
• Enable metrics with TLS support.
• The project sets up the certwatcher to reload metrics certificates.
• Enable Prometheus for metrics scraping using TLS and certs rotated with certwatcher
  (default scaffold with Prometheus, [METRICS-WITH-CERTS][PROMETHEUS-WITH-CERTS] all enabled)

All works as expected when running with a single replica.

🐛 The issue seems to occur only after modifying the controller Deployment to use replicas: 2. From your description, it sounds like:

• Prometheus stops being able to scrape metrics from the non-leader pod after cert rotation (due to outdated or invalid certs).
• But metrics continue to work correctly on the leader.

Is that accurate? Or do you still have access to metrics via the leader, and the non-leader errors are just noise in the logs?

---

🤔 A Few Follow-up Questions & Thoughts

a) If you keep the manager at a single replica (i.e., no manual change to replicas: 2), does everything behave as expected, with proper cert rotation and availability via the leader?

b) If so, then perhaps the best HA pattern would be to use the LeaderElectionReleaseOnCancel = true option in the manager config. This allows graceful leadership handoff without needing multiple concurrently running manager instances.

c) That said, since controller-runtime allows us to configure TLS for metrics and supports setting up a cert watcher, we could reasonably expect this to work regardless of leader status. Should it properly work when we have two replicas?

kubebuilder/testdata/project-v4/cmd/main.go

Lines 160 to 185 in 6790714

	// If the certificate is not specified, controller-runtime will automatically
	// generate self-signed certificates for the metrics server. While convenient for development and testing,
	// this setup is not recommended for production.
	//
	// TODO(user): If you enable certManager, uncomment the following lines:
	// - [METRICS-WITH-CERTS] at config/default/kustomization.yaml to generate and use certificates
	// managed by cert-manager for the metrics server.
	// - [PROMETHEUS-WITH-CERTS] at config/prometheus/kustomization.yaml for TLS certification.
	if len(metricsCertPath) > 0 {
	setupLog.Info("Initializing metrics certificate watcher using provided certificates",
	"metrics-cert-path", metricsCertPath, "metrics-cert-name", metricsCertName, "metrics-cert-key", metricsCertKey)
	var err error
	metricsCertWatcher, err = certwatcher.New(
	filepath.Join(metricsCertPath, metricsCertName),
	filepath.Join(metricsCertPath, metricsCertKey),
	)
	if err != nil {
	setupLog.Error(err, "to initialize metrics certificate watcher", "error", err)
	os.Exit(1)
	}
	metricsServerOptions.TLSOpts = append(metricsServerOptions.TLSOpts, func(config *tls.Config) {
	config.GetCertificate = metricsCertWatcher.GetCertificate
	})
	}

kubebuilder/testdata/project-v4/cmd/main.go

Lines 187 to 205 in 6790714

	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
	Scheme: scheme,
	Metrics: metricsServerOptions,
	WebhookServer: webhookServer,
	HealthProbeBindAddress: probeAddr,
	LeaderElection: enableLeaderElection,
	LeaderElectionID: "da1d9c86.testproject.org",
	// LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily
	// when the Manager ends. This requires the binary to immediately end when the
	// Manager is stopped, otherwise, this setting is unsafe. Setting this significantly
	// speeds up voluntary leader transitions as the new leader don't have to wait
	// LeaseDuration time first.
	//
	// In the default scaffold provided, the program ends immediately after
	// the manager stops, so would be fine to enable this option. However,
	// if you are doing or is intended to do any operation such as perform cleanups
	// after the manager stops then its usage might be unsafe.
	// LeaderElectionReleaseOnCancel: true,
	})

So perhaps this highlights an improvement opportunity in controller-runtime — to make the metrics TLS watcher function correctly across replicas, not just the leader. It may make sense to raise this for further discussion with maintainers if it turns out to be an architectural limitation. @sbueringer @alvaroaleman @vincepri Could you please give a hand on this one? WDYT?

Thanks again for surfacing this — looking forward to your thoughts!

[camilamacedo86]

HI @hrak

Please let us know if you can check the above comment.
Could you help us by providing the steps to reproduce the scenario with the latest version?

[sbueringer]

To be honest, I'm not aware of any issues that this doesn't work on non-leader replicas in controller-runtime.
We setup the GetCertificate func here: https://github.com/kubernetes-sigs/controller-runtime/blob/3b0b9958086ff68d94d070da3c9436742a5a1e6b/pkg/metrics/server/server.go#L301

I have no indication that any of this code depends on the replica being the leader

[sbueringer]

Also I don't know why kubebuilder is setting the GetCertificate func explicitly, does the CR implementation not work for kube-builder?

[camilamacedo86]

Also I don't know why kubebuilder is setting the GetCertificate func explicitly, does the CR implementation not work for kube-builder?

Thank you for the input 🥇

We explicitly set GetCertificate here because Kubebuilder supports custom cert paths to inform the certs for webhooks and metrics as usage of cert-manager for the certs.

If you see a better way to handle this—or something we should change—please let us know!
We'd love to improve the default scaffold to support all configs and follow controller-runtime best practices.

Please let me know how we should be doing so.
Here is the main: https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/cmd/main.go
How do you suggest we change it to allow all the possible combinations still, and fail back in c+r default behaviour as much as possible?

[DustinChaloupka]

We have identified this as an issue we are having as well.

as only the leader's webhook server will be active and handling requests.

I do not think this is accurate. The validating/mutating webhook configuration specifies the Service to match against and send validating/mutating requests to. If you have multiple Pods running, it should be sending it to all of them.

Could you confirm whether everything works as expected after the non-leader becomes leader?

We confirmed this is the case.

2025/08/11 14:22:08 http: TLS handshake error from 127.0.0.1:33316: remote error: tls: unknown certificate authority
2025/08/11 14:45:33 http: TLS handshake error from 10.10.10.10:41054: EOF
2025/08/11 14:45:56 http: TLS handshake error from 10.10.10.8:50400: EOF

# Leader pod was deleted here
I0811 14:48:36.147961       1 leaderelection.go:271] successfully acquired lease my-namespace/84d4f733.example.com
2025-08-11T14:48:36Z	DEBUG	events	my-operator-controller-manager-6548cfc97c-nwm4d_f20f36ea-2e97-4741-a744-31ef085fd9fa became leader	{"type": "Normal", "object": {"kind":"Lease","namespace":"my-namespace","name":"84d4f733.example.com","uid":"d546473d-9597-4dde-bbee-8c4436c29bf6","apiVersion":"coordination.k8s.io/v1","resourceVersion":"1412594917"}, "reason": "LeaderElection"}
2025-08-11T14:48:36Z	INFO	controller-runtime.certwatcher	Starting certificate poll+watcher	{"interval": "10s"}
2025-08-11T14:48:36Z	INFO	controller-runtime.certwatcher	Starting certificate poll+watcher	{"interval": "10s"}

This should be reproducible in a new Kubebuilder project after enabling webhooks, port-forwarding to the webhook port to capture the current certificate, and then recreating the webhook configuration so it has a new certificate. Only the leader should update at that point, verifiable by checking the webhook port certificate again.

Only the leader pod runs the manager, which includes components like the metrics server and certwatcher. So when you add the certwatcher with mgr.Add(...), it only runs in the leader, and that’s why only it detects and reloads cert changes.

Non-leader pods stay idle—they don’t start the manager or any runnables, including the certwatcher.

This all seems to match this understanding. Because of the webhook configuration routing all requests to all replicas though, this seems like the certwatchers should not be added with mgr.Add(...).

[alvaroaleman]

as only the leader's webhook server will be active and handling requests

Yes that is incorect, all replicas will serve webhooks, leader or not.

At least in the way controller-runtime uses the cert-watcher I don't see why it would not work on non-leader replicas, as it is constructed and started inside the webhook server: https://github.com/kubernetes-sigs/controller-runtime/blob/27041639576376da329b9968948eae1dbc2c9777/pkg/webhook/server.go#L207-L217

I don't really know what kubebuilder does. If it adds it to the manager it will indeed not work, as its not a LeaderElectionRunnable/Doesn't declare that it does not need leader election and the default is to require leader election.

Can someone point me on some OSS code that has this issue?

[alvaroaleman]

Can someone point me on some OSS code that has this issue?

Nevermind, it is already in the previous comments. This change here in controller-runtime will make it run on non-leader election runnables as well: kubernetes-sigs/controller-runtime#3273
People can also wrap the thing to make it a leader election runnable in the meantime.

And to be clear, this works correctly if you are using stock controller-runtime, it just doesn't work correctly with the way kubebuilder sets things up which is why IMHO this is not a c-r bug.

[DustinChaloupka]

Thanks @alvaroaleman ! I did not realize the mgr.Add(...) did other things with the configured components, so was trying out removing that, but obviously did not fully work as expected. Thanks for the upstream change!

[sbueringer]

Also I don't know why kubebuilder is setting the GetCertificate func explicitly, does the CR implementation not work for kube-builder?

Thank you for the input 🥇

We explicitly set GetCertificate here because Kubebuilder supports custom cert paths to inform the certs for webhooks and metrics as usage of cert-manager for the certs.

If you see a better way to handle this—or something we should change—please let us know! We'd love to improve the default scaffold to support all configs and follow controller-runtime best practices.

Please let me know how we should be doing so. Here is the main: master/testdata/project-v4/cmd/main.go How do you suggest we change it to allow all the possible combinations still, and fail back in c+r default behaviour as much as possible?

I'm not sure which use cases you are trying to address that are not possible otherwise. In general you could set CertDir / CertName / KeyName on metricsserver.Options instead of using them to setup a certwatcher. The same should apply to webhook.Options

i.e. instead of

	if len(metricsCertPath) > 0 {
		setupLog.Info("Initializing metrics certificate watcher using provided certificates",
			"metrics-cert-path", metricsCertPath, "metrics-cert-name", metricsCertName, "metrics-cert-key", metricsCertKey)

		var err error
		metricsCertWatcher, err = certwatcher.New(
			filepath.Join(metricsCertPath, metricsCertName),
			filepath.Join(metricsCertPath, metricsCertKey),
		)
		if err != nil {
			setupLog.Error(err, "to initialize metrics certificate watcher", "error", err)
			os.Exit(1)
		}

		metricsServerOptions.TLSOpts = append(metricsServerOptions.TLSOpts, func(config *tls.Config) {
			config.GetCertificate = metricsCertWatcher.GetCertificate
		})
	}

you could do this

	if len(metricsCertPath) > 0 {
		setupLog.Info("Initializing metrics certificate watcher using provided certificates",
			"metrics-cert-path", metricsCertPath, "metrics-cert-name", metricsCertName, "metrics-cert-key", metricsCertKey)
		metricsServerOptions.CertDir = metricsCertPath
		metricsServerOptions.CertName = metricsCertName
		metricsServerOptions.KeyName = metricsCertKey
	}

[camilamacedo86]

Hi @sbueringer,

Thanks for the suggestion and for outlining the alternative. I did take a look at it, but the main blocker is that switching to CertDir/CertName/KeyName directly (as in your example) would mean not using CertWatcher—and that would remove the automatic certificate rotation functionality.

That’s why Kubebuilder currently wires things up this way:
https://github.com/kubernetes-sigs/controller-runtime/blob/2c7b0ba7a51f275722638637f1316a0ae454a113/pkg/certwatcher/certwatcher.go#L44

What we’re trying to achieve

• Use real certs from cert-manager for webhooks/metrics.
• Watch (rotate) certs automatically.

CertWatcher is what allows us to handle updates to the certificate files at runtime without restarting the process, which is important when certs are rotated by cert-manager or other tooling. Without it, any rotation would require a manual restart to pick up the new certs.

So while your suggested approach works for static certs, it wouldn’t cover the rotation use case we need to support.

OR if we do directly

// Metrics
metricsServerOptions.CertDir  = metricsCertPath
metricsServerOptions.CertName = metricsCertName
metricsServerOptions.KeyName  = metricsCertKey
// No custom mgr.Add(certwatcher)

Will the contriller-runtime handle the rotation internally?