sec: Don't block for update on OIDC service start#28608
Merged
michael-redpanda merged 1 commit intoNov 18, 2025
Conversation
Upon OIDC service start, it will attempt to get the certificates from the IdP. If the service is unable to communicate with the IdP, this process times out after 5 seconds. If this occurs during cluster start up, this will prevent leadership election from occurring, which is especially troublesome with the controller. This has led to a cascade of other failures (such as being unable to assign the cluster ID in the metrics report service). This change starts the update in the background to permit the rest of startup to continue without being blocked. Signed-off-by: Michael Boquard <michael@redpanda.com>
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR addresses a blocking issue during OIDC service startup that could prevent controller leadership election. Previously, the OIDC service would synchronously wait for certificate updates from the IdP during startup, causing a 5-second timeout if the IdP was unreachable. This blocking behavior during cluster startup prevented critical operations like leadership election and cluster ID assignment.
Key Changes:
- Changed OIDC service startup to spawn the update operation asynchronously instead of blocking on it
- Removed the gate holder acquisition in favor of
ssx::spawn_with_gateto manage the background task lifecycle
BenPope
reviewed
Nov 18, 2025
| auto holder = _gate.hold(); | ||
| co_await update(); | ||
| ssx::spawn_with_gate(_gate, [this] { return update(); }); | ||
| return ss::now(); |
Contributor
There was a problem hiding this comment.
Is it just the 5s delay that causes the failed startup? I.e., does that lead to a timeout elsewhere?
Contributor
Author
There was a problem hiding this comment.
There's a timeout in config_manager awaiting controller leadership. This timeout is preventing controller leadership election from forming.
Collaborator
Retry command for Build#76558please wait until all jobs are finished before running the slash command |
BenPope
approved these changes
Nov 18, 2025
Collaborator
CI test resultstest results on build#76558
|
Contributor
Author
Collaborator
|
/backport v25.3.x |
Collaborator
|
/backport v25.2.x |
Collaborator
|
/backport v25.1.x |
This was referenced Nov 18, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upon OIDC service start, it will attempt to get the certificates from the IdP. If the service is unable to communicate with the IdP, this process times out after 5 seconds. If this occurs during cluster start up, this will prevent leadership election from occurring, which is especially troublesome with the controller. This has led to a cascade of other failures (such as being unable to assign the cluster ID in the metrics report service).
This change starts the update in the background to permit the rest of startup to continue without being blocked.
Backports Required
Release Notes
Bug Fixes