Add health endpoint with readiness and liveness probes #416

ChristianGeie · 2025-08-15T05:52:02Z

This PR introduces a comprehensive health check endpoint (/healthz) to significantly improve the robustness and reliability of the sidecar when running in a Kubernetes environment. It allows for proper readiness and liveness checks, enabling better lifecycle management and self-healing capabilities.

Key Features

New /healthz Endpoint: A new HTTP endpoint is available on port 8080 (configurable via the HEALTH_PORT environment variable)
Readiness Probe:
- The sidecar now reports as "ready" (HTTP 200) only after the initial synchronization of all configured resources is complete
- This prevents the main application container from starting or receiving traffic prematurely, ensuring all configuration files are present at startup
Liveness Probe:
- The probe continuously monitors the sidecar's health by checking two critical conditions:
  - Kubernetes API Contact: Verifies that the sidecar has had successful contact with the Kubernetes API within the last 60 seconds
  - Watcher Process Health: Ensures that all internal watcher subprocesses are running correctly
- If any check fails, the probe fails, signaling Kubernetes to restart the container

Enhancements

Reduced Log Noise: Access logs for frequent /healthz requests are automatically filtered out to keep application logs clean and focused
Fail-Fast on Process Death: The main process now exits immediately if a critical watcher subprocess dies, ensuring a prompt restart by Kubernetes

Testing

The CI pipeline has been enhanced with new tests to validate this functionality:

A test to confirm the Uvicorn health server starts successfully
A liveness test that simulates a watcher process failure and asserts that Kubernetes restarts the pod as expected
A Kubernetes Config load test for Sleep and Watch based sidecar

This feature makes the sidecar more production-ready and easier to operate reliably within Kubernetes.

add mark_ready() after initial write to container filesystem

* Sending URL request need a log Info event: Requiring debug log level to see url request is either unhelpful or too noisy. Also mirrors script running log event. * When logging of url request "None" was reported if defaulting to GET method. * Useful to see payload in event og POST. * Removal of ellipsis in config load. * A missing, non-mandatory, option (folder annotation) shouldn't result in a warning.

Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.3...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [kubernetes](https://github.com/kubernetes-client/python) from 32.0.1 to 33.1.0. - [Release notes](https://github.com/kubernetes-client/python/releases) - [Changelog](https://github.com/kubernetes-client/python/blob/v33.1.0/CHANGELOG.md) - [Commits](kubernetes-client/python@v32.0.1...v33.1.0) --- updated-dependencies: - dependency-name: kubernetes dependency-version: 33.1.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [logfmter](https://github.com/josheppinette/python-logfmter) from 0.0.9 to 0.0.10. - [Release notes](https://github.com/josheppinette/python-logfmter/releases) - [Changelog](https://github.com/josheppinette/python-logfmter/blob/main/HISTORY.md) - [Commits](josheppinette/python-logfmter@v0.0.9...v0.0.10) --- updated-dependencies: - dependency-name: logfmter dependency-version: 0.0.10 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 4 to 5. - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@v4...v5) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]>

especially for SSLCertVerificationError

to avoid returning "none" which would lead to an AttributeError in the caller code. add exception handling for SSLError and RetryError add debug log message for dummy response

in _get_file_data_and_name() to avoid AttributeError

…master

This reverts commit d1dc57c.

This reverts commit 12eaa68.

This reverts commit 7f0399e.

Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4...v5) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]>

…ation

…quest function

…al None responses

…zation and add a check for 5xx error handling

… instead of existence

…zation with more detailed error messages and add a check for empty 500.txt files

…eness-Probes

The Kubernetes client configuration was not being reliably initialized in the forked processes used by the WATCH and SLEEP methods. This was due to a call to a non-existent function `_ensure_kube_config_in_child` which was left over from a previous refactoring. This change replaces the broken call with a call to the correct `_initialize_kubeclient_configuration` function at the start of the `list_resources` and `_watch_resource_iterator` functions. This ensures that a valid Kubernetes client configuration is loaded in every process, resolving potential connectivity issues with the API server. The unused and broken helper function `_init_kube_in_child_if_needed` has been removed.

This commit enhances the integration test suite to verify that the Kubernetes client is correctly configured in both WATCH and SLEEP modes. - A new 'sidecar-sleep' pod is added to the test resources, configured to run in SLEEP mode. - The 'build_and_test' workflow is updated to include this new pod in the test run. - A new verification step, "Verify K8s Config Loading", is added. This step dynamically retrieves the ClusterIP of the Kubernetes API service and checks the logs of both the WATCH mode pod ('sidecar') and the SLEEP mode pod ('sidecar-sleep') to ensure they connect to the correct API server address.

Christian Geie and others added 30 commits April 28, 2025 12:38

add fastapi and uvicorn to requirements

245781e

Implement FastAPI health check endpoint and server startup

f2166cd

add call of start_health_server() to beginning of main()

dab83e6

add mark_ready() after initial write to container filesystem

call mark_ready() after first event happend, but only once

e0df26d

add pod with readyness and liveness probes

321e9bc

deploy container in test scenario

8b2b263

remove fishy ready check for test deployment

bf632ae

upgrade python:alpine3.21 to python:alpine3.22 to fix vulnerabilities

79fd799

downgrade kubernetes lib to 32.0.1

908fc13

add exception handling while loading k8s configuration

8a3d3f5

especially for SSLCertVerificationError

return a response object in any cases

50a6fed

to avoid returning "none" which would lead to an AttributeError in the caller code. add exception handling for SSLError and RetryError add debug log message for dummy response

add check for 'none' response

e919464

in _get_file_data_and_name() to avoid AttributeError

update github-tag-action to version 1.73.0 and set default branch to …

6b10fc8

…master

Revert "return a response object in any cases"

80b8007

This reverts commit d1dc57c.

Revert "add exception handling while loading k8s configuration"

a91bcc1

This reverts commit 12eaa68.

Revert "add check for 'none' response"

c9850ba

This reverts commit 7f0399e.

Add SSL certificate error handling during Kubernetes client initializ…

36a8791

…ation

Add error handling for SSL verification and request retries in the re…

ca5638c

…quest function

Improve request handling in _get_file_data_and_name to manage potenti…

7df22d5

…al None responses

Improve the verification of sidecar files after the initial synchroni…

f7c7fd9

…zation and add a check for 5xx error handling

extend gitignore

69cddce

Improve verification of 5xx error handling by checking for empty file…

4641e99

… instead of existence

Add error logging for HTTP errors in request function

603a6d3

Improve the verification of sidecar-5xx files after initial synchroni…

028fb2e

…zation with more detailed error messages and add a check for empty 500.txt files

ChristianGeie and others added 18 commits October 13, 2025 10:10

Merge branch 'master' into Add-Health-Endpoint-with-Readiness-and-Liv…

b3353c8

…eness-Probes

add timeout for installing and update configmaps and secrets step

1dd9597

add function to to (re)initialize kube config in every child process

9155ecd

before every API-object we initialize child processes

4a196c6

remove color output from univcorn

6eef2f2

adjust needed log messages for container

1624a68

fix intentions

ed08c54

fix intentions

5c296a6

increase expected log file count

0f1b20e

add function to check min log count and adjust tests

7c67a1b

comment out check

bde2ccf

remove line

5a977bb

increase wait duration for liveness probe

c15a863

change test sidecar pod name for liveness probe check

2ed5c24

move k8s client config in separate file

2a5b8f9

be sure watcher has a client config

5d7345e

fixing some intentions

f813d98

fixing typo

3a46b05

ChristianGeie mentioned this pull request Oct 28, 2025

wyn_load_config: - move k8s client initialization into own file, so watcher subprocesses can have the correct client #434

Closed

This was linked to issues Oct 28, 2025

1.30.10+ Breaks kubernetes api connection #431

Closed

Is 1.31.0 stable load? #435

Closed

Christian Geie added 3 commits October 28, 2025 10:42

fixing typo

2850707

ChristianGeie added enhancement New feature or request test new or better tests github_actions Pull requests that update Github_actions code python Pull requests that update Python code labels Oct 28, 2025

ChristianGeie merged commit 397a806 into master Oct 28, 2025
12 checks passed

Infinoid mentioned this pull request Nov 24, 2025

sidecar causes error listen tcp :8080: bind: address already in use perses/helm-charts#138

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add health endpoint with readiness and liveness probes #416

Add health endpoint with readiness and liveness probes #416

Uh oh!

ChristianGeie commented Aug 15, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add health endpoint with readiness and liveness probes #416

Add health endpoint with readiness and liveness probes #416

Uh oh!

Conversation

ChristianGeie commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ChristianGeie commented Aug 15, 2025 •

edited

Loading