1.30.10+ Breaks kubernetes api connection

[NickyHeuperman]

1.30.9 still works correctly.

After updating to 1.30.10 or 1.30.11 requests fail with the following logs

{"time": "2025-10-14T08:33:57.420929+00:00", "level": "INFO", "msg": "Starting collector"} {"time": "2025-10-14T08:33:57.421394+00:00", "level": "INFO", "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory"} {"time": "2025-10-14T08:33:57.421884+00:00", "level": "INFO", "msg": "Loading incluster config..."} {"time": "2025-10-14T08:33:57.424052+00:00", "level": "INFO", "msg": "Config for cluster api at 'https://192.168.20.254:6443' loaded."} {"time": "2025-10-14T08:33:57.424398+00:00", "level": "INFO", "msg": "Unique filenames will not be enforced."} {"time": "2025-10-14T08:33:57.424671+00:00", "level": "INFO", "msg": "5xx response content will not be enabled."} {"time": "2025-10-14T08:34:09.934155+00:00", "level": "WARNING", "msg": "Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fceb16cbe00>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource&timeoutSeconds=60&watch=True"} {"time": "2025-10-14T08:34:09.935806+00:00", "level": "WARNING", "msg": "Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fceb0e52990>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource&timeoutSeconds=60&watch=True"} {"time": "2025-10-14T08:34:09.937058+00:00", "level": "WARNING", "msg": "Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fceb0e52c10>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource&timeoutSeconds=60&watch=True"} {"time": "2025-10-14T08:34:09.938434+00:00", "level": "ERROR", "msg": "MaxRetryError when calling kubernetes: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource&timeoutSeconds=60&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fceb0eb1220>: Failed to establish a new connection: [Errno 111] Connection refused'))\n"}

What's strange here is that connections seem to be tried to localhost:80 instead of the kubernetes api endpoint

[ChristianGeie]

Hi @NickyHeuperman thx for your report.
It's a known issue with following root cause: Since 1.30.10 processes relatet to configmaps and secrets are running separately and each of these processes requires its own kube config. I did not sufficiently ensure that this was always available. So, with no valid kubeconfig, connection fall back to defaults and try localhost:80. I am already working on a fix.

[ChristianGeie]

@NickyHeuperman Can you confirm that v1.30.9 works for you?

[NickyHeuperman]

Indeed 1.30.9 works

[ChristianGeie]

thx a lot. I mark release 1.30.11 and 1.30.10 as bugged.

[Apocrathia]

I can also confirm that downgrading from v1.30.11 to v1.30.9 also resolved this issue for me. https://gitlab.com/Apocrathia/homelab/-/commit/c4f3efd249efdc3a7c71de19d9aaf8ce5440e08f

[muhlba91]

i have a funny case with 1.30.11.

i am running this sidecar once as an init container with method list and then as an actual sidecar with method watch. (see https://github.com/muhlba91/homelab-home-cluster-applications/blob/main/common/core/gatus/app/values.yaml)

In 1.30.10 this works flawlessy but with 10.30.11 the actual sidecar using watch container gets killed with OOM. i tried incrasing the memory already but the same happens.
the funny part is that looking at the changes made between these releases, i wouldn't be able to spot anything which causes this behaviour.
it's also consisten across 2 clusters with the same configuration.