Skip to content

lightkube doesnot work with httpx 0.28.x #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hemanthnakkina opened this issue Dec 2, 2024 · 12 comments
Closed

lightkube doesnot work with httpx 0.28.x #78

hemanthnakkina opened this issue Dec 2, 2024 · 12 comments

Comments

@hemanthnakkina
Copy link

Running lightkube get/list commands failed with Unauthorized error.

Ran a sample from documentation:

from lightkube import Client
from lightkube.resources.core_v1 import Node

client = Client()
for node in client.list(Node):
    print(node.metadata.name)

Error message:

Traceback (most recent call last):
  File "/home/ubuntu/venv/lib/python3.10/site-packages/lightkube/core/generic_client.py", line 235, in raise_for_status
    resp.raise_for_status()
  File "/home/ubuntu/venv/lib/python3.10/site-packages/httpx/_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '401 Unauthorized' for url 'https://10.121.193.103:6443/api/v1/nodes'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/venv/lib/python3.10/site-packages/lightkube/core/generic_client.py", line 318, in list
    cont, chunk = self.handle_response("list", resp, br)
  File "/home/ubuntu/venv/lib/python3.10/site-packages/lightkube/core/generic_client.py", line 251, in handle_response
    self.raise_for_status(resp)
  File "/home/ubuntu/venv/lib/python3.10/site-packages/lightkube/core/generic_client.py", line 237, in raise_for_status
    raise transform_exception(e)
lightkube.core.exceptions.ApiError: Unauthorized

Tried the same by downgrading httpx package to 0.27.x and it worked.

There are ssl related changes to httpx 0.28.0 which might cause this issue, see release notes
https://github.com/encode/httpx/releases/tag/0.28.0

@hemanthnakkina hemanthnakkina changed the title lightkube doesnot work with http 0.28.x lightkube doesnot work with httpx 0.28.x Dec 2, 2024
@gtsystem
Copy link
Owner

gtsystem commented Dec 5, 2024

Hi, what authentication method are you using? Can you post the corresponding configuration section (properly anonymized)?
I cannot reproduce in my setup and it doesn't seems lightkube is using any of the removed features of 0.28 (app and proxies). There are few deprecations that will need to be addressed, but they are just warning at the moment and so it should still work.

@gtsystem
Copy link
Owner

gtsystem commented Dec 5, 2024

There seems to be actually a bug in httpx v0.8.0 that will be fixed in 0.28.1: encode/httpx#3442

@gtsystem
Copy link
Owner

gtsystem commented Dec 5, 2024

I released v0.15.6 to temporary avoid httpx version 0.28.0. Fixing the deprecation will require first compatiblity of RESPX lundberg/respx#277 with the changes to 0.28 and also the release of httpx 0.28.1 encode/httpx#3445

@nsklikas
Copy link

nsklikas commented Dec 9, 2024

httpx==0.28.1 was released last week, the workaround on setup.py does not work (perhaps it should be httpx<0.28.0?).

@gtsystem
Copy link
Owner

gtsystem commented Dec 9, 2024

0.28.1 was supposed to fix that issue, that's why I kept it as a valid dependency version. Is the issue you encounter still reproducible with 0.28.1? If so, please provide your auth configuration

@logileifs
Copy link

I have a fresh install of lightkube (version 0.15.6) which brings in httpx 0.28.1 as dependency and I'm also having this issue.

$ kubectl config current-context
k3d-test
$ kubectl get nodes
NAME                STATUS   ROLES                  AGE     VERSION
k3d-test-agent-0    Ready    <none>                 7d22h   v1.30.6+k3s1
k3d-test-server-0   Ready    control-plane,master   7d22h   v1.30.6+k3s1
from lightkube import Client
from lightkube.resources.core_v1 import Node

client = Client()
for node in client.list(Node):
    print(node.metadata.name)

Traceback (most recent call last):
  File "/Users/logileifsson/tmp/registrator/__pypackages__/3.11/lib/lightkube/core/generic_client.py", line 235, in raise_for_status
    resp.raise_for_status()
  File "/Users/logileifsson/tmp/registrator/__pypackages__/3.11/lib/httpx/_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '401 Unauthorized' for url 'https://0.0.0.0:57271/api/v1/nodes'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/logileifsson/tmp/registrator/__pypackages__/3.11/lib/lightkube/core/generic_client.py", line 318, in list
    cont, chunk = self.handle_response("list", resp, br)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/logileifsson/tmp/registrator/__pypackages__/3.11/lib/lightkube/core/generic_client.py", line 251, in handle_response
    self.raise_for_status(resp)
  File "/Users/logileifsson/tmp/registrator/__pypackages__/3.11/lib/lightkube/core/generic_client.py", line 237, in raise_for_status
    raise transform_exception(e)
lightkube.core.exceptions.ApiError: Unauthorized
client.config
SingleConfig(context_name='k3d-test', context=Context(cluster='k3d-test', user='admin@k3d-test', namespace=None), cluster=Cluster(server='https://0.0.0.0:57271', certificate_auth=None, certificate_auth_data='LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUzTXpNeU1qZ3hOVFV3SGhjTk1qUXhNakF6TVRJeE5UVTFXaGNOTXpReE1qQXhNVEl4TlRVMQpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUzTXpNeU1qZ3hOVFV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFTQW9odmVFQUVCRUpuQjBoRFJXak9KN1gvVDZTWHhFSk4vV3hPS0Vac2YKd2wzRlI0bGFqM0RtL2JPZFM4ano0SkxmcnNyVDRBbll5bEF2bFBmRVNPSkdvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVWxqNk9pbTZUbUFFak95b28rWk1LClBCQWVIWUF3Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQUxoVnZwVnNvdXBaZjdOb2N1RXF4S0JEenZwdCtwNVIKMU5QTjdwQmJnUWFhQWlFQXliQmVRMFNnem9iZ3V0VUwxaklHTjNiMVpNcDBySWtkK3ljTW1qWlFLeEE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K', insecure=False), user=User(exec=None, username=None, password=None, token=None, auth_provider=None, client_cert=None, client_cert_data='LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrRENDQVRlZ0F3SUJBZ0lJYzN3UEt6SVErS2d3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOek16TWpJNE1UVTFNQjRYRFRJME1USXdNekV5TVRVMU5Wb1hEVEkxTVRJdwpNekV5TVRVMU5Wb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJMYjFac2lwbTV3ZENTeisKaUFKd1J0K2FqUGVNZ2lwaHJYWDE3Q1ZqYmhPeTllZkVHMW5LNzlDYS9ybTRNenFYQitLcElsa2Y4cWtxSWJOZwpBOXFTakYyalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCU2R1QmxscjNTN2JZNXA5MlphdWZqZUtZTkY3ekFLQmdncWhrak9QUVFEQWdOSEFEQkUKQWlCMFkzNFAvbTF4U1A1Z0pIUWpwNmJ6Z09jUGVkQlFCR01EdmxxWjNwZjUrUUlnTjZYekMvY1FvUXcwSC9OTQpJY0xCUUNNOUtiR0dqTDJSQWdmajg3eDJRSWM9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdFkyeHAKWlc1MExXTmhRREUzTXpNeU1qZ3hOVFV3SGhjTk1qUXhNakF6TVRJeE5UVTFXaGNOTXpReE1qQXhNVEl4TlRVMQpXakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwWlc1MExXTmhRREUzTXpNeU1qZ3hOVFV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFUOXFSR3lBNTZ5cE1ySytKU1BJSllld2Z4L0hRNldKdnNrZDVYUzhtbWgKVkszVFdyS0lVV2Qvb3pjYWZhNWdLRFhkaXVEWEM0OFBFQVRCVUtneFBXMFVvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVW5iZ1paYTkwdTIyT2FmZG1Xcm40CjNpbURSZTh3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnQXdDUDJEZmYrallLZjltVmFBOXdLQ2hNZUR6NlpHbU8KUUlnbnQ2ZUo0MUVDSVFDSDhmbUkybjZNVlFSd3MzUk9SZTVJMWdKU0FEbTlQVVVhMWxmVVc5cUt6QT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K', client_key=None, client_key_data='LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUM4dzR5SjlCU0hMdTIyeTlTbHhoOEhYZENFeUxGbHZiVkJMU2xacnl0VkFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFdHZWbXlLbWJuQjBKTFA2SUFuQkczNXFNOTR5Q0ttR3RkZlhzSldOdUU3TDE1OFFiV2NydgowSnIrdWJnek9wY0g0cWtpV1IveXFTb2hzMkFEMnBLTVhRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo='), fname=PosixPath('/Users/logileifsson/.kube/config'))

orfeas-k added a commit to canonical/charmed-kubeflow-chisme that referenced this issue Dec 11, 2024
@logileifs
Copy link

And I can confirm that downgrading httpx to 0.27.2 fixes the issue

@gtsystem
Copy link
Owner

Thanks for the details, this is going to be useful for the investigation. For now I just released v0.15.7 with httpx<0.28.0.

@ca-scribner
Copy link
Collaborator

I'm still experiencing this issue with lightkube 0.16.2 (and 0.16.1). I don't understand why though.

With lightkube <=0.16.0 and httpx==0.27.2, it works fine:

uv pip freeze                       
anyio==4.8.0
certifi==2024.12.14
h11==0.14.0
httpcore==1.0.7
httpx==0.27.2
idna==3.10
lightkube==0.16.0
lightkube-models==1.32.0.8
pyyaml==6.0.2
respx==0.22.0
sniffio==1.3.1

python -c "import lightkube; from lightkube.resources.core_v1 import Namespace; print(list(lightkube.Client().list(Namespace)))"
# expected output

But if I bump lightkube with httpx held constant, or bump httpx to >=0.28 (or bump both), I get this error:

Traceback (most recent call last):
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
        pool_request.request
    )
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    raise exc
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 78, in handle_request
    stream = self._connect(request)
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 156, in _connect
    stream = stream.start_tls(**kwargs)
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpcore/_backends/sync.py", line 154, in start_tls
    with map_exceptions(exc_map):
         ~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/scribs/.local/share/uv/python/cpython-3.13.0rc2-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: CA cert does not include key usage extension (_ssl.c:1020)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import lightkube; from lightkube.resources.core_v1 import Namespace; print(list(lightkube.Client().list(Namespace)))
                                                                               ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/lightkube/core/generic_client.py", line 317, in list
    resp = self.send(req)
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/lightkube/core/generic_client.py", line 271, in send
    return self._client.send(req, stream=stream)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
        request,
    ...<2 lines>...
        history=[],
    )
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
        request,
        follow_redirects=follow_redirects,
        history=history,
    )
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
         ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/scribs/.local/share/uv/python/cpython-3.13.0rc2-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/home/scribs/.global_venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: CA cert does not include key usage extension (_ssl.c:1020)

So for some reason, the fix in 0.16.2 is not working for me

@gtsystem
Copy link
Owner

gtsystem commented Jan 9, 2025

@ca-scribner, have you seen this issue elastic/elasticsearch#117769 ? The error message and the python version matches the stack trace above. Maybe you need to fix your certificate?

The default ssl context changed a bit from 0.27.x to 0.28.x:
Before: https://github.com/encode/httpx/blob/0.27.2/httpx/_config.py#L159
After: https://github.com/encode/httpx/blob/0.28.1/httpx/_config.py#L40

@ca-scribner
Copy link
Collaborator

ty @gtsystem, I had not seen this and I think you're right. I'm naive about this stuff, but reading python/cpython#107361 reinforces your point

What I think I'm seeing is that microk8s (the k8s I was using) does not generate rfc 5280 compliant certs (or at least, the certs it generates somehow disagrees with python/cpython#107361). I was using python 3.13 when hitting this, so that seems consistent.

If I drop back to python 3.12, things work fine for me. So seems like the combination of microk8s's certs+python 3.13 leads to my issue

@ca-scribner
Copy link
Collaborator

for anyone interested, my repro steps were (sorry if there's a typo, wrote these from memory after the fact):

lxc launch ubuntu:24.04 microk8s-1p31
lxc shell microk8s-1p31

then from inside the container

snap install microk8s --channel 1.31-strict/stable
mkdir -p ~/.kube
microk8s config > ~/.kube/config
# install uv (so swapping python is easy)
curl -LsSf https://astral.sh/uv/install.sh | sh
# source whatever uv says to source to activate it
uv python install 3.13
uv python install 3.12

#test with 3.12
uv venv -p 3.12 .venv_3.12
source .venv_3.12/bin/activate
uv pip install lightkube
python -c "import lightkube; from lightkube.resources.core_v1 import Namespace; print(list(lightkube.Client().list(Namespace)))"
# output will be a list of your namespaces

# test with 3.13
uv venv -p 3.13 .venv_3.13
source .venv_3.13/bin/activate
uv pip install lightkube
python -c "import lightkube; from lightkube.resources.core_v1 import Namespace; print(list(lightkube.Client().list(Namespace)))"
# output will be the above error

gboutry pushed a commit to gboutry/sunbeam-charms that referenced this issue Jan 15, 2025
During restarts, the k8s services of type loadbalancer
can get diferent IPs and the integration tests tries to
communicate with older IPs.
Add a step after deploying the bundle to add loadbalancer
annotations to services of type loadbalancer so that
IPs persist during restart of pods.

Pin httpx version in tox.ini due to bug [1] in lightkube

[1] gtsystem/lightkube#78

Change-Id: I013ec1c1e9dbac3ae86c57abcd9d87a3b99f6e82
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants