public.ecr.aws/lambda/python:3.9 incompatible with latest botocore

[thyer]

Related to #8 - with botocore introducing changes in auth.py, any docker image using lamba/python:3.9 as a base and adding botocore via pip or poetry will see the error Unable to import module 'my_module': cannot import name 'resolve_auth_type' from 'botocore.auth' (/var/runtime/botocore/auth.py). This is due to the base image's version of /var/lang/lib/python3.9/site-packages/botocore/__init__.py not having the latest changes.

A minimal repro as follows:

FROM public.ecr.aws/lambda/python:3.9
COPY . ${LAMBDA_TASK_ROOT}
RUN pip install awscli
RUN pip install botocore==1.35.7
RUN pip install boto3==1.35.7
CMD ["my_module.handle_request"]

Where my_module.handle_request simply calls the newly added function
from botocore.auth import resolve_auth_type

[thyer]

Note that as mentioned at the time of the last issue, wiping them from the image is a suitable workaround, but it provides a ticking time bomb for anybody relying on a stable production environment. I would strongly encourage a fix that does not simply bring the latest botocore into the base image, but rather addresses the /var/runtime/ vs. /var/lang/lib/python3.9/ shadowing.

[jtuliani]

Hi @thyer . Prior to Python 3.11, /var/runtime took precedence over the site-packages directory. This resulted in the module being loaded from the runtime instead of their pip installation. Switching the base image to Python 3.11 should resolve the issue. For details, see our blog post.

Also, from the above, it looks like your module installation didn't copy your pip-installed version into ${LAMBDA_TASK_ROOT}, so your pip-installed version wasn't actually being used.