Skip to content

Thumbor 6.5.2 #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
eba5093
add .gitignore
leviwilson Sep 1, 2018
36d4640
add Dockerfile for building thumbor
leviwilson Sep 1, 2018
b750ff4
get around pip.req issue with pip upgrade
leviwilson Sep 1, 2018
daa9d3b
include make in the Dockerfile
leviwilson Sep 1, 2018
0fd15e5
bump thumbor to 6.5.2
leviwilson Sep 1, 2018
ffcdbec
bump tc_aws to fix Botocore import error
leviwilson Sep 1, 2018
48aaef3
closer; just need to find the right base amazonlinux to pin
leviwilson Sep 1, 2018
d14a347
lock the yum repo down as well
leviwilson Sep 2, 2018
c1bb993
bump botocore and tornado_botocore to latest
leviwilson Sep 2, 2018
a2e9c4c
enable epel on amazonlinux 2
leviwilson Sep 2, 2018
79b7521
compile nss instead of openssl
leviwilson Sep 2, 2018
36c7d3f
add ImageMagick
leviwilson Sep 2, 2018
7d51222
move ImageMagick up a little
leviwilson Sep 2, 2018
58882ef
response :arrow_right: thumbor_response
leviwilson Sep 2, 2018
4e23e8a
Revert "bump botocore and tornado_botocore to latest"
leviwilson Sep 2, 2018
3387878
add pngcrush, libjpeg* and gifsicle
leviwilson Sep 2, 2018
58948c6
add wget so mozjpeg can be added
leviwilson Sep 3, 2018
9800e89
install optipng and pngquant to the docker image
leviwilson Sep 3, 2018
110d6a0
add an ENTRYPOINT / CMD to the Dockerfile to perform the build-s3-dis…
leviwilson Sep 3, 2018
844a73a
remove .vimrc from the cache
leviwilson Sep 3, 2018
33e08e8
update the README.md to reflect leveraging docker to perform the buil…
leviwilson Sep 3, 2018
3627e5a
remove the MAINTAINER line
leviwilson Sep 3, 2018
6bbc037
specify pip._internal.req vs. pip.req
leviwilson Sep 3, 2018
0462412
ignore deployment/dist not deployment
leviwilson Sep 3, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
deployment/dist
46 changes: 46 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
FROM amazonlinux:2017.03.1.20170812

# lock yum to the same repository version
RUN sed -i 's/releasever=.*/releasever=2017.03/g' /etc/yum.conf
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So that the build is repeatable, I locked the releasever to 2017.03 vs. latest. In particular, the issue was when building the pycurl library from source since that when you yum install libcurl-devel if it pulls latest, when the lambda function runs you get an error that the linked version of libcurl differs from the compiled version of libcurl.

To get around that, I just pinned the yum repository to that version.


# base requirements
RUN yum install yum-utils zip -y && \
yum-config-manager --enable epel && \
yum install wget git libpng-devel libcurl-devel gcc python27-devel libjpeg-devel -y

# enable epel on Amazon Linux 2
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

# ImageMagick
RUN yum install -y ImageMagick-devel

# Other libraries
RUN yum install -y pngcrush libjpeg* gifsicle

# optipng
RUN wget http://dl.fedoraproject.org/pub/epel/6/x86_64/Packages/o/optipng-0.7.6-6.el6.x86_64.rpm && \
yum localinstall optipng-0.7.6-6.el6.x86_64.rpm -y && rm optipng*rpm

# pngquant
RUN wget http://dl.fedoraproject.org/pub/epel/6/x86_64/Packages/l/libimagequant-2.5.2-5.el6.x86_64.rpm && \
yum localinstall libimagequant-2.5.2-5.el6.x86_64.rpm -y && rm libimagequant*rpm && \
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/Packages/p/pngquant-2.5.2-5.el6.x86_64.rpm && \
yum localinstall pngquant-2.5.2-5.el6.x86_64.rpm -y && rm pngquant*rpm
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was not able to find optipng or pngquant in yum search. In order to get around errors in including those when running the build script, I grabbed them directly from here. Now when I run it doesn't complain in the build script at the end.

zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip pngquant
zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip jpegtran
zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip optipng
zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip pngcrush
zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip gifsicle
zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip mozjpeg
zip -q -g /lambda/deployment/dist/env/../serverless-image-handler.zip imgmin

When I didn't have these dependencies, the output would say something to the effect of nothing to do here (when it couldn't find the dependencies).


# pip
RUN alias sudo='env PATH=$PATH' && \
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
python get-pip.py && rm get-pip.py && \
pip install --upgrade setuptools && \
pip install --upgrade virtualenv

# pycurl
RUN yum install -y nss-devel
ENV PYCURL_SSL_LIBRARY=nss
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pycurl is built from source. I also has to specify nss as the SSL library. Originally when I tried openssl, when it ran in lambda it complained that the compiled version used openssl but the runtime used nss.


RUN mkdir /lambda
VOLUME /lambda
WORKDIR /lambda/deployment

ENTRYPOINT ["./build-s3-dist.sh"]
CMD ["source-bucket-base-name"]
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This defaults the bucket to source-bucket-base-name; passing any other value when you docker run will override this.

40 changes: 29 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,30 +2,48 @@
A solution to dynamically handle images on the fly, utilizing Thumbor (thumbor.org).
Published version, additional details and documentation are available here: https://aws.amazon.com/answers/web-applications/serverless-image-handler/

## OS/Python Environment Setup
## Docker Environment Setup
In order to build the package locally, you'll need to build the docker image. In order to do so, run the following command:

```bash
sudo yum-config-manager --enable epel
sudo yum update -y
sudo yum install git libpng-devel libcurl-devel gcc python-devel libjpeg-devel -y
sudo pip install --upgrade pip
alias sudo='sudo env PATH=$PATH'
sudo pip install --upgrade setuptools
sudo pip install --upgrade virtualenv
docker build -t serverless-image-handler .
```

This will build a docker image that has the following properties:

* pinned to the `amazonlinux` version that [Lambda runs on](https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html)
* pinned yum repository to `releasever=2017.03`
* this is important when building `pycurl` so that the compiled version of libcurl does not differ from the runtime version on the Lambda AMI
* has all of the base requirements installed
* libpng
* libjpeg
* pngcrush
* gifsicle
* optipng
* pngquant

## Building Lambda Package
To build the Lambda package, use the `serverless-image-handler` image that was built earlier. The following command will build the deployment packages:

```bash
cd deployment
./build-s3-dist.sh source-bucket-base-name
docker run -it --rm --volume $PWD:/lambda serverless-image-handler source-bucket-base-name
```
source-bucket-base-name should be the base name for the S3 bucket location where the template will source the Lambda code from.

`source-bucket-base-name` should be the base name for the S3 bucket location where the template will source the Lambda code from.
The template will append '-[region_name]' to this value.
For example: ./build-s3-dist.sh solutions
The template will then expect the source code to be located in the solutions-[region_name] bucket

## CF template and Lambda function
Located in deployment/dist

```
deployment/dist/
├── [2.0M] serverless-image-handler-custom-resource.zip
├── [6.4M] serverless-image-handler-ui.zip
├── [ 23K] serverless-image-handler.template
└── [ 48M] serverless-image-handler.zip
```

***

Expand Down
2 changes: 1 addition & 1 deletion source/image-handler-custom-resource/setup.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# coding: utf-8

from setuptools import setup, find_packages
from pip.req import parse_requirements
from pip._internal.req import parse_requirements
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pip >= 10 does not expose pip.req and instead it was moved to _internal.


setup(
name='image_handler_custom_resource',
Expand Down
2 changes: 1 addition & 1 deletion source/image-handler/lambda_function.py
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ def request_thumbor(original_request, session):

def process_thumbor_responde(thumbor_response, vary):
if thumbor_response.status_code != 200:
return response_formater(status_code=response.status_code)
return response_formater(status_code=thumbor_response.status_code)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe using response here was a mistake; at one point when running my lambda function when I had an issue with tc_aws it hit this code path and blew up saying that response wasn't a global.

I believe thumbor_response is what was intended here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this was fixed in #34 as well

if vary:
vary = thumbor_response.headers['vary']
content_type = thumbor_response.headers['content-type']
Expand Down
10 changes: 5 additions & 5 deletions source/image-handler/setup.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# coding: utf-8

from setuptools import setup, find_packages
from pip.req import parse_requirements
from pip._internal.req import parse_requirements

tests_require = [
'mock',
Expand All @@ -23,11 +23,11 @@
'': ['*.conf'],
},
install_requires=[
'botocore==1.3.7',
'tornado_botocore==1.0.2',
'botocore==1.8',
'tornado_botocore==1.3.2',
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

botocore, tornado_botocore as well as tc_aws needed to be bumped to get around the pip >= 10 issue

'requests_unixsocket>=0.1.5',
'thumbor>=6.4.2',
'tc_aws==6.0.3',
'thumbor>=6.5.2',
'tc_aws==6.2.10',
'opencv-python==3.2.0.6'
],
extras_require={
Expand Down