Skip to content

breaking: Change Model server to Torchserve for PyTorch Inference #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 30 commits into from
Jul 31, 2020
Merged

breaking: Change Model server to Torchserve for PyTorch Inference #79

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
8225190
Change Model server tp Torchserve for PytTorch Inference
dk19y Jul 10, 2020
900b406
PR Feedback - Replace MMS with TS
dk19y Jul 15, 2020
9bb211a
Fixed flake errors
dk19y Jul 17, 2020
1abf0e3
Fix env module usage from sagemaker-inference-toolkit
dk19y Jul 17, 2020
cf38f47
Fix unit test failures
dk19y Jul 17, 2020
8d51e40
PR Feedback
dk19y Jul 27, 2020
d16ed07
Fix flakes issues
dk19y Jul 27, 2020
a4eee3e
Update src/sagemaker_pytorch_serving_container/torchserve.py
dk19y Jul 27, 2020
0ae09ff
Update src/sagemaker_pytorch_serving_container/torchserve.py
dk19y Jul 27, 2020
f0629ae
Update src/sagemaker_pytorch_serving_container/torchserve.py
dk19y Jul 27, 2020
270489b
Update src/sagemaker_pytorch_serving_container/torchserve.py
dk19y Jul 27, 2020
299d6cb
Update src/sagemaker_pytorch_serving_container/torchserve.py
dk19y Jul 27, 2020
ed641f8
Update src/sagemaker_pytorch_serving_container/torchserve.py
dk19y Jul 27, 2020
292cf5a
Update src/sagemaker_pytorch_serving_container/torchserve.py
dk19y Jul 27, 2020
19f0da0
PR Feedback
dk19y Jul 27, 2020
9ccc715
Update test/unit/test_model_server.py
dk19y Jul 27, 2020
b565428
Update src/sagemaker_pytorch_serving_container/torchserve.py
dk19y Jul 29, 2020
9262b48
PR Feedback - Unit Test
dk19y Jul 29, 2020
2be5474
Update test_model_server.py
dk19y Jul 29, 2020
22567d0
Update test_model_server.py
dk19y Jul 29, 2020
5dada25
Update framework version to 1.6.0
dk19y Jul 29, 2020
9cc1574
Update DLC Containers
dk19y Jul 29, 2020
b0e1d5f
Update version in buildspec for Codebuild
dk19y Jul 30, 2020
07556d6
Update 1.5.0 Images for CodeBuild
dk19y Jul 30, 2020
b37fcd3
Skip EIA Test for 1.5.0
dk19y Jul 30, 2020
5d48150
Skipped EIA Test
dk19y Jul 31, 2020
811e171
Retrigger build
dk19y Jul 31, 2020
c2cc29e
Update src/sagemaker_pytorch_serving_container/torchserve.py
dk19y Jul 31, 2020
6ab7850
Update Logger
dk19y Jul 31, 2020
0dd3199
Update logeger
dk19y Jul 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
0 artifacts/mms-entrypoint.py → artifacts/ts-entrypoint.py
View file
Open in desktop
File renamed without changes.
6 changes: 3 additions & 3 deletions src/sagemaker_pytorch_serving_container/serving.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@
from subprocess import CalledProcessError

from retrying import retry
from sagemaker_inference import model_server
from sagemaker_inference import torchserve

from sagemaker_pytorch_serving_container import handler_service

HANDLER_SERVICE = handler_service.__name__
HANDLER_SERVICE = handler_service.__file__


def _retry_if_error(exception):
Expand All @@ -32,7 +32,7 @@ def _start_model_server():
# there's a race condition that causes the model server command to
# sometimes fail with 'bad address'. more investigation needed
# retry starting mms until it's ready
model_server.start_model_server(handler_service=HANDLER_SERVICE)
torchserve.start_model_server(handler_service=HANDLER_SERVICE)


def main():
Expand Down
14 changes: 9 additions & 5 deletions test/container/1.5.0/Dockerfile.pytorch
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,29 @@ FROM pytorch/pytorch:1.5-cuda10.1-cudnn7-runtime
LABEL com.amazonaws.sagemaker.capabilities.accept-bind-to-port=true
LABEL com.amazonaws.sagemaker.capabilities.multi-models=true

ARG MMS_VERSION=1.0.8
ARG TS_VERSION=0.1.1

ENV SAGEMAKER_SERVING_MODULE sagemaker_pytorch_serving_container.serving:main
ENV TEMP=/home/model-server/tmp

RUN apt-get update \
&& apt-get install -y --no-install-recommends software-properties-common \
&& add-apt-repository ppa:openjdk-r/ppa \
&& apt-get update \
&& apt-get install -y --no-install-recommends \
libgl1-mesa-glx \
libglib2.0-0 \
libsm6 \
libxext6 \
libxrender-dev \
openjdk-8-jdk-headless \
openjdk-11-jdk \
&& rm -rf /var/lib/apt/lists/*

RUN conda install -c conda-forge opencv==4.0.1 \
&& ln -s /opt/conda/bin/pip /usr/local/bin/pip3

RUN pip install mxnet-model-server==$MMS_VERSION
RUN pip install torchserve==$TS_VERSION \
&& pip install torch-model-archiver==$TS_VERSION

COPY dist/sagemaker_pytorch_inference-*.tar.gz /sagemaker_pytorch_inference.tar.gz
RUN pip install --no-cache-dir /sagemaker_pytorch_inference.tar.gz && \
Expand All @@ -31,11 +35,11 @@ RUN useradd -m model-server \
&& mkdir -p /home/model-server/tmp \
&& chown -R model-server /home/model-server

COPY artifacts/mms-entrypoint.py /usr/local/bin/dockerd-entrypoint.py
COPY artifacts/ts-entrypoint.py /usr/local/bin/dockerd-entrypoint.py
COPY artifacts/config.properties /home/model-server

RUN chmod +x /usr/local/bin/dockerd-entrypoint.py

EXPOSE 8080 8081
ENTRYPOINT ["python", "/usr/local/bin/dockerd-entrypoint.py"]
CMD ["mxnet-model-server", "--start", "--mms-config", "/home/model-server/config.properties"]
CMD ["torchserve", "--start", "--ts-config", "/home/model-server/config.properties", "--model-store", "/home/model-server/"]
6 changes: 2 additions & 4 deletions test/unit/test_serving.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,9 @@
from mock import patch


@patch('sagemaker_inference.model_server.start_model_server')
@patch('sagemaker_inference.torchserve.start_model_server')
def test_hosting_start(start_model_server):
from sagemaker_pytorch_serving_container import serving

serving.main()

start_model_server.assert_called_with(
handler_service='sagemaker_pytorch_serving_container.handler_service')
start_model_server.assert_called()