Skip to content

Fix invalid paddle binary file path #3421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Yancey0623 merged 10 commits into from
Aug 16, 2017
Merged

Fix invalid paddle binary file path #3421

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
dfc86dd
Fix invalid paddle binary file path
Yancey0623 Aug 11, 2017
98255dc
update
Yancey0623 Aug 14, 2017
a87b82c
update
Yancey0623 Aug 14, 2017
d6bd503
merge develop branch
Yancey0623 Aug 15, 2017
98b35ab
fix paddle bin directory
Yancey0623 Aug 15, 2017
d98e2d5
update install doc
Yancey0623 Aug 15, 2017
5c926b0
update
Yancey0623 Aug 15, 2017
a872e33
update
Yancey0623 Aug 16, 2017
ec9ac91
update
Yancey0623 Aug 16, 2017
3565518
update
Yancey0623 Aug 16, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 3 additions & 23 deletions paddle/scripts/docker/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,25 +128,6 @@ EOF
/woboq/indexgenerator/codebrowser_indexgenerator $WOBOQ_OUT
fi

# generate deb package for current build
# FIXME(typhoonzero): should we remove paddle/scripts/deb ?
if [[ ${WITH_DEB:-ON} == "ON" ]]; then
cat <<EOF
========================================
Generating .deb package ...
========================================
EOF
set +e
cpack -D CPACK_GENERATOR='DEB' -j `nproc` ..
err_code=$?
if [ ${err_code} -ne 0 ]; then
# cat error logs if cpack failed.
cat /paddle/build/_CPack_Packages/Linux/DEB/PreinstallOutput.log
exit ${err_code}
fi
set -e
fi

cat <<EOF
========================================
Generate /paddle/build/Dockerfile ...
Expand All @@ -166,14 +147,13 @@ EOF
fi

cat >> /paddle/build/Dockerfile <<EOF
# Use different deb file when building different type of images
ADD *.deb /
ADD python/dist/*.whl /
# run paddle version to install python packages first
RUN apt-get update &&\
apt-get install -y wget python-pip && pip install -U pip && \
dpkg -i /*.deb ; apt-get install -f -y && \
pip install /*.whl; apt-get install -f -y && \
apt-get clean -y && \
rm -f /*.deb && \
rm -f /*.whl && \
paddle version
${DOCKERFILE_CUDNN_DSO}
${DOCKERFILE_GPU_ENV}
Expand Down
39 changes: 16 additions & 23 deletions paddle/scripts/submit_local.sh.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,6 @@ if [ -z "${PADDLE_NO_STAT+x}" ]; then
fi
fi


MYDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

if [ ! -z "${DEBUGGER}" ]; then
echo "Using debug command ${DEBUGGER}"
fi
Expand Down Expand Up @@ -91,34 +88,30 @@ else:
sys.exit(0)
EOF

if [ $? -eq 1 ]; then # Older version installed, or not installed at all
echo "First time run paddle, need to install some python dependencies."
# setuptools normalizes package version, so we need to use normalized
# package version for paddle python package
PYTHON_PADDLE_VERSION=$(python -c 'import packaging.version
import setuptools
print str(packaging.version.Version("@PADDLE_VERSION@"))
' 2>/dev/null)
BASEDIR=$(dirname "$0")
pip install ${BASEDIR}/../opt/paddle/share/wheels/*-${PYTHON_PADDLE_VERSION}-*.whl
if [ $? -ne 0 ]; then
echo "pip install wheels failed. "
echo "Please use 'sudo paddle' at the first time you use PaddlePaddle"
echo "PaddlePaddle will install some python dependencies automatically."
exit 1
# find the paddle executable module path
PYTHON_SITEPACKAGES_PATHES=$(python -c "import site; print ';'.join(site.getsitepackages())")
IFS=';' items=($PYTHON_SITEPACKAGES_PATHES)
for item in ${items[@]}; do
if [ -f $item/usr/local/opt/paddle/bin/paddle_trainer ]; then
PADDLE_BIN_PATH=$item/usr/local/opt/paddle/bin
Copy link
Contributor

@typhoonzero typhoonzero Aug 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PADDLE_BIN_PATH should also check if binaries are under /usr/local/opt/...

As mentioned by @tensor-tang, you can change the binaries installation dir in setup.py.in but either way is OK, it's up to you.

Copy link
Contributor Author

@Yancey0623 Yancey0623 Aug 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the comment #3421 (comment), may be date_files does not support the real absolute path...

PADDLE_BIN_PATH should also check if binaries are under /usr/local/opt/...

I'm not sure why we also need to check /usr/local/opt/..., *.whl will never install binary file under /usr/local/opt/...

Copy link
Contributor

@tensor-tang tensor-tang Aug 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I try to correct this path in other PR #3461
It may fix your issue, if works.

Copy link
Contributor Author

@Yancey0623 Yancey0623 Aug 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, thanks @tensor-tang , it works correctly. But for my test, the directory should be configured as opt/paddle/bin and the binary files will be installed under /usr/local/opt/paddle/bin.

Copy link
Contributor

@tensor-tang tensor-tang Aug 14, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

configured as opt/paddle/bin and the binary files will be installed under /usr/local/opt/paddle/bin.

which means your sys.prefix is /usr/local? That's weird.
Or some var were set in your env, before pip install ?
Could you please check print sys.prefix? I tried login as root and user. The result are both /usr.

BTW my OS CentOS7.2.

Copy link
Contributor Author

@Yancey0623 Yancey0623 Aug 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, it's also confusing for me, print sys.prefix will print /usr.

root@6a4bc12ae305:/# python -c "import sys; print sys.prefix"
/usr

I executed pip install *.whl in production Docker image, the base image is ubuntu:16.04.

And if the data_files was configured as local/opt/paddle/bin, the binary files would be installed under /usr/local/local/opt/paddle/bin, the duplicate local directory.

Copy link
Contributor

@tensor-tang tensor-tang Aug 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you try pip install --prefix='/usr' *.whl in docker?

Copy link
Contributor Author

@Yancey0623 Yancey0623 Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, if I specify the --prefix='/usr', paddle will be installed under /usr.

Copy link
Contributor

@tensor-tang tensor-tang Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Yancey1989

break
fi
echo "Python dependencies are installed."
done

if [ -z $PADDLE_BIN_PATH ]; then
echo "can not find paddle executable file."
exit 1
fi

case "$1" in
"train")
${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_trainer ${@:2}
${DEBUGGER} $PADDLE_BIN_PATH/paddle_trainer ${@:2}
;;
"merge_model")
${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_merge_model ${@:2}
${DEBUGGER} $PADDLE_BIN_PATH/paddle_merge_model ${@:2}
;;
"pserver")
${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_pserver_main ${@:2}
${DEBUGGER} $PADDLE_BIN_PATH/paddle_pserver_main ${@:2}
;;
"dump_config")
python -m paddle.utils.dump_config ${@:2}
Expand All @@ -127,7 +120,7 @@ case "$1" in
python -m paddle.utils.make_model_diagram ${@:2}
;;
"usage")
$MYDIR/../opt/paddle/bin/paddle_usage ${@:2}
$PADDLE_BIN_PATH/paddle_usage ${@:2}
;;
"version")
version
Expand Down