Skip to content

Fix invalid paddle binary file path #3421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Yancey0623 merged 10 commits into from
Aug 16, 2017
Merged

Fix invalid paddle binary file path #3421

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
dfc86dd
Fix invalid paddle binary file path
Yancey0623 Aug 11, 2017
98255dc
update
Yancey0623 Aug 14, 2017
a87b82c
update
Yancey0623 Aug 14, 2017
d6bd503
merge develop branch
Yancey0623 Aug 15, 2017
98b35ab
fix paddle bin directory
Yancey0623 Aug 15, 2017
d98e2d5
update install doc
Yancey0623 Aug 15, 2017
5c926b0
update
Yancey0623 Aug 15, 2017
a872e33
update
Yancey0623 Aug 16, 2017
ec9ac91
update
Yancey0623 Aug 16, 2017
3565518
update
Yancey0623 Aug 16, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 46 additions & 25 deletions doc/getstarted/build_and_install/build_from_source_en.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ As a simple example, consider the following:

1. **BLAS Dependencies(optional)**

CMake will search BLAS libraries from system. If not found, OpenBLAS will be downloaded, built and installed automatically.
CMake will search BLAS libraries from the system. If not found, OpenBLAS will be downloaded, built and installed automatically.
To utilize preinstalled BLAS， you can simply specify MKL, OpenBLAS or ATLAS via `MKL_ROOT`, `OPENBLAS_ROOT` or `ATLAS_ROOT`.

```bash
Expand Down Expand Up @@ -131,9 +131,9 @@ As a simple example, consider the following:
To build GPU version, you will need the following installed:

1. a CUDA-capable GPU
2. A supported version of Linux with a gcc compiler and toolchain
2. A supported version of Linux with a GCC compiler and toolchain
3. NVIDIA CUDA Toolkit (available at http://developer.nvidia.com/cuda-downloads)
4. NVIDIA cuDNN Library (availabel at https://developer.nvidia.com/cudnn)
4. NVIDIA cuDNN Library (available at https://developer.nvidia.com/cudnn)

The CUDA development environment relies on tight integration with the host development environment,
including the host compiler and C runtime libraries, and is therefore only supported on
Expand Down Expand Up @@ -162,16 +162,27 @@ mkdir build && cd build

Finally, you can build and install PaddlePaddle:

```bash
# you can add build option here, such as:
cmake .. -DCMAKE_INSTALL_PREFIX=<path to install>
# please use sudo make install, if you want to install PaddlePaddle into the system
make -j `nproc` && make install
# set PaddlePaddle installation path in ~/.bashrc
export PATH=<path to install>/bin:$PATH
# install PaddlePaddle Python modules.
sudo pip install <path to install>/opt/paddle/share/wheels/*.whl
```
- Install PaddlePaddle with `make install`
Copy link
Contributor

@typhoonzero typhoonzero Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must inform that choose one of these two methods, not both.

Copy link
Contributor Author

@Yancey0623 Yancey0623 Aug 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can use pip install instead of make install, so I deleted the make install

```bash
# you can add build option here, such as:
cmake .. -DCMAKE_INSTALL_PREFIX=<path to install>
# please use sudo make install, if you want to install PaddlePaddle into the system
make -j `nproc` && make install
# set PaddlePaddle installation path in ~/.bashrc
export PATH=<path to install>/bin:$PATH
# install PaddlePaddle Python modules.
sudo pip install <path to install>/opt/paddle/share/wheels/*.whl
```
Copy link
Contributor

@typhoonzero typhoonzero Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

export PATH=<path to install>/bin:$PATH
 +    # install PaddlePaddle Python modules.
 +    sudo pip install <path to install>/opt/paddle/share/wheels/*.whl

are no longer needed.


- Install PaddlePaddle with `pip install`
```bash
# buid PaddlePaddle
cmake ..
make -j `nproc`
# install PaddlePaddle modules, you can specify the customer install path with --prefix
sudo pip install python/dist/*.whl --prefix=<path to install>
```

## <span id="centos">Build on Centos 7</span>

### Install Dependencies
Expand All @@ -192,9 +203,9 @@ sudo pip install <path to install>/opt/paddle/share/wheels/*.whl
To build GPU version, you will need the following installed:

1. a CUDA-capable GPU
2. A supported version of Linux with a gcc compiler and toolchain
2. A supported version of Linux with a GCC compiler and toolchain
3. NVIDIA CUDA Toolkit (available at http://developer.nvidia.com/cuda-downloads)
4. NVIDIA cuDNN Library (availabel at https://developer.nvidia.com/cudnn)
4. NVIDIA cuDNN Library (available at https://developer.nvidia.com/cudnn)

The CUDA development environment relies on tight integration with the host development environment,
including the host compiler and C runtime libraries, and is therefore only supported on
Expand Down Expand Up @@ -223,13 +234,23 @@ mkdir build && cd build

Finally, you can build and install PaddlePaddle:

```bash
# you can add build option here, such as:
cmake3 .. -DCMAKE_INSTALL_PREFIX=<path to install>
# please use sudo make install, if you want to install PaddlePaddle into the system
make -j `nproc` && make install
# set PaddlePaddle installation path in ~/.bashrc
export PATH=<path to install>/bin:$PATH
# install PaddlePaddle Python modules.
sudo pip install <path to install>/opt/paddle/share/wheels/*.whl
```
- Install PaddlePaddle with `make install`
```bash
# you can add build option here, such as:
cmake3 .. -DCMAKE_INSTALL_PREFIX=<path to install>
# please use sudo make install, if you want to install PaddlePaddle into the system
make -j `nproc` && make install
# set PaddlePaddle installation path in ~/.bashrc
export PATH=<path to install>/bin:$PATH
# install PaddlePaddle Python modules.
sudo pip install <path to install>/opt/paddle/share/wheels/*.whl
```
Copy link
Contributor

@typhoonzero typhoonzero Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

+    # set PaddlePaddle installation path in ~/.bashrc
 +    export PATH=<path to install>/bin:$PATH
 +    # install PaddlePaddle Python modules.
 +    sudo pip install <path to install>/opt/paddle/share/wheels/*.whl

are no longer needed.

Copy link
Contributor

@typhoonzero typhoonzero Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File ubuntu_install_en.rst and ubuntu_install_cn.rst seems no longer needed.

Copy link
Contributor Author

@Yancey0623 Yancey0623 Aug 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what ubuntu_install_en.rst and ubuntu_install_cn.rst used for, maybe @luotao1 knows more?


- Install PaddlePaddle with `pip install`
```bash
# buid PaddlePaddle
cmake3 ..
make -j `nproc`
# install PaddlePaddle modules, you can specify the customer install path with --prefix
sudo pip install python/dist/*.whl --prefix=<path to install>
```
26 changes: 3 additions & 23 deletions paddle/scripts/docker/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,25 +128,6 @@ EOF
/woboq/indexgenerator/codebrowser_indexgenerator $WOBOQ_OUT
fi

# generate deb package for current build
# FIXME(typhoonzero): should we remove paddle/scripts/deb ?
if [[ ${WITH_DEB:-ON} == "ON" ]]; then
cat <<EOF
========================================
Generating .deb package ...
========================================
EOF
set +e
cpack -D CPACK_GENERATOR='DEB' -j `nproc` ..
err_code=$?
if [ ${err_code} -ne 0 ]; then
# cat error logs if cpack failed.
cat /paddle/build/_CPack_Packages/Linux/DEB/PreinstallOutput.log
exit ${err_code}
fi
set -e
fi

cat <<EOF
========================================
Generate /paddle/build/Dockerfile ...
Expand All @@ -166,14 +147,13 @@ EOF
fi

cat >> /paddle/build/Dockerfile <<EOF
# Use different deb file when building different type of images
ADD *.deb /
ADD python/dist/*.whl /
# run paddle version to install python packages first
RUN apt-get update &&\
apt-get install -y wget python-pip && pip install -U pip && \
dpkg -i /*.deb ; apt-get install -f -y && \
pip install /*.whl; apt-get install -f -y && \
apt-get clean -y && \
rm -f /*.deb && \
rm -f /*.whl && \
paddle version
${DOCKERFILE_CUDNN_DSO}
${DOCKERFILE_GPU_ENV}
Expand Down
29 changes: 5 additions & 24 deletions paddle/scripts/submit_local.sh.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,7 @@ if [ -z "${PADDLE_NO_STAT+x}" ]; then
fi
fi


MYDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
PADDLE_BIN_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

if [ ! -z "${DEBUGGER}" ]; then
echo "Using debug command ${DEBUGGER}"
Expand Down Expand Up @@ -93,34 +92,16 @@ else:
sys.exit(0)
EOF

if [ $? -eq 1 ]; then # Older version installed, or not installed at all
echo "First time run paddle, need to install some python dependencies."
# setuptools normalizes package version, so we need to use normalized
# package version for paddle python package
PYTHON_PADDLE_VERSION=$(python -c 'import packaging.version
import setuptools
print str(packaging.version.Version("@PADDLE_VERSION@"))
' 2>/dev/null)
BASEDIR=$(dirname "$0")
pip install ${BASEDIR}/../opt/paddle/share/wheels/*-${PYTHON_PADDLE_VERSION}-*.whl
if [ $? -ne 0 ]; then
echo "pip install wheels failed. "
echo "Please use 'sudo paddle' at the first time you use PaddlePaddle"
echo "PaddlePaddle will install some python dependencies automatically."
exit 1
fi
echo "Python dependencies are installed."
fi

case "$1" in
"train")
${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_trainer ${@:2}
${DEBUGGER} $PADDLE_BIN_PATH/paddle_trainer ${@:2}
;;
"merge_model")
${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_merge_model ${@:2}
${DEBUGGER} $PADDLE_BIN_PATH/paddle_merge_model ${@:2}
;;
"pserver")
${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_pserver_main ${@:2}
${DEBUGGER} $PADDLE_BIN_PATH/paddle_pserver_main ${@:2}
;;
"dump_config")
python -m paddle.utils.dump_config ${@:2}
Expand All @@ -129,7 +110,7 @@ case "$1" in
python -m paddle.utils.make_model_diagram ${@:2}
;;
"usage")
$MYDIR/../opt/paddle/bin/paddle_usage ${@:2}
$PADDLE_BIN_PATH/paddle_usage ${@:2}
;;
"version")
version
Expand Down
12 changes: 6 additions & 6 deletions python/setup.py.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,14 @@ if '${CMAKE_SYSTEM_PROCESSOR}' not in ['arm', 'armv7-a', 'aarch64']:
setup_requires+=["opencv-python"]

# the prefix is sys.prefix which should always be usr
paddle_bin_dir = 'local/opt/paddle/bin'
paddle_bin_dir = 'opt/paddle/bin'
Copy link
Contributor

@tensor-tang tensor-tang Aug 15, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should cause difference path when use cmake install.
cmake install : /usr/local/opt/paddle/bin
pip install : /usr/opt/paddle/bin

Copy link
Contributor Author

@Yancey0623 Yancey0623 Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tensor-tang , but maybe it doesn't matter for the different path?
If users want to install paddle to a customer directory, he/she could set --prefix=<customer directory>, but in the production Docker image, we just make it running correctly.

BTW, the directory in date_files wouldn't be configured local/opt/paddle/bin, this will caused paddle be installed under /usr/local/local/opt/paddle/bin

Copy link
Contributor Author

@Yancey0623 Yancey0623 Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tensor-tang , paddle_bin_files is executable files, so I move these to scripts int setup.py.

Copy link
Contributor

@typhoonzero typhoonzero Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think make install and pip install will install to different default location doesn't harm using.

  1. Users could either use make install or pip install to install and then use paddle after that.
  2. The default install path is determined by cmake and pip, we shouldn't interfere with it. Let cmake and pip do their work.
  3. Maybe add some documentation about how to use --prefix parameter when installing, and the default installation behaviour?

Copy link
Contributor

@tensor-tang tensor-tang Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but maybe it doesn't matter for the different path?

I am not sure about this, maybe @luotao1 @typhoonzero can answer this?

Maybe add some documentation about how to use --prefix parameter when installing, and the default installation behaviour?

Totally agreed~

Anyway, I am OK with the both ways, just giving a method for this case.
You can change them as you wish. I think the efficient way is the best~
Thanks @Yancey1989 @typhoonzero ~

Copy link
Contributor Author

@Yancey0623 Yancey0623 Aug 15, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done with the document for installing PaddlePaddle.
And thanks @typhoonzero and @tensor-tang !!

paddle_bins = ['${PADDLE_BINARY_DIR}/paddle/scripts/paddle_usage',
'${PADDLE_BINARY_DIR}/paddle/trainer/paddle_trainer',
'${PADDLE_BINARY_DIR}/paddle/trainer/paddle_merge_model',
'${PADDLE_BINARY_DIR}/paddle/pserver/paddle_pserver_main']
'${PADDLE_BINARY_DIR}/paddle/pserver/paddle_pserver_main',
'${PADDLE_BINARY_DIR}/paddle/scripts/paddle']

paddle_rt_lib_dir = 'local/lib'
paddle_rt_lib_dir = 'lib'
Copy link
Contributor

@tensor-tang tensor-tang Aug 15, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this change would impact "pip install " without docker for user.
Since the unit test for this has not ready yet @luotao1

Anyway, if /usr/lib and /usr/local/lib are both in LD_LIBRARY_PATH, then it won't be a problem for users.

paddle_rt_libs = [] if '${MKL_SHARED_LIBS}'== '' else '${MKL_SHARED_LIBS}'.split(';')

setup(name='paddlepaddle',
Expand All @@ -50,8 +51,7 @@ setup(name='paddlepaddle',
'paddle.v2.framework.proto': '${PADDLE_BINARY_DIR}/paddle/framework',
'py_paddle': '${PADDLE_SOURCE_DIR}/paddle/py_paddle'
},
scripts=['${PADDLE_BINARY_DIR}/paddle/scripts/paddle'],
scripts=paddle_bins,
distclass=BinaryDistribution,
data_files=[(paddle_bin_dir, paddle_bins),
(paddle_rt_lib_dir, paddle_rt_libs)]
data_files=[(paddle_rt_lib_dir, paddle_rt_libs)]
)