Fix invalid paddle binary file path #3421

Yancey0623 · 2017-08-11T10:10:17Z

typhoonzero · 2017-08-11T11:02:18Z

paddle/scripts/submit_local.sh.in

 ' 2>/dev/null)
    BASEDIR=$(dirname "$0")
-    pip install ${BASEDIR}/../opt/paddle/share/wheels/*-${PYTHON_PADDLE_VERSION}-*.whl
+    pip install ${BASEDIR}/../../opt/paddle/share/wheels/*-${PYTHON_PADDLE_VERSION}-*.whl


If we all switch to install using pip install this may not be needed?

typhoonzero · 2017-08-11T11:06:26Z

paddle/scripts/submit_local.sh.in

 case "$1" in
    "train")
-        ${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_trainer ${@:2}
+        ${DEBUGGER} $MYDIR/../../opt/paddle/bin/paddle_trainer ${@:2}


If install with pip install, the paddle script will be under /usr/local/bin/paddle but binaries are under somewhere like: /usr/lib/python2.7/site-packages/usr/opt/paddle/bin.

So the correct way is to check both where paddle installed and the python package installed, and find where the binaries are.

Just for your reference:
I build and installed with root, then pip install whl, and got paddle in /usr/bin and the binaries are in /usr/lib64/python2.7/site-packages/usr/local/opt/paddle/bin/.

Since paddle is installed with scripts, however other binaries are installed with data_files in setup.py.in.
I thought maybe them could be installed with the same way, then it should be much easier to find the correct binaries.

Thanks @typhoonzero and @tensor-tang , I think the binary file paddle_train/paddle_pserver_main/... is placed under two different directories in the Docker image just because we installed the deb package while building production Docker image.For now we can install paddle with whl instead of deb package, I'll update this PR and please review after a moment.

@tensor-tang , I tried to modify data_files to the absolute path /usr/local/opt/paddle, but maybe date_files will always be installed under python packages directory, here is the document from https://docs.python.org/2.7/distutils/setupscript.html#installing-additional-files:

Each file name in files is interpreted relative to the setup.py script at the top of the package source distribution.

typhoonzero · 2017-08-14T06:54:20Z

paddle/scripts/submit_local.sh.in

+IFS=';' items=($PYTHON_SITEPACKAGES_PATHES)
+for item in ${items[@]}; do
+    if [ -f $item/usr/local/opt/paddle/bin/paddle_trainer ]; then
+        PADDLE_BIN_PATH=$item/usr/local/opt/paddle/bin


PADDLE_BIN_PATH should also check if binaries are under /usr/local/opt/...

As mentioned by @tensor-tang, you can change the binaries installation dir in setup.py.in but either way is OK, it's up to you.

As the comment #3421 (comment), may be date_files does not support the real absolute path...

PADDLE_BIN_PATH should also check if binaries are under /usr/local/opt/...

I'm not sure why we also need to check /usr/local/opt/..., *.whl will never install binary file under /usr/local/opt/...

I try to correct this path in other PR #3461
It may fix your issue, if works.

Cool, thanks @tensor-tang , it works correctly. But for my test, the directory should be configured as opt/paddle/bin and the binary files will be installed under /usr/local/opt/paddle/bin.

configured as opt/paddle/bin and the binary files will be installed under /usr/local/opt/paddle/bin.

which means your sys.prefix is /usr/local? That's weird.
Or some var were set in your env, before pip install ?
Could you please check print sys.prefix? I tried login as root and user. The result are both /usr.

BTW my OS CentOS7.2.

Yep, it's also confusing for me, print sys.prefix will print /usr.

root@6a4bc12ae305:/# python -c "import sys; print sys.prefix" /usr

I executed pip install *.whl in production Docker image, the base image is ubuntu:16.04.

And if the data_files was configured as local/opt/paddle/bin, the binary files would be installed under /usr/local/local/opt/paddle/bin, the duplicate local directory.

Can you try pip install --prefix='/usr' *.whl in docker?

Yep, if I specify the --prefix='/usr', paddle will be installed under /usr.

Thanks @Yancey1989

tensor-tang · 2017-08-15T05:14:37Z

python/setup.py.in

 from setuptools import setup, Distribution
 class BinaryDistribution(Distribution):
    def has_ext_modules(foo):
        return True


This should cause difference path when use cmake install.
cmake install : /usr/local/opt/paddle/bin
pip install : /usr/opt/paddle/bin

Thanks @tensor-tang , but maybe it doesn't matter for the different path?
If users want to install paddle to a customer directory, he/she could set --prefix=<customer directory>, but in the production Docker image, we just make it running correctly.

BTW, the directory in date_files wouldn't be configured local/opt/paddle/bin, this will caused paddle be installed under /usr/local/local/opt/paddle/bin

@tensor-tang , paddle_bin_files is executable files, so I move these to scripts int setup.py.

I think make install and pip install will install to different default location doesn't harm using.

Users could either use make install or pip install to install and then use paddle after that.

The default install path is determined by cmake and pip, we shouldn't interfere with it. Let cmake and pip do their work.

Maybe add some documentation about how to use --prefix parameter when installing, and the default installation behaviour?

but maybe it doesn't matter for the different path?

I am not sure about this, maybe @luotao1 @typhoonzero can answer this?

Maybe add some documentation about how to use --prefix parameter when installing, and the default installation behaviour?

Totally agreed~

Anyway, I am OK with the both ways, just giving a method for this case.
You can change them as you wish. I think the efficient way is the best~
Thanks @Yancey1989 @typhoonzero ~

Done with the document for installing PaddlePaddle.
And thanks @typhoonzero and @tensor-tang !!

tensor-tang · 2017-08-15T07:42:19Z

python/setup.py.in

+               '${PADDLE_BINARY_DIR}/paddle/scripts/paddle']

-paddle_rt_lib_dir = 'local/lib'
+paddle_rt_lib_dir = 'lib'


Maybe this change would impact "pip install " without docker for user.
Since the unit test for this has not ready yet @luotao1

Anyway, if /usr/lib and /usr/local/lib are both in LD_LIBRARY_PATH, then it won't be a problem for users.

luotao1 · 2017-08-15T08:16:51Z

去掉两个local后，我在本地测是可以的。import paddle, py_paddle, paddle.v2.framework.core都可以。
安装路径在下面：

 /home/luotao/.jumbo/bin/paddle
  /home/luotao/.jumbo/bin/paddle_merge_model
  /home/luotao/.jumbo/bin/paddle_pserver_main
  /home/luotao/.jumbo/bin/paddle_trainer
  /home/luotao/.jumbo/bin/paddle_usage
  /home/luotao/.jumbo/lib/libiomp5.so
  /home/luotao/.jumbo/lib/libmkldnn.so
  /home/luotao/.jumbo/lib/libmkldnn.so.0
  /home/luotao/.jumbo/lib/libmklml_intel.so
  /home/luotao/.jumbo/lib/python2.7/site-packages/paddle/__init__.py
  /home/luotao/.jumbo/lib/python2.7/site-packages/paddle/__init__.pyc
  /home/luotao/.jumbo/lib/python2.7/site-packages/paddle/proto/DataConfig_pb2.py
  /home/luotao/.jumbo/lib/python2.7/site-packages/paddle/proto/DataConfig_pb2.pyc
...

tensor-tang · 2017-08-15T08:21:50Z

That's clean and perfect!
Thx @luotao1.

Yancey0623 · 2017-08-15T09:32:54Z

Thanks @luotao1 !

typhoonzero · 2017-08-15T10:58:27Z

doc/getstarted/build_and_install/build_from_source_en.md

-# install PaddlePaddle Python modules.
-sudo pip install <path to install>/opt/paddle/share/wheels/*.whl
-```
+- Install PaddlePaddle with `make install`


Must inform that choose one of these two methods, not both.

I think we can use pip install instead of make install, so I deleted the make install

typhoonzero · 2017-08-15T10:59:03Z

doc/getstarted/build_and_install/build_from_source_en.md

+    export PATH=<path to install>/bin:$PATH
+    # install PaddlePaddle Python modules.
+    sudo pip install <path to install>/opt/paddle/share/wheels/*.whl
+    ```


export PATH=<path to install>/bin:$PATH + # install PaddlePaddle Python modules. + sudo pip install <path to install>/opt/paddle/share/wheels/*.whl

are no longer needed.

typhoonzero · 2017-08-15T10:59:43Z

doc/getstarted/build_and_install/build_from_source_en.md

+    export PATH=<path to install>/bin:$PATH
+    # install PaddlePaddle Python modules.
+    sudo pip install <path to install>/opt/paddle/share/wheels/*.whl
+    ```


+ # set PaddlePaddle installation path in ~/.bashrc + export PATH=<path to install>/bin:$PATH + # install PaddlePaddle Python modules. + sudo pip install <path to install>/opt/paddle/share/wheels/*.whl

are no longer needed.

File ubuntu_install_en.rst and ubuntu_install_cn.rst seems no longer needed.

I'm not sure what ubuntu_install_en.rst and ubuntu_install_cn.rst used for, maybe @luotao1 knows more?

typhoonzero

LGTM++

maybe we can remove ubuntu_install_en.rst in another PR.

luotao1 · 2017-08-16T04:54:51Z

@Yancey1989

I think that ubuntu_install_en.rst and ubuntu_install_cn.rst can be removed.

I think we can use pip install instead of make install, so I deleted the make install

why you delete make install? If users build Paddle from source, they should use make install and then pip install.

Yancey0623 · 2017-08-16T05:17:00Z

Thanks @luotao1 , now all libraries and binary files are packaged into the whl pakcage, so maybe we can use pip install to install PaddlePaddle? If i don't understand enough, please told me ... thanks again.

tensor-tang · 2017-08-16T05:27:57Z

For common users, they can directly pip install the released whl.
For developers, they could download, build and make install from the released source code.

I think we should give this choice to users and developers.

Yancey0623 · 2017-08-16T06:34:41Z

@tensor-tang , for users, I think he/she can pip install from pypi, or use production Dcoker image directly, and only develops would like to download/develop/build/install PaddlePaddle. But i aggree with you, I will retain the two partment of pip install and make install, thanks again!

Yancey0623 · 2017-08-16T07:04:12Z

I create a new issue #3524 to discuss whether we can retain make install, we can discuss that one.
I will merge this PR and recover the document about installing PaddlePaddle, because of we need to fix executing paddle train/pserver failed in the latest Docker image,

Fix invalid paddle binary file path

dfc86dd

Yancey0623 requested a review from typhoonzero August 11, 2017 10:10

typhoonzero requested changes Aug 11, 2017

View reviewed changes

update

98255dc

typhoonzero reviewed Aug 14, 2017

View reviewed changes

tensor-tang mentioned this pull request Aug 14, 2017

add shared mkl lib in whl #3461

Merged

Yancey0623 added 2 commits August 14, 2017 17:50

update

a87b82c

merge develop branch

d6bd503

tensor-tang reviewed Aug 15, 2017

View reviewed changes

fix paddle bin directory

98b35ab

tensor-tang reviewed Aug 15, 2017

View reviewed changes

Yancey0623 added 2 commits August 15, 2017 17:14

update install doc

d98e2d5

update

5c926b0

typhoonzero reviewed Aug 15, 2017

View reviewed changes

update

a872e33

typhoonzero approved these changes Aug 16, 2017

View reviewed changes

Yancey0623 added 2 commits August 16, 2017 15:08

update

ec9ac91

update

3565518

Yancey0623 merged commit 0f86881 into PaddlePaddle:develop Aug 16, 2017

Yancey0623 deleted the fix_paddle_path branch August 16, 2017 08:09

luotao1 mentioned this pull request Aug 30, 2017

remove unused ubuntu Debian install doc #3744

Closed

heavengate pushed a commit to heavengate/Paddle that referenced this pull request Aug 16, 2021

[MOT] add jde fairmot deploy doc (PaddlePaddle#3421)

d214d9e

Fix invalid paddle binary file path #3421

Fix invalid paddle binary file path #3421

Uh oh!

Conversation

Yancey0623 commented Aug 11, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tensor-tang Aug 14, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tensor-tang Aug 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Yancey0623 Aug 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tensor-tang Aug 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luotao1 commented Aug 15, 2017

Uh oh!

tensor-tang commented Aug 15, 2017

Uh oh!

Yancey0623 commented Aug 15, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

typhoonzero left a comment

Choose a reason for hiding this comment

tensor-tang Aug 14, 2017 •

edited

Loading

tensor-tang Aug 15, 2017 •

edited

Loading

Yancey0623 Aug 15, 2017 •

edited

Loading

tensor-tang Aug 15, 2017 •

edited

Loading