Installing spaCy with OpenMP support on OS X

[mikepb]

I have successfully built and installed spaCy 0.100.7 with OpenMP on OS X for brew installed python. For the impatient among us, here are instructions on how to do it yourself.

Choose your adventure:

1. Install python:
   • brew install python
   • Other distributions such as Anaconda may work as well. Please let us know if you are successful!
2. (choose one):
   • llvm-3.8
   • clang-omp
   • GCC (Warning: segfauts Crash when built with GCC and OpenMP on OS X #266)
3. Edit setup.py, then build and test spacy
4. Install spacy

Note that these instructions are for brew installed python.

Option: LLVM-3.8

1. Download the LLVM-3.8 binaries:

   • Download page: http://llvm.org/releases/download.html#3.8.0
   • Direct link for MacOS binaries: http://llvm.org/releases/3.8.0/clang+llvm-3.8.0-x86_64-apple-darwin.tar.xz

2. Install the binaries:

   For example, to install at /opt/llvm38:

   tar xJf clang+llvm-3.8.0-x86_64-apple-darwin.tar.xz
   sudo mkdir -p /opt
   sudo mv clang+llvm-3.8.0-x86_64-apple-darwin /opt/llvm38

3. Tell pip to use clang-3.8:

   export CC=/opt/llvm38/bin/clang
   export CXX=/opt/llvm38/bin/clang++
   export PATH=/opt/llvm38/bin:$PATH
   export C_INCLUDE_PATH=/opt/llvm38/include:$C_INCLUDE_PATH
   export CPLUS_INCLUDE_PATH=/opt/llvm38/include:$CPLUS_INCLUDE_PATH
   export LIBRARY_PATH=/opt/llvm38/lib:$LIBRARY_PATH
   export DYLD_LIBRARY_PATH=/opt/llvm38/lib:$DYLD_LIBRARY_PATH

Option: Clang-OMP

1. Install clang-omp Homebrew:

   brew install clang-omp

2. Tell pip to use clang-omp:

   export CC=clang-omp
   export CXX=clang-omp
   export PATH=/usr/local/bin:$PATH
   export C_INCLUDE_PATH=/usr/local/include/libiomp:$C_INCLUDE_PATH
   export CPLUS_INCLUDE_PATH=/usr/local/include/libiomp:$CPLUS_INCLUDE_PATH
   export LIBRARY_PATH=/usr/local/lib:$LIBRARY_PATH
   export DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH

Option: GCC

1. Install GCC via Homebrew:

   brew install gcc --without-multilib

   The --without-multilib option is required for OpenMP support.

2. Tell pip to use GCC:

   export CC=gcc-5
   export CXX=g++-5

WARNING: Compiling with GCC as of spaCy 0.100.5 may result in a segfault (#266).

Required: edit setup.py and install

Follow the 'Compile from source' instructions from spaCy documentation, with the following adjustments.

git clone https://github.com/honnibal/spaCy.git
cd spaCy
git checkout 0.100.6 # or 'master' if you wish

Edit setup.py lines 88-90 to enable OpenMP:

# if not sys.platform.startswith('darwin'):
compile_options['other'].append('-fopenmp')
link_options['other'].append('-fopenmp')

Now continue with the install instuctions as per the documentation.

virtualenv .env && source .env/bin/activate
export PYTHONPATH=`pwd`
pip install -r requirements.txt
python setup.py clean
pip install -e .
python -m spacy.en.download
pip install pytest
py.test spacy/tests/

To install spaCy outside of virtualenv and/or outside the source directory:

1. Deactivate virtualenv using the deactivate command.
2. Run pip install . in the source directory.
3. For clang/llvm, add the appropriate library path:
   • clang-omp: export DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH
   • llvm38: export DYLD_LIBRARY_PATH=/opt/llvm38/lib:$DYLD_LIBRARY_PATH

Relevant resources:

• http://docs.cython.org/src/userguide/parallelism.html
• https://clang-omp.github.io
• http://stackoverflow.com/questions/35134681/installing-openmp-on-mac-os-x-10-11
• http://stackoverflow.com/questions/27273302/cython-conditional-compile-based-on-external-value-given-via-setuptools
• http://stackoverflow.com/questions/16549893/programatically-testing-for-openmp-support-from-a-python-setup-script

Thank you for the help @honnibal @henningpeters @gushecht !

History:

• May 9 2016: Add instructions for LLVM-3.8 and use deactivate virtualenv
  command.
• use pip install -e . instead of python setup.py build_ext --inplace
• add instructions for installing outside of virtualenv.
• Update for 0.100.6

[henningpeters]

Thanks for the pointers and making it work.

I am not a big fan of these silent up/downgrades depending on what dependencies you had installed. PIL comes to my mind as a particular bad example of this style. I would rather prefer passing some args/envs to setup.py and fail if dependencies are not available or ship a separate package. We'll definitely look into making this more accessible.

[mikepb]

LLVM since version 3.7 added native OpenMP support, so the best solution may just be to wait until Apple releases the next version of XCode this summer (or thereabouts).

[henningpeters]

I can confirm clang-omp to work. We had another thread some time ago about compilation problems with gcc on OSX (#237). Its unfortunate, but there is very little support on detecting compilers with setuptools and even if you could do that reliably you might still be out of luck in case the user's Python got built with non-gcc flags (AFAIK they are not visible from within setup.py). What I want to say is: supporting gcc on osx is really tough while not being that much appreciated in case it does work.

[mikepb]

Great news on clang-omp. I've been experimenting with many native extensions for Python, R, and NodeJS. I've had so many issues with native extensions and compilers that I now just install whichever compiler works best... Once Apple updates XCode with LLVM 3.7, there will be much less need for supporting GCC on OS X.

Practically speaking, the clang-omp solution works for me and I'm okay leaving it at that for now. I'm sure you guys have plenty of other interesting improvements to make to spaCy!

[gushecht]

Hello again @mikepb

I got to python setup.py build_ext --inplace and then it failed. Screenshot attached. Many thanks, as always.

Also, for what it's worth, I found that I had to downgrade to Python 2.7.9 from 2.7.11

[mikepb]

I see you are using the Anaconda Python. I wrote the instructions for the homebrew Python. What is the compiler used to build Anaconda Python?

Just running the interpreter on my machine printed this message:

Python 2.7.11 (default, Feb 18 2016, 14:32:04) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

If you are using GGC-built Python, try using GCC to build spacy instead:

brew install gcc --without-multilib
export CC=gcc-5
export CXX=g++-5

I've been told that the gcc --without-multilib option is required for the OpenMP support used by spacy.

If you are successful, please let us know!

[mikepb]

Remember to reset your build environment as well. Running python setup.py clean then using a new terminal session should do the trick.

[gushecht]

Update: tried with GCC and successfully got the segfault :)

So then I tried with homebrew Python. Please note that I was getting an issue trying to clone the repo after running the export statements, so I revised the order a little:

brew install python
brew install clang-omp
git clone https://github.com/honnibal/spaCy.git
cd spaCy
git checkout 0.100.5
# Made the changes you specified to setup.py
export CC=clang-omp
export CXX=clang-omp
export PATH=/usr/local/bin:$PATH
export C_INCLUDE_PATH=/usr/local/include/libiomp:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=/usr/local/include/libiomp:$CPLUS_INCLUDE_PATH
export LIBRARY_PATH=/usr/local/lib:$LIBRARY_PATH
export DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH
virtualenv .env && source .env/bin/activate
export PYTHONPATH=`pwd`
pip install -r requirements.txt
python setup.py clean
python setup.py build_ext --inplace

And then....when downloading the English pack

(.env) Guss-MacBook-Pro:spaCy gushecht$ python -m spacy.en.download
/Users/gushecht/spaCy/.env/bin/python: dlopen(spacy/tokenizer.so, 2): Symbol not found: __ZTINSt8ios_base7failureE
  Referenced from: spacy/tokenizer.so
  Expected in: dynamic lookup

Any thoughts?

Perhaps at this point it makes more sense for me to spin up a Linux box on AWS and bypass the issue.

[mikepb]

I actually downloaded the English pack using pip-installed spacy and symlinked the directory. I'm on a slow connection and neglected to test that command. Symlinking the data dir might work.

I've used the spot instances for cheap, but expect your instance to be killed during peak hours.

[henningpeters]

As an alternative to python -m spacy.en.download you can also run python -m sputnik --name spacy install en. But a symbol not found error looks like your installation didn't finish properly or is in a half-baked state. Can you run python setup.py clean and install again?

Btw: python setup.py build_ext --inplace is outdated. Please run pip install -e . for a development install.

[gushecht]

Hey all, looks like pip install -e . was the magic formula!

For final reference:

brew install python

brew install clang-omp

git clone https://github.com/honnibal/spaCy.git

cd spaCy
git checkout 0.100.5
# Make the specified changes to setup.py

export CC=clang-omp
export CXX=clang-omp
export PATH=/usr/local/bin:$PATH
export C_INCLUDE_PATH=/usr/local/include/libiomp:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=/usr/local/include/libiomp:$CPLUS_INCLUDE_PATH
export LIBRARY_PATH=/usr/local/lib:$LIBRARY_PATH
export DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH

virtualenv .env && source .env/bin/activate

export PYTHONPATH=`pwd`
pip install -r requirements.txt

python setup.py clean

pip install -e .

Thanks so much to each of you.

[mikepb]

Thanks for the update 👍 I've updated the issue description to use pip install -e .

[mikepb]

Updated instructions with LLVM-3.8 option for native OpenMP support.