ARM cross-compilation (tl;dr: use proper SPIR target)

[lissyx]

As mentionned in mozilla/DeepSpeech#1346 I'm currently investigating how much we can rely on the OpenCL VC4CL [https://github.com/doe300/VC4CL#opencl-support] driver to leverage RPi3's GPU.

So far, I built successfully the driver with a linaro cross-compiler and vc4c's testsuite somehow works. I could also verify that comptecpp_info can at least see the things.

Now I am facing a dumb issue: how to cross-compile for ARM from SYCL branches. We have setup to cross-compile for ARM and ARMv8 on https://github.com/mozilla/tensorflow, so I blindly did a configure step referencing the ARM version of ComputeCpp:

echo "" | TF_NEED_GCP=0 TF_NEED_GDR=0 TF_NEED_HDFS=0 TF_NEED_S3=0 TF_NEED_JEMALLOC=1 TF_ENABLE_XLA=0 TF_NEED_MKL=0 TF_NEED_VERBS=0 TF_NEED_MPI=0 TF_NEED_CUDA=0 TF_NEED_OPENCL_SYCL=1 TF_NEED_COMPUTECPP=1 COMPUTECPP_TOOLKIT_PATH=../ComputeCpp-CE-0.7.0-Ubuntu-14.04-ARM_32/ TF_USE_DOUBLE_SYCL=0 TF_USE_HALF_SYCL=0 ./configure

And then, I built it:

bazel build --config=sycl -s -j 96 --config=monolithic  --config=rpi3 --config=rpi3_opt -c opt --copt=-fvisibility=hidden --copt=-DCTC_DISABLE_OMP --verbose_failures //native_client:libdeepspeech.so //native_client:deepspeech_utils

This do build an ARM lib, linked with libComputeCpp.so. But at runtime, it does not seems like it runs OpenCL.

Now, I've also stumbled upon TF_SYCL_CROSS_TOOLCHAIN and TF_SYCL_CROSS_TOOLCHAIN_NAME, but they lack of documentation, and trying to use them do fail:

$ echo "" | TF_NEED_GCP=0 TF_NEED_GDR=0 TF_NEED_HDFS=0 TF_NEED_S3=0 TF_NEED_JEMALLOC=1 TF_ENABLE_XLA=0 TF_NEED_MKL=0 TF_NEED_VERBS=0 TF_NEED_MPI=0 TF_NEED_CUDA=0 TF_NEED_OPENCL_SYCL=1 TF_NEED_COMPUTECPP=1 COMPUTECPP_TOOLKIT_PATH=ComputeCpp-CE-0.7.0-Ubuntu-14.04-ARM_32/ TF_SYCL_CROSS_TOOLCHAIN=gcc-linaro-4.9.4-2017.01-x86_64_arm-linux-gnueabihf/bin/ TF_SYCL_CROSS_TOOLCHAIN_NAME=arm-linux-gnueabihf- TF_USE_DOUBLE_SYCL=0 TF_USE_HALF_SYCL=0 ./configure 
$ bazel build --config=sycl -s -j 96 --config=monolithic -c opt --copt=-fvisibility=hidden --copt=-DCTC_DISABLE_OMP --verbose_failures //native_client:libdeepspeech.so //native_client:deepspeech_utils
[...]
  ComputeCpp-CE-0.7.0-Ubuntu-14.04-ARM_32/bin/compute -ffunction-sections -fdata-sections -fPIE -fno-omit-frame-pointer -Wall -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections -DGEMMLOWP_ALLOW_SLOW_SCALAR_FALLBACK '-DDISABLE_SKINNY=1' '-fvisibility=hidden' -DCTC_DISABLE_OMP '-std=c++11' -fsycl-ih-last -sycl-driver -Xclang -cl-denorms-are-zero -Xclang -cl-fp32-correctly-rounded-divide-sqrt -Xclang -cl-mad-enable -sycl-target spir64 '-DTENSORFLOW_USE_SYCL=1' '-DEIGEN_USE_SYCL=1' '-DEIGEN_HAS_C99_MATH=1' '-DEIGEN_HAS_CXX11_MATH=1' -Wno-unused-variable -Wno-unused-const-variable '-DTENSORFLOW_SYCL_NO_HALF=1' '-DTENSORFLOW_SYCL_NO_DOUBLE=1' -MD -MF bazel-out/k8-opt/bin/external/double_conversion/_objs/double-conversion/external/double_conversion/double-conversion/diy-fp.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/double_conversion/_objs/double-conversion/external/double_conversion/double-conversion/diy-fp.pic.o' -fPIC -iquote external/double_conversion -iquote bazel-out/k8-opt/genfiles/external/double_conversion -iquote external/bazel_tools -iquote bazel-out/k8-opt/genfiles/external/bazel_tools -isystem external/double_conversion -isystem bazel-out/k8-opt/genfiles/external/double_conversion -isystem external/bazel_tools/tools/cpp/gcc3 -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -no-canonical-prefixes -c external/double_conversion/double-conversion/diy-fp.cc -o bazel-out/k8-opt/bin/external/double_conversion/_objs/double-conversion/external/double_conversion/double-conversion/diy-fp.pic.o)
ComputeCpp-CE-0.7.0-Ubuntu-14.04-ARM_32/bin/compute: 1 ComputeCpp-CE-0.7.0-Ubuntu-14.04-ARM_32/bin/compute: Syntax error: word unexpected (expecting ")")

which seems expected, given ComputeCpp-CE-0.7.0-Ubuntu-14.04-ARM_32/bin/compute++: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=df02ab122bb64fc87724de838f7d5a45b8e3f1a5, not stripped

So, what step am I missing to be able to cross-compile ?

[lissyx]

For context, I'm using current https://github.com/lukeiwanski/tensorflow/tree/integration/1.8

[lissyx]

Okay, first mistake on my side, it looks like I should --config=sycl_arm instead of --config=sycl ?

[lissyx]

Using --config=sycl_arm fails like that: ERROR: Toolchain identifier '' for cpu 'armeabi' is illegal (does not match '[a-zA-Z_][\.\- \w]*')

[lissyx]

Okay, so it seems TF_SYCL_CROSS_TOOLCHAIN and TF_SYCL_CROSS_TOOLCHAIN_NAME are not being put into the action_env, and that's why sycl_configure.bzl fails to identify cross-compilation. I'm now able to start the build with proper SYCL-level cross-compilation. It seems I still have to pass a ComputeCpp SDK that can run on my system, and not the ARM one.

Also, proper value for TF_SYCL_CROSS_TOOLCHAIN_NAME=arm-linux-gnueabihf because that should be a valid clang triplet, and TF_SYCL_CROSS_TOOLCHAIN=gcc-linaro-4.9.4-2017.01-x86_64_arm-linux-gnueabihf/. Now, waiting for the build to go to the end :)

[lissyx]

Looks like clang sees different things when cross-compiling. Currently stuck with those errors on protobuf, never had any issue cross-compiling for armv7 and armv8 with gcc 4.9.4 and gcc 7.2.1:

$ grep 'error: ' build.log 
external/protobuf_archive/src/google/protobuf/arena_impl.h:280:55: error: cast from pointer to smaller type 'google::protobuf::internal::AtomicWord' (aka 'int') loses information
external/protobuf_archive/src/google/protobuf/metadata_lite.h:142:9: error: cast from pointer to smaller type 'intptr_t' (aka 'int') loses information
external/protobuf_archive/src/google/protobuf/metadata_lite.h:142:9: error: cast from pointer to smaller type 'intptr_t' (aka 'int') loses information
external/protobuf_archive/src/google/protobuf/metadata_lite.h:76:15: error: no matching member function for call to 'PtrValue'
external/protobuf_archive/src/google/protobuf/metadata_lite.h:137:12: error: cast from pointer to smaller type 'intptr_t' (aka 'int') loses information
external/protobuf_archive/src/google/protobuf/metadata_lite.h:137:12: error: cast from pointer to smaller type 'intptr_t' (aka 'int') loses information
external/protobuf_archive/src/google/protobuf/metadata_lite.h:142:9: error: cast from pointer to smaller type 'intptr_t' (aka 'int') loses information
external/protobuf_archive/src/google/protobuf/metadata_lite.h:142:9: error: cast from pointer to smaller type 'intptr_t' (aka 'int') loses information
external/protobuf_archive/src/google/protobuf/metadata_lite.h:76:15: error: no matching member function for call to 'PtrValue'

[lissyx]

To summup:

• is it expected I should use COMPUTECPP_TOOLKIT_PATH=ComputeCpp-CE-0.7.0-Ubuntu-16.04-x86_64/ when cross-compiling for ARM ?
• is it expected that action_env flags TF_SYCL_CROSS_TOOLCHAIN and TF_SYCL_CROSS_TOOLCHAIN_NAME are not being passed from configure and that I need to specify them ?
• According to my described steps, should I expect to see the build to complete?

[Rbiessy]

Hello lissyx,

Our cross-compilation is still experimental and is indeed not documented anywhere yet. Thank you for your interest in it.

You seem to be really close to getting it working. Here are a few tips that should help you:

• you should create a new computecpp folder that contains everything from ComputeCpp-CE-0.7.0-Ubuntu-16.04-x86_64 except for libComputeCpp.so which should be replaced by the one from ComputeCpp-CE-0.7.0-Ubuntu-16.04-ARM_64.
• you have to export TF_SYCL_CROSS_TOOLCHAIN and TF_SYCL_CROSS_TOOLCHAIN_NAME to your environment before running configure.
• you will also probably have to export CC_OPT_FLAGS="-march=armv8-a".

On a side note if you are using the tip of dev/amd_gpu you will need to update to ComputeCpp CE 0.8.0 but that is not needed for cross-compiling.

Hope this helps!

[lissyx]

Thanks @Rbiessy for your feedback. So it means with my current setup, it should work upto the link stage (because currently my libComputeCpp.so is amd64 and not armv7).

I did pass TF_SYCL_CROSS_TOOLCHAIN TF_SYCL_CROSS_TOOLCHAIN_NAME when running ./configure, but they were not being picked Is it possible this might result in misconfiguration ?

I'm targetting ARMv7 and not ARMv8 so I guess I need to use -march=armv7-a as we do already for our other targets :)

[lissyx]

BTW i'm using lukeiwanski/integration/1.8, is that right with ComputeCpp v0.7.0 ?

[Rbiessy]

Yes it should fail to link.
I am not sure what you mean by not being picked by configure? But since you specified --config=sycl-arm it would have failed immediately if these were wrong.
Oh right, it should (cross-)compile but I would recommend CE 0.8.0 for integration/1.8.

[lissyx]

Would you be able to share which toolchain you are using on your side @Rbiessy ? And which ARM target ?

[lissyx]

@Rbiessy Well, when I say not being picked by configure I mean I run it with those in the env:

TF_SYCL_CROSS_TOOLCHAIN=xxx TF_SYCL_CROSS_TOOLCHAIN_NAME=yyy ./configure

But then they are not written into .tf_configure.bazelrc, and thus at the bazel build step, they are not there. Now, it's likely if you export, that they might get seen at the bazel build time, but it's a bit weird since other variables do not :-). Not a huge issue IMHO.

I'll give a try to ComputeCpp 0.8.0, but I'm not sure. It feels like there's some mess in the include directories. I'm trying that with GCC 4.9.4 from Linaro: https://releases.linaro.org/components/toolchain/binaries/4.9-2017.01/arm-linux-gnueabihf/gcc-linaro-4.9.4-2017.01-x86_64_arm-linux-gnueabihf.tar.xz

[Rbiessy]

I tried the cross-compilation with TF_SYCL_CROSS_TOOLCHAIN=gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu and TF_SYCL_CROSS_TOOLCHAIN_NAME=aarch64-linux-gnu (64bit ARMv8). Also the mali OpenCL driver requires spirv64 which can be specified with TF_SYCL_BITCODE_TARGET if needed.
That reminds me that you will have some issues with python. You have to make sure to download all the TF python dependencies installed on your machine for the targeted architecture. Then create a symlink in your toolchain like so: ln -s /usr/include/aarch64-linux-gnu/python2.7/ path/to/gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu/aarch64-linux-gnu/libc/usr/include/aarch64-linux-gnu/ (for my setup).

I see, if I remember correctly it was because we didn't want to introduce even more questions specific to ComputeCpp in the configure.

Finally it is pretty much guaranteed that you will have more issues specific to 32bit architectures. We plan to work on that at some point.

[lissyx]

Thanks! We make no use of Python, so we should be good on that side. I'll verify build with armv8 as well, to check if I'm doing something wrong or if that is just the current status of the build support :)