cmake: make pre-built library discoverable via CMake configuration files

[ConnorBaker]

Similar to the work done in mitsuba-renderer/nanothread#7, this PR adds the ability to make drjit-core discoverable via CMake instead of requiring projects vendor it in-tree.

Additionally, it allows testing through CMake by using CTests

[ConnorBaker]

@wjakob are you able to run the CI test again to see if the latest works? If it doesn't can you post the failure so I can troubleshoot it?

[njroussel]

Hi @ConnorBaker

If you log in as a guest, you should be able to see the build log.
(I'm unsure why it only ran the Linux Clang build though, instead of the full matrix of OS x compilers.)

[ConnorBaker]

After adding nanothread to the export set, I'm getting this error:

[06:34:11] :	 [Step 2/3] [47/55] Building CXX object tests/CMakeFiles/test_vcall.dir/vcall.cpp.o
[06:34:12] :	 [Step 2/3] [48/55] Linking CXX shared library libdrjit-core.so
[06:34:12] :	 [Step 2/3] FAILED: libdrjit-core.so 
[06:34:12] :	 [Step 2/3] : && /usr/bin/clang++-10 -fPIC -stdlib=libc++ -D_LIBCPP_VERSION -fcolor-diagnostics -O3 -DNDEBUG -flto=thin  -stdlib=libc++ -shared -Wl,-soname,libdrjit-core.so -o libdrjit-core.so CMakeFiles/drjit-core.dir/src/log.cpp.o CMakeFiles/drjit-core.dir/src/strbuf.cpp.o CMakeFiles/drjit-core.dir/src/var.cpp.o CMakeFiles/drjit-core.dir/src/op.cpp.o CMakeFiles/drjit-core.dir/src/malloc.cpp.o CMakeFiles/drjit-core.dir/src/registry.cpp.o CMakeFiles/drjit-core.dir/src/util.cpp.o CMakeFiles/drjit-core.dir/src/cuda_api.cpp.o CMakeFiles/drjit-core.dir/src/cuda_core.cpp.o CMakeFiles/drjit-core.dir/src/cuda_tex.cpp.o CMakeFiles/drjit-core.dir/src/cuda_eval.cpp.o CMakeFiles/drjit-core.dir/src/optix_api.cpp.o CMakeFiles/drjit-core.dir/src/optix_core.cpp.o CMakeFiles/drjit-core.dir/src/llvm_api.cpp.o CMakeFiles/drjit-core.dir/src/llvm_memmgr.cpp.o CMakeFiles/drjit-core.dir/src/llvm_core.cpp.o CMakeFiles/drjit-core.dir/src/llvm_mcjit.cpp.o CMakeFiles/drjit-core.dir/src/llvm_orcv2.cpp.o CMakeFiles/drjit-core.dir/src/llvm_eval.cpp.o CMakeFiles/drjit-core.dir/src/io.cpp.o CMakeFiles/drjit-core.dir/src/eval.cpp.o CMakeFiles/drjit-core.dir/src/vcall.cpp.o CMakeFiles/drjit-core.dir/src/loop.cpp.o CMakeFiles/drjit-core.dir/src/init.cpp.o CMakeFiles/drjit-core.dir/src/api.cpp.o CMakeFiles/drjit-core.dir/resources/kernels.c.o  -Wl,-rpath,/drjit-core/build/ext/nanothread:  ext/nanothread/libnanothread.so  liblz4.a  libxxhash.a  -ldl  -lpthread && :
[06:34:12] :	 [Step 2/3] /usr/bin/ld: liblz4.a(lz4.c.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
[06:34:12] :	 [Step 2/3] clang: error: linker command failed with exit code 1 (use -v to see invocation)

Full log is at https://gist.github.com/ConnorBaker/ac5b8e16cdc3714673b540a215ae1b8b

Is it possible the environment already has lz4 in the environment?

I guess, more generally, should use of system-provided dependencies be gated behind an option? I know that PyTorch, for example, uses USE_SYSTEM_* flags.

[ConnorBaker]

I've added DRJIT_USE_SYSTEM_* flags to try to guard the functionality behind options. Also seemed to have resolved the build errors by adding lz4 and xxhash as shared libraries, specifically.

[ConnorBaker]

@njroussel would you be able to take a look at why CI isn't running for any of the other platforms?

[wjakob]

Hi @ConnorBaker -- an important meta-comment: we're in the midst of a huge refactor of drjit-core, drjit, and mitsuba. I fear the work you are doing here will all need to be redone once we're done with this refactor (it will end up with the nanobind branch here and nanobind_v2 in the main Dr.Jit repository taking the replacing the master one). Would it make more sense to postpone this project for a bit?

[ConnorBaker]

@wjakob I've been checking the other branches every few hours because I am very excited about the refactor! How stable would you say the CMake files are? I don't mind contributing to the work-in-progress branches; otherwise, I'm fine with holding off for a bit.

I hope the refactor is going well!