Not building with cuQuantum SDK (and cirq)

[rugantio]

I built qsim locally from this repo, also modifing the Makefile in order to build the cuQuantum files:

TARGETS = qsim-custatevec
TESTS = run-custatevec-tests

In fact in the local folder I have a "qsim_custatevec.cpython-39-x86_64-linux-gnu.so" file, but when I do "pip install ." it does not copy over to "/usr/lib/python3.9/site-packages/qsimcirq/"

If I instantiate a simulator that tries to use custatevec it does not find it:

gpu_options = qsimcirq.QSimOptions(use_gpu=True, gpu_mode=1)
qsim_simulator = qsimcirq.QSimSimulator(qsim_options=gpu_options)

results in a "ValueError: cuStateVec GPU execution requested, but not supported. If your device has GPU support and the NVIDIA cuStateVec library is installed, you may need to compile qsim locally."

Everything works fine with just CUDA11 (gpu_mode=0)

[rugantio]

I noticed that the Makefile tests fail, so it it not building correctly. It might be a problem with GCC 11?

Here is the make log:

[rugantio@boris qsim]$ echo $CUQUANTUM_DIR && make qsim-custatevec && make run-custatevec-tests
/home/rugantio/Downloads/cuquantum
make -C apps/ qsim-custatevec
make[1]: Entering directory '/home/rugantio/Downloads/qsim/apps'
nvcc -o ./qsim_base_custatevec.x qsim_base_custatevec.cu -O3 -I/home/rugantio/Downloads/cuquantum/include -L/home/rugantio/Downloads/cuquantum/lib64 -lcustatevec -lcublas
make[1]: Leaving directory '/home/rugantio/Downloads/qsim/apps'
make -C tests/ custatevec-tests
make[1]: Entering directory '/home/rugantio/Downloads/qsim/tests'
git submodule update --init --recursive googletest
mkdir -p /home/rugantio/Downloads/qsim/tests/googletest/googletest/make
cd /home/rugantio/Downloads/qsim/tests/googletest/googletest/make &&    cmake .. && make
CMake Warning at CMakeLists.txt:54 (project):
  VERSION keyword not followed by a value or was followed by a value that
  expanded to nothing.


-- The CXX compiler identification is GNU 11.1.0
-- The C compiler identification is GNU 11.1.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
CMake Deprecation Warning at CMakeLists.txt:56 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Found PythonInterp: /usr/bin/python (found version "3.9.7") 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Configuring done
-- Generating done
-- Build files have been written to: /home/rugantio/Downloads/qsim/tests/googletest/googletest/make
make[2]: Entering directory '/home/rugantio/Downloads/qsim/tests/googletest/googletest/make'
make[3]: Entering directory '/home/rugantio/Downloads/qsim/tests/googletest/googletest/make'
make[4]: Entering directory '/home/rugantio/Downloads/qsim/tests/googletest/googletest/make'
make[4]: Leaving directory '/home/rugantio/Downloads/qsim/tests/googletest/googletest/make'
make[4]: Entering directory '/home/rugantio/Downloads/qsim/tests/googletest/googletest/make'
[ 25%] Building CXX object CMakeFiles/gtest.dir/src/gtest-all.cc.o
[ 50%] Linking CXX static library lib/libgtest.a
make[4]: Leaving directory '/home/rugantio/Downloads/qsim/tests/googletest/googletest/make'
[ 50%] Built target gtest
make[4]: Entering directory '/home/rugantio/Downloads/qsim/tests/googletest/googletest/make'
make[4]: Leaving directory '/home/rugantio/Downloads/qsim/tests/googletest/googletest/make'
make[4]: Entering directory '/home/rugantio/Downloads/qsim/tests/googletest/googletest/make'
[ 75%] Building CXX object CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
[100%] Linking CXX static library lib/libgtest_main.a
make[4]: Leaving directory '/home/rugantio/Downloads/qsim/tests/googletest/googletest/make'
[100%] Built target gtest_main
make[3]: Leaving directory '/home/rugantio/Downloads/qsim/tests/googletest/googletest/make'
make[2]: Leaving directory '/home/rugantio/Downloads/qsim/tests/googletest/googletest/make'
nvcc -o ./hybrid_custatevec_test.x hybrid_custatevec_test.cu -I/home/rugantio/Downloads/qsim/tests/googletest/googletest/include -L/home/rugantio/Downloads/qsim/tests/googletest/googletest/make/lib -lgtest -O3 -I/home/rugantio/Downloads/cuquantum/include -L/home/rugantio/Downloads/cuquantum/lib64 -lcustatevec -lcublas
nvcc -o ./statespace_custatevec_test.x statespace_custatevec_test.cu -I/home/rugantio/Downloads/qsim/tests/googletest/googletest/include -L/home/rugantio/Downloads/qsim/tests/googletest/googletest/make/lib -lgtest -O3 -I/home/rugantio/Downloads/cuquantum/include -L/home/rugantio/Downloads/cuquantum/lib64 -lcustatevec -lcublas
nvcc -o ./qtrajectory_custatevec_test.x qtrajectory_custatevec_test.cu -I/home/rugantio/Downloads/qsim/tests/googletest/googletest/include -L/home/rugantio/Downloads/qsim/tests/googletest/googletest/make/lib -lgtest -O3 -I/home/rugantio/Downloads/cuquantum/include -L/home/rugantio/Downloads/cuquantum/lib64 -lcustatevec -lcublas
qtrajectory_testfixture.h(40): warning #177-D: variable "p2" was declared but never referenced
          detected during instantiation of "void qsim::TestBitFlip(const Factory &) [with Factory=qsim::Factory<float>]" 
qtrajectory_custatevec_test.cu(55): here

qtrajectory_testfixture.h(89): warning #177-D: variable "p3" was declared but never referenced
          detected during instantiation of "void qsim::TestGenDump(const Factory &) [with Factory=qsim::Factory<float>]" 
qtrajectory_custatevec_test.cu(59): here

qtrajectory_testfixture.h(119): warning #177-D: variable "p3" was declared but never referenced
          detected during instantiation of "void qsim::TestGenDump(const Factory &) [with Factory=qsim::Factory<float>]" 
qtrajectory_custatevec_test.cu(59): here

../lib/channels_cirq.h(414): warning #177-D: variable "p2" was declared but never referenced
          detected during:
            instantiation of "qsim::Cirq::Channel<fp_type> qsim::Cirq::BitFlipChannel<fp_type>::Create(unsigned int, unsigned int) const [with fp_type=float]" 
../lib/circuit_noisy.h(63): here
            instantiation of "qsim::NoisyCircuit<Gate> qsim::MakeNoisy<Gate,ChannelFactory>(unsigned int, std::vector<Gate, std::allocator<Gate>>::const_iterator, std::vector<Gate, std::allocator<Gate>>::const_iterator, const ChannelFactory &) [with Gate=qsim::Gate<float, qsim::Cirq::GateKind>, ChannelFactory=qsim::Cirq::BitFlipChannel<float>]" 
../lib/circuit_noisy.h(88): here
            instantiation of "qsim::NoisyCircuit<Gate> qsim::MakeNoisy(unsigned int, const std::vector<Gate, std::allocator<Gate>> &, const ChannelFactory &) [with Gate=qsim::Gate<float, qsim::Cirq::GateKind>, ChannelFactory=qsim::Cirq::BitFlipChannel<float>]" 
qtrajectory_testfixture.h(623): here
            instantiation of "void qsim::TestUncomputeFinalState(const Factory &) [with Factory=qsim::Factory<float>]" 
qtrajectory_custatevec_test.cu(75): here

nvcc -o ./simulator_custatevec_test.x simulator_custatevec_test.cu -I/home/rugantio/Downloads/qsim/tests/googletest/googletest/include -L/home/rugantio/Downloads/qsim/tests/googletest/googletest/make/lib -lgtest -O3 -I/home/rugantio/Downloads/cuquantum/include -L/home/rugantio/Downloads/cuquantum/lib64 -lcustatevec -lcublas
make[1]: Leaving directory '/home/rugantio/Downloads/qsim/tests'
make -C tests/ run-custatevec-tests
make[1]: Entering directory '/home/rugantio/Downloads/qsim/tests'
for exe in ./hybrid_custatevec_test.x ./statespace_custatevec_test.x ./qtrajectory_custatevec_test.x ./simulator_custatevec_test.x; do if ! ./$exe; then exit 1; fi; done
././hybrid_custatevec_test.x: error while loading shared libraries: libcustatevec.so.0: cannot open shared object file: No such file or directory
make[1]: *** [Makefile:43: run-custatevec-tests] Error 1
make[1]: Leaving directory '/home/rugantio/Downloads/qsim/tests'
make: *** [Makefile:72: run-custatevec-tests] Error 2

[95-martin-orion]

pip install . uses the CMake build path. Changes to Makefiles or .so files generated by running make will not affect it. (FWIW, the logs above appear to only be warnings - do you know if the tests pass with the warnings disabled?)

That said, if you have the cuQuantum SDK installed, the CMake build process should automatically take cuQuantum into account. To pipe CUQUANTUM_DIR into CMake, you may need to export it by running:

export CUQUANTUM_DIR=[PATH_TO_CUQUANTUM_SDK]

(where [PATH_TO_CUQUANTUM_SDK] is replaced with the path to your cuQuantum installation)

[rugantio]

@95-martin-orion have you read the log at all? the CUQUANTUM_DIR variable is set to "/home/rugantio/Downloads/cuquantum", you can see that it was correctly parsed.
The "make qsim-custatevec" contains only warnings that's right, this is the problem, that I'm referring to, because the tests that are supposed to check that everything was fine are running into errors (see the last 6 lines). why is libcustatevec.so.0 not detected?

[leofang]

@rugantio Try adding the cuQuantum libraries to your LD_LIBRARY_PATH, something like:

export LD_LIBRARY_PATH=$CUQUANTUM_DIR/lib64:$LD_LIBRARY_PATH

[95-martin-orion]

RE: CUQUANTUM_DIR, I'm just verifying the build conditions. Running CUQUANTUM_DIR=[PATH_TO_CUQUANTUM] would also cause echo $CUQUANTUM_DIR to print the expected path, but unless it was exported the variable may not be available to CMake or other tools.

My read on the errors is that make -C tests/ custatevec-tests stopped short of generating the .so file due to the warnings it encountered, which caused the subsequent errors during make -C tests/ run-custatevec-tests. Disabling warnings or using make -k {...} to keep going after warnings may resolve the issue.

Adding @sergeisakov for his experience running these tests, and input on fixing warnings.

[rugantio]

@leofang thank you leo, linking the loader did solve the issue related to "make run-custatevec-tests", which now works fine and returns, correctly:

[----------] Global test environment tear-down
[==========] 20 tests from 2 test suites ran. (25545 ms total)
[  PASSED  ] 20 tests.

However there is still something wrong with the .so file installation (which at this point we can safely say that is independent of the tests). As I mentioned, I can find the qsim_custatevec.cpython-39-x86_64-linux-gnu.so in the local qsimcirc folder (@95-martin-orion so it seems that the ADD_SUBDIRECTORY(pybind_interface/custatevec) in the CMakeLists is executed correctly?), but then it is not moved to the python environment.
I can verify it also by running

import importlib
importlib.import_module("qsimcirq.qsim_custatevec")

results in a ModuleNotFoundError: No module named 'qsimcirq.qsim_custatevec'