"Caffe2: Cannot find cuDNN library" solved

I was receiving the following build error about Caffe2: Cannot find cuDNN library

(cp2) c2@rtx1:~/src/spconv$ python setup.py bdist_wheel
running bdist_wheel
running build
running build_py
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/spconv
copying spconv/identity.py -> build/lib.linux-x86_64-3.6/spconv
copying spconv/__init__.py -> build/lib.linux-x86_64-3.6/spconv
copying spconv/pool.py -> build/lib.linux-x86_64-3.6/spconv
copying spconv/ops.py -> build/lib.linux-x86_64-3.6/spconv
copying spconv/tables.py -> build/lib.linux-x86_64-3.6/spconv
copying spconv/test_utils.py -> build/lib.linux-x86_64-3.6/spconv
copying spconv/modules.py -> build/lib.linux-x86_64-3.6/spconv
copying spconv/functional.py -> build/lib.linux-x86_64-3.6/spconv
copying spconv/conv.py -> build/lib.linux-x86_64-3.6/spconv
creating build/lib.linux-x86_64-3.6/spconv/utils
copying spconv/utils/__init__.py -> build/lib.linux-x86_64-3.6/spconv/utils
running build_ext
Release
|||||CMAKE ARGS||||| ['-DCMAKE_PREFIX_PATH=/home/c2/anaconda3/envs/cp2/lib/python3.6/site-packages/torch', '-DPYBIND11_PYTHON_VERSION=3.6', '-DSPCONV_BuildTests=OFF', '-DPYTORCH_VERSION=10600', '-DCMAKE_CUDA_FLAGS="--expt-relaxed-constexpr" -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/c2/src/spconv/build/lib.linux-x86_64-3.6/spconv', '-DCMAKE_BUILD_TYPE=Release']
-- The CXX compiler identification is GNU 7.5.0
-- The CUDA compiler identification is NVIDIA 11.1.105
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda-11.1/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found CUDA: /usr/local/cuda (found version "10.1") 
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 10.1
-- Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH) 
CMake Warning at /home/c2/anaconda3/envs/cp2/lib/python3.6/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:109 (message):
  Caffe2: Cannot find cuDNN library.  Turning the option off
Call Stack (most recent call first):
  /home/c2/anaconda3/envs/cp2/lib/python3.6/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /home/c2/anaconda3/envs/cp2/lib/python3.6/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
  CMakeLists.txt:22 (find_package)


-- Autodetected CUDA architecture(s):  7.5
-- Added CUDA NVCC flags for: -gencode;arch=compute_75,code=sm_75
CMake Error at /home/c2/anaconda3/envs/cp2/lib/python3.6/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message):
  Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN
  libraries.  Please set the proper cuDNN prefixes and / or install cuDNN.
Call Stack (most recent call first):
  /home/c2/anaconda3/envs/cp2/lib/python3.6/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
  CMakeLists.txt:22 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/c2/src/spconv/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log".
See also "/home/c2/src/spconv/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "setup.py", line 108, in <module>
    zip_safe=False,
  File "/home/c2/anaconda3/envs/cp2/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/home/c2/anaconda3/envs/cp2/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/c2/anaconda3/envs/cp2/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/home/c2/anaconda3/envs/cp2/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/c2/.local/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 192, in run
    self.run_command('build')
  File "/home/c2/anaconda3/envs/cp2/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/c2/anaconda3/envs/cp2/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/c2/anaconda3/envs/cp2/lib/python3.6/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/home/c2/anaconda3/envs/cp2/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/c2/anaconda3/envs/cp2/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "setup.py", line 48, in run
    self.build_extension(ext)
  File "setup.py", line 91, in build_extension
    subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
  File "/home/c2/anaconda3/envs/cp2/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/home/c2/src/spconv', '-DCMAKE_PREFIX_PATH=/home/c2/anaconda3/envs/cp2/lib/python3.6/site-packages/torch', '-DPYBIND11_PYTHON_VERSION=3.6', '-DSPCONV_BuildTests=OFF', '-DPYTORCH_VERSION=10600', '-DCMAKE_CUDA_FLAGS="--expt-relaxed-constexpr" -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/c2/src/spconv/build/lib.linux-x86_64-3.6/spconv', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.

And fixed it by pointing CMake to the appropriate directories in CMakeCache.txt (for me it’s in build/temp.linux-x86_64-3.6/CMakeCache.txt). To do this look for the following lines in your CMakeCache.txt and change them to your cuda / cudnn paths, for example:

//Folder containing NVIDIA cuDNN header files
CUDNN_INCLUDE_DIR:FILEPATH=/home/c2/anaconda3/envs/cp2/include

//Path to a file.
CUDNN_INCLUDE_PATH:PATH=/home/c2/anaconda3/envs/cp2/include

//Path to the cudnn library file (e.g., libcudnn.so)
CUDNN_LIBRARY:FILEPATH=/home/c2/anaconda3/envs/cp2/lib/libcudnn.so

//Path to a library.
CUDNN_LIBRARY_PATH:FILEPATH=/home/c2/anaconda3/envs/cp2/lib/libcudnn.so

I’m using Python 3.6, Anaconda, spconv@f22dd9 from Oct 27, 2020, pytorch 1.6, cuda 10.1, and Ubuntu 18.04.

Note that I also installed the same version of cuda (you can have multiple!) on my system per this guide so that I get nvcc which is missing in the Anaconda cuda installation. To see what version of cuda your anaconda pytorch is using run

conda activate your-env
conda list | grep cuda

You may also need to install cudnn with conda

conda install cudnn

Also, you can point CMake to the version of cuda and nvcc you want with the environment variable CUDA_BIN_PATH, i.e.:

export CUDA_BIN_PATH=/usr/local/cuda-10.1

Now my build looks like this

If you want to do a clean build, run the following but note that it will delete your CMakeCache.txt changes and you’ll need to redo them.

python setup.py clean --all