cudaErrorSystemNotReady: system not yet initialized

output of python -c 'import cupy; cupy.show_config()':

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/berceanu/miniconda3/envs/fbpic/lib/python3.7/site-packages/cupy/__init__.py", line 812, in show_config
    sys.stdout.write(str(_cupyx.get_runtime_info()))
  File "/home/berceanu/miniconda3/envs/fbpic/lib/python3.7/site-packages/cupyx/runtime.py", line 174, in get_runtime_info
    return _RuntimeInfo()
  File "/home/berceanu/miniconda3/envs/fbpic/lib/python3.7/site-packages/cupyx/runtime.py", line 107, in __init__
    cupy.cuda.cublas.CUBLASError)
  File "/home/berceanu/miniconda3/envs/fbpic/lib/python3.7/site-packages/cupyx/runtime.py", line 23, in _eval_or_error
    return func()
  File "/home/berceanu/miniconda3/envs/fbpic/lib/python3.7/site-packages/cupyx/runtime.py", line 106, in <lambda>
    cupy.cuda.device.get_cublas_handle()),
  File "cupy/cuda/device.pyx", line 44, in cupy.cuda.device.get_cublas_handle
  File "cupy/cuda/device.pyx", line 45, in cupy.cuda.device.get_cublas_handle
  File "cupy/cuda/device.pyx", line 27, in cupy.cuda.device._get_device
  File "cupy/cuda/runtime.pyx", line 229, in cupy.cuda.runtime.getDevice
  File "cupy/cuda/runtime.pyx", line 201, in cupy.cuda.runtime.check_status
cupy.cuda.runtime.CUDARuntimeError: cudaErrorSystemNotReady: system not yet initialized

Conditions
- OS/Platform: Ubuntu 16.04.6 LTS
- CUDA version: 10.0, installed via conda install cudatoolkit=10.0
- CuPy version: 7.0.0, installed via pip install cupy-cuda100 in my conda env

nvidia-smi works properly, and shows NVIDIA-SMI 430.64 Driver Version: 430.64 CUDA Version: 10.1

Issue Analytics

State:
Created 4 years ago
Comments:13 (5 by maintainers)

Top GitHub Comments

2reactions

berceanucommented, Apr 1, 2020

Did you try https://www.supermicro.com/support/faqs/faq.cfm?faq=31029

0reactions

JordanCheneycommented, Apr 1, 2020

@berceanu I did, that changed the return code from an 802 to an 83 on my machine. 83 is not documented in the CUDA API so I’m not sure what that means. I opened an issue on the developer forum that details everything I’ve tried.

Top Results From Across the Web

Error 802: system not yet initialized CUDA 11.3

Error 802 generally indicates your system requires the NVLink/NVSwitch fabric manager, and you have not installed it.

NVIDIA GPU driver fails to initialize - IBM

Solution. This issue can occur due to your GPU driver library not being successfully installed when you first created your GPU device plug-in....

Bug#1016595: cuda: CUDA Error : system not yet initialized

CUDA Error : system not yet initialized - Unable to enum CUDA GPUs: unknown error - No OpenCL platforms found - No avaiable...

cuda runtime error (802) : system not yet initialized ...

cudaGetDeviceProperties returned 802 -> system not yet initialized CUDA error at bandwidthTest.cu:256 code=802(cudaErrorSystemNotReady) ...

PyTorch-v1.7.1+cu110 - CUDA initialization error

Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized (Triggered ......