cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version

Question

I get the following error when l run tensorflow in GPU.

2018-09-15 18:56:51.011724: E tensorflow/core/common_runtime/direct_session.cc:158] Internal: cudaGetDevi         


        

Answer 1

In the case I just solved, it was updating the GPU driver to the latest and installing the cuda toolkit. Your error is telling you your CUDA driver version is too old. I believe the nvcc version we were seeing was 7.5, and you have 7.3.

I think all you will have to do is: sudo apt install nvidia-cuda-toolkit then reboot.

Below are the steps I took for the problem where the libcuda.so.1 file could not be found.

First, the ppa was added and a newer GPU driver installed:

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-390

After adding the ppa, it showed options for driver versions, and 390 was the latest 'stable' version that was shown.

Then install the cuda toolkit:

sudo apt install nvidia-cuda-toolkit

Then reboot:

sudo reboot

It updated the drivers to a newer version than the 390 originally installed in the first step (it was 410; this was a p2.xlarge instance on AWS).