TensorFlow: Blas GEMM launch failed

Question

When I\'m trying to use TensorFlow with Keras using the gpu, I\'m getting this error message:

C:\\Users\\nicol\\Anaconda3\\envs\\tensorflow\\lib\\site-packag         


        

Answer 1

Try running the sample program simpleCUBLAS (it comes with CUDA) to test your CUBLAS installation and see if it works.

In my case (I am using Ubuntu) I had to reinstall CUDA to solve this issue. After I did that, simpleCUBLAS passed the test.

For some reason I started running into the same issue after a while, and I found that cleaning the directory .nv (inside my home folder) resolved the issue, and simpleCUBLAS test passed again.

Answer 2

For me, closing and restarting my processes using Python worked.

I tried a few things here but they didn't work. For example, the

    os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

line. I think it is because I am using a newer version of Keras and Tensorflow. A lot of what I have read on the internet, include the official Keras tutorial, does not work because of version conflicts.

But I saw a couple of posts about more than one Python process running. So I shut down Jupyter, Anaconda, and PyCharm, and restarted everything. And then the error went away. It may or may not be what fixes is for you, but it is worth a try.

Answer 3

I was stuck with this for days and I was able to get rid of this error eventually.

I had the wrong versions of tensorflow and cuda installed in my PC. Just make sure you have the right versions of tf,cuda and cudnn installed.

https://i.stack.imgur.com/Laiii.png

Use this link for reference.

Answer 4

I was using Jupyter Lab, but must have had the GPU locked from another TensofFlow program that I ran earlier. After killing Jupyter Lab and restarting it, the error went away.

Answer 5

I was getting exactly the same error message. I realized that there was an error with my CUDA installation, specifically with the cuBLAS library.

You can check if yours has the same problem by running the sample program simpleCUBLAS (it comes with the CUDA installation, you will probably find it in the CUDA home folder:$CUDA_HOME\samples\7_CUDALibraries\simpleCUBLAS)

Try running this program. If the test fails, you have a problem with your CUDA installation. You should try to reinstall it. That's how I solved the same problem here.

Renaming cublas64_10.dll to cublas64_100.dll may be a solution.

Answer 6

I ran into this problem when trying to run several servers that use a model to serve predictions. As I wasn't training a model but simply using it, the difference between using GPU or CPU was minor. For this specific case, the issue can be avoided by forcing Tensorflow to use the CPU by "hiding" the GPU.

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"  # Force TF to use only the CPU