TensorFlow: Blas GEMM launch failed

Question

When I\'m trying to use TensorFlow with Keras using the gpu, I\'m getting this error message:

C:\\Users\\nicol\\Anaconda3\\envs\\tensorflow\\lib\\site-packag         


        

Answer 1

This worked for me on TensorFlow 2.1.0 (per: https://www.tensorflow.org/api_docs/python/tf/config/experimental/set_memory_growth)

import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU') 
tf.config.experimental.set_memory_growth(physical_devices[0], True)

Answer 2

It's a simple fix, but it was a nightmare to figure it all out

On Windows I found the Keras install in Anaconda3\Lib\site-packages\keras

sources:

https://www.tensorflow.org/guide/using_gpu

https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py

Find the following in your keras/tensorflow_backend.py file you'll add config.gpu_options.allow_growth= True in both places

if _SESSION is None:
            if not os.environ.get('OMP_NUM_THREADS'):
                config = tf.ConfigProto(allow_soft_placement=True)
                config.gpu_options.allow_growth=True
            else:
                num_thread = int(os.environ.get('OMP_NUM_THREADS'))
                config = tf.ConfigProto(intra_op_parallelism_threads=num_thread,
                                        allow_soft_placement=True)
                config.gpu_options.allow_growth=True
            _SESSION = tf.Session(config=config)
        session = _SESSION

Answer 3

For me it was a runaway ipynb script which I thought I had terminated but was actually still running, thus my GPU was in use and this error appeared

Answer 4

I had a similar kind of error while inferencing on a Tensorflow model , Fixed that issue by downgrading Tensorflow from 2.1 to 1.14. Initially, I checked the GPU usage, it took all the GPU memory and couldn't able to perform the inference and found the following exception:

InternalError: 2 root error(s) found.
(0) Internal: Blas GEMM launch failed : a.shape=(86494, 257), b.shape=(257, 64), m=86494, n=64, k=257 [[{{node log_mel_features/MatMul}}]]
(1) Internal: Blas GEMM launch failed : a.shape=(86494, 257), b.shape=(257, 64), m=86494, n=64, k=257 [[{{node log_mel_features/MatMul}}]] [[log_mel_features/Log/_769]] Bellow are the commands I used:

pip uninstall tensorflow-gpu
pip install tensorflow-gpu==1.14

Answer 5

This Answer is much related to Tensorflow:

Sometimes Tensorflow fails at creation in Windows.

Restarting the notebook using gpu solves it in most cases

If it doesnt then try restarting the notebook after adding these options in your code.

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.9)

tf.Session(config=tf.ConfigProto(gpu_options=gpu_options,allow_soft_placement=True)

I never had such error while using Keras But try restarting your notebook

Answer 6

I have got the same error,lucky,I have got it fixed. my error is: the last time,I open the tensorflow sess = tf.Session(),but I forgot close the session.

so I open the terminal, type command:

ps -aux | grep program_name

find the PID,and type command kill the PID:

kill -9 PID

Ok,the GPU is realase.