Why is TF Keras inference way slower than Numpy operations?

Question

I\'m working on a reinforcement learning model implemented with Keras and Tensorflow. I have to do frequent calls to model.predict() on single inputs.

While testing infe

Answer 1

The memory leak issue still seems to persist in Keras. The following lines of code mentioned in that issue did the trick for me:

import ... as K
import gc

model = ....
del model
K.clear_session()
gc.collect()

Answer 2

A little late, but maybe useful for someone:

Replace model.predict(X) with model.predict(X, batch_size=len(X))

That should do it.

Answer 3

Are you running your Keras model (with TensorFlow backend) in a loop? If so, Keras has a memory leak issue identified here: LINK

In this case you have to import the following:

import keras.backend.tensorflow_backend
import tensorflow as tf

from keras.backend import clear_session

Finally, you have to put the following at the end of every iteration of a loop after you're done doing your computations:

clear_session()
if keras.backend.tensorflow_backend._SESSION:
    tf.reset_default_graph()
    keras.backend.tensorflow_backend._SESSION.close()
    keras.backend.tensorflow_backend._SESSION = None

This should help you free up memory at the end of every loop and eventually, make the process faster. I hope this helps.