What is the difference between CuDNNLSTM and LSTM in Keras?

Question

In Keras, the high-level deep learning library, there are multiple types of recurrent layers; these include LSTM (Long short term memory) and CuD

Answer 1

TL;DR; The difference is 15x speed up in model training time!

Setup Steps

Dependencies

Performance Benchmark: Comparison of the standard test machines.
1 iteration of Training on 612235 samples.

keras.layers.LSTM Intel i5-4690 CPU only: 612235/612235 [==============================] - 3755s 6ms/step - loss: 2.7339 - acc: 0.5067 - val_loss: 2.1149 - val_acc: 0.6175

GTX:950 & Intel i5-4690: 612235/612235 [==============================] - 1417s 2ms/step - loss: 2.7007 - acc: 0.5137 - val_loss: 2.0983 - val_acc: 0.6199

2.5x gain with GPU.

GTX:970 & Intel i5-4690: 612235/612235 [==============================] - 1322s 2ms/step - loss: 1.9214 - acc: 0.6442 - val_loss: 1.8808 - val_acc: 0.6461

Ignorable gain with powerful GPU.

RTX 2070 & Intel i7-9700K: 612235/612235 [==============================] - 1012s 2ms/step - loss: 2.7268 - acc: 0.5111 - val_loss: 2.1162 - val_acc: 0.6234

Very minimal gain even with awesome HW upgrades!!!

keras.layers.CuDNNLSTM RTX 2070 & Intel i7-9700K: 612235/612235 [==============================] - 69s 112us/step - loss: 1.9139 - acc: 0.6437 - val_loss: 1.8668 - val_acc: 0.6469

54x gain over CPU!
15x gain over traditional(non Cuda) LSTM implementation!