How to calculate the number of parameters of an LSTM network?

Question

Is there a way to calculate the total number of parameters in a LSTM network.

I have found a example but I\'m unsure of how correct this is or If I have understood i

Answer 1

I think it would be easier to understand if we start with a simple RNN.

Let's assume that we have 4 units (please ignore the ... in the network and concentrate only on visible units), and the input size (number of dimensions) is 3:

The number of weights is 28 = 16 (num_units * num_units) for the recurrent connections + 12 (input_dim * num_units) for input. The number of biases is simply num_units.

Recurrency means that each neuron output is fed back into the whole network, so if we unroll it in time sequence, it looks like two dense layers:

and that makes it clear why we have num_units * num_units weights for the recurrent part.

The number of parameters for this simple RNN is 32 = 4 * 4 + 3 * 4 + 4, which can be expressed as num_units * num_units + input_dim * num_units + num_units or num_units * (num_units + input_dim + 1)

Now, for LSTM, we must multiply the number of of these parameters by 4, as this is the number of sub-parameters inside each unit, and it was nicely illustrated in the answer by @FelixHo