How to calculate the number of parameters of an LSTM network?

Question

Is there a way to calculate the total number of parameters in a LSTM network.

I have found a example but I\'m unsure of how correct this is or If I have understood i

Answer 1

The others have pretty much answered it. But just for further clarification, on creating an LSTM layer. The number of params is as follows:

No of params= 4*((num_features used+1)*num_units+ num_units^2)

The +1 is because of the additional bias we take.

Where the num_features is the num_features in your input shape to the LSTM: Input_shape=(window_size,num_features)

Answer 2

I think it would be easier to understand if we start with a simple RNN.

Let's assume that we have 4 units (please ignore the ... in the network and concentrate only on visible units), and the input size (number of dimensions) is 3:

The number of weights is 28 = 16 (num_units * num_units) for the recurrent connections + 12 (input_dim * num_units) for input. The number of biases is simply num_units.

Recurrency means that each neuron output is fed back into the whole network, so if we unroll it in time sequence, it looks like two dense layers:

and that makes it clear why we have num_units * num_units weights for the recurrent part.

The number of parameters for this simple RNN is 32 = 4 * 4 + 3 * 4 + 4, which can be expressed as num_units * num_units + input_dim * num_units + num_units or num_units * (num_units + input_dim + 1)

Now, for LSTM, we must multiply the number of of these parameters by 4, as this is the number of sub-parameters inside each unit, and it was nicely illustrated in the answer by @FelixHo

Answer 3

No - the number of parameters of a LSTM layer in Keras equals to:

params = 4 * ((size_of_input + 1) * size_of_output + size_of_output^2)

Additional 1 comes from bias terms. So n is size of input (increased by the bias term) and m is size of output of a LSTM layer.

So finally :

4 * (4097 * 256 + 256^2) = 4457472

Answer 4

image via this post

num_params = [(num_units + input_dim + 1) * num_units] * 4

num_units + input_dim: concat [h(t-1), x(t)]

+ 1: bias

* 4: there are 4 neural network layers (yellow box) {W_forget, W_input, W_output, W_cell}

model.add(LSTM(units=256, input_dim=4096, input_length=16))

[(256 + 4096 + 1) * 256] * 4 = 4457472

PS: num_units = num_hidden_units = output_dims

Answer 5

Formula expanding for @JohnStrong :

4 means we have different weight and bias variables for 3 gates (read / write / froget) and - 4-th - for the cell state (within same hidden state). (These mentioned are shared among timesteps along particular hidden state vector)

4 * lstm_hidden_state_size * (lstm_inputs_size + bias_variable + lstm_outputs_size) 

as LSTM output (y) is h (hidden state) by approach, so, without an extra projection, for LSTM outputs we have :

lstm_hidden_state_size = lstm_outputs_size 

let's say it's d :

d = lstm_hidden_state_size = lstm_outputs_size 

Then

params = 4 * d * ((lstm_inputs_size + 1) + d) = 4 * ((lstm_inputs_size + 1) * d + d^2)

Answer 6

LSTM Equations (via deeplearning.ai Coursera)