How to apply a different dense layer for each timestep in Keras

问题

I know that applying a TimeDistributed(Dense) applies the same dense layer over all the timesteps but I wanted to know how to apply different dense layers for each timestep. The number of timesteps is not variable.

P.S.: I have seen the following link and can't seem to find an answer

回答1:

You can use a LocallyConnected layer.

The LocallyConnected layer words as a Dense layer connected to each of kernel_size time_steps (1 in this case).

from tensorflow import keras
from tensorflow.keras.layers import *
from tensorflow.keras.models import Model

sequence_length = 10
n_features = 4

def make_model():
  inp = Input((sequence_length, n_features))
  h1 = LocallyConnected1D(8, 1, 1)(inp)
  out = Flatten()(h1)
  model = Model(inp, out)
  model.compile('adam', 'mse')
  return model

model = make_model()
model.summary()

Per summary the number of variables used by the LocallyConnected layer is (output_dims * (input_dims + bias)) * time_steps or (8 * (4 + 1)) * 10 = 400.

Wording it another way: the locally connected layer above behaves as 10 different Dense layers each connected to its time step (because we choose kernel_size as 1). Each of these blocks of 50 variables, is a weights matrix of shape (input_dims, output_dims) plus a bias vector of size (output_dims).

Also note that given an input_shape of (sequence_len, n_features), Dense(output_dims) and Conv1D(output_dims, 1, 1) are equivalent.

i.e. this model:

def make_model():
  inp = Input((sequence_length, n_features))
  h1 = Conv1D(8, 1, 1)(inp)
  out = Flatten()(h1)
  model = Model(inp, out)

and this model:

def make_model():
  inp = Input((sequence_length, n_features))
  h1 = Dense(8)(inp)
  out = Flatten()(h1)
  model = Model(inp, out)

Are the same.

来源：https://stackoverflow.com/questions/56884285/how-to-apply-a-different-dense-layer-for-each-timestep-in-keras