How to apply Layer Normalisation in LSTMCell

妖精的绣舞 提交于 2019-12-24 19:00:25

问题


I want to apply Layer Normalisation to recurrent neural network while using tf.compat.v1.nn.rnn_cell.LSTMCell.

There is a LayerNormalization class but how should I apply this in LSTMCell. I am using tf.compat.v1.nn.rnn_cell.LSTMCell because I want to use projection layer. How should I achieve Normalisation in this case.

class LM(tf.keras.Model):
  def __init__(self, hidden_size=2048, num_layers=2):
    super(LM, self).__init__()
    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.lstm_layers = []
    self.proj_dim = 640
    for i in range(self.num_layers):
        name1 = 'lm_lstm'+str(i)
        self.cell = tf.compat.v1.nn.rnn_cell.LSTMCell(2048, num_proj=640)
        self.lstm_layers.append(tf.keras.layers.RNN(self.cell, return_sequences=True, name=name1))

  def call(self, x):
    for i in range(self.num_layers):
      output = self.lstm_layers[i](x)
      x = output
    state_h = ""
    return x, state_h

回答1:


It depends whether you want to apply the normalization at cell level or at layer level - I'm not sure which one is the correct way to do it - the paper doesn't specify it. Here is an older implementation that you might use for inspiration.

To normalize at cell level, you probably need to create a custom RNNCell and implement the normalization there.

P.S. You might also be able to apply LayerNormalization to the output of RNN, for example like shown below, but you'll need to think carefully about whether it has the desired effect, especially given the variable shapes inherent to sequence models.

self.lstm_layers.append(tf.keras.layers.RNN(self.cell, return_sequences=True, name=name1))
self.lstm_layers.append(tf.keras.layers.LayerNormalization())


来源:https://stackoverflow.com/questions/59369146/how-to-apply-layer-normalisation-in-lstmcell

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!