Using Dropout with Keras and LSTM/GRU cell

冷暖自知 提交于 2019-12-03 07:33:38

The recurrent layers perform the same repeated operation over and over.

In each timestep, it takes two inputs:

  • Your inputs (a step of your sequence)
  • Internal inputs (can be states and the output of the previous step, for instance)

Note that the dimensions of the input and output may not match, which means that "your input" dimensions will not match "the recurrent input (previous step/states)" dimesions.

Then in every recurrent timestep there are two operations with two different kernels:

  • One kernel is applied to "your inputs" to process and transform it in a compatible dimension
  • Another (called recurrent kernel by keras) is applied to the inputs of the previous step.

Because of this, keras also uses two dropout operations in the recurrent layers. (Dropouts that will be applied to every step)

  • A dropout for the first conversion of your inputs
  • A dropout for the application of the recurrent kernel

So, in fact there are two dropout parameters in RNN layers:

  • dropout, applied to the first operation on the inputs
  • recurrent_dropout, applied to the other operation on the recurrent inputs (previous output and/or states)

You can see this description coded either in GRUCell and in LSTMCell for instance in the source code.


What is correct?

This is open to creativity.

You can use a Dropout(...) layer, it's not "wrong", but it will possibly drop "timesteps" too! (Unless you set noise_shape properly or use SpatialDropout1D, which is currently not documented yet)

Maybe you want it, maybe you dont. If you use the parameters in the recurrent layer, you will be applying dropouts only to the other dimensions, without dropping a single step. This seems healthy for recurrent layers, unless you want your network to learn how to deal with sequences containing gaps (this last sentence is a supposal).

Also, with the dropout parameters, you will be really dropping parts of the kernel as the operations are dropped "in every step", while using a separate layer will let your RNN perform non-dropped operations internally, since your dropout will affect only the final output.

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!