Loading a video dataset (Keras)

Question

I\'m trying to implement an LRCN/C(LSTM)RNN to classify emotions in videos. My dataset structure is split in two folders - \"train_set\" and \"valid_set\". When you open, e

Answer 1

1. flow_from_directory is made for images, not movies. It will not understand your directory structure and will not create a "frames" dimension. You need your own generator (usually better to implement a keras.utils.Sequence)

2. You can only load into batches if :

   • you load movies one by one due to their different lengths
   • you pad your videos with empty frames to make them all have the same length

3. Same as 1.

4. If you make your own generator implementing a keras.utils.Sequence(), the safety will be kept as long as your implementation knows what is each movie.

5. It would shuffle images if you were loading images

6. TimeDistributed allows data with an extra dimension at index 1. Example: a layer that usually takes (batch_size, ...other dims...) will take (batch_size, extra_dim, ...other dims...). This extra dimension may mean anything, not necessarily time, and it will remain untouched.

   • Recurrent layers don't need this (unless you really want an extra dimension there for unusual reasons), they already consider the index 1 as time.
   • CNNs will work exactly the same, for each image, but you can organize your data in the format (batch_size, video_frames, height, width, channels)