Calling “fit” multiple times in Keras

Question

I\'ve working on a CNN over several hundred GBs of images. I\'ve created a training function that bites off 4Gb chunks of these images and calls fit over each of th

Answer 1

For datasets that do not fit into memory, there is an answer in the Keras Documentation FAQ section

You can do batch training using model.train_on_batch(X, y) and model.test_on_batch(X, y). See the models documentation.

Alternatively, you can write a generator that yields batches of training data and use the method model.fit_generator(data_generator, samples_per_epoch, nb_epoch).

You can see batch training in action in our CIFAR10 example.

So if you want to iterate your dataset the way you are doing, you should probably use model.train_on_batch and take care of the batch sizes and iteration yourself.

One more thing to note is that you should make sure the order in which the samples you train your model with is shuffled after each epoch. The way you have written the example code seems to not shuffle the dataset. You can read a bit more about shuffling here and here