Choosing number of Steps per Epoch

Question

If I want to train a model with train_generator, is there a significant difference between choosing

• 10 Epochs with 500 Steps each

and

• 100

Answer 1

Steps per epoch does not connect to epochs.

Naturally what you want if to 1 epoch your generator pass through all of your training data one time. To achieve this you should provide steps per epoch equal to number of batches like this:

steps_per_epoch = int( np.ceil(x_train.shape[0] / batch_size) )

as from above equation the largest the batch_size, the lower the steps_per_epoch.

Next you will choose epoch based on chosen validation. (choose what you think best)

Answer 2

The Steps per epoch denote the number of batches to be selected for one epoch. If 500 steps are selected then the network will train for 500 batches to complete one epoch. If we select the large number of epochs it can be computational

Answer 3

Based on what you said it sounds like you need a larger batch_size, and of course there are implications with that which could impact the steps_per_epoch and number of epochs.

To solve for jumping-around

• A larger batch size will give you a better gradient and will help to prevent jumping around
• You may also want to consider a smaller learning rate, or a learning rate scheduler (or decay) to allow the network to "settle in" as it trains

Implications of a larger batch-size

• Too large of a batch_size can produce memory problems, especially if you are using a GPU. Once you exceed the limit, dial it back until it works. This will help you find the max batch-size that your system can work with.
• Too large of a batch size can get you stuck in a local minima, so if your training get stuck, I would reduce it some. Imagine here you are over-correcting the jumping-around and it's not jumping around enough to further minimize the loss function.

When to reduce epochs

• If your train error is very low, yet your test/validation is very high, then you have over-fit the model with too many epochs.
• The best way to find the right balance is to use early-stopping with a validation test set. Here you can specify when to stop training, and save the weights for the network that gives you the best validation loss. (I highly recommend using this always)

When to adjust steps-per-epoch

• Traditionally, the steps per epoch is calculated as train_length // batch_size, since this will use all of the data points, one batch size worth at a time.
• If you are augmenting the data, then you can stretch this a tad (sometimes I multiply that function above by 2 or 3 etc. But, if it's already training for too long, then I would just stick with the traditional approach.