Deeplearning4j: Iterations, Epochs, and ScoreIterationListener

Question

Good afternoon to everybody.

I'm quite new to the Deepleaning4j library and there are a couple of stuff that are still unclear to me. The concept of "epoch" is not new, thus, it is clear that it represents a full-cycle on the training set. My first doubt is related to the concept of "iteration". What is an iteration over the training set? Does it correspond to the analysis of a mini-batch number of training instances or to something else?

In my code, I set ".iterations(1)"; however, when I run my code I see a lot of:

... ScoreIterationListener - Score at iteration XX is yy.yyyyyy"

So, if I set ".iterations(1)", why do I continue to see values of XX greater than 1? Are there, maybe, some differences between the idea of "iteration" as a network configuration parameter and what "iteration" means for the ScoreIterationListener class?

Thanks everybody for any answer or link to useful information.

Best, Mauro.

Answer 1

The DeepLearning4J documentation has some good insight, especially with respect to the difference between an epoch and an iteration.

According to DL4J's documentation:

"An iteration is simply one update of the neural net model’s parameters. Not to be confused with an epoch which is one complete pass through the dataset. Many iterations can occur before an epoch is over. Epoch and iteration are only synonymous if you update your parameters once for each pass through the whole dataset; if you update using mini-batches, they mean different things. Say your data has 2 minibatches: A and B. .numIterations(3) performs training like AAABBB, while 3 epochs looks like ABABAB."

With respect to your question and as referenced by this excerpt, if you set .iterations(1) and had only one batch, the iteration would be synonymous with 1 epoch, or one pass through the complete dataset. However, if you update using mini-batches, an epoch and an iteration would differ slightly -- an iteration would result in AAABBB, rather than an epoch, which would produce ABABAB (referenced by the example above).

Hopefully this answer and the documentation linked answers your question!

P.S. I apologize for the late reply; I stumbled on this question very recently!