how important is the loss difference between training and validation data at the beginning when training a neuronal network?

问题

Short question: Is the difference between validation and training loss at the beginning of the training (first epochs) a good indicator for the amount of data that should be used? E.g would it be a good method to increase the amount of data until the difference at the beginning is as small as possible? It would save me time and computation.

backround: I am working on a neuronal network that overfits very fast. The best result after applying many different techniques like dropouts, batch normalization, reducing learning rate, reducing batch size, increasing variety of data, reducing layers, increasing filter sizes ..... is still very bad. While the training loss decreases very well, validation loss overfits too early(with too early I mean, the desired loss is not reached, it should be many times less) Since the training with my dataset ~200 samples took 24 hours for 50 epochs, I was hoping to find a way to fight against overfitting with all the methods I described above, before increasing the amount of data. Because nothing helped I am at the point of increasing the amount of data. I am thinking about how much data could be enough for my network to eliminate overfitting. I know that this it is not easy to answer because it depends on the complexity of the data and the task I am trying to solve.. therefore I try to generalize my question to:

回答1:

Short answer to the short question: No

explanation: There's a correlation between (train_loss - val_loss) and the amount of data you need to train your model, but there's a bunch of other factors that could be the source of the big (train_loss - val_loss). For example, your network architecture is too small, and therefor your model quickly overfits. Or, your validation set doesn't reflect the training data. Or your learning rate is too big. Or...

So my recommendation: formulate your problem in another SO question, and ask "what might I be doing wrong?"

来源：https://stackoverflow.com/questions/60461069/how-important-is-the-loss-difference-between-training-and-validation-data-at-the