Keras cifar10 example validation and test loss lower than training loss

Question

I\'m playing around with the cifar10 example from Keras which you can find here. I\'ve recreated the model (i.e., not same file but everything else pretty much the same) and you

Answer 1

This is a rare phenomenon but it happens from time to time. There are several reasons why this might be the case:

• smaller datasets have smaller intrinsic variance so this means that your model properly captures patterns inside of your data and train error is greater simply because the inner variance of training set is greater then validation set.
• Simple accident - this might also occur - that your split is suitable for such behaviour.

Answer 2

Marcin's answer is good. There are also another few big reasons for high training error:

• Dropout. Dropout layers are "on" in training, but they will be turned "off" (skipped) when doing validation and testing. This is automatic and it is by design. Dropout harms training error slightly. This is to be expected. Dropout layers are actually helpful in deep neural nets for regularization despite the additional training challenges. Most deep neural nets probably use dropout.

• Learning rate too high. It's like throwing a coin into a glass. It can jump out when thrown too hard.