Keras cifar10 example validation and test loss lower than training loss

Question

I\'m playing around with the cifar10 example from Keras which you can find here. I\'ve recreated the model (i.e., not same file but everything else pretty much the same) and you

Answer 1

Marcin's answer is good. There are also another few big reasons for high training error:

• Dropout. Dropout layers are "on" in training, but they will be turned "off" (skipped) when doing validation and testing. This is automatic and it is by design. Dropout harms training error slightly. This is to be expected. Dropout layers are actually helpful in deep neural nets for regularization despite the additional training challenges. Most deep neural nets probably use dropout.

• Learning rate too high. It's like throwing a coin into a glass. It can jump out when thrown too hard.