Data augmentation in test/validation set?

Question

It is common practice to augment data (add samples programmatically, such as random crops, etc. in the case of a dataset consisting of images) on both training and test set, or

Answer 1

This answer on stats.SE makes the case for applying crops on the validation / test sets so as to make that input similar the the input in the training set that the network was trained on.

Answer 2

Do it only on the training set. And, of course, make sure that the augmentation does not make the label wrong (e.g. when rotating 6 and 9 by about 180°).

The reason why we use a training and a test set in the first place is that we want to estimate the error our system will have in reality. So the data for the test set should be as close to real data as possible.

If you do it on the test set, you might have the problem that you introduce errors. For example, say you want to recognize digits and you augment by rotating. Then a 6 might look like a 9. But not all examples are that easy. Better be save than sorry.

Answer 3

The goal of data augmentation is to generalize the model and make it learn more orientation of the images, such that the during testing the model is able to apprehend the test data well. So, it is well practiced to use augmentation technique only for training sets.

Answer 4

I would argue that, in some cases, using data augmentation for the validation set can be helpful.

For example, I train a lot of CNNs for medical image segmentation. Many of the augmentation transforms that I use are meant to reduce the image quality so that the network is trained to be robust against such data. If the training set looks bad and the validation set looks nice, it will be hard to compare the losses during training and therefore assessing overfit will be complicated.

I would never use augmentation for the test set unless I'm using test-time augmentation to improve results or estimate aleatoric uncertainty.

Answer 5

In computer vision, you can use data augmentation during test time to obtain different views on the test image. You then have to aggregate the results obtained from each image for example by averaging them.

For example, given this symbol below, changing the point of view can lead to different interpretations :

Answer 6

Only on training. Data augmentation is used to increase the size of training set and to get more different images. Technically, you could use data augmentation on test set to see how model behaves on such images, but usually people don't do it.