Train Validation Test Splitting After or Before Data Augmentation?

Question

I have seen tutorials online saying that you should do data augmentation AFTER doing the train/val/test split. However, when I go online to read some research papers, I see numerous instances of authors saying that they first do data augmentation on the dataset and then split it because they don't have enough data. Is it just that these are silly mistakes, even for papers with many citations, or is this acceptable?

Example: Research paper. they say: "Among these selected 480 images, 94 images were col-lected while changing the viewing angle, including images of 30 youngapples, 32 expanding apples, and 32 ripe apples.These 480 images were then expanded to 4800 images using dataaugmentation methods, yielding the training dataset. The training da-taset is used to train the detection model. The remaining 480 images areused as the test dataset to verify the detection performance of theYOLOV3-dense model".

Answer 1

In my personal experience, that depends. Augmenting data for training purposes is valid, and can even improve performance, as you may be aware. For testing purposes, it may be valid. Let me give you two examples when that may be the case:

Facial Recognition. Imagine that you have an augmentation function that can change the face pose (left/right pose, for simplicity). You may want to include augmented pose images for testing your models robustness.

The paper you mentioned. In this case, you have apple detection. As the authors in [1] say:

Apples in orchards were detected and the growth stages of apples were judged. Since the angle and intensity of sunlight illumination varies greatly during the day, whether the neural network can process the images collected at different time of the day depends on the integrity of the training dataset. In order to enhance the richness of the experimental dataset, the collected images were pre-processed in terms of colour, brightness, rotation, and image definition.

After this brief introduction, the authors proceed into discussing the augmentation types they used for enhancing the richness of the dataset. As for the case of Facial Recognition, augmenting the test data follows the same idea of having a diverse testing data.

References

[1] Tian, Y., Yang, G., Wang, Z., Wang, H., Li, E., & Liang, Z. (2019). Apple detection during different growth stages in orchards using the improved YOLO-V3 model. Computers and electronics in agriculture, 157, 417-426.

Answer 2

I personally can’t think of any good reason to apply data augmentation before splitting the dataset, though one may exist. The issue is that if you augment first and split later, you risk introducing unwanted correlations between your training and test datasets.

In the paper you linked it sounds as if the training data is all procedurally derived from the test set, which makes it hard to trust the test accuracy. How do you know the network is doing any more than picking up some basic (non-generalizable) similarity between the various augmentations and their source input?