Is creating dataset only by augmentation a bad practice?

Question

I wonder if creating data set only by augmentation base images is a bad practice.

I mean the situation when you have to train net to predict really simple patterns, for example printed-like digits. And all digits from specific group looks basically the same, for example all one's look the same and so on. The only difference is rotation/translation etc. in the image.

Is it bad way to create data set by taking digit image and randomly rotate, translate and maybe erode/dilate it?

My intuition tells me that something's wrong with that approach, but I cannot find any reason why it should be wrong.

Answer 1

Data augmentation is usually rotating, cropping and translating images. And this makes sense if your network could meet these kind of images.

If I take a digit recognition like LeNet, it is useless to complicate the task of the network by forcing it to learn rotated digits, which could lead to a more complex architecture and training and less accuracy in the task. Another example I could think of is human pose recognition (openpose project). As we humans usually stands with our feet on the bottom of the image and the head on top, openpose project didn't use rotation of images on the dataset.

So I would say data augmentation is a great tool (especially when we lack data), but I would only use it when the augmented data could be met when performing the task. If the digits are always oriented and placed on the middle of the image, it doesn't make much sense to use translations and rotations on the data set to augment it unless we really lack data.

In your example, it does make sense to me to create the dataset using data augmentation.

Answer 2

Outside of using a generated dataset to study machine learning, the typical purpose of a trained machine learning model is to process new inputs from some source.

For a model to be effective, the training data set inputs and new inputs should be taken from the same distribution. The loss function used in training, combined with cross-validation to measure and maintain generalisation, will have ensured that the most accurate results occur for a population of inputs that is similar to the training data.

The further that inputs stray from being like the training data, in any aspect, then the more likely that outputs from a trained model are inaccurate. This may include over-generalising - if you set much higher ranges for some variations - e.g. far more dilate and erode that would be seen in practice, then the neural network weights will be tuned to allow for this data and may score worse on your target data even though it will appear to score well overall on the training data. That is because the measurements for loss and accuracy from the more realistic generation will be diluted by measurements from training data that has no relevance to the real-world problem you are trying to solve. Maybe it will be OK, maybe worse, maybe even better - however your measurements of loss and accuracy during training will not tell you.

So there is significant danger in relying on a generated-only dataset for training. If any aspect of the simulated inputs does not match how the system will be used in practice, the impact is likely to be felt in terms of reduced accuracy.

For your digits example, you should consider where the "real" digits will come from later, and try to ensure that your data generation takes into account any complications, variations, imperfections that will occur when collecting the data. For instance, if the real digits are scanned from paper, then take a look at some typical scanned images, and check how close your generated data is to them.

If you can obtain a limited number of "real" values, perhaps not enough for training, but enough to get some accuracy statistics from, then consider using them for test and cross-validation phases. Remember, that using them for test should be done sparingly, and not used to select between models with similar results, but only to establish a rough estimate of accuracy at the end of training. Whilst using any for cross-validation may help select a model that generalises the best between the generated data set and reality, but precludes using the same examples for test.