Why does data augmentation using synthetic data generated by one model improve the performance of another model?

Question

I understand from articles like this one that synthetic data generated by one model based on real data can improve the performance of a second model. Can anyone help me understand the intuition behind why this works? Specifically, why not directly use the real data to train the second model? Since the data generated by the first model is at most as good as the real data used for that model, won't the performance be actually better if the real data are directly used to train the second model?

Answer 1

The reasoning behind synthetic data is the same behind classic data augmentation,so the goal is to increase the amount of training instances to improve generalization.

The difference with classic data augmentation techniques is that classic data augmentation rely on linear transformations like flipping or rotation that do not provide as much variation as we would like. Synthetic data instead are usually generated with generative models, which sample new data from a posterior distribution learned from real data. This difference is huge. Imagine augmenting data with a flipped pictures of the same dog flipped around vs augmenting the data with a pictures of different dogs not present in your current real data.

Now the problem of course is that training generative model to produce good samples is as hard. Training a GAN on MNIST or other toy datasets is rather easy, doing so on real datasets with much less instances and all sort of imperfections (picture taken with different cameras to give an example) is many times a rabbit hole in itself. So you always want to do a good list of pros and cons before deciding to invest time in training a model just for synthetic data generation.

On the other hand, it's not impossible to find synthetic datasets ready to be used out of the shelf. In that case exploring the data to check they fit your use case is usually a no brainer and time worth spending for sure. An example is SynthCity, a dataset for point cloud segmentation, I found it very handy for a project I was working on where we didn't have lot of point clouds to start with.

Update

Why not feeding the real data used to train the generative model directly?

First answer is that you always feed the real data as well, i.e. you never train only on synthetic data, the point again is to augment you initial dataset, not replacing it. So you move from a situation like this:

real data + naive augmentation -> model

to this:

real data + synthetic data + naive augmentation -> model

Second answer is that not always synthetic data are generated trough machine learning, there are situations in which you can use analytic methods to produce them (for example fluid simulations trough differential equations). So in this case you're not even starting from real data at all.