What is the difference between feature extraction with or without data augmentation?

Question

Here's an extract from Chollet's book "Deep Learning with Python" about using pre-trained CNN to predict class from a photo set (p. 146):

At this point, there are two ways you could proceed:

• Running the convolutional base over your dataset, recording its output to a Numpy array on disk, and then using this data as input to a standalone, densely connected classifier similar to those you saw in part 1 of this book. This solution is fast and cheap to run, because it only requires running the convolutional base once for every input image, and the convolutional base is by far the most expensive part of the pipeline. But for the same reason, this technique won’t allow you to use data augmentation.

• Extending the model you have (conv_base) by adding Dense layers on top, and running the whole thing end to end on the input data. This will allow you to use data augmentation, because every input image goes through the convolutional base every time it’s seen by the model. But for the same reason, this technique is far more expensive than the first.

The first method is called (1) and the second is (2).

If I use data augmentation to expand my data set, then could (1) be as good ad (2)? If no, why?

Answer 1

There are two ways that you could perform data augmentation:

• Up front, by expanding the input dataset into a larger one, performing a range of changes to each input then storing the result. This appears to be what you are suggesting.

• Just in time, by sampling from possible augmentations on each epoch, or even per sample when building a mini-batch. This appears to be what Chollet is suggesting.

Chollet's approach allows for augmentation to include finer degrees of augmentation that are different each time an input is considered, e.g. rotations of any angle, selecting a slightly different area from a larger image each time. For your approach you could consider the same set of augmentations but they would have to be "frozen in" at the time of building a dataset, and you would not be able to consider all possible variations for each image because it would make the dataset too large.

Both approaches are valid, and both would be called data augmentation. Chollet's approach obtains better re-use of image samples in the long term, and may result in better generalisation in the final trained network. Your approach may allow for more efficient use of CPU time to obtain a result that passes a threshold in accuracy. In some cases the difference between approaches may be minor compared to that caused by other changes in hyperparameters.