How does rotating an image and adding new 'rotated classes' prevent overfitting?

Question

From Meta-Learning with Memory-Augmented Neural Networks in section 4.1:

To reduce the risk of overfitting, we performed data augmentation by randomly translating and rotating character images. We also created new classes through 90◦, 180◦ and 270◦ rotations of existing data.

I can maybe see how rotations could reduce overfitting by allowing the model to generalize better. But if augmenting the training images through rotations prevents overfitting, then what is the purpose of adding new classes to match those rotations? Wouldn't that cancel out the augmentation?

Answer 1

How can data augmentation reduce overfitting?

You write that you can already maybe see how data augmentation can help prevent overfitting in general, but it sounds a bit uncertain and it's still asked in the title of the question, so I'll address this first:

Generally, when we use Machine Learning for classification problems, we would ideally learn a classifier that can perform well on a population. An example of a population would be: the set of all handwritten characters in the entire world. Generally, we don't have that complete population available for training, we only have a (much smaller) training dataset. If a training set is large enough, it might be a good approximation of the true population we're interested in (a "dense sampling" of the space we're interested in), but it's still just that; an approximation.

We say that a learning algorithm is overfitting if it performs singificantly better on the training set than it is on the population (which we generally approximate again using a separate test set).

Now, data augmentation (like adding rotations / translations of images in the training set to the training set) can help combat overfitting because it bridges the gap between training set and population. The population (all handwritten characters in the entire world) will likely include characters at various offsets from the middle (e.g. translations) and at various rotations. So, data augmentation is simply adding more examples (and possibly more varied examples) to our training set, which importantly are considered to be a part of the population we're interested in. If, for example, the population we are interested in were only the set of all handwritten characters at a specific position in the image (e.g., centered), then augmenting the dataset by adding various translations would not help; we'd be adding instances that are outside the population we want to learn about.

---

Why doesn't adding extra classes for rotations cancel out augmentations?

There are two possible explanations I can come up with:

1. Maybe the "extra-class" rotations are different from the "data augmentation" rotations.

Here is the exact quote that's relevant from the paper:

"To reduce the risk of overfitting, we performed data augmentation by randomly translating and rotating character images. We also created new classes through 90◦, 180◦ and 270◦ rotations of existing data."

That first sentence is not 100% clear in my opinion. I imagine the translations they use for data augmentation are relatively small (e.g. offsets of a few pixels), so maybe the rotations they use for data augmentation are also only "small" rotations (for example, between -10◦ and +10◦). The "larger" rotations (multiples of 90◦) described in the second sentence may then no longer be a part of the "data augmentation to reduce the risk of overfitting" in the first sentence; they're simply parts of a different action performed to increase the number of classes in the dataset (and, I imagine, for each of these larger rotations they may again perform "smaller rotations" for data augmentations).

This explanation is kind of hypothetical though, it's not 100% clear from the paper exactly what they mean here in my opinion.

2. "Overfitting" can have a slightly different interpretation in the case of one-shot learning than in traditional learning.

Note that this paper is about "one-shot learning", where the goal is to be able to classify accurately after being presented only a single example ("one shot") of a never-before-seen class. In such one-shot problems, you could in some sense say that an algorithm might "overfit" to the "distribution of classes" if it can only perform one-shot learning well on a certain set of similar classes, but not on others.

For example, if you only train one-shot learning on a set of handwritten characters that are "upright" (close to 0 rotation), your algorithm might be able to perform well in terms of one-shot learning when presented with new classes (new handwritten characters) that are also upright, but might be incapable of proper one-shot learning when presented with new classes (new handwritten characters) that are upside-down.

Answer 2

Over-fitting in the context of convergence in a neural network can have many causes. When the model implied in the design of the network is not well fitted for the task, the network may still converge within the time frame allowed and the example set presented but it will take more time and a greater number of examples than necessary, and the reliability and accuracy of the trained circuit may be far below what could be achievable with a solid design.

Gross over-fitting can be one of the causes of decreased reliability. A more slight over-fit will exhibit accuracy somewhat diminished from the accuracy found by the end of training.

This is why various designs have emerged with functionally specific circuit simulations between more general multi-layer perceptron networks.

• Convolution kernels
• Rotations
• Other basic translations
• Hash lookups
• Other patterned circuits that remove burden from general convergence

In the case of rotation, the convergence on an optima angle in one specialized layer or longitudinal stack element can remove considerable burden and allow overall convergence with fewer general activation layers, using fewer examples, and with a significantly more reliable and accurate result.

Consider what perceptrons must do to rotate an image arbitrarily. They must wire what is essentially rotational trigonometry into the parameters of everything that is orientation-dependent within the network, creating what is essentially a pliable helix, possibly in many locations within the trained network. Creating the pliable helix functionality, parameterized in advance of training and carefully handling back-propagation to adjust to its existence, drastically reduces the complexity of convergence.

If done well, over-fitting will be much less of an issue. If done poorly, there could be worse over-fitting or other problems such as non-convergence.

In summary, the best practice is to leave to general network training what must, by its nature, be complex but handle with specific functionality what is well understood and for which mathematical and algorithmic approaches already exist.