# How data augmentation like rotation affects the quality of detection?

I'm using an object detection neural network and I employ data augmentation to increase a little my small dataset. More specifically I do rotation, translation, mirroring and rescaling.

I notice that rotating an image (and thus it's bounding box) changes its shape. This implies an erroneous box for elongated boxes, for instance on the augmented image (right image below) the box is not tightly packed around the left player as it was on the original image.

The problem is that this kind of data augmentation seems (in theory) to hamper the network to gain precision on bounding boxes location as it loosens the frame.

Are there some studies dealing with the effect of data augmentation on the precision of detection networks? Are there systems that prevent this kind of thing?

(Obviously, it seems advisable to use small rotation angles)

The problem is that this kind of data augmentation seems (in theory) to hamper the network to gain precision on bounding boxes location as it loosens the frame.

Yes, it is clear from your examples that the bounding boxes become wider. Generally, including large amounts of data like this in your training data will mean that your network will also have a tendency to learn slightly larger bounding boxes. Of course, if the majority of your training data still has tight boxes, it should stell tend towards learning those... but likely slightly wider ones than if the training data did not include these kinds of rotations.

Are there some studies dealing with the effect of data augmentation on the precision of detection networks? Are there systems that prevent this kind of thing?

(Obviously, it seems advisable to use small rotation angles)

I do not personally work directly in the area of computer vision really, so I'm not sufficiently familiar with the literature to point you to any references on this particular issue. Based on my own intuition, I can recommend:

1. Using relatively small rotation angles, as you also already suggested yourself. The bounding boxes will become a little bit wider than in the original dataset, but not by too much.
2. Using rotation angles that are a multiple of $$90^\circ$$. Note that if you rotate a bounding box by a multiple of $$90^\circ$$, the rotated bounding boxes become axis-aligned and your problem disappears again, they'll become just as tight as the bounding boxes in the unrotated image. Of course, you can also combine this suggestion with the previous one, and use rotation angles in, for example, $$[85^\circ, 95^\circ]$$.
3. Apply larger rotations primarily in images that only have bounding boxes that are approximately "square". From looking at your image, I get the impression that the problem of bounding boxes becoming wider after rotations is much more severe when you have extremely wide or thin bounding boxes (with one dimension much greater than the other). When the original bounding box is square, there still will be some widening after rotation, but not nearly as much, so the problem may be more acceptable in such cases.

Tilt Tolerance in the Simple Case of Letters

An educated English speaking person can look at the letter P and recognize it as a capital p. If that person tilts their head before looking at it, the letter tilted is also recognizable as a capital p.

This is because some translation tolerance is contained in shape recognition. Adult human vision exhibits the ability to read tilted letters, words, and word sequences, with diminishing speed and accuracy as the tilt angle increases beyond $$\, \pi / 8 \,$$ to the point where the text is nearly unrecognizable when upside down unless the person learns to recognize text upside down through practice. This is separately true of text that is horizontally mirrored, vertically mirrored, and rotated by $$\, \pi \,$$ radians.

The capability to learn translation tolerance is contained in DNA expression that constructs the vision system, but there is likely a developmental component involving variance in viewing angle that is also required. It is believed, but not yet proven, that this is a function of the neck, body, and page angles during the period where children learn to read. Tilt the body or neck relative to the page and the letter angle changes.

$$\ell = p - n - b$$

We cannot mistreat children by forcing them to develop reading skills in a world where $$\ell$$ is constant to prove the hypothesis, but it is likely that without variance in $$\ell$$, the tilt tolerance of the resulting teenage reader would be narrow.

Types of Visual Translation

There are translation types that require separate consideration. They are not all treated the same way in human vision, flying insect vision, or what a vision system engineer may wish to be capabilities of the system. There may be DNA differences between these types, although that is not known as of this writing, but there are certainly AI system and process design differences.

• Tilts typical of heads and cameras in casual terrestrial circumstances
• Tilts typical of athletes, jugglers, astronauts, circus performers, pilots, flight systems, climbers, missiles, and countermeasures
• Vertical mirroring
• Horizontal mirroring
• Magnified capture, which requires probabilistic projection of form for scene recognition when objects are truncated by the frame
• Wide angle capture, which requires interpolative magnification for detailed recognition

Providing simulations of these in digital processing is obviously common and central to this question. It is useful to note that the difference between the casual and extreme tilt is greater than the difference between horizontal and vertical mirroring in most processing scenarios. Software zooming produces an interpolation problem and software un-zooming produces a projection problem.

Trigonometric perspective distorts and stereo vision effects resulting from parallax is a poorly investigated form of pre-processing of training data. It is worth mentioning, since it is obviously part of a child's visual learning experience and likely has much to do with depth perception in 3D scenes and thus hand-eye coordination, navigation around obstacles, and athletic ability.

Data Augmentation

What is colloquially called data augmentation is the simulation of uncaptured visual scenarios through trivial manipulations of captured ones. Although the practice in visual system training is not equivalent to additionally captured experience, these trivial manipulations have some advantages. A more symmetric result is produced because the simulated additional experience is symmetric.

In a way, this simulates self-discipline regarding symmetry of training for some scenarios, like a person who learns to bat left and right handed recognizes ball trajectory symmetrically.

In other cases no benefit results, such as reading English. The need to read books in a mirror does not often arise.

In robotic vision, one might consider horizontal translation as one of the trivial manipulations that would lead to ambidextrous coordination, but this is where the types of translations diverge significantly in the way they correspond to AI topology and process design.

Special Case of Mirroring

Mirroring in either dimension is already built into the way kernels, fully connected layers, and other artificial network topologies work. No benefit can be obtained through data augmentation for robotics through simple horizontal and vertical flips that can't be obtained by appropriate transforming the learned parameter tensors after learning occurred. This post processing of training information saves significant computing resource and can drastically reduce wait time for development stakeholders.

No such post processing translations can be reliably or accurately accomplished with tilting, and changes in magnification because the pixels do not have a one to one correspondence given a rectangular frame.

Special Case of Rotation

An array of rotations will almost always improve the time-independent quality of detection, but the cost may make the time-relative quality lower, since the cost permissible may be limited. Whether the added time to train with an array of rotations produces a good return on investment depends on several things, including but not limited to these.

• Whether training time is critical
• Whether there is sufficient data for robust training
• Whether tilt tolerance is critical

With the above understood, we can return to the four transformations mentioned in the question.

• Translation (zeroth degree Cartesian)
• Mirroring (first degree Cartesian)
• Rescaling (first degree Cartesian)

In the images shown, there may be others. One obvious one for soccer is color replacement to change the colors of jerseys. There may also be value in converting the entire RGB representation of pixels to HSVA (hue, saturation, value, alpha).

Skewing and spherical translations may assist with 3D processing further downstream in robotic applications.

Frames in Transformations

With the exception of magnification and mirroring, the frame of the capture produces a misalignment of the input and output bounding rectangles during transformation that leads to indeterminate pixel values.

Rotating an image (and thus it's bounding box) changes its shape. This implies an erroneous box for elongated boxes, for instance on the augmented image (right image below) the box is not tightly packed around the left player as it was on the original image.

With human eyes, each field of view is a circle. That obviously transforms radially more nicely than a rectangle, but the problem still exists because the nose bridge produces a near parabolic interference on the inner side of each eye. Closing one eye and modifying cognitive focus allows that feature to come into view. At deeper depths, the visual networks of the brain learn to dismiss the nose unless a bug or some mark is on it or it appears injured.

With rectangular capture, the issue is augmented.

Are there systems that prevent this kind of thing?

The most basic way to handle mirroring is in the post processing of the learned parameter tensor. It is not common for systems to address undefined pixels in other transformations. There are approaches to system design that can be used effectively.

The way to handle overlap between negative before and positive after frames in computer vision,

$$\, \lnot \, \mathcal{I} \, \land \, \mathcal{O} \, \text{,} \,$$

where $$\mathcal{I}$$ is the transformation input and $$\mathcal{O}$$ is the transformation output, is with an additional channel. The alpha channel of the pixel matrix in conjunction with a constant default color or gray value is the common method. This is useful with rotation, translation, de-magnification, skew, spherical transformations, or any other transformations that produce unassigned pixels in the resulting matrix. To use this method, such transformations must have the ability to mark the pixel with a transparency flag represented in this non color and non gray scale channel.

From an information theory perspective, convergence then includes knowledge of the lack of knowledge at that pixel's location. The default color or gray value does hamper the training without this pixel level knowledge of obscurity, just as what is behind the human nose known not to be known and the color of the nose does not interfere with learning to read or catch a ball. With the absence of pixel knowledge passed into the network, that hampering is effectively mitigated in most scenarios.

Additional Questions in the Question Body

There are a few other questions that should be addressed directly.

Are there some studies dealing with the effect of data augmentation on the precision of detection networks?

Since AI is an interdisciplinary field, the answer is definitely, "Yes," but those studies usually appear in cognitive science and neurological imaging areas of academic publishing, and it is an active area of research. The auto industry's research departments have taken interest in biology, neurology, and cognition for this reason.

It seems advisable to use small rotation angles.

Again, the return on investment is the criteria to consider. The smaller the rotational angle increment used to produce the simulation of tilted camera, head, body, page, or playing field experiences that were not captured, the more training data is simulated for the maximum variance of tilt chosen. Just as is generally the case, huge amounts of data increase reliability and accuracy but also increase training resource requirement in either hardware, O.S. resources, or wait time.

Too large a rotational angle increment and there will be recognition gaps in the distribution of reliability and accuracy as a function of tilt angle.

Formulaic Determination of Increment Size

It is possible to calculate the best tilt angle increment given the following criteria.

• Minimum degradation in reliability between training angles for a given recognition task
• Minimum degradation in reliability between training angles for a given recognition task

Such could also be theoretically worked out for the other kinds of transformations. Review of the PAC (probably approximately correct) learning framework theory may be helpful for those interested in this vein of research. The development of an actual formula to use could be of considerable value to practitioners, but it would likely be a two year project to develop the theory and verify it empirically using public data sets, something worthy of a PhD thesis for the right student.