How data augmentation like rotation affects the quality of detection?

Question

I'm using an object detection neural network and I employ data augmentation to increase a little my small dataset. More specifically I do rotation, translation, mirroring and rescaling.

I notice that rotating an image (and thus it's bounding box) changes its shape. This implies an erroneous box for elongated boxes, for instance on the augmented image (right image below) the box is not tightly packed around the left player as it was on the original image.

The problem is that this kind of data augmentation seems (in theory) to hamper the network to gain precision on bounding boxes location as it loosens the frame.

Are there some studies dealing with the effect of data augmentation on the precision of detection networks? Are there systems that prevent this kind of thing?

Thank you in advance!

(Obviously, it seems advisable to use small rotation angles)

Answer 1

The problem is that this kind of data augmentation seems (in theory) to hamper the network to gain precision on bounding boxes location as it loosens the frame.

Yes, it is clear from your examples that the bounding boxes become wider. Generally, including large amounts of data like this in your training data will mean that your network will also have a tendency to learn slightly larger bounding boxes. Of course, if the majority of your training data still has tight boxes, it should stell tend towards learning those... but likely slightly wider ones than if the training data did not include these kinds of rotations.

Are there some studies dealing with the effect of data augmentation on the precision of detection networks? Are there systems that prevent this kind of thing?

(Obviously, it seems advisable to use small rotation angles)

I do not personally work directly in the area of computer vision really, so I'm not sufficiently familiar with the literature to point you to any references on this particular issue. Based on my own intuition, I can recommend:

1. Using relatively small rotation angles, as you also already suggested yourself. The bounding boxes will become a little bit wider than in the original dataset, but not by too much.
2. Using rotation angles that are a multiple of $90^\circ$. Note that if you rotate a bounding box by a multiple of $90^\circ$, the rotated bounding boxes become axis-aligned and your problem disappears again, they'll become just as tight as the bounding boxes in the unrotated image. Of course, you can also combine this suggestion with the previous one, and use rotation angles in, for example, $[85^\circ, 95^\circ]$.
3. Apply larger rotations primarily in images that only have bounding boxes that are approximately "square". From looking at your image, I get the impression that the problem of bounding boxes becoming wider after rotations is much more severe when you have extremely wide or thin bounding boxes (with one dimension much greater than the other). When the original bounding box is square, there still will be some widening after rotation, but not nearly as much, so the problem may be more acceptable in such cases.